Freenas® 11.2-U3 User Guide

Total Page:16

File Type:pdf, Size:1020Kb

Freenas® 11.2-U3 User Guide FreeNAS® 11.2-U3 User Guide FreeNAS® is © 2011-2019 iXsystems FreeNAS® and the FreeNAS® logo are registered trademarks of iXsystems FreeBSD® is a registered trademark of the FreeBSD Foundation Written by users of the FreeNAS® network-attached storage operating system. Version 11.2 Copyright © 2011-2019 iXsystems (https://www.ixsystems.com/) CONTENTS Welcome ......................................................... 8 Typographic Conventions ................................................ 10 1 Introduction 11 1.1 New Features in 11.2 ............................................... 11 1.2 Changes Since 11.2 ................................................ 13 1.2.1 RELEASE-U1 ................................................ 14 1.2.2 U2 ..................................................... 14 1.2.3 U3 ..................................................... 15 1.3 Path and Name Lengths ............................................. 15 1.4 Hardware Recommendations .......................................... 16 1.4.1 RAM .................................................... 16 1.4.2 The Operating System Device ...................................... 17 1.4.3 Storage Disks and Controllers ...................................... 18 1.4.4 Network Interfaces ............................................ 18 1.5 Getting Started with ZFS ............................................. 19 2 Installing and Upgrading 20 2.1 Getting FreeNAS® ................................................. 20 2.2 Preparing the Media ............................................... 20 2.2.1 On FreeBSD or Linux ........................................... 21 2.2.2 On Windows ............................................... 21 2.2.3 macOS ................................................... 21 2.3 Performing the Installation ............................................ 22 2.4 Installation Troubleshooting ........................................... 29 2.5 Upgrading ..................................................... 30 2.5.1 Caveats .................................................. 30 2.5.2 Initial Preparation ............................................ 30 2.5.3 Upgrading Using the ISO ......................................... 31 2.5.4 Upgrading From the GUI ......................................... 34 2.5.5 If Something Goes Wrong ........................................ 34 2.5.6 Upgrading a ZFS Pool .......................................... 36 2.6 Virtualization .................................................... 37 2.6.1 VirtualBox ................................................. 37 2.6.2 VMware ESXi ............................................... 48 3 Booting 55 3.1 Obtaining an IP Address ............................................. 56 3.2 Logging In ..................................................... 57 3.3 Initial Configuration ................................................ 58 4 Account 59 4.1 Groups ....................................................... 59 4.2 Users ........................................................ 62 3 5 System 66 5.1 Information .................................................... 66 5.2 General ....................................................... 67 5.3 Boot ......................................................... 70 5.3.1 Mirroring the Boot Device ........................................ 72 5.4 Advanced ...................................................... 73 5.4.1 Autotune ................................................. 75 5.4.2 Self-Encrypting Drives .......................................... 75 5.5 Email ........................................................ 77 5.6 System Dataset .................................................. 78 5.7 Tunables ...................................................... 79 5.8 Update ....................................................... 82 5.8.1 Preparing for Updates .......................................... 82 5.8.2 Updates and Trains ........................................... 82 5.8.3 Checking for Updates .......................................... 83 5.8.4 Applying Updates ............................................. 83 5.8.5 Manual Updates ............................................. 84 5.9 Cloud Credentials ................................................. 84 5.10 Alerts ........................................................ 87 5.11 Alert Services ................................................... 88 5.11.1 How it Works ............................................... 89 5.12 CAs ......................................................... 90 5.13 Certificates ..................................................... 92 5.14 Support ....................................................... 95 6 Tasks 98 6.1 Cloud Sync ..................................................... 98 6.1.1 Cloud Sync Example ........................................... 101 6.2 Cron Jobs ...................................................... 103 6.3 Init/Shutdown Scripts ............................................... 105 6.4 Rsync Tasks .................................................... 106 6.4.1 Rsync Module Mode ........................................... 110 6.4.2 Rsync over SSH Mode .......................................... 110 6.5 S.M.A.R.T. Tests .................................................. 113 7 Network 116 7.1 Global Configuration ............................................... 116 7.2 Interfaces ..................................................... 118 7.3 IPMI ......................................................... 120 7.4 Link Aggregations ................................................. 122 7.4.1 LACP, MPIO, NFS, and ESXi ....................................... 122 7.4.2 Creating a Link Aggregation ....................................... 123 7.5 Network Summary ................................................ 126 7.6 Static Routes .................................................... 126 7.7 VLANs ........................................................ 126 8 Storage 128 8.1 Swap Space .................................................... 128 8.2 Volumes ...................................................... 128 8.2.1 Volume Manager ............................................. 129 8.2.1.1 Encryption ............................................ 131 8.2.1.2 Encryption Performance .................................... 131 8.2.1.3 Manual Setup .......................................... 132 8.2.1.4 Extending a ZFS Volume .................................... 133 8.2.2 Change Permissions ........................................... 134 8.2.3 Create Dataset .............................................. 136 8.2.3.1 Deduplication .......................................... 138 8.2.3.2 Compression ........................................... 139 8.2.4 Create zvol ................................................ 139 8.2.5 Import Disk ................................................ 141 8.2.6 Import Volume .............................................. 142 8.2.6.1 Importing an Encrypted Pool .................................. 142 8.2.7 View Disks ................................................. 143 8.2.8 Volumes .................................................. 145 8.2.8.1 Managing Encrypted Volumes ................................. 147 8.2.8.2 Additional Controls for Encrypted Volumes ......................... 148 8.2.9 View Multipaths .............................................. 151 8.2.10 Replacing a Failed Drive ......................................... 151 8.2.10.1 Replacing an Encrypted Drive ................................. 153 8.2.10.2 Removing a Log or Cache Device ............................... 153 8.2.11 Replacing Drives to Grow a ZFS Pool .................................. 153 8.2.12 Adding Spares .............................................. 154 8.3 Periodic Snapshot Tasks ............................................. 154 8.4 Replication Tasks ................................................. 156 8.4.1 Examples: Common Configuration ................................... 156 8.4.1.1 Alpha (Source) .......................................... 156 8.4.1.2 Beta (Destination) ........................................ 157 8.4.2 Example: FreeNAS® to FreeNAS® Semi-Automatic Setup ...................... 157 8.4.3 Example: FreeNAS® to FreeNAS® Dedicated User Replication ................... 159 8.4.4 Example: FreeNAS® to FreeNAS® or Other Systems, Manual Setup ................ 160 8.4.4.1 Encryption Keys ......................................... 160 8.4.5 Replication Options ........................................... 163 8.4.6 Replication Encryption .......................................... 164 8.4.7 Limiting Replication Times ........................................ 164 8.4.8 Troubleshooting Replication ...................................... 164 8.4.8.1 SSH ................................................ 164 8.4.8.2 Compression ........................................... 165 8.4.8.3 Manual Testing .......................................... 165 8.5 Resilver Priority .................................................. 165 8.6 Scrubs ....................................................... 166 8.7 Snapshots ..................................................... 169 8.7.1 Browsing a snapshot collection ..................................... 171 8.8 VMware-Snapshot ................................................. 171 9 Directory Services 173 9.1 Active Directory .................................................. 173 9.1.1 Troubleshooting Tips ........................................... 177 9.1.2 If the
Recommended publications
  • Introduzione Al Mondo Freebsd
    Introduzione al mondo FreeBSD Corso avanzato Netstudent Netstudent http://netstudent.polito.it E.Richiardone [email protected] maggio 2009 CC-by http://creativecommons.org/licenses/by/2.5/it/ The FreeBSD project - 1 ·EÁ un progetto software open in parte finanziato ·Lo scopo eÁ mantenere e sviluppare il sistema operativo FreeBSD ·Nasce su CDROM come FreeBSD 1.0 nel 1993 ·Deriva da un patchkit per 386BSD, eredita codice da UNIX versione Berkeley 1977 ·Per problemi legali subisce un rallentamento, release 2.0 nel 1995 con codice royalty-free ·Dalla release 5.0 (2003) assume la struttura che ha oggi ·Disponibile per x86 32 e 64bit, ia64, MIPS, ppc, sparc... ·La mascotte (Beastie) nasce nel 1984 The FreeBSD project - 2 ·Erede di 4.4BSD (eÁ la stessa gente...) ·Sistema stabile; sviluppo uniforme; codice molto chiaro, ordinato e ben commentato ·Documentazione ufficiale ben curata ·Licenza molto permissiva, spesso attrae aziende per progetti commerciali: ·saltuariamente esterni collaborano con implementazioni ex-novo (i.e. Intel, GEOM, atheros, NDISwrapper, ZFS) ·a volte no (i.e. Windows NT) ·Semplificazione di molte caratteristiche tradizionali UNIX Di cosa si tratta Il progetto FreeBSD include: ·Un sistema base ·Bootloader, kernel, moduli, librerie di base, comandi e utility di base, servizi tradizionali ·Sorgenti completi in /usr/src (~500MB) ·EÁ giaÁ abbastanza completo (i.e. ipfw, ppp, bind, ...) ·Un sistema di gestione per software aggiuntivo ·Ports e packages ·Documentazione, canali di assistenza, strumenti di sviluppo ·i.e. Handbook,
    [Show full text]
  • El-Arte-De-La-Linea-De-Comandos
    Traducido por Lyx Drymer Maxxcan Zorro Fanta Suggie y Frangor de El Binario 2 Indice´ general 0.1. El arte de la l´ınea de comandos . .4 0.1.1. Meta . .4 0.1.2. Fundamentos . .5 0.1.3. De uso diario . .6 0.1.4. Procesamiento de archivos y datos . .8 0.1.5. Depuracion´ del sistema . 10 0.1.6. Comandos concatenados . 11 0.1.7. Oscuro pero util´ . 13 0.1.8. Solo MacOS . 15 0.1.9. Mas´ recursos . 16 0.1.10. Advertencia . 16 0.1.11. Licencia . 16 3 4 INDICE´ GENERAL 0.1. El arte de la lınea´ de comandos La soltura del uso de la consola es una destreza a menudo abandonada y considerada arcaica, pero mejorara´ tu flexibilidad y productividad como si fueras un i ngeniero de una forma obvia y sutil. Esta es una seleccion´ de notas y consejos de como usar la l´ınea de comandos de consola que encontre´ util´ cuando trabajaba en Linux. Algunos consejos son basicos,´ y otros bastante espec´ıficos, sofiscitados, u .oscuros”. Esta pagina´ no es larga, pero si usas y recuerdas todos los puntos, sabras´ lo suficiente. Figura 1: curl -s ‘https://raw.githubusercontent.com/jlevy/the-art-of-command- line/master/README.md’q j egrep -o ‘nw+’ j tr -d “’ j cowsay -W50 0.1.1. Meta Objetivo: Esta gu´ıa es tanto para el principiante como para el experimentado. Los objeti- vos son amplitud (todo importa), especificidad (dar ejemplos concretos del uso mas´ comun),´ y brevedad (evitar lo que no sea esencial o que se puedan encontrar facil-´ mente en otro lugar).
    [Show full text]
  • GELI Boot Booting from Encrypted Disks on Freebsd
    GELI Boot Booting from Encrypted Disks on FreeBSD Allan Jude -- ScaleEngine Inc. [email protected] twitter: @allanjude Introduction Allan Jude ● 13 Years as FreeBSD Server Admin ● FreeBSD src/doc committer (focus: ZFS, bhyve, ucl, xo) ● Co-Author of “FreeBSD Mastery: ZFS” and “FreeBSD Mastery: Advanced ZFS” with Michael W. Lucas (For sale in the hallway) ● Architect of the ScaleEngine CDN (HTTP and Video) ● Host of BSDNow.tv & TechSNAP.tv Podcasts ● Use ZFS for large collections of videos, extremely large website caches, mirrors of PC-BSD pkgs and RaspBSD ● Single Handedly Manage Over 1000TB of ZFS Storage Overview ● Do a lot of work with ZFS ● Helped build the ZFS bits of the installer ● Integrated ZFS Boot Environments ● Created ZFS Boot Env. Menu ● ZFS Boot Env. do not work with GELI ● Booting from GELI encrypted pool requires creating an unencrypted “boot pool” with the kernel and GELI module ● Boot Environments are awesome, you should use them too I Have Written A Thing ● I am a very novice C programmer ● Implemented a minimal version of GELI in the gpt{,zfs}boot (UFS and ZFS) bootcodes ● Took a lot of time to understand the existing bootcode and how it works ● Took a lot of learning about C ● The existing boot code is terrible, and needs much love, too much copy-pasta ● Had to navigate many obstacles ● but, it works! How Do Computers Even Work? ● BIOS reads the 512 bytes MBR ● Consists of 446 byte bootstrap program, and partition table (4 entries) ● This bootstrap is then executed (boot0.S) ● It examines the partition table and finds the active partition, reads the first 512 bytes ● This is boot1.
    [Show full text]
  • Nxadmin CLI Reference Guide Unity Iv Contents
    HYPER-UNIFIED STORAGE nxadmin Command Line Interface Reference Guide NEXSAN | 325 E. Hillcrest Drive, Suite #150 | Thousand Oaks, CA 91360 USA Printed Thursday, July 26, 2018 | www.nexsan.com Copyright © 2010—2018 Nexsan Technologies, Inc. All rights reserved. Trademarks Nexsan® is a trademark or registered trademark of Nexsan Technologies, Inc. The Nexsan logo is a registered trademark of Nexsan Technologies, Inc. All other trademarks and registered trademarks are the property of their respective owners. Patents This product is protected by one or more of the following patents, and other pending patent applications worldwide: United States patents US8,191,841, US8,120,922; United Kingdom patents GB2466535B, GB2467622B, GB2467404B, GB2296798B, GB2297636B About this document Unauthorized use, duplication, or modification of this document in whole or in part without the written consent of Nexsan Corporation is strictly prohibited. Nexsan Technologies, Inc. reserves the right to make changes to this manual, as well as the equipment and software described in this manual, at any time without notice. This manual may contain links to web sites that were current at the time of publication, but have since been moved or become inactive. It may also contain links to sites owned and operated by third parties. Nexsan is not responsible for the content of any such third-party site. Contents Contents Contents iii Chapter 1: Accessing the nxadmin and nxcmd CLIs 15 Connecting to the Unity Storage System using SSH 15 Prerequisite 15 Connecting to the Unity
    [Show full text]
  • Institutionalizing Freebsd Isolated and Virtualized Hosts Using Bsdinstall(8), Zfs(8) and Nfsd(8)
    Institutionalizing FreeBSD Isolated and Virtualized Hosts Using bsdinstall(8), zfs(8) and nfsd(8) [email protected] @MichaelDexter BSDCan 2018 Jails and bhyve… FreeBSD’s had Isolation since 2000 and Virtualization since 2014 Why are they still strangers? Institutionalizing FreeBSD Isolated and Virtualized Hosts Using bsdinstall(8), zfs(8) and nfsd(8) Integrating as first-class features Institutionalizing FreeBSD Isolated and Virtualized Hosts Using bsdinstall(8), zfs(8) and nfsd(8) This example but this is not FreeBSD-exclusive Institutionalizing FreeBSD Isolated and Virtualized Hosts Using bsdinstall(8), zfs(8) and nfsd(8) jail(8) and bhyve(8) “guests” Application Binary Interface vs. Instructions Set Architecture Institutionalizing FreeBSD Isolated and Virtualized Hosts Using bsdinstall(8), zfs(8) and nfsd(8) The FreeBSD installer The best file system/volume manager available The Network File System Broad Motivations Virtualization! Containers! Docker! Zones! Droplets! More more more! My Motivations 2003: Jails to mitigate “RPM Hell” 2011: “bhyve sounds interesting...” 2017: Mitigating Regression Hell 2018: OpenZFS EVERYWHERE A Tale of Two Regressions Listen up. Regression One FreeBSD Commit r324161 “MFV r323796: fix memory leak in [ZFS] g_bio zone introduced in r320452” Bug: r320452: June 28th, 2017 Fix: r324162: October 1st, 2017 3,710 Commits and 3 Months Later June 28th through October 1st BUT July 27th, FreeNAS MFC Slips into FreeNAS 11.1 Released December 13th Fixed in FreeNAS January 18th 3 Months in FreeBSD HEAD 36 Days
    [Show full text]
  • GELI (8) Freebsd System Manager's Manual GELI (8) NAME Geli
    GELI (8) FreeBSD System Manager’s Manual GELI (8) NAME geli — control utility for cryptographic GEOM class SYNOPSIS To compile GEOM ELI into your kernel, place the following lines in your kernel configuration file: device crypto options GEOM_ELI Alternately, to load the GEOM ELI module at boot time, place the following line in your loader.conf(5): geom_eli_load="YES" Usage of the geli(8) utility: geli init [ -bPv][ -a algo ][ -i iterations ][ -K newkeyfile][ -l keylen ][ -s sectorsize ] prov geli label - an alias for init geli attach [ -dpv ][ -k keyfile ] prov geli detach [ -fl] prov . geli stop - an alias for detach geli onetime [ -d][ -a algo][ -l keylen][ -s sectorsize ] prov . geli setkey [ -pPv ][ -i iterations ][ -k keyfile][ -K newkeyfile][ -n keyno] prov geli delkey [ -afv ][ -n keyno] prov geli kill [ -av][ prov . ] geli backup [ -v] prov file geli restore [ -v] file prov geli clear [ -v] prov . geli dump [ -v] prov . geli list geli status geli load geli unload DESCRIPTION The geli utility is used to configure encryption on GEOM providers. The following is a list of the most important features: • Utilizes the crypto(9) framework, so when there is crypto hardware available, geli will make use of it automatically. • Supports many cryptographic algorithms (currently AES, Blowfish and 3DES). • Can create a key from a couple of components (user entered passphrase, random bits from a file, etc.). • Allows to encrypt the root partition - the user will be asked for the passphrase before the root file system is mounted. • The passphrase of the user is strengthened with: B. Kaliski, PKCS #5: Password-Based Cryptography Specification, Version 2.0.
    [Show full text]
  • ZFS Boot Environments Reloaded NLUUG ZFS Boot Environments Reloaded
    ZFS Boot Environments Reloaded NLUUG ZFS Boot Environments Reloaded Sławomir Wojciech Wojtczak [email protected] vermaden.wordpress.com twitter.com/vermaden https://is.gd/BECTL ntro !"#$%##%#& ZFS Boot Environments Reloaded NLUUG What is ZFS Boot Environment? Its bootable clone%sna(shot of the working system. What it is' !"#$%##%#& ZFS Boot Environments Reloaded NLUUG What is ZFS Boot Environment? Its bootable clone%sna(shot of the working system. ● In ZFS terminology its clone of the snapshot. ZFS dataset → ZFS dataset@snapshot → ZFS clone (origin=dataset@snapshot) What it is' !"#$%##%#& ZFS Boot Environments Reloaded NLUUG What is ZFS Boot Environment? Its bootable clone%sna(shot of the working system. ● In ZFS terminology its clone of the snapshot. ZFS dataset → ZFS dataset@snapshot → ZFS clone (origin=dataset@snapshot) ● In ZFS (as everywhere) sna(shot is read onl). What it is' !"#$%##%#& ZFS Boot Environments Reloaded NLUUG What is ZFS Boot Environment? Its bootable clone%sna(shot of the working system. ● In ZFS terminology its clone of the snapshot. ZFS dataset → ZFS dataset@snapshot → ZFS clone (origin=dataset@snapshot) ● In ZFS (as everywhere) sna(shot is read onl). ● In ZFS clone can be mounted read write (and you can boot from it). What it is' !"#$%##%#& ZFS Boot Environments Reloaded NLUUG What is ZFS Boot Environment? Its bootable clone%sna(shot of the working system. ● In ZFS terminology its clone of the snapshot. ZFS dataset → ZFS dataset@snapshot → ZFS clone (origin=dataset@snapshot) ● In ZFS (as everywhere) sna(shot is read onl). ● In ZFS clone can be mounted read write (and you can boot from it).
    [Show full text]
  • Smaller Flight Data Recorders{
    Available online at http://docs.lib.purdue.edu/jate Journal of Aviation Technology and Engineering 2:2 (2013) 45–55 Smaller Flight Data Recorders{ Yair Wiseman and Alon Barkai Bar-Ilan University Abstract Data captured by flight data recorders are generally stored on the system’s embedded hard disk. A common problem is the lack of storage space on the disk. This lack of space for data storage leads to either a constant effort to reduce the space used by data, or to increased costs due to acquisition of additional space, which is not always possible. File compression can solve the problem, but carries with it the potential drawback of the increased overhead required when writing the data to the disk, putting an excessive load on the system and degrading system performance. The author suggests the use of an efficient compressed file system that both compresses data in real time and ensures that there will be minimal impact on the performance of other tasks. Keywords: flight data recorder, data compression, file system Introduction A flight data recorder is a small line-replaceable computer unit employed in aircraft. Its function is recording pilots’ inputs, electronic inputs, sensor positions and instructions sent to any electronic systems on the aircraft. It is unofficially referred to as a "black box". Flight data recorders are designed to be small and thoroughly fabricated to withstand the influence of high speed impacts and extreme temperatures. A flight data recorder from a commercial aircraft can be seen in Figure 1. State-of-the-art high density flash memory devices have permitted the solid state flight data recorder (SSFDR) to be implemented with much larger memory capacity.
    [Show full text]
  • The Pillars of Lossless Compression Algorithms a Road Map and Genealogy Tree
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 6 (2018) pp. 3296-3414 © Research India Publications. http://www.ripublication.com The Pillars of Lossless Compression Algorithms a Road Map and Genealogy Tree Evon Abu-Taieh, PhD Information System Technology Faculty, The University of Jordan, Aqaba, Jordan. Abstract tree is presented in the last section of the paper after presenting the 12 main compression algorithms each with a practical This paper presents the pillars of lossless compression example. algorithms, methods and techniques. The paper counted more than 40 compression algorithms. Although each algorithm is The paper first introduces Shannon–Fano code showing its an independent in its own right, still; these algorithms relation to Shannon (1948), Huffman coding (1952), FANO interrelate genealogically and chronologically. The paper then (1949), Run Length Encoding (1967), Peter's Version (1963), presents the genealogy tree suggested by researcher. The tree Enumerative Coding (1973), LIFO (1976), FiFO Pasco (1976), shows the interrelationships between the 40 algorithms. Also, Stream (1979), P-Based FIFO (1981). Two examples are to be the tree showed the chronological order the algorithms came to presented one for Shannon-Fano Code and the other is for life. The time relation shows the cooperation among the Arithmetic Coding. Next, Huffman code is to be presented scientific society and how the amended each other's work. The with simulation example and algorithm. The third is Lempel- paper presents the 12 pillars researched in this paper, and a Ziv-Welch (LZW) Algorithm which hatched more than 24 comparison table is to be developed.
    [Show full text]
  • Introduzione Al Mondo Freebsd Corso Avanzato
    Introduzione al mondo FreeBSD corso Avanzato •Struttura •Installazione •Configurazione •I ports •Gestione •Netstudent http://netstudent.polito.it •E.Richiardone [email protected] •Novembre 2012 •CC-by http://creativecommons.org/licenses/by/3.0/it/ The FreeBSD project - 1 • E` un progetto software open • Lo scopo e` mantenere e sviluppare il sistema operativo FreeBSD • Nasce su CDROM come FreeBSD 1.0 nel 1993 • Deriva da un patchkit per 386BSD, eredita codice da UNIX versione Berkeley 1977 • Per problemi legali subisce un rallentamento, release 2.0 nel 1995 con codice royalty-free • Dalla release 4.0 (2000) assume la struttura che ha oggi • Disponibile per x86 32 e 64bit, ia64, MIPS, ppc, sparc... • La mascotte (Beastie) nasce nel 1984 The FreeBSD project - 2 • Erede di 4.4BSD (e` la stessa gente...) • Sistema stabile; sviluppo uniforme; codice molto chiaro, ordinato e ben commentato • Documentazione ufficiale ben curata • Licenza molto permissiva, spesso attrae aziende per progetti commerciali: • saltuariamente progetti collaborano con implementazioni ex-novo (i.e. Intel, GEOM, NDISwrapper, ZFS, GNU/Linux emulation) • Semplificazione di molte caratteristiche tradizionali UNIX Di cosa si tratta Il progetto FreeBSD include: • Un sistema base • Bootloader, kernel, moduli, librerie di base, comandi e utility di base, servizi tradizionali • Sorgenti completi in /usr/src (~500MB) • E` gia` completo (i.e. ipfw, ppp, bind, ...) • Un sistema di gestione per software aggiuntivo • Ports e packages • Documentazione, canali di assistenza, strumenti
    [Show full text]
  • The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on Iot Nodes in Smart Cities
    sensors Article The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on IoT Nodes in Smart Cities Ammar Nasif *, Zulaiha Ali Othman and Nor Samsiah Sani Center for Artificial Intelligence Technology (CAIT), Faculty of Information Science & Technology, University Kebangsaan Malaysia, Bangi 43600, Malaysia; [email protected] (Z.A.O.); [email protected] (N.S.S.) * Correspondence: [email protected] Abstract: Networking is crucial for smart city projects nowadays, as it offers an environment where people and things are connected. This paper presents a chronology of factors on the development of smart cities, including IoT technologies as network infrastructure. Increasing IoT nodes leads to increasing data flow, which is a potential source of failure for IoT networks. The biggest challenge of IoT networks is that the IoT may have insufficient memory to handle all transaction data within the IoT network. We aim in this paper to propose a potential compression method for reducing IoT network data traffic. Therefore, we investigate various lossless compression algorithms, such as entropy or dictionary-based algorithms, and general compression methods to determine which algorithm or method adheres to the IoT specifications. Furthermore, this study conducts compression experiments using entropy (Huffman, Adaptive Huffman) and Dictionary (LZ77, LZ78) as well as five different types of datasets of the IoT data traffic. Though the above algorithms can alleviate the IoT data traffic, adaptive Huffman gave the best compression algorithm. Therefore, in this paper, Citation: Nasif, A.; Othman, Z.A.; we aim to propose a conceptual compression method for IoT data traffic by improving an adaptive Sani, N.S.
    [Show full text]
  • The Illusion of Space and the Science of Data Compression And
    The Illusion of Space and the Science of Data Compression and Deduplication Bruce Yellin EMC Proven Professional Knowledge Sharing 2010 Bruce Yellin Advisory Technology Consultant EMC Corporation [email protected] Table of Contents What Was Old Is New Again ......................................................................................................... 4 The Business Benefits of Saving Space ....................................................................................... 6 Data Compression Strategies ....................................................................................................... 9 Data Compression Basics ....................................................................................................... 10 Compression Bakeoff .............................................................................................................. 13 Data Deduplication Strategies .................................................................................................... 16 Deduplication - Theory of Operation ....................................................................................... 16 File Level Deduplication - Single Instance Storage ................................................................. 21 Fixed-Block Deduplication ....................................................................................................... 23 Variable-Block Deduplication .................................................................................................. 24 Content-Aware Deduplication
    [Show full text]