FreeBSD Enterprise Storage Polish BSD User Group Welcome 2020/02/11 FreeBSD Enterprise Storage Sławomir Wojciech Wojtczak [email protected] vermaden.wordpress.com twitter.com/vermaden bsd.network/@vermaden https://is.gd/bsdstg FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 What is Enterprise Storage? The wikipedia.org/wiki/enterprise_storage page tells nothing about enterprise. Actually just redirects to wikipedia.org/wiki/data_storage page. The other wikipedia.org/wiki/computer_data_storage page also does the same. The wikipedia.org/wiki/enterprise is just meta page with lin s. FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 Common Charasteristics o Enterprise Storage ● Category that includes ser$ices/products designed &or !arge organizations. ● Can handle !arge "o!umes o data and !arge num%ers o sim#!tano#s users. ● 'n$olves centra!ized storage repositories such as SA( or NAS de$ices. ● )equires more time and experience%expertise to set up and operate. ● Generally costs more than consumer or small business storage de$ices. ● Generally o&&ers higher re!ia%i!it'%a"aila%i!it'%sca!a%i!it'. FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 EnterpriCe or EnterpriSe? DuckDuckGo does not pro$ide search results count +, Goog!e search &or enterprice word gi$es ~ 1 )00 000 results. Goog!e search &or enterprise word gi$es ~ 1 000 000 000 results ,1000 times more). ● /ost dictionaries &or enterprice word sends you to enterprise term. ● Given the *+,CE o& many enterprise solutions it could be enterPRICE 0 ● 0 or enterpri$e as well +. ● When in doubt just use S $ersion 1 Enterprise. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 2020/02/11 Internal So!#tions - Filesystems ● UFS 1 classic%mature/traditional small memory &ootprint U('2 3lesytem. ○ UFS with So&t Updates (SU) allows Snapshots and dump(8)/restore(8) features. ○ UFS with Journaled So&t Updates (SU+J) with ultra fast fsck(8) process. ● /FS 1 modern pooled U('2 storage. ○ Stable 56S - based on FreeBSD Base System !FS i plementaiton. ○ 7atest 56S - based on !oL/!oF #penZFS repository (use FreeBSD $orts). ○ Differences - http://open-zfs.org/wiki/Feature_/lags - detailed infor ation. ● F01%E212 1 6reeBSD maintains BSD licensed 6AT%!2T8 3lesystem implementations. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 2020/02/11 Internal So!#tions - Frameworks ● GE34 1 6reeBSD's /odular Disk Transformation 6ramewor . ○ Access/control o& classes through use o& pro$iders/de$ices in /dev directory. ○ Pro$ides $arious storage related &eatures and utilites: ◦ Software %&'D(/%&'D)/%&'D)(/%&'D*/%&'D+ con,-urations. ◦ .ransparent encryption of underlyin- de/ices with 01"'/GDB1 (li2e LU3S). ◦ .ransparent ,lesyte 4ournalin- for &56 ,lesyste with GJ#U%5&". ◦ 17port block de/ice o/er networ2 with 01#8 0&.1 de/ices (li2e 5FS for bloc2). ● FUSE 1 BSD licensed 6US! 3lesystem implementation with :.;4 1 :.83 protocol support. ○ Details 1 https://freebsd.org/news/status/report-2019-04-2019-06.html#FUSE ○ Classic 1 NTFS3G%e>6AT%!2T2%!2T3%!2T<%26S%?6S%/TP%BT)6S%7inu>7@/%... ○ Virtual 1 s3fs/gphoto&s/mp=&s/rar8&s%sqlfs/ssh&s/union&s/wi ipediafs/... FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 2020/02/11 Internal So!#tions - 0vailabi!it' ● 50S1 1 ?ighly A$ailable Storage ,li e DRBD). ○ .ransparent stora-e across se/eral achines connected o/er .9$/'$ networ2. ○ 9an be understood as network based %&'D1 ( irror). ○ :ith FreeBSD's 9&%$ and devd(8) allows hi-hly a/ailable stora-e cluster. ● C0+* 1 #ommon Address )edundancy Protocol. ○ &llows ultiple hosts to share the sa e '$ address and <irtual =ost 'D (<=ID). ○ $ro/ides hi-h a/ailability for one or ore ser/ices. ○ $ro/ides >oatin- shared hi-hly a/ailable '$ address. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 U6S 2020/02/11 .FS with So t Updates (S.7 ● /ature classic fi!esystem with $ery sma!! memory ootprint. ● Supports 1+,4 nati$ely which allows ef3cient data deletion on SSDs. ● Supports read on!y snaphots ,not a$ailable in SUJ mode.. ● Neither 4ournaling ,SU4. nor So&t Updates ,SU. guarantees no data will be lost. ● They ,SU4%SU. ma e sure that 8!esystem metadata wi!! remain consistent. ● Ad$antage o& SU%SUJ is that 3lesystem can be mo#nted immediate!y a ter crash. ● U6S ,with SU%SU4. requires traditional fsck(8) in background to ma e it clean. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 U6S 2020/02/11 .FS with Jo#rna!ed So t Updates (S.97 ● Primary purpose was to e!iminate long 8!esystem check times with fsck(8). ● SU4 journal ,.sujournal. logs on!' two inconsistencies possible in SU+ ○ &llocated but unreferenced blocks. ○ Incorrectly hi-h link counts (includin- unreferenced inodes). ● 200 GB data on dis takes 1 second with SUJ under fsck(8). ○ Sa e dis2 with SU only (w/o Journalin-) ta2es ?@ inutes ()A(0 ti es ore). ● 10 1B data on disk takes 1 min#te with SUJ under fsck(8). ○ Sa e dis2 with SU only (w/o Journalin-) took appro7i ately A; hours. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 U6S 2020/02/11 Why UFS in 2020? ● U6S has both ragments and %!ocks+ ○ Files s aller than bloc2size can be stored in indi/idual fra- ents. ○ 9an create filesystem opti iBed for s all files () 3B) with (8 3B) blocksiBe. ● U6S can be grown on!ine with growfs(8) at boot time or anytime at system wor . ● So&t Updates is great &or apps/data%ases with their own log ,like PostgreSB7.+ ○ Usin- Journalin- would log e/erythin- twice - SU passes data throu-h. ● So&t Updates has interesting property regarding short !i"ed 6temporar'7 8!es+ ○ 9reate file + write data to it + delete it (in short ti e span). ○ 5either data nor etadata from this ,le will e/er touch filesystem. ● Netflix uses U6S in 8;8; &or their content storage on 6reeBSD. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 56S 2020/02/11 /FS - Zettabyte File System ● /odern pooled storage. ● Always consistent on1dis state 1 no fsck(8) needed. ● Snapshots ,read only. and clones ,read write.. ● Pro$ides end1to1end data integrity with checks#ms. ● ?a$e se! -hea!ing &eatures. ● ?a$e built1in redundancy. ● Scalable design and $'namic striping. ● @ariable bloc siDe. ● Builtin rep!ication%compression%encr'ption%$edup!ication. ● Possible to add read cache as 78A)# ,8nd 7e$el o& Adaptive Replacement Cache.. ● Possible to add write cache as 5'7 ,ZFS Intent Log.. ● Simple administration 1 two simple zfs(8) and zpool(8) commands. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 56S 2020/02/11 /FS - Common 4yths ● /yth E1 1 !ots o RAM is nee$e$. ○ %&8 is only cache for ZFS (called A%9) and its siBe can be tuned down to e/en 1(8B for example. ○ Use vfs.zfs.arc_min and vfs.zfs.arc_max in /boot/loader.conf ,le. ○ Use kern.maxvnodes in /etc/sysctl.conf file if needed to li it for sure. ○ ' ha/e used 2TB ZFS mirror with 5)?%&8 and it was roc2 stable for se/eral years. ● /yth E8 1 ECC +04 must be use$. ○ &ll filesystems benefit from EC9 R&8 and ZFS is no different here. ○ !FS without 199 R&8 is safer then other filesystems with EC9 R&8 (chec2sums). ● /yth E3 1 %ad or !aptop/$esktop. ○ Sin-le dis2 de/ices still benetif from snapshots/clones/chec2sums/co pression/deduplication. ○ !FS allows bulletproof up-rades with !FS Boot En/ironments - https://is.gd/BECTL - ore here. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 G!F/ 2020/02/11 GE34 Idea ● G!F/ is all about layers. ● 7i e ogres or onions. ● !>amples o& G!F/ layers below. ● A. 56S on G!7' ,encryption. on GPT ,pA. partition. ● B. 6AT38 on G!7' on G4FU)(A7 ,journaling. on /BR ,s1) partition%slice. B. FILESYSTEM FAT32 A. FILESYSTEM ZFS ENCRYPTION /dev/da0s1.journal.eli ENCRYPTION /dev/ada0p1.eli JOURNAL /dev/da0s1.journal GPT PARTITION /dev/ada0p1 MBR PARTITION /dev/da0s1 RAW DEVICE /dev/ada0 RAW DEVICE /dev/da0 FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 G!F/ 2020/02/11 GE34 Classes/*ro"iders (1/2) ● C0CHE /sbin/gcache Fptional read cache &or G!F/ )A'D3 graid3(8) class. ● C3:C01 /sbin/gconcat #oncat multiple de$ices into one $irtual de$ice. ● DBE /sbin/gbde G!F/ based disk encryption ,older.. ● E;, /sbin/geli Bloc de$ice disk encryption ,modern.. ● (01E /sbin/ggate* !>port block de$ice o$er networ ,like (6S &or block.. ● 93.+:0; /sbin/gjournal Generic bloc de$ice le$el journal pro$ider. ● ;0BE; /sbin/glabel /anual and automatic labeliDation pro$ider. ● 4,++3+ /sbin/gmirror /irror ,)A'DA. pro$ider. ● 43.:1<E+ /sbin/gmountver Queues '%F requests and waits &or pro$ider. ● 4.;1,*015 /sbin/gmultipath De$ice multipath con3guration pro$ider. FreeBSD Enterprise Storage Polish BSD User Group 'nternal Solutions 1 G!F/ 2020/02/11 GE34 Classes/*ro"iders (2/2) ● :3* /sbin/gnop Pro$ider to e>ample emulate dif&erent blocksiDe. ● *0+1 /sbin/gpart Partition ,BSD%/B)%GPT%0. G!F/ de$ice pro$iders. ● +0,D /sbin/graid So&tware RA'D management ,'ntel/4/icron%Si'%Promise%0.. ● +0,D= /sbin/graid3 RA'D= pro$ider. ● +0,D) sysutils/graid5 RA'D5 pro$ider ,a$ailable &rom 6reeBSD Ports.. ● SCHED /sbin/gsched Change scheduling policy o& requests going to pro$ider.