Lightning Talk FreeBSD storage performance

Benjeman Meekhof – AGLT2 USATLAS Tier2/Tier3 Workshop University of Michigan , Ann Arbor May 27th, 2008 Background y Config: Dell PE2950, 2 Perc6/e RAID controllers, 4x15PD in MD1000 shelves. y RAID setup: Two hardware RAID6 volumes,30 PD each, 128k stripe , readahad , write-cache on. Devices mfid0 and mfid2 in OS. y Hardware devices strippggped in software using “gstripe” – g for GEOM, disk framework enabling RAID and FS independent journals.

2 FreeBSD Storage Performance 5/27/2008

y By default, FreeBSD’s UFS2 filesystem uses “soft updates” to ensure filesystem consistency in the event of incomplete meta-data operations. y Meta-data = directories, files, other constructs…data about where the data is for any filesystem. If corrupted, filesystem is unusable. y Soft updates can be described as “ordered meta-data updates” - avoid writing meta- data twice. y More complicated but more efficient than a journaled system. y Meta-data changes are kept and sorted in memory before going to disk. On -disk meta-data is possibly several minutes behind running system in heavy usage. y journaling FS writes to journal on disk then to filesystem on disk - two hardware writes compared to only one using soft updates. y After crash, filesystem is consistent and usable but needs fsck in background to possibly free space marked as “used” (no metadata in filesystem, but the disk blocks were written with data and marked used) .

3 FreeBSD Storage Performance 5/27/2008 Tuning and Testing y Testing done with 128K software stripe over two hardware based volumes using 128K stripe in hardware : y Command: “gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2” y Bad results with default newfs (UFS2) settings y Tried many software stripe settings with no effect(-s)

FreeBSD striped UFS2 default settings (16384 blocksize) 350000 300000 250000 Initial write 200000 KB/s Rewrite 150000 Read 100000 Re-read 50000 0 12345678910111213 IOZone threads

4 FreeBSD Storage Performance 5/27/2008 Final Results y Finally the one key was to increase UFS2 blocksize to 65536 bytes with all other params equal: y gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 y newfs -U -b 65536 /dev/stripe/test y Strange read anomalies around 4 threads…why???

FreeBSD strippped UFS2, soft updates, 65536 FS blocksize

700000 Initial write 600000 Rewrite 500000 Read 400000 Re-read KB/s 300000 200000 100000 0 1 2 3 4 5 6 7 8 9 10 11 12 13

IOZone threads 5 FreeBSD Storage Performance 5/27/2008 Other Notes • GEOM journal was tested but did not perform better in generalhl than t he same filesystems usifding soft updates. • ZFS is built in but experimental in 7.0-Release – should be able to outperform UFS2 • FreeBSD people are supportive, and answers to postings on -ppyerformance list confirmed my results with UFS2 even on other hardware. • Basically, wait until ZFS is stable to see what FreeBSD can dihihdo in high-performance storage – thlhese results are the best you can hope for with UFS2 for now.

6 FreeBSD Storage Performance 5/27/2008