Is Open Source good enough? A deep study of Swift and Ceph performance [email protected] 11/2013 Agenda • Self introduction • Ceph Block service performance • Swift Object Storage Service performance • Summary 2 Self introduction • Jiangang Duan (段建钢) • Open source Cloud Software team in Intel @ Shanghai • OpenStack related performance • Blog: http://software.intel.com/en-us/user/335367/track This is the team work: • Chendi Xue, Chen Zhang, Jian Zhang, Thomas Barnes, Xinxin Shu, Xiaoxi Chen, Yuan Zhou, Ziyuan Wang, etc. 3 Agenda • Self introduction • Ceph Block service performance • Swift Object Storage Service performance • Summary 4 Ceph Test Environment V V V V V V V V V M M … M M M … M M M … M 1 2 n 1 2 n 1 2 n Compute1 Compute2 … Compute5 10Gb NIC OSD1 OSD2 OSD3 OSD4 10x 1TB 7200rpm HDD 10x 1TB 7200rpm HDD 10x 1TB 7200rpm HDD 10x 1TB 7200rpm HDD 3x 200GB DC3700 3x 200GB DC3700 3x 200GB DC3700 3x 200GB DC3700 • Full 10Gb network • 4 Xeon UP server for Ceph cluster: • 16GB memory each • 10x 1TB SATA HDD for data through LSI9205 HBA (JBOD), each is parted into 1 partition for OSD daemon • 3x SSD for journal directly connected with SATA controller, 20GB for each OSD 5 Ceph Test-Bed Configuration: Software Ceph cluster Network tuning OS Ubuntu 12.10 • Enable Jumbo Frame (1496 bytes -> 8000 bytes) Kernel 3.6.6 XFS tuning Ceph Cuttle Fish 0.61.2 • osd mount options xfs = FileSystem XFS rw,noatime,inode64,logbsize=256k,delaylog • osd mkfs options xfs = -f -i size=2048 Client host Ceph Tuning OS Ubuntu 12.10 • Default replication setting (2 replicas), 4096 pgs. Kernel 3.5.0-22 • Enlarge read ahead size to 2048 Kb for OSD OpenStack Folsom (2012.2) disks. • Max Sync Interval 5s - > 10s Client VM • Large Queue and Large Bytes • OSD OP Thread 2 -> 20 OS Ubuntu 12.10 • Turn off FileStore Flusher Kernel 3.5.0-22 • Sync Flush False -> True (to avoid a OSD crash bug) 6 Methodology • Storage Interface: • RBD (aka volume) is attached to one virtual machine via QEMU RBD driver • Workload • FIO to simulate 4 IO pattern: • 64k Sequential Read/Write (SR and SW) with 60MB/sec max per VM • 4k Random Read/Write (RR and RW) with 100IOPS max per VM • Run Rules • Volume is pre-allocated before testing with dd • Drop OSD page cache before each measurement • Duration 400 secs warm-up, 600 secs data collection • 60GB per Volume, increase Stress VM number to get full load line • QoS requirement: • Less than 20ms latency for random read/write • Per VM throughput larger than 90% of the predefine threshold • 54MB/sec for SR/SW and 90IOPS for RR/RW 7 Random read performance 4KB random Read with CAP=100 IOPS,QD=2 120 5000 IOPS per VM total IOPS 4,5824500 99 99 99 99 99 4,336 100 95 4,051 4000 84 3,732 3500 80 3,341 75 68 3000 2,858 62 60 57 2500 total IOPStotal IOPS per VMperIOPS 1,980 2000 40 1500 990 1000 20 500 396 198 0 99 0 1 2 4 10 20 30 40 50 60 70 80 number of VM • Max measured value at ~4600 IOPS • When scale to 30VM, per VM IOPS drop to 95 For more complete information about performance and benchmark results, visit 8 Performance Test Disclosure Random write performance 4KB Random Write with CAP=100 IOPS,QD=2 120 50 throughtput latency 48 45 100 100 100 100 100 100 100 41 40 82 36 35 80 66 30 30 56 60 24 25 48 total IOPStotal IOPS per VMperIOPS 42 20 40 15 10 20 5 3 3 3 3 3 3 0 0 VM=1 VM=2 VM=4 VM=10 VM=20 VM=30 VM=40 VM=50 VM=60 VM=70 VM=80 number of VM • Peak at ~3291 IOPS • When scale to 40VM, per VM VW drop to 82 IOPS per VM For more complete information about performance and benchmark results, visit 9 Performance Test Disclosure Sequential read performance 64KB sequential read QD = 64, CAP=60MB/s 70 3000 per VM BW total BW 2,759 60 60 60 60 60 60 2,671 60 2,575 57 2500 2,428 2,263 50 49 43 2000 1,799 40 38 34 1500 30 1,200 total total BW (MB/s) per per VM BW (MB/s) 1000 20 600 500 10 240 120 0 60 0 1 2 4 10 20 30 40 50 60 70 80 The number of VM • Max measured value at ~2759MB/s • When scale to 40VM, per VM VW drop to 57MB/sec For more complete information about performance and benchmark results, visit 10 Performance Test Disclosure Sequential write performance 64KB Sequential Write QD = 64, CAP=60MB/s 70 1600 per VM BW total BW 1,487 1,487 60 60 60 60 1,457 60 1,424 1400 60 1,350 1,363 1,197 1200 50 45 1000 40 36 800 30 30 600 25 600 total total BW (MB/s) per per VM BW (MB/s) 21 20 17 400 240 10 200 120 60 0 0 1 2 4 10 20 30 40 50 60 70 80 The number of VM • Max at 1487MB/sec • When scale to 30VM, per VM BW drop to 45MB/sec per VM For more complete information about performance and benchmark results, visit 11 Performance Test Disclosure Latency analysis 20ms 4ms • Latency is strongly related to size of QD • Random is good enough. Sequential still has improve room. For more complete information about performance and benchmark results, visit 12 Performance Test Disclosure IO trace analysis for Sequential read • Ceph converts the logical sequential to random disk access • Read ahead can help, but not enough For more complete information about performance and benchmark results, visit 13 Performance Test Disclosure Full SSD performance • 4x 200GB SSD with 8 OSD (60GB data + 10GB journal each) • Good enough full SSD performance. • ~55K with 1ms latency; ~80K with 2ms, peak 85K with 3ms For more complete information about performance and benchmark results, visit 14 Performance Test Disclosure Ceph Data Summary In theory Max in theory Network workload pattern measureable throughput Consider QoS Disk throughput Efficiency throughput througput (IOPS and MB/s) (IOPS and MB/s) 4KB random read CAP=100 4582 2858 3600 31250 79% 4KB random write CAP=100 3357 3000 4000 31250 75% 64KB seq read QD=64 CAP=60 2759 2263 6400 4000 57% 64KB seq write QD=64 CAP=60 1487 1197 3200 4000 37% Assume 160MB/s for SR/SW and 90IOPS for RR and 200IOPS for RW and 40Gb Network BW limit total capacity total throughput 60GB Volume number 60GB Volume number HW (GB) (IOPS) consider capacity consider performance 40x 1TB with replica=2 20000 2800 333.3 30 3 x 200GB SSD with replica=2 150 85000 2.5 850 Consider Random IO only • Random performance is pretty good. • Sequential performance has room to do better. • Challenges and opportunities: • Throttle both BW and IOPS to better support multiple tenant case • Mix SSD and HDD together to achieve a balanced capacity/performance. For more complete information about performance and benchmark results, visit 15 Performance Test Disclosure Agenda • Self introduction • Ceph Block service performance • Swift Object Storage Service performance • Summary 16 H/W Configuration • Proxy Node x2 – CPU = 2 * Intel Xeon E5 2.6GHz (8C/16T) – Memory = 64GB, 8x 8G DDR3 1333MHz – NIC = Intel 82599 Dual port 10GbE – OS = CentOS 6.4 (2.6.32-358.14.1) – BIOS = Turbo on, EIST on, CPU C-state ON • Storage Node x10 – CPU = 1* Intel Xeon E3 3.5GHz (4C/8T) – Memory =16 GB, 4x 4G DDR3 1333MHz – NIC = 1 * Intel 82576 Quad port 1GbE (4*1Gb) – Disk = 12* 1TB 7200rpm SATA, 1x 64GB SSD (3Gb/s) for account/container zone – OS = CentOS 6.4 (2.6.32-358.14.1) • Network – 1x 10Gb to in/out of Proxy – 4x 1Gb for each storage node 17 Software Configuration • Swift 1.9.0 • Memcached Server • Swift Proxy Server – cache-size = 2048MB – max-concurrent-connections = 20480 – log-level = warning – workers = 64 – account/container-recheck = 80640s • Swift Account/Container Server – log-level = warning – workers = 16 • Swift Object Server – log-level = warning – workers = 32 • Swift replicator/auditor/updater – default 18 Test Methodology COSBench is an Intel developed Benchmark to measure Cloud Object Storage Service performance. https://github.com/intel-cloud/cosbench 19 Swift Performance Overview Object- Worker- Avg- #con x # obj RW-Mode Throughput Bottleneck Size Count ResTime -- -- -- ms op/s -- Read 640 51.77 12359.84 Proxy CPU = 100%, SN CPU = 72% 128KB Write 640 213.12 3001.65 Proxy CPU=47%, SN CPU = 96% 100x100 Read 160 699.3 228.78 Proxy CPU = 61%, SN CPU = 55%, Proxy NIC 9.6Gb/s 10MB Write 80 1020.8 78.37 Proxy CPU = 15%, SN CPU = 59%, Proxy NIC 9.6Gb/s Object- Worker- Avg- #con x # obj RW-Mode Throughput Bottleneck Size Count ResTime -- -- -- ms op/s -- Read 640 248.56 2574.78 Proxy CPU = 25% SN CPU = 52%, disk latency 22ms 10000*10000 128KB Write 80 62.75 1272.97 Proxy CPU = 21%, SN CPU = 57%, disk latency 18ms Read 160 699.17 228.83 Proxy CPU = 56%, SN CPU = 62%, Proxy NIC 9.6Gb/s 10000*100 10MB Write 320 4096.3 78.12 Proxy CPU = 16%, SN CPU = 81%, Proxy NIC 9.6Gb/s • For large object, bottleneck is front end network • 80% and 58% performance degradation in many-small-objects case For more complete information about performance and benchmark results, visit 20 Performance Test Disclosure Problem : large-scale test 128K read 80+% drops • Different IO pattern – large-scale : 34 KB per op – small-scale: 122KB per op • Blktrace shows lots of IO generated by XFS file system – Meta data cannot be cached with 10k * 10k objects – Frequent disk access for that info For more complete information about performance and benchmark results, visit 21 Performance Test Disclosure Option 1 : Using larger memory to cache Inodes – Set vfs_cache_pressure =1 and run “ls –R /srv/node” to preload the cache – Close the gap less than 10% although this is expensive option.