Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Report: ZFS on Linux Performance Evaluation
Norbert Schramm
Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik Fakultät für Mathematik, Informatik und Naturwissenschaften Universität Hamburg
2016-03-30
Norbert Schramm Report: ZFS on Linux Performance Evaluation 1 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Agenda
1 Introduction
2 Comparing ZFS
3 ZFS on Linux
4 Lustre on ZFS
5 Comparison
Norbert Schramm Report: ZFS on Linux Performance Evaluation 2 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Goal of the Project
ZFS on Linux: Linux-Version of original ZFS by Sun stable-released in April 2013 benefits against traditional file systems supported by Lustre since May 2013 How good is ZFS on Linux Compared to other ZFS-Versions?
Norbert Schramm Report: ZFS on Linux Performance Evaluation 3 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Tested ZFS on 3 Operating Systems OpenIndiana (based on Illumos, Former Solaris) FreeBSD Linux (Ubuntu) Hardware Core-i5 2500k (4x3.3 GHz), 8 GB RAM, 1 TB WD Black Xeon E3110 (2x3.0 GHz), 8 GB RAM, 1 TB WD Black Xeon X5677 (4x3.46 GHz), 32 GB RAM, 136 GB 15K SAS-Drive Benchmark bonnie++
Norbert Schramm Report: ZFS on Linux Performance Evaluation 4 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Figure: Core i5, bonnie++, ZFS initial
Norbert Schramm Report: ZFS on Linux Performance Evaluation 5 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Ubuntu uses relatime also available on ZFS (initial: disabled)
Figure: Core i5, atime optimisation
Norbert Schramm Report: ZFS on Linux Performance Evaluation 6 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
bonnie++ Data Compression Rate: > 130x
Figure: Core i5, LZ4 Compression
Norbert Schramm Report: ZFS on Linux Performance Evaluation 7 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Figure: Core i5, Deduplication
Norbert Schramm Report: ZFS on Linux Performance Evaluation 8 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Figure: Core-i5, Compression % Deduplication
Norbert Schramm Report: ZFS on Linux Performance Evaluation 9 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Figure: Core i5, Comparison Write
Norbert Schramm Report: ZFS on Linux Performance Evaluation 10 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Comparing ZFS
Figure: Comparison 3 Processor Generations with LZ4 Compression
Norbert Schramm Report: ZFS on Linux Performance Evaluation 11 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
ZoL Analysis
recompiled Linux-Kernel with LOCK_STAT Using wrstat for detailed analysis lock_stat oprofile (debug kernel needed) /proc file system analyzed run of bonnie++
Norbert Schramm Report: ZFS on Linux Performance Evaluation 12 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Example: ZFS no Compression/Dedup
Figure: ZFS initial: RAM Usage
Norbert Schramm Report: ZFS on Linux Performance Evaluation 13 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Example: ZFS no Compression/Dedup
Figure: ZFS initial: CPU States
Norbert Schramm Report: ZFS on Linux Performance Evaluation 14 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Example: ZFS no Compression/Dedup
Figure: ZFS initial: Locks, when ARC is saturated
Norbert Schramm Report: ZFS on Linux Performance Evaluation 15 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: Preparation
Test Setup 3 Nodes GBit Uplink to central Switch Xeon X5560 4(8)x2.8 GHz, 12 GB RAM, 2TB HDD for ZFS nehalem1: MGS, MDS, OSS nehalem2, nehalem3: OSS Benchmark serial: bonnie++ parallel: ior (with mpi) Operating System CentOS 7
Norbert Schramm Report: ZFS on Linux Performance Evaluation 16 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: bonnie++
Figure: Bonnie++ Benchmark on Lustre on ZFS-Basis
Norbert Schramm Report: ZFS on Linux Performance Evaluation 17 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: bonnie++
Problem: Data transfered uncompressed over the Network
Figure: Network Limiting Problem
Norbert Schramm Report: ZFS on Linux Performance Evaluation 18 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: ior
6x ior (2 of each node) Dedup very good LZ4 fastests, Gzip slowest
Norbert Schramm Figure: Ior-BenchmarkReport: ZFS on auf Linux Lustre Performance auf Evaluation ZFS-Basis 19 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: ior
Bursty Traffic when ARC is full Compressratio of ior-data: ca. 3.9x more work on HDD
Figure: Network and HDD on nehalem1 while ior-run with ZFS-Compression
Norbert Schramm Report: ZFS on Linux Performance Evaluation 20 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Lustre on ZFS: ior
slow write performance thus parallel access txg_sync while no data is transfered ZIL is located on disk, slows disk down
Figure: Network and HDD on nehalem1 while ior-run without ZFS-Compression
Norbert Schramm Report: ZFS on Linux Performance Evaluation 21 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
ZoL is comparable to other versions (functionality, performance) actual Version: 0.6.5.6 Version 1.0 when zvols are implementet Uses a lot of Locks while caching, can possibly slow down separate Device for L2ARC and SLOG can improve performance Lustre can get benefits from ZFS (e.g. Compression)
Norbert Schramm Report: ZFS on Linux Performance Evaluation 22 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison
Thank you for your attention. Questions?
Norbert Schramm Report: ZFS on Linux Performance Evaluation 23 / 23