Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Report: ZFS on Linux Performance Evaluation Norbert Schramm Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik Fakultät für Mathematik, Informatik und Naturwissenschaften Universität Hamburg 2016-03-30 Norbert Schramm Report: ZFS on Linux Performance Evaluation 1 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Agenda 1 Introduction 2 Comparing ZFS 3 ZFS on Linux 4 Lustre on ZFS 5 Comparison Norbert Schramm Report: ZFS on Linux Performance Evaluation 2 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Goal of the Project ZFS on Linux: Linux-Version of original ZFS by Sun stable-released in April 2013 benefits against traditional file systems supported by Lustre since May 2013 How good is ZFS on Linux Compared to other ZFS-Versions? Norbert Schramm Report: ZFS on Linux Performance Evaluation 3 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Tested ZFS on 3 Operating Systems OpenIndiana (based on Illumos, Former Solaris) FreeBSD Linux (Ubuntu) Hardware Core-i5 2500k (4x3.3 GHz), 8 GB RAM, 1 TB WD Black Xeon E3110 (2x3.0 GHz), 8 GB RAM, 1 TB WD Black Xeon X5677 (4x3.46 GHz), 32 GB RAM, 136 GB 15K SAS-Drive Benchmark bonnie++ Norbert Schramm Report: ZFS on Linux Performance Evaluation 4 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Figure: Core i5, bonnie++, ZFS initial Norbert Schramm Report: ZFS on Linux Performance Evaluation 5 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Ubuntu uses relatime also available on ZFS (initial: disabled) Figure: Core i5, atime optimisation Norbert Schramm Report: ZFS on Linux Performance Evaluation 6 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS bonnie++ Data Compression Rate: > 130x Figure: Core i5, LZ4 Compression Norbert Schramm Report: ZFS on Linux Performance Evaluation 7 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Figure: Core i5, Deduplication Norbert Schramm Report: ZFS on Linux Performance Evaluation 8 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Figure: Core-i5, Compression % Deduplication Norbert Schramm Report: ZFS on Linux Performance Evaluation 9 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Figure: Core i5, Comparison Write Norbert Schramm Report: ZFS on Linux Performance Evaluation 10 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Comparing ZFS Figure: Comparison 3 Processor Generations with LZ4 Compression Norbert Schramm Report: ZFS on Linux Performance Evaluation 11 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison ZoL Analysis recompiled Linux-Kernel with LOCK_STAT Using wrstat for detailed analysis lock_stat oprofile (debug kernel needed) /proc file system analyzed run of bonnie++ Norbert Schramm Report: ZFS on Linux Performance Evaluation 12 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Example: ZFS no Compression/Dedup Figure: ZFS initial: RAM Usage Norbert Schramm Report: ZFS on Linux Performance Evaluation 13 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Example: ZFS no Compression/Dedup Figure: ZFS initial: CPU States Norbert Schramm Report: ZFS on Linux Performance Evaluation 14 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Example: ZFS no Compression/Dedup Figure: ZFS initial: Locks, when ARC is saturated Norbert Schramm Report: ZFS on Linux Performance Evaluation 15 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: Preparation Test Setup 3 Nodes GBit Uplink to central Switch Xeon X5560 4(8)x2.8 GHz, 12 GB RAM, 2TB HDD for ZFS nehalem1: MGS, MDS, OSS nehalem2, nehalem3: OSS Benchmark serial: bonnie++ parallel: ior (with mpi) Operating System CentOS 7 Norbert Schramm Report: ZFS on Linux Performance Evaluation 16 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: bonnie++ Figure: Bonnie++ Benchmark on Lustre on ZFS-Basis Norbert Schramm Report: ZFS on Linux Performance Evaluation 17 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: bonnie++ Problem: Data transfered uncompressed over the Network Figure: Network Limiting Problem Norbert Schramm Report: ZFS on Linux Performance Evaluation 18 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: ior 6x ior (2 of each node) Dedup very good LZ4 fastests, Gzip slowest Norbert Schramm Figure: Ior-BenchmarkReport: ZFS on auf Linux Lustre Performance auf Evaluation ZFS-Basis 19 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: ior Bursty Traffic when ARC is full Compressratio of ior-data: ca. 3.9x more work on HDD Figure: Network and HDD on nehalem1 while ior-run with ZFS-Compression Norbert Schramm Report: ZFS on Linux Performance Evaluation 20 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Lustre on ZFS: ior slow write performance thus parallel access txg_sync while no data is transfered ZIL is located on disk, slows disk down Figure: Network and HDD on nehalem1 while ior-run without ZFS-Compression Norbert Schramm Report: ZFS on Linux Performance Evaluation 21 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison ZoL is comparable to other versions (functionality, performance) actual Version: 0.6.5.6 Version 1.0 when zvols are implementet Uses a lot of Locks while caching, can possibly slow down separate Device for L2ARC and SLOG can improve performance Lustre can get benefits from ZFS (e.g. Compression) Norbert Schramm Report: ZFS on Linux Performance Evaluation 22 / 23 Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Thank you for your attention. Questions? Norbert Schramm Report: ZFS on Linux Performance Evaluation 23 / 23.