RHEL 7 Performance Tuning Joe Mario Senior Principal Software Engineer Sept 22, 2016 1 RED HAT CONFIDENTIAL #rhconvergence Agenda ● The RH performance team ● Problem areas where we spend most of our time: ● System tuning ● Numa issues DividerBackup Slide ● RHEL 7 performance enhancements ● Where to learn more. ● Backup slides – if interest and time permits. 2 RED HAT CONFIDENTIAL #rhconvergence •Performance Engineering Team 30+ Engineers around the globe Striving for best performance for - Micro-Benchmarks - Applications/Benchmarks - Application Scaling - OS Scaling - New performance feature support - Partner support - Customer support ... 3 RED HAT CONFIDENTIAL #rhconvergence RHEL Platform(s) Performance Coverage Benchmarks Application Performance ● CPU – linpack, lmbench ● Linpack MPI, SPEC CPU, SPECjbb 05/13 ● Memory – lmbench, McCalpin STREAM ● AIM 7 – shared, filesystem, db, ● Disk IO – iozone, fio – SCSI, FC, iSCSI compute ● Filesystems – iozone, ext3/4, xfs, gfs2, ● Database: DB2, Oracle 11/12, gluster Sybase 15.x , MySQL, MariaDB, ● Networks – netperf – 10/40Gbit, PostgreSQL, Mongo Infiniband/RoCE, Bypass ● OLTP – Bare Metal, KVM, RHEV-M clusters – TPC-C/Virt ● Bare Metal, RHEL6/7 KVM ● DSS – Bare Metal, KVM, RHEV-M, ● White box AMD/Intel, with our OEM IQ, TPC-H/Virt partners ● SPECsfs NFS ● TPC-C, TPC-H based worloads for database testing ● SAP – SLCS, SD, HANA ● YCSB and SPECvirt for virtualizaton testing 4 RED HAT CONFIDENTIAL Red#rhconvergence Hat Confidential RHEL Performance Evolution RHEL5 RHEL6 RHEL7 RH Cloud Static Hugepages Transparent Tuned - RHEV tuned Hugepages throughput- profile CPU Sets performance Tuned - Choose (default) RHEL OSP7 Ktune on/of Profile Tuned, NUMA, Automatic NUMA- SR-IOV CPU Affinity NUMAD - balancing kernel (taskset) userspace scheduler RHEL Atomic Host, Atomic Ent NUMA Pinning cgroups Containers/Docker (numactl) OpenShift v3 irqbalance - Irqbalance - irqbalance NUMA enhanced NUMA enhanced CloudForms 5 RED HAT CONFIDENTIAL #rhconvergence Agenda Problem areas where we spend most of our time. First: System tuning – with “tuned” DividerBackup Slide 6 RED HAT CONFIDENTIAL #rhconvergence Performance Metrics Latency==Speed. Throughput==Bandwidth Latency – Speed Limit Throughput – Bandwidth - # lanes in Highway - Ghz of CPU, Memory PCI - Width of data path / cachelines - Small transfers, disable - Bus Bandwidth, QPI links, PCI 1-2-3 aggregation – TCP nodelay - Network 1 / 10 / 40 Gb – aggregation, NAPI - Dataplane optimization DPDK - Fiberchannel 4/8/16, SSD, NVME Drivers What is “tuned” ? • Tuning profile delivery mechanism • Red Hat ships tuned profiles that improve performance for many workloads...hopefully yours! • Customize your own profile. 8 RED HAT CONFIDENTIAL #rhconvergence Is your OS is optimally tuned? What is my system currently tuned to? •# tuned-adm active Current active profile: balanced How do I change my current tuning setting? •# tuned-adm profile network-latency 9 RED HAT CONFIDENTIAL #rhconvergence Tuned (cont) # tuned-adm list Available profiles: - balanced - desktop - latency-performance - network-latency - network-throughput - powersave - throughput-performance - virtual-guest - virtual-host Current active profile: balanced 10 RED HAT CONFIDENTIAL #rhconvergence Tuned Updates for RHEL7 ● Installed by default! ● Profiles updated for RHEL7 features and characteristics ● Profiles automatically set based on installation – Desktop/Workstation: balanced profile – Server/HPC: throughput-performance profile ● Optional hook/callout capability ● Concept of Inheritance (just like httpd.conf) 11 RED HAT CONFIDENTIAL #rhconvergence Tuned: Network Latency Performance Boost Performance Latency Network Tuned: 12 Latency (Microseconds) C-state lock improves determinism, reduces jitter reduces determinism, C-state lock improves 250 100 150 200 50 0 C6 C6 C3 C1 C0 Time (1-sec intervals) (1-sec Time RED HAT CONFIDENTIAL HAT RED #rhconvergence Max Tuned: Storage Performance Boost RHEL7 File System In Cache Perf Intel I/O (iozone - geoM 1m-4g, 4k-1m) 4500 4000 c e S 3500 / B 3000 M 2500 n i t 2000 u p 1500 h g 1000 u o r 500 h T 0 ext3 ext4 xfs gfs2 not tuned tuned Larger is better 13 RED HAT CONFIDENTIAL #rhconvergence throughput-performance (RHEL7 default) • governor=performance • energy_perf_bias=performance • min_perf_pct=100 • readahead=4096 • kernel.sched_min_granularity_ns = 10000000 • kernel.sched_wakeup_granularity_ns = 15000000 • vm.dirty_background_ratio = 10 • vm.swappiness=10 14 RED HAT CONFIDENTIAL #rhconvergence Tuned: Profile Inheritance (throughput) throughput-performance governor=performance energy_perf_bias=performance min_perf_pct=100 readahead=4096 kernel.sched_min_granularity_ns = 10000000 kernel.sched_wakeup_granularity_ns = 15000000 vm.dirty_background_ratio = 10 vm.swappiness=10 network-throughput net.ipv4.tcp_rmem="4096 87380 16777216" net.ipv4.tcp_wmem="4096 16384 16777216" net.ipv4.udp_mem="3145728 4194304 16777216" 15 RED HAT CONFIDENTIAL #rhconvergence Tuned: Profile Inheritance Parents throughput-performance balanced latency-performance Children network-throughput desktop network-latency virtual-host virtual-guest Your-Web Your-DB Your-Middleware 16 RED HAT CONFIDENTIAL | Joe Mario #rhconvergence Agenda Two hottest categories where we spend most of our time: Second: NUMA issues DividerBackup Slide 17 RED HAT CONFIDENTIAL #rhconvergence Typical NUMA System Node 0 Node 1 Node 0 RAM Node 1 RAM Core 0 L3 Core 1 Core 0 L3 Core 1 Cache Cache Core 2 Core 3 Core 2 Core 3 QPI links, IO, etc. QPI links, IO, etc. Node 2 Node 3 Node 2 RAM Node 3 RAM Core 0 L3 Core 1 Core 0 L3 Core 1 Cache Cache Core 2 Core 3 Core 2 Core 3 QPI links, IO, etc. QPI links, IO, etc. 18 RED HAT CONFIDENTIAL #rhconvergence Goal: Align process memory and CPU threads within nodes Before: processes use After: processes have cpus & memory from more “localized” cpu & multiple nodes. memory usage. Node 0 Node 1 Node 2 Node 3 Node 0 Node 1 Node 2 Node 3 Process 37 Process 29 Proc Proc Proc 19 Proc 37 Process 19 29 61 Process 61 19 RED HAT CONFIDENTIAL #rhconvergence Numa Due Diligence ● Know your hardware ● lstopo ● numactl --hardware ● Install adapters “close” to the CPU that will run the critical application ● When BIOS reports locality, irqbalance handles NUMA/IRQ affinity automatically. 20 RED HAT CONFIDENTIAL #rhconvergence Numa Due Diligence (cont) ● Know your application's memory usage ● numastat -mcv <proc_name> ● Understand where processes are executing and the memory they access ● Run “top”, then enter “f”, then select “Last used cpu” field ● ps -T -o pid,tid,psr,comm ● Use process placement tools ● numactl, taskset ● mbind, set_mempolicy, sched_setaffinity, pthread_setaffinity_np ● Virtualized environments – just as important! 21 RED HAT CONFIDENTIAL #rhconvergence numastat: per-node meminfo (new) # numastat -mczs Node 0 Node 1 Total ------ ------ ------ MemTotal 65491 65536 131027 MemFree 60366 59733 120099 MemUsed 5124 5803 10927 Active 2650 2827 5477 FilePages 2021 3216 5238 Active(file) 1686 2277 3963 Active(anon) 964 551 1515 AnonPages 964 550 1514 Inactive 341 946 1287 Inactive(file) 340 946 1286 Slab 380 438 818 SReclaimable 208 207 415 SUnreclaim 173 230 403 AnonHugePages 134 236 370 22 RED HAT CONFIDENTIAL #rhconvergence numastat – per-PID mode # numastat -c java (default scheduler – non-optimal) Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Node 3 Total ------------ ------ ------ ------ ------ ----- 57501 (java) 755 1121 480 698 3054 57502 (java) 1068 702 573 723 3067 57503 (java) 649 1129 687 606 3071 57504 (java) 1202 678 1043 150 3073 ------------ ------ ------ ------ ------ ----- Total 3674 3630 2783 2177 12265 # numastat -c java (numabalance close to opt) Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Node 3 Total ------------ ------ ------ ------ ------ ----- 56918 (java) 49 2791 56 37 2933 56919 (java) 2769 76 55 32 2932 56920 (java) 19 55 77 2780 2932 56921 (java) 97 65 2727 47 2936 ------------ ------ ------ ------ ------ ----- Total 2935 2987 2916 2896 11734 23 RED HAT CONFIDENTIAL #rhconvergence Visualize CPUs via lstopo 24 RED HAT CONFIDENTIAL #rhconvergence Visualize NUMA Topology: lstopo NUMA Node 0 NUMA Node 1 How can I visualize my system's NUMA PCI Devices topology in Red Hat Enterprise Linux? https://access.redhat.com/site/solutions/62879 25 RED HAT CONFIDENTIAL #rhconvergence NUMA layout via numactl # numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 4 8 12 16 20 24 28 32 36 node 0 size: 65415 MB node 0 free: 63482 MB node 1 cpus: 2 6 10 14 18 22 26 30 34 38 node 1 size: 65536 MB node 1 free: 63968 MB node 2 cpus: 1 5 9 13 17 21 25 29 33 37 node 2 size: 65536 MB node 2 free: 63897 MB node 3 cpus: 3 7 11 15 19 23 27 31 35 39 node 3 size: 65536 MB node 3 free: 63971 MB node distances: node 0 1 2 3 0: 10 21 21 21 1: 21 10 21 21 2: 21 21 10 21 3: 21 21 21 10 26 RED HAT CONFIDENTIAL #rhconvergence RHEL NUMA Scheduler ● RHEL6 ● numactl, numastat enhancements ● numad – usermode tool, dynamically monitor, auto-tune ● RHEL7 – auto numa balancing ● Moves tasks (threads or processes) closer to the memory they are accessing. ● Moves application data to memory closer to the tasks that reference it. ● A win for most apps. ● Enable / Disable ● sysctl kernel.numabalancing={0,1} 27 RED HAT CONFIDENTIAL #rhconvergence NUMA Performance – SPECjbb2005 on DL980 Westmere EX RHEL7 Auto-Numa-Balance SPECjbb2005 multi-instance - bare metal + kvm 8 socket, 80 cpu, 1TB mem 3500000 3000000 RHEL7 – No_NUMA 2500000 Auto-Numa-Balance – BM Auto-Numa-Balance