Linux kernel and LKP dynamics Wu Fengguang <[email protected]> December 17, 2016 Outline Linux 4.x changes 0day/LKP testing Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 2 / 16 Linux 4.0 1 Live patching 2 DAX - Direct Access, for persistent memory storage 3 KASan, kernel address sanitizer 4 "lazytime" option, for better update of file timestamps 5 Multiple lower layers in overlayfs 6 Support Parallel NFS server, default to NFS v4.2 7 dm-crypt scalability improvements DAX run fs on NVM, way fitting NVM into current Linux framework KASan one of the compiler aided runtime checkers overlayfs for Docker/container; misses NFS/CIFS support Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 3 / 16 Linux 4.1 1 Ext4 encryption support 2 Experimental cluster support for MD 3 Device mapper: new target that logs writes 4 Single user support 5 Virtual GEM driver for improved software rasterizers 6 Block device for persistent memory 7 Multiprotocol Label Switching 8 BPF programs can be attached to kprobes 9 ACPI support for the ARM64 architecture ext4 crypt convenient and efficient replacement for dm-crypt/eCryptfs BPF Berkeley Packet Filter => the universal in-kernel virtual machine => Linux’s DTrace Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 4 / 16 Linux 4.2 1 New driver amdgpu for modern AMD Radeon hardware 2 Add virtio gpu driver 3 Atomic modesetting API enabled by default 4 Stacking of security modules 5 Queued spinlocks become the default spinlock implementation 6 cgroup writeback support 7 Reintroduction of the H8/300 architecture dirty thrtl inode ownership tracking enforced (only) in CFQ iosched alternative: VFS layer throttling Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 5 / 16 Linux 4.3 1 The Ext3 filesystem has been removed 2 userfaultfd(), a system call for handling page-faults in user space 3 membarrier(), a system call for issuing memory barriers on a set of threads 4 New PID controller for limiting the number of PIDs in cgroups 5 Ambient capabilities 6 Introduce idle page tracking, a more precise way to track the memory being used by applications 7 Support for IPv6 Identifier Locator Addressing 8 Network light weight tunnels 9 Virtual Routing and Forwarding (Lite) support userfaultfd copy-on-access: quick QEMU migration; volatile ranges copy-on-write: sync dirtied pages IPv6 ILA net connection migration in datacenter Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 6 / 16 Linux 4.4 1 Faster and leaner loop device with Direct I/O and Asynchronous I/O support 2 3D support in virtual GPU driver 3 LightNVM adds support for Open-Channel SSDs 4 TCP listener handling completely lockless, making TCP servers faster and more scalable 5 Journalled RAID5 MD support 6 Unprivileged eBPF + persistent eBPF programs 7 perf + eBPF integration 8 Block polling support 9 mlock2() syscall allow users to request memory to be locked on page fault LightNVM many Vendors are supporting host-based control host OS: data placement, I/O scheduling, garbage collection SSD controller: managing flash chips, capacitors, etc. Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 7 / 16 Linux 4.5 1 Copy offloading with new copy_file_range(2) system call 2 Experimental PowerPlay supports brings high performance to the amdgpu driver 3 Btrfs free space handling scalability improvements 4 Support for GCC's Undefined Behavior Sanitizer (-fsanitize=undefined) 5 Forwarded Error Correction support in the device-mapper's verity target 6 Add MADV_FREE flag to madvise(2) 7 Better epoll multithread scalability 8 cgroup unified hierarchy is considered stable 9 Performance improvements for SO_REUSEPORT UDP sockets 10 Proper control of socket memory usage in the memory controller MADV_FREE lazy MADV_DONTNEED; for user-space memory allocator cgroup v2 keep kernel sane; leave war to systemd and docker Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 8 / 16 Linux 4.6 1 USB 3.1 SuperSpeedPlus (10 Gbps) support 2 Improve the reliability of the Out Of Memory task killer 3 Support for Intel memory protection keys 4 OrangeFS, a new distributed file system 5 Kernel Connection Multiplexor, a facility for accelerating application layer protocols 6 802.1AE MAC-level encryption (MACsec) 7 BATMAN V protocol 8 dma-buf: new ioctl to manage cache coherency between CPU and GPU 9 OCFS2 online inode checker 10 Support for cgroup namespaces 11 Add support for the pNFS SCSI layout MPX handy for user space to apply restrictions to large ranges of memory KCM atomic messages over TCP; uses BPF Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 9 / 16 Linux 4.7 1 Support for Radeon RX480 GPUs 2 Parallel directory lookups 3 New 'schedutil" frequency governor 4 Histograms of events in ftrace 5 perf trace calls stack 6 Allow BPF programs to attach to tracepoints 7 EFI 'Capsule' firmware updates 8 Support for creating virtual USB Device Controllers in USB/IP 9 Android's sync_file fencing mechanism considered stable 10 LoadPin, a security module to restrict the origin of kernel modules schedutil towards Energy Aware Scheduling Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 10 / 16 FPGA Reconfigurable computing for in memory database processing search intelligent network acceleration, DPDK machine and deep learning 4G/5G base station Video analytics blockchain other analytic algorithms.... half/half interests from embedded/server Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 11 / 16 FPGA: ways to widespread 1 standalization FPGA mgr Enumeration Config and assignment Orchestration => FPGA IP app store? 2 ease of use C/C++ C based (OpenCL) VHDL/HDL ... Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 12 / 16 0day/LKP testing: numbers The most comprehensive Linux test infrastructure. 80 servers 700+ kernel trees 2000 runtime test jobs 36000 build tests per day 150000 runtime tests per day =) 800 build reports per month 60 runtime reports per month Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 13 / 16 0day/LKP testing: principles 0 effort for developers shift left: test git pushes and LKML patches aim at fixing bugs: bisect and report to the relevant people up to 1000 levarage ratio: merge to test 100% test machines utilization: cyclic jobs get the most out of test runs: dozens of monitors, monitor boot/functional/throughput/latency/power etc. regressions at same time Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 14 / 16 0day/LKP testing: current status build tests: mature full automation near 100% build and static check coverage v4.9-rc2: 0 build warnings (x86_64 allmodconfig); Android: 40,000 warnings. runtime tests: there are rooms to improve auto reporting merge failures bisect effectiveness service stability ... costly and challenging to increase coverage Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 15 / 16 Thank you! Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 16 / 16.