Linux kernel and LKP dynamics

Wu Fengguang

December 17, 2016 Outline

Linux 4.x changes

0day/LKP testing

Wu Fengguang ( OTC) and LKP dynamics 2016 Intel OTC 2 / 16 Linux 4.0

1 Live patching 2 DAX - Direct Access, for persistent memory storage 3 KASan, kernel address sanitizer 4 "lazytime" option, for better update of file timestamps 5 Multiple lower layers in 6 Support Parallel NFS server, default to NFS v4.2 7 dm-crypt scalability improvements

DAX run fs on NVM, way fitting NVM into current Linux framework KASan one of the aided runtime checkers overlayfs for /container; misses NFS/CIFS support

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 3 / 16 Linux 4.1

1 encryption support 2 Experimental cluster support for MD 3 : new target that logs writes 4 Single user support 5 Virtual GEM driver for improved software rasterizers 6 Block device for persistent memory 7 Multiprotocol Label Switching 8 BPF programs can be attached to kprobes 9 ACPI support for the ARM64 architecture

ext4 crypt convenient and efficient replacement for dm-crypt/eCryptfs BPF => the universal in-kernel => Linux’s DTrace

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 4 / 16 Linux 4.2

1 New driver amdgpu for modern AMD Radeon hardware 2 Add virtio gpu driver 3 Atomic modesetting API enabled by default 4 Stacking of security modules 5 Queued spinlocks become the default spinlock implementation 6 cgroup writeback support 7 Reintroduction of the H8/300 architecture

dirty thrtl ownership tracking enforced (only) in CFQ iosched alternative: VFS layer throttling

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 5 / 16 Linux 4.3

1 The filesystem has been removed 2 userfaultfd(), a for handling page-faults in 3 membarrier(), a system call for issuing memory barriers on a set of threads 4 New PID controller for limiting the number of PIDs in 5 Ambient capabilities 6 Introduce idle page tracking, a more precise way to track the memory being used by applications 7 Support for IPv6 Identifier Locator Addressing 8 Network light weight tunnels 9 Virtual Routing and Forwarding (Lite) support

userfaultfd copy-on-access: quick QEMU migration; volatile ranges copy-on-write: dirtied pages IPv6 ILA net connection migration in datacenter

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 6 / 16 Linux 4.4

1 Faster and leaner loop device with Direct I/O and Asynchronous I/O support 2 3D support in virtual GPU driver 3 LightNVM adds support for -Channel SSDs 4 TCP listener handling completely lockless, making TCP servers faster and more scalable 5 Journalled RAID5 MD support 6 Unprivileged eBPF + persistent eBPF programs 7 + eBPF integration 8 Block polling support 9 mlock2() syscall allow users to request memory to be locked on page fault

LightNVM many Vendors are supporting host-based control host OS: data placement, I/O , garbage collection SSD controller: managing flash chips, capacitors, etc.

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 7 / 16 Linux 4.5

1 Copy offloading with new copy_file_range(2) system call 2 Experimental PowerPlay supports brings high performance to the amdgpu driver 3 free space handling scalability improvements 4 Support for GCC's Undefined Behavior Sanitizer (-fsanitize=undefined) 5 Forwarded Error Correction support in the device-mapper's verity target 6 Add MADV_FREE flag to madvise(2) 7 Better multithread scalability 8 cgroup unified hierarchy is considered stable 9 Performance improvements for SO_REUSEPORT UDP sockets 10 Proper control of socket memory usage in the memory controller

MADV_FREE lazy MADV_DONTNEED; for user-space memory allocator cgroup v2 keep kernel sane; leave war to and docker

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 8 / 16 Linux 4.6

1 USB 3.1 SuperSpeedPlus (10 Gbps) support 2 Improve the reliability of the task killer 3 Support for Intel keys 4 OrangeFS, a new distributed 5 Kernel Connection Multiplexor, a facility for accelerating application layer protocols 6 802.1AE MAC-level encryption (MACsec) 7 BATMAN V protocol 8 dma-buf: new to manage cache coherency between CPU and GPU 9 OCFS2 online inode checker 10 Support for cgroup namespaces 11 Add support for the pNFS SCSI layout

MPX handy for user space to apply restrictions to large ranges of memory KCM atomic messages over TCP; uses BPF

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 9 / 16 Linux 4.7

1 Support for Radeon RX480 GPUs 2 Parallel directory lookups 3 New 'schedutil" frequency governor 4 Histograms of events in 5 perf trace calls stack 6 Allow BPF programs to attach to tracepoints 7 EFI 'Capsule' firmware updates 8 Support for creating virtual USB Device Controllers in USB/IP 9 Android's sync_file fencing mechanism considered stable 10 LoadPin, a security module to restrict the origin of kernel modules

schedutil towards Energy Aware Scheduling

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 10 / 16 FPGA

Reconfigurable for in memory database processing search intelligent network acceleration, DPDK machine and deep learning 4G/5G base station Video analytics blockchain other analytic algorithms....

half/half interests from embedded/server

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 11 / 16 FPGA: ways to widespread

1 standalization FPGA mgr Enumeration Config and assignment Orchestration => FPGA IP app store?

2 ease of use /C++ C based (OpenCL) VHDL/HDL ...

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 12 / 16 0day/LKP testing: numbers

The most comprehensive Linux test infrastructure.

80 servers 700+ kernel trees 2000 runtime test jobs

36000 build tests per day 150000 runtime tests per day

=⇒ 800 build reports per month 60 runtime reports per month

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 13 / 16 0day/LKP testing: principles

0 effort for developers shift left: test git pushes and LKML patches aim at fixing bugs: bisect and report to the relevant people

up to 1000 levarage ratio: merge to test 100% test machines utilization: cyclic jobs get the most out of test runs: dozens of monitors, monitor boot/functional/throughput//power etc. regressions at same time

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 14 / 16 0day/LKP testing: current status

build tests: mature full automation near 100% build and static check coverage v4.9-rc2: 0 build warnings (x86_64 allmodconfig); Android: 40,000 warnings. runtime tests: there are rooms to improve auto reporting merge failures bisect effectiveness service stability ... costly and challenging to increase coverage

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 15 / 16 Thank you!

Wu Fengguang (Intel OTC) Linux kernel and LKP dynamics 2016 Intel OTC 16 / 16