The Multikernel A new OS architecture for scalable multicore systems Andrew Baumann1 Paul Barham2 Pierre-Evariste Dagand3 Tim Harris2 Rebecca Isaacs2 Simon Peter1 Timothy Roscoe1 Adrian Schüpbach1 Akhilesh Singhania1 1 Systems Group, ETH Zurich 2 Microsoft Research, Cambridge 3 ENS Cachan Bretagne Systems Group | Department of Computer Science | ETH Zurich SOSP, 12th October 2009 Introduction How should we structure an OS for future multicore systems? I Scalability to many cores I Heterogeneity and hardware diversity 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 2 System diversity “Victoria Falls” 128 threads FB DIMMFB FB DIMM DIMM FB FBDIMM DIMM FB FB DIMM DIMM FB DIMM 16 cores 65x (two sockets) MCU MCU MCU MCU MCU MCU MCU MCU AMD Opteron (Istanbul) UltraSPARC T2 L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ processor L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ 64 threads Full Cross Bar eight cores Full Cross Bar 35x C0 C1 C2 C3 C4 C5 C6 C7 C0 C1 C2 C3 C4 C5 C6 C7 FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU FPU SPU SPU SPU SPU SPU SPU SPU SPU UltraSPARC® T1 processor 32 threads eight cores NIU Sun Niagara T2 14x Sys I/F PCIe (E-NET+) Buffer Switch Core UltraSPARC® IIIi NIU Sys I/F PCIe processor (Ethernet+) Buffer Switch Core 1x 2x 10 Power <100 W x8 @2. GHz Gigabit Ethernet 2x 10 Power <95 W x8 @ 2.0 GHz Intel Nehalem (Beckton) Gigabit Ethernet 2004 2005 2006 2007 2008 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 3 The interconnect matters Today’s 8-socket Opteron 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 4 The interconnect matters Tomorrow’s 8-socket Nehalem 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 5 The interconnectGDDR matters GDDR On-chip interconnects In-Order In-Order Multi-threaded Multi-threaded Wide SIMD Wide SIMD I$ D$ I$ D$ Fixed Function Fixed Fixed Function Fixed Display Interface Display Coherent L2 Cache Interface Display Memory ControllerMemory Memory ControllerMemory ControllerMemory Memory ControllerMemory In-Order In-Order Multi-threaded Multi-threaded Wide SIMD Wide SIMD Texture Logic Texture Texture Logic Texture I$ D$ I$ D$ System Interface System System Interface System PCIe DDR2 Controller 0 DDR2 Controller 1 SerDes SerDes PROCESSOR CACHE PCIe 0 Reg File L2 CACHE L-1I L-1D MAC/ P P P I-TLB D-TLB PHY 2 1 0 2D DMA UART, MDN TDN HPI, I2C, GbE 0 UDN IDN JTAG,SPI STN SWITCH Flexible GbE 1 I/O Flexible I/O PCIe 1 MAC/ XAUI 1 PHY MAC/ PHY SerDes SerDes DDR2 Controller 3 DDR2 Controller 2 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 6 Core diversity I Within a system: I Programmable NICs I GPUs I FPGAs (in CPU sockets) I On a single die: I Performance asymmetry I Streaming instructions (SIMD, SSE, etc.) I Virtualisation support 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 7 Summary I Increasing core counts, increasing diversity I Unlike HPC systems, cannot optimise at design time 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 8 I Proposal: structure the OS as a distributed system I Design principles: 1. Make inter-core communication explicit 2. Make OS structure hardware-neutral 3. View state as replicated The multikernel model I It’s time to rethink the default structure of an OS I Shared-memory kernel on every core I Data structures protected by locks I Anything else is a device 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 9 The multikernel model I It’s time to rethink the default structure of an OS I Shared-memory kernel on every core I Data structures protected by locks I Anything else is a device I Proposal: structure the OS as a distributed system I Design principles: 1. Make inter-core communication explicit 2. Make OS structure hardware-neutral 3. View state as replicated 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 9 Outline Introduction Motivation Hardware diversity The multikernel model Design principles The model Barrelfish Evaluation Case study: Unmap 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 10 I Decouples system structure from inter-core communication mechanism I Communication patterns explicitly expressed I Naturally supports heterogeneous cores, non-coherent interconnects (PCIe) I Better match for future hardware I . with cheap explicit message passing (e.g. Tile64) I . without cache-coherence (e.g. Intel 80-core) I Allows split-phase operations I Decouple requests and responses for concurrency I We can reason about it 1. Make inter-core communication explicit I All communication with messages (no shared state) 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 11 1. Make inter-core communication explicit I All communication with messages (no shared state) I Decouples system structure from inter-core communication mechanism I Communication patterns explicitly expressed I Naturally supports heterogeneous cores, non-coherent interconnects (PCIe) I Better match for future hardware I . with cheap explicit message passing (e.g. Tile64) I . without cache-coherence (e.g. Intel 80-core) I Allows split-phase operations I Decouple requests and responses for concurrency I We can reason about it 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 11 Message passing vs. shared memory: experiment Shared memory (move the data to the operation): I Each core updates the same memory locations (no locking) I Cache-coherence protocol migrates modified cache lines I Processor stalled while line is fetched or invalidated I Limited by latency of interconnect round-trips I Performance depends on data size (cache lines) and contention (number of cores) 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 12 Shared memory results 4 4-core AMD system ⇥ 12 SHM1 10 8 1000) × 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 13 Shared memory results 4 4-core AMD system ⇥ 12 SHM2 SHM1 10 8 1000) × 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 13 Shared memory results 4 4-core AMD system ⇥ 12 SHM4 SHM2 10 SHM1 8 1000) × 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 13 Shared memory results 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 8 1000) × 6 (no locking!) 4 Latency (cycles 2 Stalled cycles 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 13 Message passing vs. shared memory: experiment Message passing (move the operation to the data): I A single server core updates the memory locations I Each client core sends RPCs to the server I Operation and results described in a single cache line I Block while waiting for a response (in this experiment) 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 14 Message passing vs. shared memory: tradeoff 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 MSG1 8 1000) × 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 15 Message passing vs. shared memory: tradeoff 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 MSG8 8 MSG1 1000) × 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 15 Message passing vs. shared memory: tradeoff 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 MSG8 8 MSG1 1000) × 6 4 Latency (cycles Messaging 2 faster for: 4 cores 0 ≥ 2 4 6 8 10 12 14 16 4 cache lines ≥ Cores 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 15 Message passing vs. shared memory: tradeoff 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 MSG8 8 MSG1 1000) × Server 6 4 Latency (cycles 2 0 2 4 6 8 10 12 14 16 Cores Actual cost of update at server 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 15 Message passing vs. shared memory: tradeoff 4 4-core AMD system ⇥ 12 SHM8 SHM4 10 SHM2 SHM1 MSG8 8 MSG1 1000) × Server 6 4 “spare” cycles Latency (cycles if RPC was 2 split-phase 0 2 4 6 8 10 12 14 16 Cores Actual cost of update at server 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 15 I Adaptability to changing performance characteristics I Late-bind protocol and message transport implementations 2. Make OS structure hardware-neutral I Separate OS structure from hardware I Only hardware-specific parts: I Message transports (highly optimised / specialised) I CPU / device drivers 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 16 2. Make OS structure hardware-neutral I Separate OS structure from hardware I Only hardware-specific parts: I Message transports (highly optimised / specialised) I CPU / device drivers I Adaptability to changing performance characteristics I Late-bind protocol and message transport implementations 12.10.2009 The Multikernel: A new OS architecture for scalable multicore systems 16 I Required by message-passing model I Naturally supports domains that do not share memory I Naturally supports changes to the set of running cores I Hotplug, power management 3.