Logical Partitions on Many-core Processors Ramya Masti, Claudio Marforio, Kari Kostiainen, Claudio Soriente, Srdjan Capkun ETH Zurich ACSAC 2015 1 Infrastructure as a Service (IaaS) VM VM App App App App OS OS OS OS Hardware Hardware Hardware Personal platforms Shared cloud platform (IaaS) - Economies of scale - Shared resources - Security problems 2 Resource Partitioning VM VM App App OS OS Hardware Cloud platform with dedicated resources - Guaranteed performance - Secure against shared memory/CPU attacks - Example: IBM cloud 3 Resource Partitioning Logical Partition Memory VM VM Subset of a system’s resources that can run an OS/VM Processor C0 C1 C2 C3 Every logical partition needs dedicated resources Many logical partitions need platforms that have lots of resources! 4 Many-core Processors for Logical Partitioning Cores, caches VM Many-core ? processor VM Many simple cores Many logical partitions Designed for workloads that share data (no isolation!) Can many-core architectures support scalable logical partitioning? 5 Hypervisor Design Alternatives Traditional Distributed Centralized VM VM …… VM VM …… Hypervisor (H) H H …… H H VM VM …… C0 C1 ….. Cn C0 C1 ….. Cn C0 C1 ….. Cn VM Run-time VM-hypervisor interaction H - Increases hypervisor’s attack surface - Performance overhead C 6 Centralized Hypervisors Today Every core supports virtualization H VM VM …… - Hypervisor and VM modes C0 C1 C2 ….. Cn - Privileged instructions and memory addressing - Enables memory confinement Higher privilege Lower privilege But every core runs only in one mode! mode mode VM Unused functionality in every core H VM Solutions without processor virtualization 7 174 174IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 46, NO. 1, JANUARY 2011 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 46, NO. 1, JANUARY 2011 Fig. 1. Block diagram and tile architecture. Intel Single-chip Cloud Computer (SCC) Fig. 1. Block diagram and tile architecture. Memory Core controller Cache Network on chip (NoC) Messaging Buffer Network Interface Tiles Sources: Howard et al. IJSSC’10; The Future of Many-core Computing, Tim Mattson, Intel Labs 8 Fig. 2. Full-chip and tile micrograph and characteristics. quired for core-to-router communication. Each tile’s NI logic A. Core Architecture also features a Message Passing Buffer (MPB), or 16 KB of Fig. 2. Full-chip and tile micrograph and characteristics. on-die shared memory. The MPB is used to increase perfor- The core is an enhanced version of the second generation Pen- tium processor [4]. L1 instruction and data caches have been mance of a message passing programming model whereby cores upsized to 16 KB, over the previous 8 KB design, and sup- communicate through local shared memory. port 4-way set associativity and both write-through and write- Total die power is kept at a minimum by dynamically scaling quired for core-to-router communication. Each tile’s NI logic A. Core Architecture back modes for increased performance. Additionally, data cache both voltage and performance. Fine grained voltage change also features a Message Passing Buffer (MPB), or 16 KB of lineson-die have shared been modified memory. with The a MPB new status is used bit used to increase to mark perfor- The core is an enhanced version of the second generation Pen- commands are transmitted over the on-die network to a Voltage the content of the cache line as Message Passing Memory Type tium processor [4]. L1 instruction and data caches have been Regulator Controller (VRC). The VRC interfaces with two mance of a message passing programming model whereby cores (MPMT). The MPMT is introduced to differentiate between upsized to 16 KB, over the previous 8 KB design, and sup- on package voltage regulators. Each voltage regulator has a communicate through local shared memory. normal memory data and message passing data. The cache line’s port 4-way set associativity and both write-through and write- 4-rail output to supply 8 on-die voltage islands. Further power Total die power is kept at a minimum by dynamically scaling MPMT bit is determined by page table information found in the back modes for increased performance. Additionally, data cache savings are achieved through active frequency scaling at a tile core’sboth TLB voltage and must and be performance. setup properly Fine by the grained operating voltage system. change lines have been modified with a new status bit used to mark granularity. Frequency change commands are issued to a tile’s commandsThe Pentium are instruction transmitted set architecture over the on-die has been network extended to a to Voltage un-core logic, whereby frequency adjustments are processed. the content of the cache line as Message Passing Memory Type includeRegulator a new Controller instruction, (VRC). INVDMB, The used VRC to support interfaces software with two Tile performance is scalable from 300 MHz at 700 mV to 1.3 (MPMT). The MPMT is introduced to differentiate between managedon package coherency. voltage When regulators. executed, Each an INVDMB voltage instruction regulator has a GHz at 1.3 V. The on-chip network scales from 60 MHz at 550 invalidates all MPMT cache lines in a single clock cycle. Subse- normal memory data and message passing data. The cache line’s mV to 2.6 GHz at 1.3 V. The design target for nominal usage is quently,4-rail output reads or to writes supply to the 8 on-die MPMT voltage cache lines islands. are guaranteed Further power MPMT bit is determined by page table information found in the 1 GHz for tiles and 2 GHz for the 2-D network, when supplied tosavings miss and are data achieved will be fetched through or active written. frequency The instruction scaling ex- at a tile core’s TLB and must be setup properly by the operating system. by 1.1 V. Full-Chip and tile micrograph and characteristics are posesgranularity. the programmer Frequency to direct change control commands of cache are management issued to a tile’s The Pentium instruction set architecture has been extended to shown in Fig. 2. whileun-core in a logic, message whereby passing environment. frequency adjustments are processed. include a new instruction, INVDMB, used to support software Tile performance is scalable from 300 MHz at 700 mV to 1.3 managed coherency. When executed, an INVDMB instruction GHz at 1.3 V. The on-chip network scales from 60 MHz at 550 invalidates all MPMT cache lines in a single clock cycle. Subse- mV to 2.6 GHz at 1.3 V. The design target for nominal usage is quently, reads or writes to the MPMT cache lines are guaranteed 1 GHz for tiles and 2 GHz for the 2-D network, when supplied to miss and data will be fetched or written. The instruction ex- by 1.1 V. Full-Chip and tile micrograph and characteristics are poses the programmer to direct control of cache management shown in Fig. 2. while in a message passing environment. Intel SCC Architecture TILE Core Core Core - Asymmetric processor - Each core can run an independent OS NETWORK INTERFACE - Simple cores: no virtualization support Look Up Tables Look Up Tables (LUTs) - Determines the resources available to a core - Resources: other tiles, RAM and I/O Network on Chip All system resources are memory mapped 9 Intel SCC Address Translation Virtual address 0x87256199 CORE Memory Management Unit Physical address 0x72008127 NETWORK INTERFACE Look Up Tables (LUTs) System-wide address On-tile Off-tile LUT Registers On-tile I/O Memory at DRAM Memory Another Tile 10 Problem: Lack of Isolation TILE Core Core NETWORK INTERFACE Look Up Tables Every core can Network on Chip change its memory access configuration! Off-tile memory (DRAM) 11 Solution Intuition TILE TILE Master CoreCore Core Core Core X X X NETWORK INTERFACE NETWORK INTERFACE Look Up Tables Look Up Tables Network on Chip Master core configures memory confinement for all coresOff-tile memory (DRAM) 12 Contributions 1. Hardware change to Intel SCC that enables logical partitioning - isolation on Network on Chip - emulate in software 2. Custom hypervisor - small TCB and attack surface 3. Cloud architecture (IaaS) - implementation and evaluation 13 Cloud Architecture (IaaS) Computing Node Crypto engine (HSM) H VM VM HW-virtualized Intel SCC peripherals Management Service User Management Storage VM Management Service 14 Operation example: VM startup Helper Hypervisor VM Computing VM node Storage Core Master Core Core Service Crypto Encrypt and Intel SCC Engine install VM 1. Boot 2. Start 3. Fetch VM 4. Decrypt VM 5. Assign core and configure LUTs 6. Start 15 Recap of Main Properties No processor virtualization ◦ Beneficial for high number of cores Small TCB ◦ Hypervisor implementation size 3.4K LOC ◦ Tolerance to compromise of other cloud components Reduced VM interaction ◦ Small attack surface ◦ No virtualization overhead 16 Comparison No Processor Small TCB Reduced VM Virtualization Interaction Xen (>100K LOC) ✓ ✗ ✗ HypeBIOS (4K LOC) ✗ ✓ ✗ Merge? NoHype (>100K LOC) ✗ ✗ ✓ } Our solution (3.4K LOC) ✓ ✓ ✓ 17 Summary VM Many-core ? H processor VM Yes! (with minor modifications) Intel SCC case study - Isolation on Network on Chip (NoC) 18 Thank you! Ramya Masti, Claudio Marforio, Kari Kostiainen, Claudio Soriente, Srdjan Capkun 19.