Virtualizing Io Through the Io Memory Management Unit (Iommu) Andy Kegel, Paul Blinzer, Arka Basu, Maggie Chan Asplos 2016 What This Tutorial Will and Will Not Cover
Total Page:16
File Type:pdf, Size:1020Kb
VIRTUALIZING IO THROUGH THE IO MEMORY MANAGEMENT UNIT (IOMMU) ANDY KEGEL, PAUL BLINZER, ARKA BASU, MAGGIE CHAN ASPLOS 2016 WHAT THIS TUTORIAL WILL AND WILL NOT COVER Definition of “IO” or “Device” or “IO Device” : ‒ Traditional IO includes GPU for graphics, NIC, storage controller, USB controller, etc. ‒ New IO (accelerators) includes general-purpose computation on a GPU (GPGPU), encryption accelerators, digital signal processors, etc. Two Parts in Virtualizing an IO Device ‒ Device specific: Virtual instances of device ‒ Virtual functions and Physical function in devices (PCIE® SR-IOV, MR-IOV) ‒ System defined: IO Memory Management Unit or IOMMU ‒ Virtualizing DMA accesses (Address Translation and Protection) ‒ Virtualizing Interrupts (Interrupt Remapping and Virtualizing) 2 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 WHAT THIS TUTORIAL WILL AND WILL NOT COVER Definition of “IO” or “Device” or “IO Device” : ‒ Traditional IO includes GPU for graphics, NIC, storage controller, USB controller, etc. ‒ New IO (accelerators) includes general-purpose computation on a GPU (GPGPU), encryption accelerators, digital signal processors, etc. Two Parts in Virtualizing an IO Device ‒ Device specific: Virtual instances of device ‒ Virtual functions and Physical function in devices (PCIE® SR-IOV, MR-IOV) ‒ System defined: IO Memory Management Unit or IOMMU ‒ Virtualizing DMA accesses (Address Translation and Protection) ‒ Virtualizing Interrupts (Interrupt Remapping and Virtualizing) Focus 3 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 AGENDA MOTIVATION & INTRODUCTION What is IOMMU? -- Andy Kegel USE CASES & Where can IOMMU help? -- Paul Blinzer DEMOSTRATION INTERNALS How does IOMMU work? -- Arka Basu, Maggie Chan RESEARCH Research Opportunities and Discussion – Arka Basu 4 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Core Core IO Device IO Device MMU MMU Memory 5 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Core Core IO Device IO Device Virtual Addresses MMU MMU Memory 6 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Core Core IO Device IO Device Virtual Protection Addresses Check MMU MMU Physical Addresses Memory 7 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check MMU MMU Physical Addresses Memory 8 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Setup Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check MMU MMU Physical Addresses Memory 9 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Setup Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check Physical DMA MMU MMU Addresses Request Physical Addresses Memory 10 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Setup Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check Physical DMA MMU MMU Addresses Request Physical Addresses Wrong location Memory 11 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Setup Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check Physical DMA MMU MMU Addresses Request Physical Addresses No protection from malicious devices --> “DMA Attack” (e.g., FinSpy) Wrong location No protection from buggy device driver Memory Side channel attack – leak information 12 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA BY IO NO SYSTEM VIRTUALIZATION Setup Device Driver Core Core IO Device IO Device Virtual Protection Addresses Check Physical DMA MMU MMU Addresses Request Physical Addresses No protection from malicious devices --> “DMA Attack” (e.g., FinSpy) Wrong location No protection from buggy device driver Memory Side channel attack – leak information Needs hardware enforced memory 13 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 protection MOTIVATION: VIRTUAL MACHINES ARE TRENDING Tremendous growth in virtualization in server Efficient access to IO under virtualization is important Source: IDC Server Virtualization, MCS 2012 14 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 BACKGROUND: TRANSLATIONS IN VIRTUALIZED SYSTEM Guest OS 0 Guest OS 1 Hypervisor (a.k.a. VMM) Hardware – CPU, Memory, IO 15 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 BACKGROUND: TRANSLATIONS IN VIRTUALIZED SYSTEM Guest Applications Guest Applications Guest Virtual Address (GVA) Guest OS 0 Guest OS 1 Hypervisor (a.k.a. VMM) Hardware – CPU, Memory, IO 16 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 BACKGROUND: TRANSLATIONS IN VIRTUALIZED SYSTEM Guest Applications Guest Applications Guest Virtual Address (GVA) Guest OS 0 Guest OS 1 Managed by Guest Physical Guest OS Address (GPA) Hypervisor (a.k.a. VMM) Hardware – CPU, Memory, IO 17 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 BACKGROUND: TRANSLATIONS IN VIRTUALIZED SYSTEM Guest Applications Guest Applications Guest Virtual Address (GVA) Guest OS 0 Guest OS 1 Managed by Guest Physical Guest OS Address (GPA) Hypervisor (a.k.a. VMM) Managed by System Physical VMM Address(SPA) Hardware – CPU, Memory, IO 18 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 BACKGROUND: TRANSLATIONS IN VIRTUALIZED SYSTEM Guest Applications Guest Applications Guest Virtual Address (GVA) Guest OS 0 Guest OS 1 Managed by Guest Physical Guest OS Address (GPA) Hypervisor (a.k.a. VMM) Managed by System Physical VMM Address(SPA) Hardware – CPU, Memory, IO Isolation across Guest OS => No access to (system) physical address from Guest OS 19 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Core Core IO Device IO Device MMU MMU Memory *SPA == “Physical Address” 20 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Guest OS 0 Guest OS 1 GVA Core Core IO Device IO Device GPA VMM MMU MMU SPA Memory *SPA == “Physical Address” 21 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Device Driver Setup No access to Physical Guest OS 0 Guest OS 1 Address GVA Core Core IO Device IO Device GPA VMM MMU MMU SPA Memory *SPA == “Physical Address” 22 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Device Driver Setup Guest OS 0 Guest OS 1 GVA Core Core IO Device IO Device GPA VMM Setup MMU MMU SPA Every DMA operation mediated by VMM Memory *SPA == “Physical Address” 23 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Device Driver Setup Guest OS 0 Guest OS 1 GVA Core Core IO Device IO Device GPA VMM Setup Physical DMA MMU MMU Addresses Operation SPA Every DMA operation mediated by VMM Memory *SPA == “Physical Address” 24 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Device Driver Setup Guest OS 0 Guest OS 1 GVA Core Core IO Device IO Device GPA VMM Setup Physical DMA MMU MMU Addresses Operation SPA Every DMA operation mediated by VMM Often ~30% performance overhead Memory *SPA == “Physical Address” 25 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL DMA IN VIRTUAL MACHINES VIRTUALIZED SYSTEM Device Driver Setup Guest OS 0 Guest OS 1 GVA Core Core IO Device IO Device GPA VMM Setup Physical DMA MMU MMU Addresses Operation SPA Every DMA operation mediated by VMM Often ~30% performance overhead Memory Virtual address translation for DMA *SPA == “Physical Address” 26 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 INTRODUCTION OF IOMMU: THE LOGICAL VIEW Core Core IO Device IO Device MMU MMU Memory 27 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 INTRODUCTION OF IOMMU: THE LOGICAL VIEW IOMMU Driver Sets up IOMMU hardware Core Core IO Device IO Device Hardware that MMU MMU intercepts DMA IOMMU transactions Key capabilities: 1. Memory protection for DMA 2. Virtual address translation for DMA Memory 28 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT NON-VIRTUALIZED SYSTEM Core Core IO Device IO Device MMU MMU Memory 29 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT NON-VIRTUALIZED SYSTEM Setup IRQ # + Core id Device Driver Core Core IO Device IO Device MMU MMU Memory 30 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT NON-VIRTUALIZED SYSTEM Setup IRQ # + Core id Device Driver Core Core IO Device IO Device APIC APIC IRQ # MMU MMU Memory 31 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT VIRTUALIZED SYSTEM Guest OS 0 Core Core IO Device IO Device VMM MMU MMU Memory 32 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT VIRTUALIZED SYSTEM Guest OS 0 Core Core IO Device IO Device IRQ # + Core i VMM Setup MMU MMU Memory 33 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT VIRTUALIZED SYSTEM Guest OS 0 Guest OS migration Core Core IO Device IO Device IRQ # + Core i VMM Setup MMU MMU Memory 34 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT VIRTUALIZED SYSTEM Guest OS 0 Guest OS migration Core Core IO Device IO Device IRQ # + Core i VMM Setup MMU MMU Memory 35 IOMMU TUTORIAL @ ASPLOS | 3RD APRIL 2016 MOTIVATION: TRADITIONAL IO INTERRUPT VIRTUALIZED SYSTEM Guest OS 0 Guest OS migration