IOMMU protection against I/O attacks: A vulnerability and a proof-of-concept Benoît Morgan, Eric Alata, Vincent Nicomette, Mohamed Kaâniche
To cite this version:
Benoît Morgan, Eric Alata, Vincent Nicomette, Mohamed Kaâniche. IOMMU protection against I/O attacks: A vulnerability and a proof-of-concept. Journal of the Brazilian Computer Society, Springer Verlag, 2018, 24, pp.Article number: 2. 10.1186/s13173-017-0066-7. hal-02877143
HAL Id: hal-02877143 https://hal.archives-ouvertes.fr/hal-02877143 Submitted on 22 Jun 2020
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Morgan et al.
RESEARCH IOMMU protection against I/O attacks: A vulnerability and a proof-of-concept
BenoˆıtMorgan1,2*, Eric´ Alata1,2, Vincent Nicomette1,2 and Mohamed Kaˆaniche1
performance levels and hardware services, more so- phisticated architectures have emerged with multiple input/output communication channels. In particular, normalized communication buses have been specified Abstract to allow tier manufacturers to complement bare ar- I/O attacks have received increasing attention chitectures with complex peripherals. Then, to relieve during the last decade. These attacks are the host processor from performing certain data copies performed by malicious peripherals that make read and operations, DMA cycles have been added to these or write accesses to DRAM memory or to memory external buses, allowing peripherals to perform read- embedded in other peripherals, through DMA /write accesses to other peripherals and RAM memory (Direct Memory Access) requests. Some protection segments. These communication channels raise serious mechanisms have been implemented in modern security concerns as they offer opportunities to attack- architectures to face these attacks. A typical ers to corrupt the system and the hosted applications example is the IOMMU (Input Output Memory using some malicious peripherals. To cope with such Management Unit). However, such mechanisms attacks, also called I/O attacks, some hardware pro- may not be properly configured and used by the tection components, such as the IOMMU, have been firmware and the operating system. This paper included in modern computers. describes a design weakness that we discovered in In order to really take advantage of these security the configuration of an IOMMU and a possible components, they have to be properly configured and exploitation scenario that would allow a malicious activated by the firmware and the kernel at boot time. peripheral to bypass the underlying protection The security of the boot process is crucial, as weakness mechanism. The exploitation scenario is at this stage may lead to a serious security flaw, de- implemented for Intel architectures, with a PCI spite the reliable design of these components. To the Express peripheral FPGA, based on Intel best of our knowledge, the security of the boot pro- specifications and Linux source code analysis. cess has not been thoroughly investigated in the liter- Finally, as a proof of concept, a Linux rootkit ature. This paper focusses on the security analysis of based on the attack presented in this paper is the IOMMU activation process at boot time. In par- implemented. ticular, it is shown, that even if this component has Keywords: Security; IOMMU; Firmware; Linux; been introduced 10 years ago, some serious security Vulnerability; Attack concerns may be raised about its actual efficiency to prevent malicious I/O attacks. Briefly, this paper high- lights a novel attack scenario that is related to the fact that the IOMMU configuration tables are initialized in 1 Introduction a DRAM region which is not protected from DMA Historically, early personal computers and their pe- at startup. As a consequence, a malicious peripheral ripherals were mostly designed and built by the same may modify these tables just before the activation of company. The peripherals used to be much less com- the IOMMU by the hardware. To illustrate the feasibil- plex than today (microcode, firmwares, etc.) and the ity of this scenario, a proof-of-concept is implemented processor manufacturers used to trust the peripher- and presented in this paper. als. However, with the increasing demand for higher A preliminary description of the vulnerabilities and *Correspondence: [email protected] the exploitation scenario of the IOMMU was presented 1Laboratoire d’Analyse et d’Architecture des Syst`emes(LAAS-CNRS), 7 in [1]. In this paper, more technical details are pro- avenue du Colonel Roche, 31400 Toulouse, France 2INSA Toulouse, 135 avenue de Rangueil, 31400 Toulouse, France vided, in particular, regarding the related work, the Full list of author information is available at the end of the article description of the proof-of-concept, and the experi- Morgan et al. Page 2 of 10
ment carried out to illustrate the IOMMU vulnera- CPU bility and its exploitation. The potential impact of the
Host bridge identified vulnerability and the main limitations are {0x0; 0xfff} [00:00.0] RAM also discussed.
Endpoint Endpoint This paper is organized as follows. The next two DMAR 0 DMAR 2 [01:00.0] [00.01.0] Sections describe fundamental components of the ar- {0x1000; 0x1fff} DMAR 1 {0x2000; 0x2fff} {01:yy.z} chitecture involved in the identified design weakness. {0x1000; 0x1fff} RAM data Root Complex Section 2 presents some basic technical background RAM read command about PCI Express bus and communications that are {02-03:yy.z} Read(0xffc, 03:00.0) {0x3000; 0x4fff} Switch Read completion (03:00.0, 00:00.0) used to perform DMA, while Section 3 presents the {addr1; addr2} Device address range IOMMU component as well as some of its internals, {xx:yy.z} Bridge id range {02:yy.z} {03:yy.z} that are necessary for the reader to understand the {0x3000; 0x3fff} {0x4000; 0x4fff} vulnerability and its potential exploitation. Section 4 Endpoint Endpoint briefly presents some examples of I/O attacks. Section [02:00.0] [03:00.0] 5 describes the vulnerability that we discovered in the {0x3000; 0x3fff}{0x4000; 0x4fff} configuration of the IOMMU and a scenario illustrat- Figure 1 Memory read message routing ing its possible exploitation. Section 7 proposes some countermeasures to cope with this vulnerability. Fi- nally, Section 8 concludes and discusses future work. contains a destination address and a device requester 2 Technical background id: the destination of the corresponding memory read This section presents basic background concepts re- completion (response) is the associated requester id. lated to the PCI Express bus and communications that PCI Express messages are therefore routed by address are useful to understand the rest of the paper. or id. To route the messages correctly, the bridges are configured by the host to know which ids and mem- 2.1 PCI Express bus ory ranges are responsible for downstream communica- Several bus specifications like Industry Standard tions. Figure 1 illustrates the routing of a memory read Architecture or Peripheral Component Interconnect message targetting address 0xffc from device 03:00.0 (PCI) have been implemented to support the com- and its associated completion from the host bridge to munications between the CPU and the peripherals. the requester device. Today, the PCI Express bus is used in most personal At boot time, the devices are not yet configured and computers and servers. There are three main types they do not know their own identifier. Indeed, the of PCI Express devices. The root of the bus hierar- manufacturer does not know the bus on which the chy, called the root complex, is connected to the CPU device will be plugged. However, when the firmware thanks to the host bridge and to first level PCI Ex- looks for all available devices (using the assembler in- press children devices. These devices can be endpoints struction mov to read the PCI Express space mapped (so-called peripherals in the paper) and bridges. A in memory), each memory access is processed by the bridge connects two different logical bus domains with host bridge. The host bridge then translates this ac- an upstream and a downstream port. cess into a PCI Express configuration request which is routed to the corresponding bus. In particular, this 2.2 Communications request contains the identifier of the contacted device. Each PCI Express device is identified with a PCI log- ical bus, a device and a function identifier (id), noted If this device is available in the system, it receives this bus:dev.fun. This identification is used to route PCI request and then knows its identifier. This step is im- Express messages between devices. portant to allow a device to communicate. The receiver of a message is either identified by its PCI Express peripherals are able, through DMA re- identifier or by an address. Thus, an address has two quests, to access other peripheral memories and the purposes. Either, it corresponds to an element in the main RAM, even in the kernel memory regions, with- main memory and the memory controller redirects the out any control of the processor. This raises a major corresponding access to the DRAM or it corresponds security concerns if the peripheral is controlled by an to a register of another device and the memory con- attacker. To mitigate this threat, Intel has integrated troller redirects the corresponding access to the device. the IOMMU hardware component, to filter PCI Ex- In the latter case, the registers of the device are said press messages. This component is presented in the memory mapped. For instance, a memory read message next section. Morgan et al. Page 3 of 10
15 2378 0 47 3839 2930 21 111220 0 Bus Device Function requester id Offset Virtual Address
RTAR « 4 « 4 « 3 « 3 « 3 « 3
GCMDR Root Context PML4 PDPT PD PT + + + + + + + Entry Entry entry entry entry entry
Physical Root table Context table PML4 table PDP table PD table Page table Address
IOMMU DRAM Device to domain mapping Address translation configuration 1 2 registers
Figure 2 DMA Remapping with 48 bits virtual addresses in 4kB pages
3 IOMMU internals – DMA Remapping 3.2 IOMMU configuration This section presents the services provided by the An architecture can contain several IOMMUs, each IOMMU and describes how it is configured. one dedicated to a subset of buses (Figure 1). DMAR units translate and filter requests according to the pro- tection domain assigned to the emitter device. A pro- 3.1 Peripherals address space Virtualization tection domain is defined by a set of translation poli- IOMMUs are designed to virtualize the memory space cies. The process is divided into two phases. The first and the interrupts of the peripherals. Memory virtual- one identifies the protection domain assigned to the ization is implemented in the so-called DMA Remap- emitter device. This phase, called device to domain ping units (DMAR) of the IOMMU. The DMAR has mapping, is conceptually similar to an address trans- been designed to simplify address space association lation but instead, it associates PCI identifiers to ad- and coherency between the drivers and the hardware. dress translation domains. In the second phase, called Indeed, device drivers use a different address space address translation, the addresses used by peripher- than the peripherals because they process virtual ad- als memory accesses are translated by DMAR, before dresses configured by the kernel. When it is necessary crossing the host bridge (Figure 1). This translation to pinpoint a specific location in the DRAM to con- is similar to the one carried out by the cores Memory figure DMA, the drivers have to translate virtual ad- Management Unit (MMU). Access controls are appli- dresses to the physical address space used by the de- cable in the two translation phases. vice. This operation increases the design complexity Each DMAR unit can be configured separately in of device drivers and micro kernels/hypervisors. Us- a configuration page and a tree structure (Figure 2), ing DMAR, the peripherals can share the same virtual the latter placed in DRAM. This configuration page address mappings as the drivers (at least for trans- contains the main control register called Global Com- lation). As a consequence, the drivers can configure mand Register (GCMDR) which is designed to acti- virtual addresses into peripheral DMA configuration vate the translation mechanism thanks to the Transla- tion Enable bit (TE) [2, section 10.4]. It also contains registers. Furthermore, since DMAR has a two level a pointer to the tree structures root (i.e. the root en- address translation mechanism, the same strategy can try of the root table in the Figure 2), named the Root even be used within virtual machines and pass-through Table Address Register (RTAR). The location of the drivers. Consequently, virtualized MMU mappings can configuration page of a DMAR is identified by means be easily matched to IOMMU mappings by the hyper- of a dedicated register in the memory controller, set visors. The IOMMU also brings similar advantages as by the firmware. The identifier of a PCI Express mes- the MMU in the cores. It is for example possible to de- sage sender is used to index the first two tables of the tect faulty devices accessing unmapped physical pages tree structure (the root table and the context table) because of software or hardware bugs. in the first phase. The resolved address is then used to From the security point of view, the filtering feature pinpoint the structures for the second phase. These of DMAR allows to immediately cope with the largest structures are indexed with the destination address part of legacy I/O attacks, if properly configured. The of the PCI Express message. The result is the phys- next section introduces the basic concepts of DMAR ical address of the translation. During the handling configuration which are necessary to understand our of these tables, a dedicated bit indicates if the access attack. is granted or forbidden. Finally, it is noteworthy that Morgan et al. Page 4 of 10
the second translation phase can be deactivated (pass security policy. However, the update cannot be com- through mode) thanks to the translation type field of mitted at each change of the DMA memory map be- a context entry. cause that will severely degrade the system perfor- Unfortunately, like other hardware or software sys- mances. Indeed, this operation consumes an average tems, these mechanisms may be inefficient if the im- of 2000 CPU cycles in Intel Sandy Bridge architecture. plementation and the configuration are not correct. In This is why the IOTLB flush is deferred by the kernel, the following, we briefly present some examples of such and performed at a lower frequency than DMA map- vulnerabilities before describing in detail the novel vul- ping changes in the system. This performance scaling nerability we discovered as well as the exploitation ex- technique opens potential time intervals during which ample we implemented. a buffer is still available to a peripheral despite the fact that it had been reallocated. To our knowledge, 4 I/O vulnerabilities and related work this vulnerability has not yet been exploited to date. Many I/O attacks have been presented in the litera- In 2011, Wojtczuk et al. succeded to attack Xen ture [3, 4, 5, 6, 7]. In this paper, we focus on DMA hypervisor, with an active IOMMU, using malicious attacks. These attacks were described in several stud- Message Signaled Interrupts (MSI) [11]. The main ies. In particular, [8] demonstrates that an outsider can idea is to reproduce the obsolete bouncing I/O attack compromise an entire system by exploiting a vulnera- in which software could use peripherals capability to bility in one of its network cards. Fortunately, most of DMA in order to modify kernel memory space. To by- these vulnerabilities have been fixed with the integra- pass the protection, authors uses the capability of de- tion of the IOMMU. vices, here an intel e1000 network controller, to set an As presented in the previous section, a well config- arbitrary interrupt vector into an MSI, to remotely fire ured DMAR unit is theoretically able to enforce an an hypercall. If the malicious hypercall is issued when expected access control policy on a DMA accessible the virtual machine is executing a specific system call, memory. The access control policy is given by the which is controlled by unprivileged malicious applica- DMAR domain and a domain is associated to a device tion code, it is possible to execute a remote buffer over- through device to domain mapping. Consequently, in flow with hypervisor execution privileges. order to be efficient, the DMAR unit needs to identify Even if most of these attacks are today inefficient precisely each DMA capable device. in recent architectures, they highlight the fact that However, in [9], the authors take benefit of the col- the configuration of the IOMMU by the firmware and location of PCI Express and PCI bus to exploit a IOMMU Linux driver in recent kernels presents new weakness in the filtering performed by the IOMMU. weaknesses that could be exploited by an attacker. Indeed, the PCI to PCI Express bridge uses its own More recently, authors of [12] presented a boot time PCI id as the requester id when translating PCI read- DMA attack allowing an attacker to recover macOS /write cycles to PCI Express messages, acting like a FileVault2 passwords, using a thunderbolt device. This proxy. Therefore, all the devices behind the PCI Ex- attack leverages the same vulnerability we had previ- press to PCI bridge are sharing the identity of the ously introduced in [13] and [1], which allows DMA bridge from the IOMMU point of view, and so the accesses at boot time before the activation of any pro- same domain. They exploited this vulnerability with a tection mechanism. Also, this vulnerability has been malicious firewire controller plugged behind the same highlighted in a developers mailing list of the core- bridge of a legitimate network card, to inject malicious boot x86 firmware project [14]. The next section de- Ethernet frames into an IP kernel stack. Finally, they tails this vulnerability and presents an attack scenario consequently succeeded to corrupt an operating sys- in which one can directly bypass IOMMU protection tem ARP cache by injecting customized ARP reply against I/O Attacks. packets. The authors of [10] argue that, for performance rea- 5 Bypassing IOMMU sons, kernels are unable to enforce a strict IOMMU This section describes some weaknesses discovered in security policy regarding DMA buffers management. the firmware and the Linux kernel and discusses when Buffers shared between a driver and a peripheral for and how these weaknesses can be exploited to by- DMA are allocated and deallocated at a given fre- pass DMAR. The observations presented in this sec- quency. Each time a DMA buffer is allocated or freed, tion are based on the Intel documentation, but also on the IOMMU configuration has to be updated. To the information collected empiricaly with the hyper- commit configuration changes, the CPU has to flush visor and FPGA malicious peripheral prototypes pre- the IOMMU translation caches called Input/Output sented in the next Section. The experiments performed Translation Lookaside Buffer (IOTLB) to update the in our study are based on Dell machine precision Morgan et al. Page 5 of 10
T1700, with the follwoing technical details: firmware bridges configuration contain the Bus Master Enable A06 (12/05/2013); Linux 4.3.4 (with IOMMU INTEL=y); (BME) bit which controls the upstream communica- Intel i7-4770 processor; chipset Intel PCH c220; boot- tions, i.e., if the messages generated by children devices loader grub 2.02.beta2. are allowed to go upstream and so perform DMA. As a consequence, devices can initiate DMA requests 5.1 Attack assumptions long before IOMMU configuration and kernel loading. Two preconditions are needed for the attacker to be successful: the first one concerns the configuration of 5.3 Linux and Intel IOMMU driver DMA weaknesses the target machine and the second one concerns the To fulfill our attack, we also exploited a weakness access to the PCI Express bus. in the Intel IOMMU Linux driver drivers/iommu/ As regards the first precondition, we assume that the intel-iommu.c. The latter makes the IOMMU con- attacker has a precise knowledge of the hardware and figuration vulnerable during the boot up process. In kernel version of the victim machine, either directly, order to understand the details, it is necessary to re- through physical access, or indirectly by fingerprinting. view the boot sequence on a Linux machine. This assumption is realistic in many situations, e.g., in big companies or professional working environments, 5.3.1 IOMMU configuration where, most of the time, homogeneous and standard After the firmware operations, the kernel is uncom- machines are deployed, to simplify administration and pressed and loaded by the bootloader (GNU GRUB management tasks. in our experiment). Linux kernel modules and drivers As regards the access to the PCI Express bus, this are then loaded and initialized. DMAR is configured by can be achieved either by plugging a malicious periph- one of these drivers. The Intel IOMMU driver creates eral, or by remotely corrupting a peripheral of the vic- the translation structures and writes them in the main tim machine using for instance the attack presented in DRAM. It builds the address translation domains and [8]). device to domain mapping before copying the root ta- ble pointer into the associated register. Finally, the 5.2 A firmware induced vulnerability driver activates DMAR by setting the TE bit of the The attack we present in this paper first relies on a GCMD Register. Let us note that these structures are vulnerability of the firmware used in the architecture. stored in memory in areas not protected by any security mechanism. 5.2.1 Device configuration At startup, IOMMUs are deactivated. During the boot 5.3.2 Cache policy time configuration, the firmware scans the PCI config- The vulnerability we discovered is exploitable also be- uration space to discover and initialize vital devices, cause Linux flushes cache lines (L1 / L2 and Last Level like the main video controller, loading and executing Cache) after every table entry modification, to ensure its embedded firmware. Read and write accesses to the integrity of the structure in memory. Consequently, the configuration space generate PCI Express config- the time period during which the entire DMAR con- uration messages that are sent to the targeted device. figuration exists in memory is maximized. Devices know their PCI ids after the first scan, as ex- plained in Section 2.2. Therefore, the peripherals are 5.3.3 Physical memory space able to generate valid messages at the early firmware We noticed that as long as the machine hardware con- execution stage, long before the execution of the boot- figuration is not modified, the physical address of the loader and then of the operating system kernel. At the DMAR root table is not changed. This property sim- end of the execution of the firmware, the control is plifies the exploitation presented in this paper, pre- given to the bootloader and to the kernel. venting the attacker from searching DMAR structures into the physical memory space. 5.2.2 PCI Express bridges early configuration Nowadays, DMA accesses are granted early by firmwares 5.4 A vulnerability window to all peripherals, mainly for compatibility reasons. Considering the previous observations, Figure 3 The reason is that modern kernels, in our case Linux, highlights the time periods during which some mali- do not seem to know how to deal with PCI Express cious peripherals can initiate DMA requests. Two im- bridges configuration. This is a common fact that portant time windows are identified. The vulnerability we have verified on our target firmware architecture window represents the time interval during which pe- and which is also widely acknowledged in open source ripherals are able to perform DMA. It starts right after firmware projects [14]. More precisely, PCI Express device address association by the firmware and ends Morgan et al. Page 6 of 10
CPU init. firmware Linux IOMMU driver app. First, the configuration of the system, at startup, al- lows all peripherals to write to DRAM. Thus, the mali- configuration write activation cious peripheral can easily produce a malicious context vulnerability window denied table in the preselected free memory page (step 0, Fig- ure 4), with all entries set to pass through. This step write window must be performed at the beginning of startup, during the vulnerability window. Figure 3 Vunlerability and write time windows
Core Corrupted device intel-iommu.c [05.00.0] 1 after IOMMU activation in the IOMMU driver acti- 3 Malicious pass through Root table context table vation phase. Right after IOMMU activation, DMA 0 is denied for malicious peripherals. The write window GCMDR 2 TE ∞ depicts the short interval of time during which the RTAR P IOMMU configuration is fully placed in DRAM and is 05 1 T 031 00.0 vulnerable. 10 Let us remind that, as stated in Section 5.3.2, IOMMU Root entry Context entry configuration the DMAR configuration physical address does not registers RAM change, as long as the hardware, the firmware and the kernel remain unmodified. Figure 4 Linux IOMMU driver attack implementation
5.5 Attacking DMAR service When Linux begins its execution, it first writes its This subsection presents how to exploit the vulnerabil- own root table (steps 1 and 2). The second step to ity window described in the previous section, in order be performed by the attacker is to overwrite the root to bypass IOMMU protection against I/O attacks. table entry associated to her device. If the corrupted device is connected to logical bus 5, the sixth entry 5.5.1 Prerequisites The attack scenario requires to locate the root of has to be overwritten before the IOMMU activation IOMMU configuration in memory by reading the through bit TE. RTAR register. As explained in the previous sections, To maximize the chances to actually overwrite the this address remains constant and the RTAR register root table entry during the write window, the mali- can be easily read within a Linux kernel module loaded cious peripheral floods the bus with PCI Express write at runtime. requests to the sixth entry of the root table (step ∞). The attack requires a free memory page. As the phys- Finally, Linux activates the IOMMU (step 3) with ical memory map does not change between two re- compromised DMAR structures. From now, the boots, the attacker can easily find free memory pages malicious peripheral can perform any DMA access. using the coloring technique to see if any software has modified the page during the boot process. One can 6 Attack proof-of-concept implementation color a page during the preboot phase, using an UEFI In order to validate our intuition, we performed prelim- custom application, and can read it after the boot pro- inary experiments and developed a proof-of-concept. cess using the dd command on /dev/mem file for exam- For that purpose, we first used a tiny bare-metal re- ple. cursive hypervisor library called Abyme that we de- veloped. Then we ran the real attack with a corrupted 5.5.2 Exploitation peripheral, simulated by a FPGA PCI Express periph- Our exploitation aims at bypassing the IOMMU mem- eral. ory protection without altering the integrity of the ker- nel itself. Fulfilling this constraint makes the attack more difficult to identify. 6.1 Preliminary hypervisor based implementation The easiest way for a malicious peripheral to access Abyme is used as a privileged tool to monitor IOMMU the DRAM is to enable pass through mode in the con- configuration events. The goal here is to perform, with text entry of the DMAR (the address translation part software, the steps (0) and (∞) of the attack imple- of Figure 2 must grant every access of the malicious pe- mentation. Since x86 hypervisors share DRAM phys- ripheral). The following steps describe a possible sce- ical address space with PCI Express peripherals, the nario to grant these accesses at the startup of the sys- simulation of the preceding steps in our hypervisor is tem. close enough to the malicious peripheral point of view. Morgan et al. Page 7 of 10
Abyme library allows the developer to register hooks net MAC, bus bridges, caches and controllers.In addi- which are functions called back when selected virtual tion, we developed the malicious PCI Express periph- machine interruptions occur. It is also possible to re- eral, PCIE-EP. This core brings host memory access configure virtual machine level memory translation, to the SOC through PCI Express memory messages. remap pages or change their access control in order With PCIE-EP, the LM32 is able to program memory to generate memory access faults and trigger the cor- reads, memory writes (with a high rate mode). Note responding hook. that this SOC is flexible and can be adapted for other To simulate the attack, our strategy is as follows. purposes, e.g., for the implementation of integrity tests We first configure our hypervisor at boot time, before in the context of a hardware assisted trusted architec- the startup of the Linux virtual machine. We register a ture as presented in [17]. hook to the virtual machine page fault event (EPT vio- lations), which is raised when virtual machine memory 6.3 Results and proof-of-concept accesses are denied. We also register a hook to virtual In order to demonstrate that the exploitation of the machine interruptions raised when the step by step ex- discovered vulnerability allows a potential attacker to ecution mode is active. Then, we unmap (read, write further take control of the host, we considered an ex- and execute credentials off) the DMAR unit 0 configu- ample of a kernel rootkit that we injected in kernel ration page, associated to external peripherals. Finally, memory through DMA requests performed by our ma- we launch the virtual machine. licious FPGA device, once it has successfully modified The goal here is to detect when the linux IOMMU the IOMMU configuration so that it can make read- driver activates the translation setting GCMDR.TE bit to /write accesses to kernel memory. 1, to perform steps (0) and (∞) with the hypervisor. The Linux IOMMU driver accesses the configuration 6.3.1 Rootkit Attack page several times. Each time an access is performed: The considered rootkit is a binary code which is in- i) a virtual machine interruption occurs; ii) the single jected in kernel memory and modifies the behavior step execution is activated; iii) if GCMDR.TE bit is to of the setuid kernel system call. According to the be written, we perform steps (0), (∞) and flush the POSIX programmer’s setuid manual page, the nor- caches; iv) we remap DMAR 0 page; v) we continue mal behavior for this system call is to modify the real virtual machine execution to let the virtual machine user uid, effective uid and others accordingly, if the execute the write access; vi) single step virtual ma- user has the rights or enough privileges to do so. Our chine interruption occurs; vii) we unmap again DMAR rootkit modifies the preceding behavior in a way that 0 page; viii) we resume virtual machine execution. each time this functionality is called, the euid (effec- With this scenario, we were able to successfully sim- tive uid) of the calling task is systematically set to 0, ulate the attack described in Figure 4. which gives root user effective privileges to the calling This preliminary experiment with our hypervisor be- process. We developed a small C code to call setuid ing successful, we decided to implement a real proof of function, that we executed by a non-root user, both concept on an FPGA PCI Express peripheral. This in the presence and in the absence of our exploitation FPGA and our implementation choices are presented of the IOMMU configuration. The short video at [1] in the following. shows the rootkit installation and use.
6.2 FPGA device implementation 6.3.2 Implementation details We based the development of our malicious periph- Linux system call implementation is located in the file eral on Milkymist System On Chip [15], which is orig- kernel/sys.c. Our attack modifies the setuid() sys- inally a video DJing open hardware project. Thanks tem call implementation. The following listing contains to the hardware flexibility of FPGAs and the thought- the most relevant parts of its code. ful modularity of hardware and software developed for Milkymist, we removed unneeded functionalities and SYSCALL_DEFINE1(setuid, uid_t, uid) added a PCI Express end point stack with minimum { effort. Milkymist is made for a custom board hosting struct user_namespace *ns = current_user_ns(); a Xilinx virtex 4 FPGA[16]. Since we needed a PCI const struct cred *old; Express connector and gigabit transceivers to develop struct cred *new; int retval; the PCI Express peripheral, we have chosen a Xilinx kuid_t kuid; ML605. Our SOC contains the original Milkymist Lattice [1]http://homepages.laas.fr/nicomett/SSTIC2016/iommu-pwn-sstic. Mico32 microprocessor (LM32), Onchip ROM, Ether- webm Morgan et al. Page 8 of 10
// (1) ffffffff810869c5: retq kuid = make_kuid(ns, uid); [..] The parameter is copied in %rbx (1). The code af- new = prepare_creds(); fectation of new is located at (2), directly followed by [..] the credential check function call (3). Then, (4) in- old = current_cred(); vokes the credential commit function and returns from [..] the system call. Finally, let us recall that Linux core // (2) kernel virtual address space is not randomized. Fur- new->fsuid = new->euid = kuid; // (3) thermore, physical addresses can be trivially deduced retval = security_task_fix_setuid(new, old, from virtual one with the following bitwise function: LSM_SETID_ID); f(a) = a ∧ ¬0xffffffff80000000. Also, we can note [..] that instruction at 0xffffffff81086998 is 4 bytes // (4) aligned, which makes DMA easier because it is also return commit_creds(new); aligned to double words. [..] Therefore, our rootkit payload is written at physical } address 0x01086998, The payload simply writes the double word 0 to the (2) affectation addresses. Then Our objective is to modify the setuid system call it jumps over (3) function call, to (4). The rootkit code to set the euid to 0 for the current process. If we payload listing is listed hereafter. look at the source code, we can notice that the euid is stored in the new structure (2), and that this mod- ffffffff81086998
between the cost and the efficiency of these counter- 7.2 Limitations measures. Also, we discussd the limitations and con- As discussed in Section 5, to be successful, the attacker straints associated with the considered attack. must have a precise knowledge of the hardware and kernel version of the victim machine, to be able to 7.1 Countermeasures predetermine the exact physical addresses of IOMMU Efficient protection solutions to cope with the attack configuration and of the required free pages. While discussed in the previous sections, and more generally such information can be easily obtained for standard boot time attacks, mainly rely on the configuration of machines, it is less realistic in the case of specific or the hardware of each computer, which is the responsi- customized platforms. bility of both the firmware and the kernel. From our point of view, even if it is challenging for Another concern that can be raised is related to the stealthiness of the attack. The attack consists in operating systems to take into account all the details [2] of all hardware platforms, they should support at least configuring the IOMMU using identity mapping in the security related features of the system bus, PCI the way that it deactivates the translation with pass- Express in our case study. We previously mentioned through mode. the behavior of the firmware regarding the BME bit In our attack scenario, either the attacker has a phys- present in the PCI Express bridges. This bit can pre- ical access to the victim machine and uses a malicious vent the unnecessary pre-boot DMA capability of pe- peripheral, or a device of the victim machine is cor- ripherals, and so avoid the vulnerability window. This rupted remotely. In the first case, the attack can be protection seems to be the best countermeasure for designed to be stealthy by construction (e.g., by hiding the attack presented in this paper. As a matter of the malicious logic in addition to the implementation fact, both kernel and firmware developers have to re- of the legitimate behavior). In the second case, two sit- think the system bus pre-boot configuration. Actually, uations can be distinguished. If the corrupted device as stated in [14], this bit is set to zero because of legacy is configured by the kernel with identity mapping op- and kernel compatibility reasons. This solution could tion, the attack will not be perceived by the victim. be easily included in the next generation of machine On the other hand, if it is not identity mapped, the firmwares. However, there is still the question about corrupted driver will probably fail and the victim ma- updating currently used firmwares. This complex and chine will have to be rebooted, though it would have expensive operation, which is under the responsibility been already permanently compromized. Nevertheless, of machine manufacturers, can take several months to the attacker can enforce the usage of identity mapping be implemented. for the corrupted device using ACPI Reserved memory Some alternative solutions can also be investigated Region Reporting Structures. to increase the effort needed by an attacker to succeed in performing the considered attack. For instance, the platform that we have studied brings additional secu- 8 Conclusion rity features: some DMA protected configurable mem- The IOMMU has been included several years ago in ory segments (The DMA Protected Range specified Intel processor architectures to provide better protec- by the processor and the Protected Memory Ranges tion against low level attacks. While this mechanism implemented in the IOMMU) [19, Vol. 2, 2.5]. Linux has proved to be efficient to cope with several I/O at- does not use these memory segments, placing IOMMU tacks, this paper shows that it can be bypassed by structures outside the protected ranges. Devices are exploiting a design weakness in its configuration by consequently able to read and write the IOMMU con- both firmwares and the Linux IOMMU driver. The figuration before its activation. These DMA protected corresponding vulnerability is discussed in this paper, memory segments are common in modern architec- a proof-of-concept and an experiment illustrating its tures and should be systematically used to set up such possible exploitation are also presented. The attack hardware protection components, such as the IOMMU. explored makes it possible for malicious peripherals to Despite these protections, the system remains vul- make read and write accesses in main memory and nerable to DMA attacks while the firmware is being to bypass the protection mechanisms embedded in the executed, in the first phase of the boot process. This weakness is due to the fact that the firmware does not [2]Basically, Linux creates two types of memory do- filter DMA. It can be exploited by a malicious periph- mains. In the first type of domain, the I/O Virtual eral to modify the code of either Linux or the firmware Addresses (IOVAs) used by the peripheral and con- itself and so prevent the activation of the IOMMU. To figured by the driver are different than the physical address this problem, it is necessary to check the in- ones (IOPAs), translated by the IOMMU. The second tegrity of software components, e.g., by using a tech- type defines IOVAs as identical as IOPAs, and is called nology like Intel TXT. identity mapping. Morgan et al. Page 10 of 10
IOMMU. We are currently studying other operating References systems (such as Windows, BSD systems) in order 1. Morgan, B., Alata, E., Nicomette, V., Kaˆaniche,M.: Bypassing iommu protection against i/o attacks. In: 2016 Seventh Latin-American to check whether this vulnerability is only related to Symposium on Dependable Computing (LADC), pp. 145–150 (2016). Linux kernel or not. In the same way, we also plan to doi:10.1109/LADC.2016.31 investigate the boot process when the Intel TXT is ac- 2. Intel Corporation: Intel R Virtualization Technology for Directed I/O. tivated to check that this technology does not suffer http://www.intel.com/content/dam/www/public/us/en/ documents/product-specifications/vt-directed-io-spec.pdf from similar weaknesses. 3. Dornseif, M.: Owned by an ipod: Firewire/1394 issues. In: PacSec (2004) 4. Becher, M., Dornseif, M., Klein, C.N.: FireWire: all your memory are Declarations belong to us. In: CanSecWest (2005) List of abbreviations 5. Maynor, D.: Dma: Skeleton key of computing and selected soap box • I/O : Input / Output rants. In: CanSecWest (2005) • (D)RAM : (Dynamic) Random-Access Memory 6. Aumaitre, D., Devine, C.: Subverting windows 7 x64 kernel with dma • DMA : Direct Memory Access attacks. sogeti esec lab (july 2010) • IOMMU : Input Output Memory Management Unit http://esec-lab.sogeti.com/dotclear/public/publications/ • PCI : Peripheral Component Interconnect 10-hitbamsterdam-dmaattacks.pdf • FPGA : Field Programmable Gate Array 7. Stewin, P., Bystrov, L.: Understanding dma malware. In: Lecture Notes • CPU : Central Processing Unit in Computer Science vol. 7591, pp. 21–41 (2012). Chap. Detection of • DMAR : DMA Remapping Intrusions and Malware and Vulnerability Assessment • MMU : Memory Management Unit 8. Duflot, L., Perez, Y.-A., Morin, B.: What if you can’t trust your • GCMDR : Global Command Register network card ? In: 14 International Conference on Recent Advances in • TE : Translation Enable bit Intrusion Detection (RAID), pp. 378–397 (2011) • RTAR : Root Table Address Register 9. Sang, F.L., Lacombe, E., Nicomette, V., Deswarte, Y.: Exploiting an • ARP : Address Resolution Protocol i/ommu vulnerability. In: Malicious and Unwanted Software • IOTLB : Input Output Translation Lookaside Buffer (MALWARE), 2010 5th International Conference On, pp. 7–14 (2010). • USB : Universal Serial Bus IEEE • TXT : Trusted Execution Technology 10. Markuze, A., Morrison, A., Tsafrir, D.: True iommu protection from • L1/L2 : x86 cache levels dma attacks: When copy is faster than zero copy. ACM SIGOPS • EPT : Extended Page Tables Operating Systems Review 50(2), 249–262 (2016) • SOC : System On Chip 11. Wojtczuk, R., Rutkowska, J.: Following the white rabbit: Software • PCIE : PCI Express attacks against intel vt-d technology. ITL: http://www. • PCIE-EP : PCIE End Point invisiblethingslab. com/resources/2011/Software% 20Attacks% 20on% • LM32 : LatticeMico32 20Intel% 20VT-d. pdf (2011) • POSIX : Portable Operating System Interface (uniX) 12. Frisk, U.: macos filevault2 password retrieval http: • BME : Bus Master Enable bit //blog.frizk.net/2016/12/filevault-password-retrieval.html 13. Morgan, B., Averlant, G., Alata, E., Nicomette, V.: Bypassing dma Ethics approval and consent to participate remapping with dma. In: Symposium sur Le S´ecurit´edes Technologies Not applicable de l’Informatino et des Communications, p. 9 (2016) 14. Coreboot Mailing List : [coreboot] DMA Protection? [AMD-Vi]. Consent for publication https://mail.coreboot.org/pipermail/coreboot/2016-November/ Not applicable 082497.html 15. S´ebastienBourdeauducq: Milkymist System On Chip. Availability of data and materials https://github.com/m-labs/milkymist Not applicable 16. S´ebastienBourdeauducq: Milkymist One Board. https://events. ccc.de/camp/2011/Fahrplan/attachments/1874_mm_ccc2011.pdf 17. Morgan, B., Alata, E., Nicomette, V., Kaaniche, M., Averlant, G.: Competing interests Design and implementation of a hardware assisted security architecture The authors declare that they have no competing interests. for software integrity monitoring. In: Dependable Computing (PRDC), 2015 IEEE 21st Pacific Rim International Symposium On, pp. 189–198 Funding (2015). doi:10.1109/PRDC.2015.46 This work is funded by the French laboratory Centre National de la 18. TianoCore. http://www.tianocore.org/ Recherche Scientifique (CNRS). TM 19. Intel Corporation: Desktop 4th Gen Intel R Core Processor Family: Datasheet, Vol. 2. Authors’ contributions http://www.intel.com/content/www/us/en/processors/core/ This paper results from a collaborative work. The order of the authors 4th-gen-core-family-desktop-vol-2-datasheet.html reflects the importance of the contribution of each author concerning the identification of the IOMMU vulnerability and the design of the experimental scenario to validate its possible exploitation. All authors read and approved the final manuscript.
Acknowledgements Not applicable
Endnotes Not applicable
Author details 1Laboratoire d’Analyse et d’Architecture des Syst`emes(LAAS-CNRS), 7 avenue du Colonel Roche, 31400 Toulouse, France. 2INSA Toulouse, 135 avenue de Rangueil, 31400 Toulouse, France.