BUDAPEST UNIVERSITY OF TECHNOLOGY AND ECONOMICS DEPARTMENT OF NETWORKED SYSTEMS AND SERVICES New Methods for Detecting Malware Infections and New Attacks against Hardware Virtualization Ph.D. Dissertation of Gabor´ Pek´ Supervisor: Levente Buttyan,´ Ph.D. Budapest, Hungary 2015 Alul´ırott Pek´ Gabor´ kijelentem, hogy ezt a doktori ertekez´ et´ magam kesz´ ´ıtettem es´ ab- ban csak a megadott forrasokat´ hasznaltam´ fel. Minden olyan reszt,´ amelyet szo´ szerint, vagy azonos tartalomban, de atfogalmazva´ mas´ forrasb´ ol´ atvettem,´ egyertelm´ uen,˝ a forras´ megadas´ aval´ megjeloltem.¨ I, the undersigned Gabor´ Pek´ hereby declare, that this Ph.D. dissertation was made by myself, and I only used the sources given at the end. Every part that was quoted word-for- word, or was taken over with the same content, I noted explicitly by giving the reference of the source. A dolgozat b´ıralatai´ es´ a ved´ esr´ ol˝ kesz´ ult¨ jegyzok˝ onyv¨ a Budapesti Muszaki˝ es´ Gazdasagtu-´ domanyi´ Egyetem Villamosmern´ oki¨ es´ Informatikai Karanak´ dek´ ani´ hivatalaban´ elerhet´ oek.˝ The reviews of the dissertation and the report of the thesis discussion are available at the Dean’s Office of the Faculty of Electrical Engineering and Informatics of the Budapest University of Technology and Economics. Budapest, :::::::::::::::::::::::: Gabor´ Pek´ iii Abstract In my dissertation, I address problems in two domains: (i) Detection of unknown malware and (ii) finding new attacks against hardware virtualization. Accordingly, this dissertation is divided into two parts. In the first part of the dissertation, I propose Membrane, a memory forensics tool to detect code injection attacks. Instead of trying to detect the code injection event itself, I focus on the changes it causes on the paging behavior of the Windows operating system. As my method focuses on the anomalies caused by code injection in paging events, I am able to detect a wide range of code injection techniques. My results indicate that on Windows XP we can detect malware behavior with 91-98% success. On Windows 7, a good detection rate is maintained except for malware injecting into explorer.exe where the success of detection decreases to 75-86%. My approach can detect stealthy malware attacks, even advanced targeted attacks using code injection. Still in the first part, I propose a new system monitoring framework that can serve as an enabler for automated malware detection on live systems. My approach takes advantage of the increased availability of hardware assisted virtualization capabilities of modern CPUs, and its basic novelty consists in launching a hypervisor layer on the live system without stopping and restarting it. This hypervisor runs at a higher privilege level than the OS itself, thus, it can be used to observe the behavior of the analyzed system in a transparent manner. For this purpose, I also propose a novel system call tracing method that is designed to be configurable in terms of transparency and granularity. In the second part of the dissertation, I shed light on VM related threats and defences by implementing, testing, and categorizing a wide range of known and unknown attacks based on directly assigned devices. I executed these attacks on an exhaustive set of VMM configurations to determine their potential impact. My experiments suggest that most of the previously known attacks are ineffective in current VMM setups. I also developed an automatic tool, called PTFuzz, to discover hardware-level problems that affect current VMMs. By using PTFuzz, I found several cases of unexpected hardware behaviour, and a major vulnerability on Intel platforms that potentially impacts a large set of machines used in the wild. These vulnerabilities affect unprivileged virtual machines that use a directly assigned device (e.g., network card) and have all the existing hardware protection mechanisms enabled. Such vulnerabilities allow either an attacker to generate a host-side interrupt or hardware faults, violating expected isolation properties. These can cause host software (e.g., VMM) halt as well as they might open the door for practical VMM exploitations. I believe that my study can help cloud providers and researchers to better understand the limitations of their current architectures to provide secure hardware virtualization and prepare for future attacks. At the same time, security practitioners make heavy use of various virtualization techniques to create sandboxing environments that provide a certain level of isolation between the host and the code being anal- ysed. However, most of these are easy to be detected and evaded. The introduction of hardware assisted vir- tualization (Intel VT and AMD-V) made the creation of novel, out-of-the-guest malware analysis platforms possible. These allow for a high level of transparency by residing completely outside the guest operating system being examined, thus conventional in-memory detection scans are ineffective. Furthermore, such an- alyzers resolve the shortcomings that stem from inaccurate system emulation, in-guest timings, privileged operations and so on. Finally, I introduce novel approaches that make the detection of hardware assisted virtualization plat- v forms and out-of-the-guest malware analysis frameworks possible. To demonstrate my concepts, I imple- mented an application framework called nEther that is capable of detecting the out-of-the-guest malware analysis framework Ether [Dinaburg et al., 2008a]. vi Kivonat A disszertaci´ omban´ ket´ problemater´ ulettel¨ foglalkozom: (i) ismeretlen kart´ ekony´ kodok´ detekcioja´ es´ (ii) uj´ tamad´ asok´ vizsgalata´ hardver virtualizaci´ o´ ellen. Ebbol˝ adod´ oan´ a disszertaci´ o´ ket´ reszre´ bonthato.´ Az elso˝ reszben´ elosz˝ or¨ egy uj´ memoria´ forenics eszkozt¨ javaslok Membrane neven,´ melynek celja´ a kod´ injekcios´ tamad´ asok´ detektal´ asa.´ Ahelyett, hogy a kodinjekci´ o´ teny´ et´ detektaln´ am,´ a Windows operaci´ os´ rendszerben jelentkezo˝ laptabla´ esemenyv´ altoz´ asokat´ vizsgalom´ meg nagy alapossaggal.´ Mivel a laptabla-´ esemenyek´ altal´ okozott anomali´ akra´ fokusz´ alok,´ szeles´ spektrumban lehet detektalni´ kul¨ onb¨ oz¨ o˝ t´ıpusu´ injekcios´ eljar´ asokat.´ Az eredmenyek´ Windows XP operaci´ os´ rendszer eseten´ 91-98%-os pontossagot´ mu- tatnak, m´ıg Windows 7 rendszereknel´ meg´ a legzajosabb folyamatok eseten´ is (pl.: explorer. exe) 75- 86% a detekcios´ pontossag.´ Az eljar´ asomnak´ kosz¨ onhet¨ oen˝ lehetseges´ ilyen t´ıpusu´ rejtozk˝ od¨ o˝ kart´ ekony´ kodokat´ detektalni,´ meg´ ha celzott´ tamad´ asb´ ol´ vett kampanyr´ ol´ is van szo.´ Ezt kovet¨ oen˝ egy uj´ rendszer-monitorozo´ eszkozt¨ javaslok, amely automatikus malware detekciot´ tesz lehetov˝ e´ eles´ gepek´ eseten.´ A megkozel¨ ´ıtesem´ a modern CPU-kban elerhet´ o˝ hardverrel tamogatott´ virtu- alizaci´ os´ eljar´ ast´ hasznalja´ ki, illetve azt a tenyt,´ hogy ezaltal´ lehetos˝ eg´ van egy hypervisort telep´ıteni a futo´ operaci´ os´ rendszer ala´ anelk´ ul,¨ hogy le kene´ a gepet´ all´ ´ıtani, vagy ujra´ kene´ azt ind´ıtani. Ez a hypervisor magasabb privilegium´ szinten fut, mint az operaci´ os´ rendszer maga. Ennek kosz¨ onhet¨ oen˝ a rendszert tran- szparensebben lehet megfigyelni, mint mas´ modszerekkel.´ Ebbol˝ a celb´ ol´ javaslok tovabb´ a´ egy uj´ rendszer- monitorozo´ eljar´ ast,´ melyet a transzparencia es´ az elemzesi´ granularitas´ menten´ lehet finomhangolni. A disszertaci´ o´ masodik´ resz´ eben´ ravil´ ag´ ´ıtok a VM-ekhez kapcsolod´ o´ fenyegetesekre´ es´ vedelmi´ megol- dasokra´ azaltal,´ hogy a kozvetlen¨ eszkozcsatol¨ assal´ kapcsolatos tamad´ asok´ szeles´ skal´ aj´ at´ tervezem meg, implementalom´ le es´ kategorizalom.´ Ezeket a tamad´ asokat´ sok kul¨ onb¨ oz¨ o˝ VMM konfiguraci´ oval´ tesztel- tem, hogy kider´ıtsem a tenyleges´ hatasukat.´ A vizsgalataim´ megmutattak,´ hogy a futtatott tamad´ asok´ egy jo´ resze´ tulajdonkeppen´ hatastalan´ a mai VMM beall´ ´ıtasok´ mellett. Kifejlesztettem azonban egy automatikus eszkozt,¨ amit PTFuzz-nak neveztem el, hogy hardver szintu,˝ VMM-hez kapcsolod´ o´ problem´ akat´ fedezzek fel. A PTFuzz seg´ıtseg´ evel´ szamos´ varatlan´ hardverviselkedest,´ tovabb´ a´ egy Intel platformhoz kothet¨ o˝ komoly ser´ ul¨ ekenys´ eget´ talaltam,´ mely rengeteg Intel chipsetre es´ ezaltal´ gepre´ hatassal´ van a mai napig. Ez a ser´ ul¨ ekenys´ eg´ minden olyan nem privilegizalt´ virtualis´ gepet´ erint,´ mely kozvetlen¨ eszkozcsatol¨ assal´ rendelkezik minden letez´ o˝ hardveres biztonsagi´ vedelem´ mellett. Egy tamad´ o´ a ser´ ul¨ ekenys´ eg´ seg´ıtseg´ evel´ hosztoldali interrupt-ot vagy hardver hibat´ tud generalni´ megsertve´ ezaltal´ az elvart´ izolaci´ os´ muk˝ od¨ est.´ Ez elobbi˝ elmeleti´ szinten tenyleges´ virtualizaci´ os´ kitor¨ est´ is lehetov˝ e´ tehet, m´ıg a masodik´ a hoszt szoftver (pl.: VMM) teljes leall´ as´ at´ idezheti´ elo.˝ Hiszem, hogy a tanulmanyom´ seg´ıt a felho˝ szolgaltat´ oknak´ es´ ku- tatoknak´ jobban megerteni´ a jelenlegi architektura´ korlatait,´ hogy biztonsagos´ hardver virtualizaci´ os´ plat- formot hozzanak letre´ es´ felkesz´ uljenek¨ jov¨ obeli˝ tamad´ asokra.´ Ugyanakkor a biztonsagi´ szakert´ ok˝ nagy mert´ ekben´ tamaszkodnak´ a kul¨ onb¨ oz¨ o˝ virtualizaci´ os´ tech- nikakra,´ hogy sandboxing kornyezeteket¨ hozzanak letre,´ melyek valamilyen szintu˝ izolaci´ ot´ biztos´ıtanak a hoszt es´ az analizalt´ kod´ koz¨ ott.¨ Azonban ezek nagy resz´ et´ nagyon konny¨ u˝ detektalni´ es´ kikerulni.¨ A hard- verrel tamogatott´ virtualizaci´ o´ bevezetese´ (Intel VT es´ AMD-V) lehetov˝ e´ tette az