Nested Virtualization: Hyper-V on KVM Ladi Prosek October 26th 2 !" #GEN&# 1. What 2. Why 3. How 2 #GEN&# 1. What is Hyper-V? What is nested virtualization( 2. Why 3. How ' #GEN&# 1. What is Hyper-V? What is nested virtualization( 2. Why should ,e care( 3. How + #GEN&# 1. What is Hyper-V? What is nested virtualization( 2. Why should ,e care( 3. How does it work( - Hyper-V ● Microso.t/s 016 *irtualization solution ● Ships as ● Microso.t Hyper-V Ser*er 3 standalone product ● Hyper-V role – )indo,s Server4 hi5her editions o. client )indo,s ● 6ype-1 hyper*isor4 root7parent partition hosts the 8ana5ement O2 6 Hyper-V (cont: " Nested *irtualization ● ;unnin5 a hypervisor in a VM )indo,s Nested VM < L2 5uest Hyper-V < L! 5uest4 L! hyper*isor Linux 7 KVM < L0 hypervisor Hard,are 1 )hy Hyper-V on KVM? ● 6estin54 develop8ent4 trainin54 demos4 <4 5eneral tinkerin5 ● May eli8inate the need .or dedicated H) ● #ll the benefits o. virtualization ,hen the ,orkload is virtualization ● Virtualization-based security 9V?2: ● Ne, in )indo,s Server 2 !6 and )indo,s ! ● Hyper-V used under the covers to protect the OS .ro8 itself 7 .ro8 8alware ● Praised by security researchers ● Hyper-V on Hyper-V works 9#zure supports nested already: ● %0pectin5 de8and for Hyper-V on KVM also ● #nd 8aybe KVM on Hyper-V as ,ell = Virtualization-based security ● Virtual Trust Level 9V6L: ● V6L is nor8al, V6L ! is secure ● 2L#6 en.orced ● Hyper-V no lon5er trusts the root partition runnin5 in V6L ● 28all a8ount of code runs in VTL ! ● Mini8al kernel @ security related 8odules ● Aser 8ode trustlets ● Device Guard prevents runnin5 unsi5ned7untrusted code ● 2L#6 en.orced )BC ● 6o5ether ,ith secure boot, IOMMA, 6PM, EEE ● Credential Guard hides crypto5raphic secrets ● 2L#6 en.orced F; ! Experiment void experiment(void) { char ret_instruction = 0xc3; void (*func_ptr)(void) = (void (*)(void))&ret_instruction; func_ptr(); } !! Experiment 9cont) !2 Experiment 9enhanced: void experiment(void) { char ret_instruction = 0xc3; void (*func_ptr)(void) = (void (*)(void))&ret_instruction; make_page_executable(&ret_instruction); func_ptr(); } !' Experiment 9enhanced4 VBS on: !+ Ho, does it work? ● Sa8e as KVM on KVM but the de*il is in the detail< ● Dssues .ound so far ● Missin5 .eatures 9*ery .e,: ● %0: &escriptor table e0its 9%CD6G;%#SONG$&6;GD&6;, %CD6G;%#SONGL&6;G6;: ● KVM bu5s 98any: ● %0: &ere.erencin5 H;' under P#E @ EP6 ● Hyper-V bu5s 9*ery .e,: ● %0: #ssu8in5 the presence o. IO#PICG;%$G%OD ● I2 KVM patches so far4 and a tiny bit of JEMU ,ork !- Kuture work ● Per.ormanceF ● )indo,s boot ti8e currently doubles a.ter enablin5 VBS :9 ● Para*irtualized .eatures as per Hyper-V 6op-Le*el Functional 2pec ● Enli5htened VMH2 ● Enli5htened M2; bit8ap ● Virtual TL? ● Please test Hyper-V L1 when making nV#$ % n&V# changes ● )indo,s Server 2 !6 evaluation a*ailable .or do,nload ● Use standard HV enli5hten8ents ● -cpu <4h*Grela0ed4h*GspinlocksL 0!...4h*G*apic,h*Gti8e ● -cpu <4-hypervisor required .or older Hyper-V *ersions !6 Debugging tips ● JEMU $&? stub ● Me8u -gdb tcp::!2'+ ● 95db: target remote localhost:!2'+ ● Hyper-V li*es in h*i06+Ee0e 9Dntel:4 h*a06+Ee0e 9#M&: ● No public sy8bols ● #ddresses chan5e due to #SL; – search .or patterns ● )indo,s kernel debu55er ● bcdedit 7hyper*isorsettin5s {serial or !'=4 settin5sO bcdedit 7set hyper*isordebu5 on bcdedit 7set hyper*isorlaunchtype auto ● ;un ,indb5 on another )indo,s VM !" Demo! !1 Summary ● Hyper-V on KVM ,orks ● ?ut e0pect rou5h ed5es, especially around per.ormance ● Virtualization-based security uses Hyper-V ● Marketin5 na8es: De*ice guard4 Credential guard ● Dnstallin5 the Hyper-V role starts an L2 != ;esources & Q# ● Ladi Prosek [email protected] ● http:77ladiproE,ordpressEco8 ● Ale0 Ionescu’s B#66LE OK 2KM #N& IUM: http:77,,,Eale0-ionescuEco87blackhat2 !-.pd. 2 .