A Binary Compatible Unikernel
Pierre Olivier*, Daniel Chiba*, Stefan Lankes+, Changwoo Min*, Binoy Ravidnran*
*Virginia Tech, +RWTH Aachen University
VEE’19 - 04/14/2019 Unikernels Presentation
Full-fledged Virtual Machine Linux distribution
Application
Libraries
OS interface Linux used Kernel Hypervisor Hardware
2/11 Unikernels Presentation
Full-fledged Virtual Machine Linux Legend : distribution Useful software Software bloat Application
Libraries
OS interface Linux used Kernel Hypervisor Hardware
2/11 Unikernels Presentation
Full-fledged Virtual Machine Linux Legend : distribution Useful software Software bloat Application Unikernel Libraries Application OS interface Linux Libraries used Kernel OS Layer Hypervisor Hardware
2/11 I single-*
I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel
I No user/kernel protection needed I Lightweight virtualization, alternative to containers
I Security advantage: small attack surface and high isolation I Per-application tailored kernel
I LibOS/Exokernel model I Reduced OS noise, increased performance
I Low system call latency
I App + kernel in ring 0, system calls are function calls
Unikernels Presentation (2)
Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor
3/11 I Lightweight virtualization, alternative to containers
I Security advantage: small attack surface and high isolation I Per-application tailored kernel
I LibOS/Exokernel model I Reduced OS noise, increased performance
I Low system call latency
I App + kernel in ring 0, system calls are function calls
Unikernels Presentation (2)
Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor
I single-*
I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel
I No user/kernel protection needed
3/11 I Per-application tailored kernel
I LibOS/Exokernel model I Reduced OS noise, increased performance
I Low system call latency
I App + kernel in ring 0, system calls are function calls
Unikernels Presentation (2)
Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor
I single-*
I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel
I No user/kernel protection needed I Lightweight virtualization, alternative to containers
I Security advantage: small attack surface and high isolation
3/11 I Reduced OS noise, increased performance
I Low system call latency
I App + kernel in ring 0, system calls are function calls
Unikernels Presentation (2)
Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor
I single-*
I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel
I No user/kernel protection needed I Lightweight virtualization, alternative to containers
I Security advantage: small attack surface and high isolation I Per-application tailored kernel
I LibOS/Exokernel model
3/11 Unikernels Presentation (2)
Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor
I single-*
I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel
I No user/kernel protection needed I Lightweight virtualization, alternative to containers
I Security advantage: small attack surface and high isolation I Per-application tailored kernel
I LibOS/Exokernel model I Reduced OS noise, increased performance
I Low system call latency
I App + kernel in ring 0, system calls are function calls
3/11 Because it is hard to port existing applications!
Unikernels The Issue
I Unikernels have plenty of benefits to bring
I Unikernels have plenty of application domains
I They are very popular in academia . . .
I . . . but why (nearly) nobody uses them in the industry?
4/11 Unikernels The Issue
I Unikernels have plenty of benefits to bring
I Unikernels have plenty of application domains
I They are very popular in academia . . .
I . . . but why (nearly) nobody uses them in the industry?
Because it is hard to port existing applications!
4/11 I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Build Unikernel (compile, binary Unikernel & link, etc.) libraries src
5/11 I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Build Unikernel (compile, binary Unikernel & link, etc.) libraries src
I Proprietary software → source code not available
5/11 I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Build Unikernel (compile, binary Unikernel & link, etc.) libraries src
I Proprietary software → source code not available I Incompatible language
5/11 I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Build Unikernel (compile, binary Unikernel & link, etc.) libraries src
I Proprietary software → source code not available I Incompatible language I Unsupported features
5/11 I Complex build toolchains HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Build Unikernel (compile, binary Unikernel & link, etc.) libraries src
I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel
5/11 HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
Unikernels The Issue: Porting to Unikernels
Application Execution src
Unikernel Build binary Unikernel & (compile, libraries src link, etc.)
I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains
5/11 Unikernels The Issue: Porting to Unikernels
I Proprietary software → source code not available
I Incompatible language
I Unsupported features
I Porting is hard, needs knowledge about both application and unikernel
I Complex build toolchains
HermiTux Solution
I A unikernel binary-compatible with Linux
I For x86-64 for now
5/11 I ELF loader convention I Load-time Stack layout I Syscalls
I Kernel adapted from HermitCore
I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
6/11 I Syscalls
I Kernel adapted from HermitCore
I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention I Load-time Stack layout
6/11 I Kernel adapted from HermitCore
I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention I Load-time Stack layout I Syscalls
6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention I Load-time Stack layout I Syscalls
I Kernel adapted Hypervisor: uHyve from HermitCore
Host: Linux kernel KVM
6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention I Load-time Stack
t layout Single-address space VM s e I Syscalls u G I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM
6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention Native I Load-time Stack
t layout
Linux s e I Syscalls App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM
6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
I ELF loader convention Native I Load-time Stack
t layout
Linux s
Hermitux e I Syscalls kernel App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM
6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
Init. stack and jump to entry point I ELF loader convention Native I Load-time Stack
t layout
Linux s
Hermitux e I Syscalls kernel App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM
6/11 I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack
Syscall t layout
Linux s
Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM support for 80+ syscalls
6/11 I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
Unikernels Overview
I Linux ABI convention:
Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack
Syscall t layout
Linux s
Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM Debug, profile support for 80+ syscalls
6/11 Unikernels Overview
I Linux ABI convention:
Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack
Syscall t layout
Linux s
Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM Debug, profile support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?
I Fast system calls and modularity
6/11 I For dynamically compiled programs:
I At runtime load a unikernel-aware Libc
I Making for system calls (fast) function calls directly into the kernel
I Automatically transformed version of Musl Libc with Coccinelle
Unikernels Fast Syscalls with Libc Substitution
I HermiTux’s syscall handler is invoked by the syscall instruction
I Reintroduce high latency for system calls due to the world switch
7/11 Unikernels Fast Syscalls with Libc Substitution
I HermiTux’s syscall handler is invoked by the syscall instruction
I Reintroduce high latency for system calls due to the world switch
I For dynamically compiled programs:
I At runtime load a unikernel-aware Libc
I Making for system calls (fast) function calls directly into the kernel
I Automatically transformed version of Musl Libc with Coccinelle
“unikernelized” Semantic patch Patch LibC (80 LoC) Coccinelle (4400 LoC) Musl LibC
7/11 Unikernels Fast Syscalls with Binary Rewriting
I What about static binaries? I (Statically) binary-rewrite syscall instructions to direct jumps to the syscall implementation
I Problem: syscall is 2 bytes long and any call/jmp instruction will be larger
Syscall binary rewriting … jmp 0x200042 (5 bytes) … nop (1 byte) mov $0, %rax (read) nop (1 byte) syscall (2 bytes) mov $3, %rdi mov $2, %esi (5 bytes) … mov $3, %rdi … Rewritten code Snippet mov %r10, %rcx Original code callq 0x200457 (read) mov $2, %esi jmp 0x400aac
8/11 Unikernels Fast Syscalls with Binary Rewriting
I What about static binaries? I (Statically) binary-rewrite syscall instructions to direct jumps to the syscall implementation
I Problem: syscall is 2 bytes long and any call/jmp instruction will be larger
0.030 null (getppid)
) 0.025 Lmbench3 s (
read
e 0.020
m write i t
n 0.015 o i t c 0.010 e x E 0.005 0.000 Linux Hermitux Hermitux Hermitux Lib. (native) Handler Rewrite Substitution
8/11 Unikernels System-call-based Modularity
I System-call based modularity
I Compile a kernel with support for only the necessary system calls I How to identify syscall needed without access to the sources?
I Use binary analysis to find out what is the value in %rax for each syscall invocation
Number of Kernel .text Program system calls size reduction Minimal 5 21.87 % Hello world 10 19.84 % PARSEC Blackscholes 15 17.05 % Postmark 26 14.36 % Sqlite 31 11.34 % Full syscalls support 64 00.00 %
9/11 Unikernels Evaluation
Legend: Low isolation (software or none) Strong isolation (EPT) Binary compatible with Linux Not Binary compatible with Linux 1000 100 100 ) ) s
Boot + Destruction time B RAM ( ) Binary Size
M B
e 10
( usage 100 M m i e ( t
g +
e 1 10 n a t z i s o o i s t
10 o U
c
y B 0.1 y u r r r a t o s n i 1 m 1 e 0.01 B e d M
I Image 650x smaller, boot time 780x faster, RAM usage 9x lower than a Linux VM!
50000 2 HermiTux OSv SET 40000 GET c
Docker Rump e s
/ 30000 s 1 t s
e 20000 u q
e 10000 R Exec. time nor-
malized to Linux 0 BT EP IS Stream Swap NBody 0 hermiTux Linux Osv Rump (Fortran) (C) (C) cluster tions (Python) (C++) (C++) Redis
10/11 Conclusion
I Porting to unikernels is hard
I Hinders their adoption in the industry
I HermiTux provides binary-compatibility with Linux applications
I HermiTux maintains unikernel benefits:
I Fast boot times, small footprints I Various techniques to get fast system calls and modularity
It’s open source, try it out! https://ssrg-vt.github.io/hermitux/
11/11