A Binary Compatible Unikernel

Pierre Olivier*, Daniel Chiba*, Stefan Lankes+, Changwoo Min*, Binoy Ravidnran*

*Virginia Tech, +RWTH Aachen University

VEE’19 - 04/14/2019 Unikernels Presentation

Full-fledged Virtual Machine distribution

Application

Libraries

OS interface Linux used Kernel Hardware

2/11 Unikernels Presentation

Full-fledged Virtual Machine Linux Legend : distribution Useful software Software bloat Application

Libraries

OS interface Linux used Kernel Hypervisor Hardware

2/11 Unikernels Presentation

Full-fledged Virtual Machine Linux Legend : distribution Useful software Software bloat Application Unikernel Libraries Application OS interface Linux Libraries used Kernel OS Layer Hypervisor Hardware

2/11 I single-*

I Single purpose: run 1 application I Single I Single binary and single address space for application + kernel

I No user/kernel protection needed I Lightweight , alternative to containers

I Security advantage: small attack surface and high isolation I Per-application tailored kernel

I LibOS/ model I Reduced OS noise, increased performance

I Low system call latency

I App + kernel in ring 0, system calls are function calls

Unikernels Presentation (2)

Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor

3/11 I Lightweight virtualization, alternative to containers

I Security advantage: small attack surface and high isolation I Per-application tailored kernel

I LibOS/Exokernel model I Reduced OS noise, increased performance

I Low system call latency

I App + kernel in ring 0, system calls are function calls

Unikernels Presentation (2)

Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor

I single-*

I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel

I No user/kernel protection needed

3/11 I Per-application tailored kernel

I LibOS/Exokernel model I Reduced OS noise, increased performance

I Low system call latency

I App + kernel in ring 0, system calls are function calls

Unikernels Presentation (2)

Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor

I single-*

I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel

I No user/kernel protection needed I Lightweight virtualization, alternative to containers

I Security advantage: small attack surface and high isolation

3/11 I Reduced OS noise, increased performance

I Low system call latency

I App + kernel in ring 0, system calls are function calls

Unikernels Presentation (2)

Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor

I single-*

I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel

I No user/kernel protection needed I Lightweight virtualization, alternative to containers

I Security advantage: small attack surface and high isolation I Per-application tailored kernel

I LibOS/Exokernel model

3/11 Unikernels Presentation (2)

Unikernel: application + dependencies + thin OS compiled as a static binary running on top of a hypervisor

I single-*

I Single purpose: run 1 application I Single process I Single binary and single address space for application + kernel

I No user/kernel protection needed I Lightweight virtualization, alternative to containers

I Security advantage: small attack surface and high isolation I Per-application tailored kernel

I LibOS/Exokernel model I Reduced OS noise, increased performance

I Low system call latency

I App + kernel in ring 0, system calls are function calls

3/11 Because it is hard to port existing applications!

Unikernels The Issue

I Unikernels have plenty of benefits to bring

I Unikernels have plenty of application domains

I They are very popular in academia . . .

I . . . but why (nearly) nobody uses them in the industry?

4/11 Unikernels The Issue

I Unikernels have plenty of benefits to bring

I Unikernels have plenty of application domains

I They are very popular in academia . . .

I . . . but why (nearly) nobody uses them in the industry?

Because it is hard to port existing applications!

4/11 I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Build Unikernel (compile, binary Unikernel & link, etc.) libraries src

5/11 I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Build Unikernel (compile, binary Unikernel & link, etc.) libraries src

I Proprietary software → source code not available

5/11 I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Build Unikernel (compile, binary Unikernel & link, etc.) libraries src

I Proprietary software → source code not available I Incompatible language

5/11 I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Build Unikernel (compile, binary Unikernel & link, etc.) libraries src

I Proprietary software → source code not available I Incompatible language I Unsupported features

5/11 I Complex build toolchains HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Build Unikernel (compile, binary Unikernel & link, etc.) libraries src

I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel

5/11 HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

Unikernels The Issue: Porting to Unikernels

Application Execution src

Unikernel Build binary Unikernel & (compile, libraries src link, etc.)

I Proprietary software → source code not available I Incompatible language I Unsupported features I Porting is hard, needs knowledge about both application and unikernel I Complex build toolchains

5/11 Unikernels The Issue: Porting to Unikernels

I Proprietary software → source code not available

I Incompatible language

I Unsupported features

I Porting is hard, needs knowledge about both application and unikernel

I Complex build toolchains

HermiTux Solution

I A unikernel binary-compatible with Linux

I For x86-64 for now

5/11 I ELF loader convention I Load-time Stack layout I Syscalls

I Kernel adapted from HermitCore

I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

6/11 I Syscalls

I Kernel adapted from HermitCore

I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention I Load-time Stack layout

6/11 I Kernel adapted from HermitCore

I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention I Load-time Stack layout I Syscalls

6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention I Load-time Stack layout I Syscalls

I Kernel adapted Hypervisor: uHyve from HermitCore

Host: Linux kernel KVM

6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention I Load-time Stack

t layout Single-address space VM s e I Syscalls u G I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM

6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention Native I Load-time Stack

t layout

Linux s e I Syscalls App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM

6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

I ELF loader convention Native I Load-time Stack

t layout

Linux s

Hermitux e I Syscalls kernel App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM

6/11 I Complete/partial support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

Init. stack and jump to entry point I ELF loader convention Native I Load-time Stack

t layout

Linux s

Hermitux e I Syscalls kernel App. u G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H Host: Linux kernel KVM

6/11 I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack

Syscall t layout

Linux s

Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM support for 80+ syscalls

6/11 I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

Unikernels Overview

I Linux ABI convention:

Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack

Syscall t layout

Linux s

Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM Debug, profile support for 80+ syscalls

6/11 Unikernels Overview

I Linux ABI convention:

Init. stack and jump to entry point I ELF loader convention handler Native I Load-time Stack

Syscall t layout

Linux s

Hermitux e I Syscalls App. u kernel G Load I Kernel adapted t Hypervisor: uHyve s from HermitCore o H I Complete/partial Host: Linux kernel KVM Debug, profile support for 80+ syscalls I How to maintain unikernel benefits without access to the application sources?

I Fast system calls and modularity

6/11 I For dynamically compiled programs:

I At runtime load a unikernel-aware Libc

I Making for system calls (fast) function calls directly into the kernel

I Automatically transformed version of Musl Libc with Coccinelle

Unikernels Fast Syscalls with Libc Substitution

I HermiTux’s syscall handler is invoked by the syscall instruction

I Reintroduce high latency for system calls due to the world switch

7/11 Unikernels Fast Syscalls with Libc Substitution

I HermiTux’s syscall handler is invoked by the syscall instruction

I Reintroduce high latency for system calls due to the world switch

I For dynamically compiled programs:

I At runtime load a unikernel-aware Libc

I Making for system calls (fast) function calls directly into the kernel

I Automatically transformed version of Musl Libc with Coccinelle

“unikernelized” Semantic patch Patch LibC (80 LoC) Coccinelle (4400 LoC) Musl LibC

7/11 Unikernels Fast Syscalls with Binary Rewriting

I What about static binaries? I (Statically) binary-rewrite syscall instructions to direct jumps to the syscall implementation

I Problem: syscall is 2 bytes long and any call/jmp instruction will be larger

Syscall binary rewriting … jmp 0x200042 (5 bytes) … nop (1 byte) mov $0, %rax (read) nop (1 byte) syscall (2 bytes) mov $3, %rdi mov $2, %esi (5 bytes) … mov $3, %rdi … Rewritten code Snippet mov %r10, %rcx Original code callq 0x200457 (read) mov $2, %esi jmp 0x400aac

8/11 Unikernels Fast Syscalls with Binary Rewriting

I What about static binaries? I (Statically) binary-rewrite syscall instructions to direct jumps to the syscall implementation

I Problem: syscall is 2 bytes long and any call/jmp instruction will be larger

0.030 null (getppid)

) 0.025 Lmbench3 s (

read

e 0.020

m write i t

n 0.015 o i t c 0.010 e x E 0.005 0.000 Linux Hermitux Hermitux Hermitux Lib. (native) Handler Rewrite Substitution

8/11 Unikernels System-call-based Modularity

I System-call based modularity

I Compile a kernel with support for only the necessary system calls I How to identify syscall needed without access to the sources?

I Use binary analysis to find out what is the value in %rax for each syscall invocation

Number of Kernel .text Program system calls size reduction Minimal 5 21.87 % Hello world 10 19.84 % PARSEC Blackscholes 15 17.05 % Postmark 26 14.36 % Sqlite 31 11.34 % Full syscalls support 64 00.00 %

9/11 Unikernels Evaluation

Legend: Low isolation (software or none) Strong isolation (EPT) Binary compatible with Linux Not Binary compatible with Linux 1000 100 100 ) ) s

Boot + Destruction time B RAM ( ) Binary Size

M B

e 10

( usage 100 M m i e ( t

g +

e 1 10 n a t z i s o o i s t

10 o U

c

y B 0.1 y u r r r a t o s n i 1 m 1 e 0.01 B e d M

I Image 650x smaller, boot time 780x faster, RAM usage 9x lower than a Linux VM!

50000 2 HermiTux OSv SET 40000 GET c

Docker Rump e s

/ 30000 s 1 t s

e 20000 u q

e 10000 R Exec. time nor-

malized to Linux 0 BT EP IS Stream Swap NBody 0 hermiTux Linux Osv Rump (Fortran) (C) (C) cluster tions (Python) (C++) (C++) Redis

10/11 Conclusion

I Porting to unikernels is hard

I Hinders their adoption in the industry

I HermiTux provides binary-compatibility with Linux applications

I HermiTux maintains unikernel benefits:

I Fast boot times, small footprints I Various techniques to get fast system calls and modularity

It’s open source, try it out! https://ssrg-vt.github.io/hermitux/

11/11