Live migration in LXD lxc move foo host2:

Tycho Andersen, Canonical Ltd.

[email protected] http://tycho.ws Who is this guy?

➔ LXD ➔ LXC ➔ CRIU ➔ Kernel The history of LXC Over 7 years of Linux system containers

2008 2009 2010 2011 2012 2013 2014 2015 2016

LXC 0.6 LXC 0.5

LXC 0.3 LXC 0.7 LX{C,D} 2.0 LXC 0.4 LXC 0.1 LXC 0.8 LXC 0.9 LXC 1.1 LXC 0.2 LXC 1.0

Beginning of Major project LXD 0.1 the LXC project rework LXD: a modern frontend for LXC

nova-lxd lxc (command line tool) your own client/script ?

LXD REST API

LXD LXD LXD LXD LXD

LXC LXC LXC LXC LXC

Linux kernel Linux kernel Linux kernel Linux kernel Linux kernel

Host A Host B Host C Host D Host ... Hypvervisor-y things lxc move host1:c1 host2:

➔ What actually happens when you type `lxc move`? lxc move host1:foo host2: lxc move host1:foo host2:

The history of CRIU Five years of checkpointing!

2006 ... 2011 2012 2013 2014 2015 2016

First kernel LXC gets LXD w/ CRIU patches support for C/R released, Virtuozzo Attempts to merge via CRIU OpenVZ in-kernel merged 7 migration based upstream on CRIU First discussion on CRIU 0.1 libcontainer gets lkml about C/R from support for C/R via Userspace CRIU

Memory page C/R in OpenVZ tracking patches Kernel merged upstream What’s the catch?

“A note on this: this is a project by various mad Russians to perform c/r mainly from userspace, with various oddball helper code added into the kernel where the need is demonstrated… However I'm less confident than the developers that it will all eventually work!” - Linus Torvalds (kernel commit 09946950) How does it work?

[lxc monitor] /var/lib/lxd/containers zesty \_ /sbin/init \_ /lib/systemd/systemd-journald \_ /lib/systemd/systemd-udevd \_ [udevadm] \_ /bin/sh /etc/init.d/apparmor start | \_ systemctl -p LoadState apparmor \_ /bin/bash /etc/init.d/ebtables start | \_ /bin/bash /etc/init.d/ebtables start | \_ systemctl -p LoadState ebtables \_ /usr/bin/python3 /usr/bin/cloud-init init How does it work?

[lxc monitor] /var/lib/lxd/containers zesty \_ 6 * fork(); setproctitle(“init”); open(); ... \_ setproctitle(“systemd-journald”); bind();... \_ setproctitle(“systemd-udevd”); ... \_ setproctitle(“[udevadm]”); exit(); \_ fork(); setproctitle(“sh”); ... | \_ open(“/etc/apparmor.d”); mmap(); ... \_ fork(); setsid(); ... | \_ fork(); setproctitle(“bash”); ... | \_ socket(AF_NETLINK, ...); setuid(); ... \_ setproctitle(“cloud-init”); ... “This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind.” - Lennart Pottering (systemd-devel, 2015)

“This gives me the creeps.” - Kees Cook (seccomp maintainer, lkml, 2015) “Can we refuse dumping selinux labels...until we understand how to properly do it?” - Pavel Emelyanov (CRIU list, May 2015)

Smoke and mirrors! Coming down the pike

● task_diag work by Andrei Vagan ● AF_UNIX and SCM_* ● userfaultfd (post-copy) by Adrain Reber && Mike Rapoport ● More LSMs and PTRACE_O_SUSPEND_LSM ● netlink ● P.haul for go Correct and Fast

Pick two Administrivia

➔ LXD ◆ Current release 2.5 ◆ 2.6 targeted for November 2016 ◆ Monthly release cadence ◆ https://linuxcontainers.org ◆ https://github.com/lxc/lxd ➔ CRIU ◆ Current stable release 2.7 “Rubber Owl” ◆ 2.8 Targeted for November 2016 ◆ Monthly release cadence ◆ http://criu.org ◆ https://github.com/xemul/criu Tycho Andersen, Canonical Ltd. [email protected] http://tycho.ws https://linuxcontainers.org/lxd https://github.com/lxc/lxd

Questions?