Container Live Migration Adrian Reber 2020, July 10 Red Hat Blog: Container migration with Podman on RHEL https://www.redhat.com/en/blog/container-migration-podman-rhel 2 DEVOOPS 2020 Agenda Use cases Details Demos Future 3 DEVOOPS 2020 Definition: Container Live Migration 4 DEVOOPS 2020 Transfer Running Container 5 DEVOOPS 2020 Serialize on Source System 6 DEVOOPS 2020 Transfer to Destination System 7 DEVOOPS 2020 Checkpoint/Restore in Userspace CRIU 8 DEVOOPS 2020 Multiple Integrations Exist 9 DEVOOPS 2020 Use Cases 10 DEVOOPS 2020 Reboot and Save State 11 DEVOOPS 2020 Container Host 12 DEVOOPS 2020 Container Host 13 DEVOOPS 2020 Container Host 14 DEVOOPS 2020 Container Host 15 DEVOOPS 2020 Quick Startup 16 DEVOOPS 2020 Container Host 17 DEVOOPS 2020 Container Container Host 18 DEVOOPS 2020 Container Container ContainerHost Container 19 DEVOOPS 2020 Container Live Migration 20 DEVOOPS 2020 Container Source Destination 21 DEVOOPS 2020 Container Container Source Destination 22 DEVOOPS 2020 Container Container Container Source Destination Container 23 DEVOOPS 2020 CRIU 24 DEVOOPS 2020 First Step: Checkpointing 25 DEVOOPS 2020 Seize Process Using ptrace() 26 DEVOOPS 2020 Collect Details From /proc/<PID>/* 27 DEVOOPS 2020 Parasite Code 28 DEVOOPS 2020 Parasite Code Most favorite part 29 DEVOOPS 2020 Parasite Code And the craziest 30 DEVOOPS 2020 Parasite Code Injected into the process 31 DEVOOPS 2020 Parasite Code Daemon waiting for commands 32 DEVOOPS 2020 Parasite Code Removed after usage 33 DEVOOPS 2020 Original Process Code To Be Checkpointed 34 DEVOOPS 2020 Original Process Code To Be Checkpointed 35 DEVOOPS 2020 Original Process Parasite Code To Be Checkpointed 36 DEVOOPS 2020 Original Process Code To Be Checkpointed 37 DEVOOPS 2020 Checkpointing Finished 38 DEVOOPS 2020 Checkpointing Finished All relevant information written 39 DEVOOPS 2020 Checkpointing Finished Target process is killed 40 DEVOOPS 2020 Checkpointing Finished Or continues to run 41 DEVOOPS 2020 Container Live Migration SELinux Linux Security Summit EU 2019 https://sched.co/Tymj 42 DEVOOPS 2020 2015: CRIU LSM support 43 DEVOOPS 2020 During checkpointing Read /proc/PID/attr/current 44 DEVOOPS 2020 During restore Write /proc/PID/attr/current 45 DEVOOPS 2020 1 if (!strstartswith(last, "unconfined_")){ 2 pr_err("Non unconfined selinux contexts not supported %s\n", last); 3 freecon(ctx); 4 return -1; 5 } 46 DEVOOPS 2020 • setsockcreatecon(3) for parasite daemon • Write to /proc/PID/attr/current • Allow dyntransition • Do not set context of threads • Allow writing to /proc/sys/kernel/ns_last_pid • Fix socket labels • Pre-create CRIU log files with appropriate labels • Fix file descriptor leaks 47 DEVOOPS 2020 Second/Last Step: Restoring 48 DEVOOPS 2020 Read Checkpoint Images 49 DEVOOPS 2020 clone() For Each PID/TID LPC: CRIU and the PID dance clone3() with Linux 5.5 https://linuxplumbersconf.org/event/4/contributions/472/ 50 DEVOOPS 2020 PID dance open() /proc/sys/kernel/ns_last_pid write() (PID - 1) to ns_last_pid close() ns_last_pid clone() getpid() 51 DEVOOPS 2020 Avoiding the PID dance (2010): eclone() https://lore.kernel.org/patchwork/patch/198220/ 52 DEVOOPS 2020 Avoiding the PID dance (2019): clone3() 53 DEVOOPS 2020 ”In general, clone3() is extensible and allows for the implementation of new features.” https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7f192e3cd316ba58c 54 DEVOOPS 2020 clone3() with set_tid 55 DEVOOPS 2020 1 struct clone_args args = {0}; 2 pid_t *set_tid; 3 set_tid[0] = 2020; 4 args.set_tid = set_tid; 5 args.set_tid_size = 1; 6 syscall(__NR_clone3, args, sizeof(struct clone_args)); 56 DEVOOPS 2020 CRIU Morphs Itself Open and position file descriptors 57 DEVOOPS 2020 CRIU Morphs Itself Map memory pages 58 DEVOOPS 2020 CRIU Morphs Itself Load security settings 59 DEVOOPS 2020 CRIU Morphs Itself Jump into restored process 60 DEVOOPS 2020 Container Live Migration 61 DEVOOPS 2020 Container Live Migration OpenVZ 62 DEVOOPS 2020 Container Live Migration Borg 63 DEVOOPS 2020 Container Live Migration LXC/LXD 64 DEVOOPS 2020 Container Live Migration Docker 65 DEVOOPS 2020 Container Live Migration Podman 66 DEVOOPS 2020 Podman: daemonless 67 DEVOOPS 2020 Podman: rootless 68 DEVOOPS 2020 Podman: Checkpoint/Restore October 2018 69 DEVOOPS 2020 Podman: Checkpoint/Restore Required runc and CRIU changes 70 DEVOOPS 2020 Podman: Container Live Migration June 2019 71 DEVOOPS 2020 Podman: Container Live Migration Required runc, CRIU, SELinux changes 72 DEVOOPS 2020 Checkpoint includes File System Changes 73 DEVOOPS 2020 1 # podman run --rm -d adrianreber/wildfly-hello 2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}" 4 10.88.0.247 5 # curl 10.88.0.247:8080/helloworld/ 6 0 7 # curl 10.88.0.247:8080/helloworld/ 8 1 9 # podman container checkpoint -l --export=/tmp/chkpt.tar.gz 10 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 11 # scp /tmp/chkpt.tar.gz rhel08:/tmp 74 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz 2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}" 4 10.88.0.247 5 # curl 10.88.0.247:8080/helloworld/ 6 2 7 # curl 10.88.0.247:8080/helloworld/ 8 3 75 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz -n hello1 2 d02feeec894d77f66cc82484fe77ae369396a85f6d05594dc156c21e685942dd 3 # podman container restore --import=/tmp/chkpt.tar.gz -n hello2 4 735efb4fee6961d3eee069beb28dde5cbc6fc46c1a32a43ecc993d04c02015b2 5 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello1 6 10.88.0.248 7 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello2 8 10.88.0.249 9 # curl 10.88.0.248:8080/helloworld/ 10 2 11 # curl 10.88.0.249:8080/helloworld/ 12 2 76 DEVOOPS 2020 Future: kubectl migrate 77 DEVOOPS 2020 Future: Non-root checkpoint/restore 78 DEVOOPS 2020 Summary • CRIU can checkpoint and restore containers • Integrated in different containers engines • Used in production • Reboot into new kernel without losing container state • Start multiple copies • Migrate running containers 79 DEVOOPS 2020 https://lisas.de/~adrian/container-live-migration-article.pdf https://asciinema.org/a/249922 https://asciinema.org/a/249918 https://lisas.de/~adrian/posts/2019-Apr-10-criu-and-selinux.html https://criu.org/Podman https://twitter.com/adrian__reber https://www.redhat.com/en/blog/container-migration-podman-rhel https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/ https://sched.co/Tymj https://linuxplumbersconf.org/event/4/contributions/472/ 80 DEVOOPS 2020 Thank you.