Container Live Migration
Adrian Reber
2020, July 10 Red Hat Blog:
Container migration with Podman on RHEL
https://www.redhat.com/en/blog/container-migration-podman-rhel
2 DEVOOPS 2020 Agenda
Use cases Details Demos Future
3 DEVOOPS 2020 Definition: Container Live Migration
4 DEVOOPS 2020 Transfer Running Container
5 DEVOOPS 2020 Serialize on Source System
6 DEVOOPS 2020 Transfer to Destination System
7 DEVOOPS 2020 Checkpoint/Restore in Userspace
CRIU
8 DEVOOPS 2020 Multiple Integrations Exist
9 DEVOOPS 2020 Use Cases
10 DEVOOPS 2020 Reboot and Save State
11 DEVOOPS 2020 Container
Host
12 DEVOOPS 2020 Container
Host
13 DEVOOPS 2020 Container
Host
14 DEVOOPS 2020 Container
Host
15 DEVOOPS 2020 Quick Startup
16 DEVOOPS 2020 Container
Host
17 DEVOOPS 2020 Container Container
Host
18 DEVOOPS 2020 Container Container
ContainerHost
Container
19 DEVOOPS 2020 Container Live Migration
20 DEVOOPS 2020 Container
Source Destination
21 DEVOOPS 2020 Container Container
Source Destination
22 DEVOOPS 2020 Container Container Container
Source Destination
Container
23 DEVOOPS 2020 CRIU
24 DEVOOPS 2020 First Step: Checkpointing
25 DEVOOPS 2020 Seize Process Using
ptrace()
26 DEVOOPS 2020 Collect Details From
/proc/
27 DEVOOPS 2020 Parasite Code
28 DEVOOPS 2020 Parasite Code Most favorite part
29 DEVOOPS 2020 Parasite Code
And the craziest
30 DEVOOPS 2020 Parasite Code Injected into the process
31 DEVOOPS 2020 Parasite Code Daemon waiting for commands
32 DEVOOPS 2020 Parasite Code Removed after usage
33 DEVOOPS 2020 Original
Process
Code
To Be
Checkpointed
34 DEVOOPS 2020 Original
Process
Code
To Be
Checkpointed
35 DEVOOPS 2020 Original
Process
Parasite Code
To Be
Checkpointed
36 DEVOOPS 2020 Original
Process
Code
To Be
Checkpointed
37 DEVOOPS 2020 Checkpointing Finished
38 DEVOOPS 2020 Checkpointing Finished
All relevant information written
39 DEVOOPS 2020 Checkpointing Finished Target process is killed
40 DEVOOPS 2020 Checkpointing Finished
Or continues to run
41 DEVOOPS 2020 Container Live Migration SELinux
Linux Security Summit EU 2019
https://sched.co/Tymj
42 DEVOOPS 2020 2015: CRIU LSM support
43 DEVOOPS 2020 During checkpointing
Read /proc/PID/attr/current
44 DEVOOPS 2020 During restore
Write /proc/PID/attr/current
45 DEVOOPS 2020 1 if (!strstartswith(last, "unconfined_")){
2 pr_err("Non unconfined selinux contexts not supported %s\n", last);
3 freecon(ctx);
4 return -1;
5 }
46 DEVOOPS 2020 • setsockcreatecon(3) for parasite daemon • Write to /proc/PID/attr/current • Allow dyntransition • Do not set context of threads • Allow writing to /proc/sys/kernel/ns_last_pid • Fix socket labels • Pre-create CRIU log files with appropriate labels • Fix file descriptor leaks
47 DEVOOPS 2020 Second/Last Step: Restoring
48 DEVOOPS 2020 Read Checkpoint Images
49 DEVOOPS 2020 clone() For Each PID/TID LPC: CRIU and the PID dance
clone3() with Linux 5.5
https://linuxplumbersconf.org/event/4/contributions/472/
50 DEVOOPS 2020 PID dance
open() /proc/sys/kernel/ns_last_pid write() (PID - 1) to ns_last_pid close() ns_last_pid clone() getpid()
51 DEVOOPS 2020 Avoiding the PID dance (2010):
eclone()
https://lore.kernel.org/patchwork/patch/198220/
52 DEVOOPS 2020 Avoiding the PID dance (2019):
clone3()
53 DEVOOPS 2020 ”In general, clone3() is extensible and allows for the implementation of new features.”
https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7f192e3cd316ba58c
54 DEVOOPS 2020 clone3() with set_tid
55 DEVOOPS 2020 1 struct clone_args args = {0};
2 pid_t *set_tid;
3 set_tid[0] = 2020;
4 args.set_tid = set_tid;
5 args.set_tid_size = 1;
6 syscall(__NR_clone3, args, sizeof(struct clone_args));
56 DEVOOPS 2020 CRIU Morphs Itself Open and position file descriptors
57 DEVOOPS 2020 CRIU Morphs Itself Map memory pages
58 DEVOOPS 2020 CRIU Morphs Itself Load security settings
59 DEVOOPS 2020 CRIU Morphs Itself Jump into restored process
60 DEVOOPS 2020 Container Live Migration
61 DEVOOPS 2020 Container Live Migration OpenVZ
62 DEVOOPS 2020 Container Live Migration Borg
63 DEVOOPS 2020 Container Live Migration LXC/LXD
64 DEVOOPS 2020 Container Live Migration
Docker
65 DEVOOPS 2020 Container Live Migration
Podman
66 DEVOOPS 2020 Podman: daemonless
67 DEVOOPS 2020 Podman: rootless
68 DEVOOPS 2020 Podman: Checkpoint/Restore
October 2018
69 DEVOOPS 2020 Podman: Checkpoint/Restore Required runc and CRIU changes
70 DEVOOPS 2020 Podman: Container Live Migration
June 2019
71 DEVOOPS 2020 Podman: Container Live Migration Required runc, CRIU, SELinux changes
72 DEVOOPS 2020 Checkpoint includes File System Changes
73 DEVOOPS 2020 1 # podman run --rm -d adrianreber/wildfly-hello
2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a
3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}"
4 10.88.0.247
5 # curl 10.88.0.247:8080/helloworld/
6 0
7 # curl 10.88.0.247:8080/helloworld/
8 1
9 # podman container checkpoint -l --export=/tmp/chkpt.tar.gz
10 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a
11 # scp /tmp/chkpt.tar.gz rhel08:/tmp
74 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz
2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a
3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}"
4 10.88.0.247
5 # curl 10.88.0.247:8080/helloworld/
6 2
7 # curl 10.88.0.247:8080/helloworld/
8 3
75 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz -n hello1
2 d02feeec894d77f66cc82484fe77ae369396a85f6d05594dc156c21e685942dd
3 # podman container restore --import=/tmp/chkpt.tar.gz -n hello2
4 735efb4fee6961d3eee069beb28dde5cbc6fc46c1a32a43ecc993d04c02015b2
5 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello1
6 10.88.0.248
7 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello2
8 10.88.0.249
9 # curl 10.88.0.248:8080/helloworld/
10 2
11 # curl 10.88.0.249:8080/helloworld/
12 2
76 DEVOOPS 2020 Future:
kubectl migrate
77 DEVOOPS 2020 Future: Non-root checkpoint/restore
78 DEVOOPS 2020 Summary
• CRIU can checkpoint and restore containers • Integrated in different containers engines • Used in production • Reboot into new kernel without losing container state • Start multiple copies • Migrate running containers
79 DEVOOPS 2020 https://lisas.de/~adrian/container-live-migration-article.pdf https://asciinema.org/a/249922 https://asciinema.org/a/249918 https://lisas.de/~adrian/posts/2019-Apr-10-criu-and-selinux.html https://criu.org/Podman https://twitter.com/adrian__reber https://www.redhat.com/en/blog/container-migration-podman-rhel https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/ https://sched.co/Tymj https://linuxplumbersconf.org/event/4/contributions/472/
80 DEVOOPS 2020 Thank you