Container Live Migration

Adrian Reber

2020, July 10 Red Hat Blog:

Container migration with Podman on RHEL

https://www.redhat.com/en/blog/container-migration-podman-rhel

2 DEVOOPS 2020 Agenda

Use cases Details Demos Future

3 DEVOOPS 2020 Definition: Container Live Migration

4 DEVOOPS 2020 Transfer Running Container

5 DEVOOPS 2020 Serialize on Source System

6 DEVOOPS 2020 Transfer to Destination System

7 DEVOOPS 2020 Checkpoint/Restore in Userspace

CRIU

8 DEVOOPS 2020 Multiple Integrations Exist

9 DEVOOPS 2020 Use Cases

10 DEVOOPS 2020 Reboot and Save State

11 DEVOOPS 2020 Container

Host

12 DEVOOPS 2020 Container

Host

13 DEVOOPS 2020 Container

Host

14 DEVOOPS 2020 Container

Host

15 DEVOOPS 2020 Quick Startup

16 DEVOOPS 2020 Container

Host

17 DEVOOPS 2020 Container Container

Host

18 DEVOOPS 2020 Container Container

ContainerHost

Container

19 DEVOOPS 2020 Container Live Migration

20 DEVOOPS 2020 Container

Source Destination

21 DEVOOPS 2020 Container Container

Source Destination

22 DEVOOPS 2020 Container Container Container

Source Destination

Container

23 DEVOOPS 2020 CRIU

24 DEVOOPS 2020 First Step: Checkpointing

25 DEVOOPS 2020 Seize Using

ptrace()

26 DEVOOPS 2020 Collect Details From

/proc//*

27 DEVOOPS 2020 Parasite Code

28 DEVOOPS 2020 Parasite Code Most favorite part

29 DEVOOPS 2020 Parasite Code

And the craziest

30 DEVOOPS 2020 Parasite Code Injected into the process

31 DEVOOPS 2020 Parasite Code Daemon waiting for commands

32 DEVOOPS 2020 Parasite Code Removed after usage

33 DEVOOPS 2020 Original

Process

Code

To Be

Checkpointed

34 DEVOOPS 2020 Original

Process

Code

To Be

Checkpointed

35 DEVOOPS 2020 Original

Process

Parasite Code

To Be

Checkpointed

36 DEVOOPS 2020 Original

Process

Code

To Be

Checkpointed

37 DEVOOPS 2020 Checkpointing Finished

38 DEVOOPS 2020 Checkpointing Finished

All relevant information written

39 DEVOOPS 2020 Checkpointing Finished Target process is killed

40 DEVOOPS 2020 Checkpointing Finished

Or continues to run

41 DEVOOPS 2020 Container Live Migration SELinux

Linux Security Summit EU 2019

https://sched.co/Tymj

42 DEVOOPS 2020 2015: CRIU LSM support

43 DEVOOPS 2020 During checkpointing

Read /proc/PID/attr/current

44 DEVOOPS 2020 During restore

Write /proc/PID/attr/current

45 DEVOOPS 2020 1 if (!strstartswith(last, "unconfined_")){

2 pr_err("Non unconfined selinux contexts not supported %s\n", last);

3 freecon(ctx);

4 return -1;

5 }

46 DEVOOPS 2020 • setsockcreatecon(3) for parasite daemon • Write to /proc/PID/attr/current • Allow dyntransition • Do not set context of threads • Allow writing to /proc/sys/kernel/ns_last_pid • Fix socket labels • Pre-create CRIU log files with appropriate labels • Fix file descriptor leaks

47 DEVOOPS 2020 Second/Last Step: Restoring

48 DEVOOPS 2020 Read Checkpoint Images

49 DEVOOPS 2020 clone() For Each PID/TID LPC: CRIU and the PID dance

clone3() with 5.5

https://linuxplumbersconf.org/event/4/contributions/472/

50 DEVOOPS 2020 PID dance

open() /proc/sys/kernel/ns_last_pid write() (PID - 1) to ns_last_pid close() ns_last_pid clone() getpid()

51 DEVOOPS 2020 Avoiding the PID dance (2010):

eclone()

https://lore.kernel.org/patchwork/patch/198220/

52 DEVOOPS 2020 Avoiding the PID dance (2019):

clone3()

53 DEVOOPS 2020 ”In general, clone3() is extensible and allows for the implementation of new features.”

https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7f192e3cd316ba58c

54 DEVOOPS 2020 clone3() with set_tid

55 DEVOOPS 2020 1 struct clone_args args = {0};

2 pid_t *set_tid;

3 set_tid[0] = 2020;

4 args.set_tid = set_tid;

5 args.set_tid_size = 1;

6 syscall(__NR_clone3, args, sizeof(struct clone_args));

56 DEVOOPS 2020 CRIU Morphs Itself Open and position file descriptors

57 DEVOOPS 2020 CRIU Morphs Itself Map memory pages

58 DEVOOPS 2020 CRIU Morphs Itself Load security settings

59 DEVOOPS 2020 CRIU Morphs Itself Jump into restored process

60 DEVOOPS 2020 Container Live Migration

61 DEVOOPS 2020 Container Live Migration OpenVZ

62 DEVOOPS 2020 Container Live Migration Borg

63 DEVOOPS 2020 Container Live Migration LXC/LXD

64 DEVOOPS 2020 Container Live Migration

Docker

65 DEVOOPS 2020 Container Live Migration

Podman

66 DEVOOPS 2020 Podman: daemonless

67 DEVOOPS 2020 Podman: rootless

68 DEVOOPS 2020 Podman: Checkpoint/Restore

October 2018

69 DEVOOPS 2020 Podman: Checkpoint/Restore Required runc and CRIU changes

70 DEVOOPS 2020 Podman: Container Live Migration

June 2019

71 DEVOOPS 2020 Podman: Container Live Migration Required runc, CRIU, SELinux changes

72 DEVOOPS 2020 Checkpoint includes File System Changes

73 DEVOOPS 2020 1 # podman run --rm -d adrianreber/wildfly-hello

2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a

3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}"

4 10.88.0.247

5 # curl 10.88.0.247:8080/helloworld/

6 0

7 # curl 10.88.0.247:8080/helloworld/

8 1

9 # podman container checkpoint -l --export=/tmp/chkpt.tar.gz

10 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a

11 # scp /tmp/chkpt.tar.gz rhel08:/tmp

74 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz

2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a

3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}"

4 10.88.0.247

5 # curl 10.88.0.247:8080/helloworld/

6 2

7 # curl 10.88.0.247:8080/helloworld/

8 3

75 DEVOOPS 2020 1 # podman container restore --import=/tmp/chkpt.tar.gz -n hello1

2 d02feeec894d77f66cc82484fe77ae369396a85f6d05594dc156c21e685942dd

3 # podman container restore --import=/tmp/chkpt.tar.gz -n hello2

4 735efb4fee6961d3eee069beb28dde5cbc6fc46c1a32a43ecc993d04c02015b2

5 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello1

6 10.88.0.248

7 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello2

8 10.88.0.249

9 # curl 10.88.0.248:8080/helloworld/

10 2

11 # curl 10.88.0.249:8080/helloworld/

12 2

76 DEVOOPS 2020 Future:

kubectl migrate

77 DEVOOPS 2020 Future: Non-root checkpoint/restore

78 DEVOOPS 2020 Summary

• CRIU can checkpoint and restore containers • Integrated in different containers engines • Used in production • Reboot into new kernel without losing container state • Start multiple copies • Migrate running containers

79 DEVOOPS 2020 https://lisas.de/~adrian/container-live-migration-article.pdf https://asciinema.org/a/249922 https://asciinema.org/a/249918 https://lisas.de/~adrian/posts/2019-Apr-10-criu-and-selinux.html https://criu.org/Podman https://twitter.com/adrian__reber https://www.redhat.com/en/blog/container-migration-podman-rhel https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/ https://sched.co/Tymj https://linuxplumbersconf.org/event/4/contributions/472/

80 DEVOOPS 2020 Thank you