Containers in Controls Workshop Rémi Voirin - CERN Accelerator Controls (BE-CO) [email protected] ICALEPCS 2019 - Brooklyn, NY

Sunday 6th October, 2019 Containers in Controls Workshop 2 Outline

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 3 Table of Contents

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 4 Just a reminder...

• This workshop is about sharing knowledge and ideas • Informal discussions • Something unclear or anything to add? Please ask questions during my presentations!

Sunday 6th October, 2019 Containers in Controls Workshop 5 Table of Contents

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 6 Where we come from: namespaces

• Introduced in 2002 in kernel 2.4.19 • Goal: isolate processes - no interference with the rest of the system • The mount namespace came first (think chroot) • There are other namespaces including: • PID (CLONE_NEWPID) • network (CLONE_NEWNET) • (CLONE_NEWNS)

Sunday 6th October, 2019 Containers in Controls Workshop 7 Where we come from: cgroups

• Cap resource usage • CPU • Memory • Disk I/O • Create a rule in a config file ("not more than 200 MB of RAM"), and then apply it to a process

Sunday 6th October, 2019 Containers in Controls Workshop 8 Missing piece

Software to manage isolated processes • LXC (2008): "light virtual machine" (hostname, ssh,...) • Docker (2013): provides an ecosystem (public and private Docker registries!)

Sunday 6th October, 2019 Containers in Controls Workshop 9 The containerisation layer

Sunday 6th October, 2019 Containers in Controls Workshop 10 Vocabulary

Change of paradigm • application (binary, rpm, deb,...) => image • (yum, apt-get,...) install, deploy + start => run / instantiate • repository => registry • download => pull • upload => push

Sunday 6th October, 2019 Containers in Controls Workshop 11 The Open Container Initiative

• Launched by the in 2015 • Goal: design standards for operating-system level virtualisation (i.e. mostly containerisation) • Created runC, now the default container runtime in Docker • Every modern container runtime is OCI-compliant

Sunday 6th October, 2019 Containers in Controls Workshop 12 Table of Contents

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 13 Benefits

• Portability: reproducible stack • Embedded shared libraries (i.e. ship the whole environment) • Only dependent on the container runtime • Easy to wipe/recreate (CI/CD) • Isolation = proper management of dependencies • Network (exposed ports) • Other services (can be shipped together) With nearly identical performance to running on bare-metal (IO, network, CPU) (source)

Sunday 6th October, 2019 Containers in Controls Workshop 14 Considerations

• Initial investment • Dramatic change in the code base: split applications into microservices • Change in CI/CD processes • HR: training! • Management of images: new service to provide • More complex stack • Root causes of problems harder to identify • Learning curve for newcomers (workflow, microservices,...) • Risk of overengineering microservice interactions

Sunday 6th October, 2019 Containers in Controls Workshop 15 Table of Contents

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 16 Initial goal - Docker only

• Let’s containerise a python web application! • We start from a simple application, run it on the system, then run it inside a container

Sunday 6th October, 2019 Containers in Controls Workshop 17 Preparing the virtual machine

• Base: Xubuntu 19.04 • The appliance is already ready, but here are simple steps to set it up # apt-get install docker.io • Download Alpine Linux image with a Python environment # docker pull python:2-alpine

Sunday 6th October, 2019 Containers in Controls Workshop 18 Demo time - Docker only

I’ll show steps here in case you’re lost

Sunday 6th October, 2019 Containers in Controls Workshop 19 Steps - Docker only

• 1. Check the python code (sudo password: workshop) $ sudo -s # cd /hands-on1˜ && less hellobasic.py • 2. Write the Dockerfile FROM python:2-alpine ADD hellobasic.py RUN chmod +x ./hellobasic.py EXPOSE 80/tcp CMD [ "./hellobasic.py" ]

Sunday 6th October, 2019 Containers in Controls Workshop 20 Steps - Docker only

• 3. Build the Docker image # docker build -t workshop/hellobasic:latest . • 4. Run the containerised application # docker run -p 80:80 workshop/hellobasic:latest Check http://localhost/ in the web browser! • Tip: use -d to detach and get back to the shell when starting the container

Sunday 6th October, 2019 Containers in Controls Workshop 21 Docker-compose

• What happens when there is a practical dependency between containers? • Let’s say we need a small database • hellobasic becomes hellodb and uses the database to display "Hello !"

Sunday 6th October, 2019 Containers in Controls Workshop 22 Preparing the virtual machine

• Install mysql-client and docker-compose # apt-get install mysql-client docker-compose • Download MySQL server 5 Docker image # docker pull mysql:5 • Prepare working directories # mkdir -p /hand-on2/sql˜ /hands-on2/python˜ /hands-on2/compose˜

Sunday 6th October, 2019 Containers in Controls Workshop 23 Demo time - Docker-compose

I’ll show steps here in case you’re lost

Sunday 6th October, 2019 Containers in Controls Workshop 24 Steps - Docker-compose

• 1. As root, complete the Docker-compose file (˜/hands-on2/compose/docker-compose.yml) python: image: workshop/hellodb:latest ports: - ’80:80’ image: mysql:5 ports: - ’3306:3306’

Sunday 6th October, 2019 Containers in Controls Workshop 25 Steps - Docker-compose

• 2. Review future database content $ less /hands-on2/sql/workshop.sql˜ • 3. Review changes in the hello python web app (note: the Docker image for hellodb is already prepared) $ less /hands-on2/python/hellodb.py˜ • 4. Launch Docker-compose # cd /hands-on2/compose˜ && docker-compose up

Sunday 6th October, 2019 Containers in Controls Workshop 26 Steps - Docker-compose

• 5. Refresh the web browser (connection reset...) • 6. Import SQL data (password: workshop) # mysql -h127.0.0.1 -u root -p < /hands-on2/sql/workshop.sql˜ • 7. Refresh the web browser, a random name will appear.

Sunday 6th October, 2019 Containers in Controls Workshop 27 Table of Contents

Just a reminder...

Defining containerisation

Impacts of containerisation

Hands-on Docker

Container security

Sunday 6th October, 2019 Containers in Controls Workshop 28 Considerations

• The risk of escaping CVE-2019-5736 (click me): file-descriptor mishandling related to /proc/self/exe allows root in a container to become root on the host! • Base images can be a risk: Alpine CVE 2019-5021 (click me). If shadow or linux-pam are there, a user with a shell can escalate. • These issues can pile up

Sunday 6th October, 2019 Containers in Controls Workshop 29 Container security strategy

• Using Docker containers implies adequate security measures • Strategy must align with accelerator/experiments/user schedules • Balance between security requirements and operations (i.e. guarantee to not disrupt processes within the organisation)

Sunday 6th October, 2019 Containers in Controls Workshop 30 Container security brainstorming Which security measures do you/would you implement regarding Docker containers within controls systems? • Container runtime (Docker) • Docker images • CI/CD ecosystem,... Process: • 5-10 minutes: Ideation • 15-20 minutes: Discuss ideas

Sunday 6th October, 2019 Containers in Controls Workshop 31 Extra slide - Container security at CERN: guidelines Ongoing design of CERN-wide policy on containerisation • Use validated Docker images from the Docker Hub or CERN-based images • Do not allow remote control of the Docker daemon (port 2375) • Use of Docker Bench (https://github.com/ docker/docker-bench-security - requires Docker CE)

Sunday 6th October, 2019 Containers in Controls Workshop 32 Extra slide - Container security at CERN: within controls

• Strict policy on use of Docker • The infrastructure team delivers base Docker images: CERN CentOS 7 (+ JDK, + development tools, + Python) • Developer workflows will be based on these Docker images • Use of external official Docker images is not defined yet • e.g. Grafana? (based on Debian) • ... and we are used to deal only with one Linux distribution!

Sunday 6th October, 2019 Containers in Controls Workshop 33 Extra slide - Container security at CERN: within controls

• Ensure refresh of Docker images • Regular accelerator technical stops • How is still an open question

Sunday 6th October, 2019 Containers in Controls Workshop 34 References

• Introducing Linux Network Namespaces (Scott Lowe) • Separation Anxiety: A Tutorial for Isolating Your System with (Mahmud Ridwan)

Sunday 6th October, 2019 Containers in Controls Workshop 35