Building a Defense-In-Depth OS Around Linux Kernel Security Improvements

CLIP OS: Building a defense-in-depth OS around Linux kernel security improvements

Timothée Ravier, Mickaël Salaün

Agence nationale de la sécurité des systèmes d’information (ANSSI)

September 26, 2018 About the ANSSI

I Agence nationale de la sécurité des systèmes d’information I French authority in the area of cyberdefence, network and information security

I We are not an intelligence agency

ANSSI CLIP OS: Building a defense-in-depth OS 2/27 Overview CLIP OS ?

I Linux distribution developed by the ANSSI I Initially only available internally I Now open source, mostly under the LGPL v2.1+ I Code and issue tracker hosted on GitHub: I Version 4: available as reference and for upstream patch contribution1 I Version 5: currently developed version, alpha status2

1https://github.com/CLIPOS-Archive 2https://github.com/CLIPOS

ANSSI CLIP OS: Building a defense-in-depth OS 4/27 Hardened OS

I Hardened Linux kernel and userspace I Conﬁned services I "Unprivileged" admin, audit and update roles: ⇒ the root account is not usable

I Automatic updates using A/B partition model (similar to Android 7+)

ANSSI CLIP OS: Building a defense-in-depth OS 5/27 Multilevel security OS

I Provide two isolated user environments: low and high I Interactions follow the Bell-LaPadula model: I Write up: upload documents from low to high I Read down: high has read only access to untrusted USB devices I Trusted write down: encrypt documents from high to write them in low

I Level high can only access network through a VPN I Per level user device assignment

ANSSI CLIP OS: Building a defense-in-depth OS 6/27 Multilevel from the end user point of view

ANSSI CLIP OS: Building a defense-in-depth OS 7/27 Admin panel: devices assignment per level

ANSSI CLIP OS: Building a defense-in-depth OS 8/27 Goal of CLIP OS I We target non-expert users I Bell-LaPadula model with two levels I We favor a defense-in-depth approach

Technical point of view I Hypervisor vs. supervisor isolation I Limited access right, even for the administrator

Diﬀerences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

ANSSI CLIP OS: Building a defense-in-depth OS 9/27 Technical point of view I Hypervisor vs. supervisor isolation I Limited access right, even for the administrator

Diﬀerences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

Goal of CLIP OS I We target non-expert users I Bell-LaPadula model with two levels I We favor a defense-in-depth approach

ANSSI CLIP OS: Building a defense-in-depth OS 9/27 Diﬀerences with Qubes OS

CLIP OS development began 5 years earlier than Qubes OS

Goal of CLIP OS I We target non-expert users I Bell-LaPadula model with two levels I We favor a defense-in-depth approach

Technical point of view I Hypervisor vs. supervisor isolation I Limited access right, even for the administrator

ANSSI CLIP OS: Building a defense-in-depth OS 9/27 Architecture

ANSSI CLIP OS: Building a defense-in-depth OS 10/27 CLIP OS 4 Linux-VServer I Linux namespaces with additional constraints I Unique container and network IDs: XIDs and NIDs

grsecurity/PaX I Kernel self-protection (e.g., memory protection, CFI) I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSM I Complement the Linux permission model I Leverage Linux-VServer and grsecurity/PaX

Hardening mechanisms

Gentoo Hardened I Hardened toolchain I Flexible patching

ANSSI CLIP OS: Building a defense-in-depth OS 12/27 grsecurity/PaX I Kernel self-protection (e.g., memory protection, CFI) I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSM I Complement the Linux permission model I Leverage Linux-VServer and grsecurity/PaX

Hardening mechanisms

Gentoo Hardened I Hardened toolchain I Flexible patching

Linux-VServer I Linux namespaces with additional constraints I Unique container and network IDs: XIDs and NIDs

ANSSI CLIP OS: Building a defense-in-depth OS 12/27 CLIP LSM I Complement the Linux permission model I Leverage Linux-VServer and grsecurity/PaX

Hardening mechanisms

Gentoo Hardened I Hardened toolchain I Flexible patching

Linux-VServer I Linux namespaces with additional constraints I Unique container and network IDs: XIDs and NIDs

grsecurity/PaX I Kernel self-protection (e.g., memory protection, CFI) I Multiple userspace hardening features (e.g., chroot, TPE)

ANSSI CLIP OS: Building a defense-in-depth OS 12/27 Hardening mechanisms

Gentoo Hardened I Hardened toolchain I Flexible patching

Linux-VServer I Linux namespaces with additional constraints I Unique container and network IDs: XIDs and NIDs

grsecurity/PaX I Kernel self-protection (e.g., memory protection, CFI) I Multiple userspace hardening features (e.g., chroot, TPE)

CLIP LSM I Complement the Linux permission model I Leverage Linux-VServer and grsecurity/PaX

ANSSI CLIP OS: Building a defense-in-depth OS 12/27 Memory (PaX) Deny writable memory to be executable, throughout the system lifetime

Devctl Enforce and extend W ⊕ X from devices to mount points

Mount points Enforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC ﬂag Enforce and extend W ⊕ X from mount points to scripts (via interpreters)

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevel isolation

ANSSI CLIP OS: Building a defense-in-depth OS 13/27 Devctl Enforce and extend W ⊕ X from devices to mount points

Mount points Enforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC ﬂag Enforce and extend W ⊕ X from mount points to scripts (via interpreters)

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevel isolation Memory (PaX) Deny writable memory to be executable, throughout the system lifetime

ANSSI CLIP OS: Building a defense-in-depth OS 13/27 Mount points Enforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC ﬂag Enforce and extend W ⊕ X from mount points to scripts (via interpreters)

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevel isolation Memory (PaX) Deny writable memory to be executable, throughout the system lifetime

Devctl Enforce and extend W ⊕ X from devices to mount points

ANSSI CLIP OS: Building a defense-in-depth OS 13/27 The O_MAYEXEC ﬂag Enforce and extend W ⊕ X from mount points to scripts (via interpreters)

Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevel isolation Memory (PaX) Deny writable memory to be executable, throughout the system lifetime

Devctl Enforce and extend W ⊕ X from devices to mount points

Mount points Enforce W ⊕ X thanks to mount options: ro ⊕ noexec

ANSSI CLIP OS: Building a defense-in-depth OS 13/27 Write ⊕ Execute policy

Avoid arbitrary code execution and persistent attacks, improve multilevel isolation Memory (PaX) Deny writable memory to be executable, throughout the system lifetime

Devctl Enforce and extend W ⊕ X from devices to mount points

Mount points Enforce W ⊕ X thanks to mount options: ro ⊕ noexec

The O_MAYEXEC ﬂag Enforce and extend W ⊕ X from mount points to scripts (via interpreters)

ANSSI CLIP OS: Building a defense-in-depth OS 13/27 O_MAYEXEC

ANSSI CLIP OS: Building a defense-in-depth OS 14/27 Container content and interaction I Tailored ﬁlesystem layouts per service I Container management with vsctl and clip-libvserver (self-jailing)

Partitioning

Hardened containers I Leverage Linux-VServer admin and watch (audit) concepts I New capability bounding sets: for root and per container I Hardened chroot

ANSSI CLIP OS: Building a defense-in-depth OS 15/27 Partitioning

Hardened containers I Leverage Linux-VServer admin and watch (audit) concepts I New capability bounding sets: for root and per container I Hardened chroot

Container content and interaction I Tailored ﬁlesystem layouts per service I Container management with vsctl and clip-libvserver (self-jailing)

ANSSI CLIP OS: Building a defense-in-depth OS 15/27 Conﬁguration example: /etc/verictl.d/chromium

/usr/.../chrome-sandbox 1002 e SETUID|SETGID|SYS_CHROOT SETUID|SETGID|SYS_CHROOT - cUP sha256 45bcbd1...

Veriexec and permissions (CLIP-LSM)

Goal I Split Linux capabilities (e.g., Fuse, unshare) I Add new permissions (e.g., network, XFRM) I Can be tied to an XID I Does not use xattr (thus independent from the ﬁlesystem)

ANSSI CLIP OS: Building a defense-in-depth OS 16/27 Veriexec and permissions (CLIP-LSM)

Goal I Split Linux capabilities (e.g., Fuse, unshare) I Add new permissions (e.g., network, XFRM) I Can be tied to an XID I Does not use xattr (thus independent from the ﬁlesystem)

Conﬁguration example: /etc/verictl.d/chromium

/usr/.../chrome-sandbox 1002 e SETUID|SETGID|SYS_CHROOT SETUID|SETGID|SYS_CHROOT - cUP sha256 45bcbd1...

ANSSI CLIP OS: Building a defense-in-depth OS 16/27 Veriexec example

ANSSI CLIP OS: Building a defense-in-depth OS 17/27 CLIP OS 5 General Linux kernel hardening

I Strict whitelist of kernel options, but easily composable sets I Paranoid command line I iommu=force, pti=on, spectre_v2=on, etc.

I Strict sysctl defaults I kernel.kptr_restrict, kernel.yama.ptrace_scope, etc.

ANSSI CLIP OS: Building a defense-in-depth OS 19/27 Security may come ﬁrst I We can handle minor compatibility breakage in our userspace I Will accept changes that upstream may reject

Interaction with upstream & KSPP I Include in-progress or ready-for-upstream patches I Integrate and validate patches in a single tree I Maintain hardening patches for latest stable kernel

Enabling Linux kernel hardening

Goals I Protecting the kernel from itself and from userspace I Include additional features for userspace I Being able to test kernel and userspace coordinated changes

ANSSI CLIP OS: Building a defense-in-depth OS 20/27 Interaction with upstream & KSPP I Include in-progress or ready-for-upstream patches I Integrate and validate patches in a single tree I Maintain hardening patches for latest stable kernel

Enabling Linux kernel hardening

Goals I Protecting the kernel from itself and from userspace I Include additional features for userspace I Being able to test kernel and userspace coordinated changes

Security may come ﬁrst I We can handle minor compatibility breakage in our userspace I Will accept changes that upstream may reject

ANSSI CLIP OS: Building a defense-in-depth OS 20/27 Enabling Linux kernel hardening

Goals I Protecting the kernel from itself and from userspace I Include additional features for userspace I Being able to test kernel and userspace coordinated changes

Security may come ﬁrst I We can handle minor compatibility breakage in our userspace I Will accept changes that upstream may reject

Interaction with upstream & KSPP I Include in-progress or ready-for-upstream patches I Integrate and validate patches in a single tree I Maintain hardening patches for latest stable kernel

ANSSI CLIP OS: Building a defense-in-depth OS 20/27 I Development status: In progress I CLIP OS status: Merged I Upstream status: Most changes unlikely to be merged upstream

Patch series: linux-hardened

Features I Memory hardening improvements, including: I better userspace ASLR I slab allocators hardening (mostly SLUB) I simpler page sanitizing I Various restrictions: TIOCSTI ioctl, perf subsystem, device timing side channels, etc. I Miscellaneous additions: more BUG_ONs, more __ro_after_init, etc.

ANSSI CLIP OS: Building a defense-in-depth OS 21/27 Patch series: linux-hardened

I Development status: In progress I CLIP OS status: Merged I Upstream status: Most changes unlikely to be merged upstream

ANSSI CLIP OS: Building a defense-in-depth OS 21/27 I Development status: Feature complete I CLIP OS status: Merged I Upstream status: Ready for upstream integration

Upstream contribution integration: Lockdown

Features I Reduce options for root to run untrusted code in kernel context

ANSSI CLIP OS: Building a defense-in-depth OS 22/27 Upstream contribution integration: Lockdown

Features I Reduce options for root to run untrusted code in kernel context

I Development status: Feature complete I CLIP OS status: Merged I Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 22/27 CLIP OS speciﬁc changes I Kept alloca()-related changes (dropped for upstream in v15)

I Development status: Feature complete I CLIP OS status: Merged I Upstream status: Ready for upstream integration

Upstream contribution integration: STACKLEAK

Features I Reduce information leaks and block attacks using uninitialized kernel stack variables: I Erase the stack before returning from system calls I Improve runtime detection of kernel stack overﬂows (e.g. Stack Clash): I Instrument calls to alloca()

ANSSI CLIP OS: Building a defense-in-depth OS 23/27 I Development status: Feature complete I CLIP OS status: Merged I Upstream status: Ready for upstream integration

Upstream contribution integration: STACKLEAK

CLIP OS speciﬁc changes I Kept alloca()-related changes (dropped for upstream in v15)

ANSSI CLIP OS: Building a defense-in-depth OS 23/27 Upstream contribution integration: STACKLEAK

CLIP OS speciﬁc changes I Kept alloca()-related changes (dropped for upstream in v15)

I Development status: Feature complete I CLIP OS status: Merged I Upstream status: Ready for upstream integration

ANSSI CLIP OS: Building a defense-in-depth OS 23/27 I Development status: Initial feature set ready I CLIP OS status: Planned I Upstream status: Work in progress

Upstream contribution: Landlock

Features I Enables seccomp-bpf -like self-sandboxing for unprivileged processes I Stackable LSM I Powered by eBPF I Dynamic ﬁlesystem access control using whitelists & blacklists I See landlock.io

ANSSI CLIP OS: Building a defense-in-depth OS 24/27 Upstream contribution: Landlock

Features I Enables seccomp-bpf -like self-sandboxing for unprivileged processes I Stackable LSM I Powered by eBPF I Dynamic ﬁlesystem access control using whitelists & blacklists I See landlock.io

I Development status: Initial feature set ready I CLIP OS status: Planned I Upstream status: Work in progress

ANSSI CLIP OS: Building a defense-in-depth OS 24/27 I Development status: Early development stage I CLIP OS status: Planned

Upstream contribution: VServer-like LSM

Features I Adds a single kernel enforced indentiﬁer for conﬁned environments I Similar in principle to VServer XID or to "Container IDs" I Inspired by the VServer patch I Integrated as a stackable LSM

ANSSI CLIP OS: Building a defense-in-depth OS 25/27 Upstream contribution: VServer-like LSM

I Development status: Early development stage I CLIP OS status: Planned

ANSSI CLIP OS: Building a defense-in-depth OS 25/27 Ongoing project I Contributions welcome I Browse the doc and the sources to ﬁnd more interesting features: docs.clip-os.org

Conclusion

Take away I Hardened Linux distro and kernel I Coordinated userspace and kernelspace I Support multilevel security

ANSSI CLIP OS: Building a defense-in-depth OS 26/27 Conclusion

Take away I Hardened Linux distro and kernel I Coordinated userspace and kernelspace I Support multilevel security

Ongoing project I Contributions welcome I Browse the doc and the sources to ﬁnd more interesting features: docs.clip-os.org

ANSSI CLIP OS: Building a defense-in-depth OS 26/27 Thanks!

clip-os.org B [email protected]

v4: github.com/CLIPOS-Archive v5: github.com/CLIPOS

We’re hiring! (but not directly for CLIP OS) Linux system security expert https://www.ssi.gouv.fr/emploi/expert-en-securite-des-systemes-linux/ I Linux kernel I initramfs I kernel command line

I DM-Verity root hash set in kernel command line I Forward error correction support (FEC) I Read only uncompressed SquashFS root ﬁlesystem

2 Minimal bootloader (gummiboot/systemd-boot) 3 EFI bundle:

4 DM-Verity partition:

Boot chain and root partition integrity protection

1 UEFI Secure Boot support: I Custom keys (i.e. not signed by Microsoft) I Requires enrollment in hardware

ANSSI CLIP OS: Building a defense-in-depth OS 28/27 I Linux kernel I initramfs I kernel command line

I DM-Verity root hash set in kernel command line I Forward error correction support (FEC) I Read only uncompressed SquashFS root ﬁlesystem

3 EFI bundle:

4 DM-Verity partition:

Boot chain and root partition integrity protection

1 UEFI Secure Boot support: I Custom keys (i.e. not signed by Microsoft) I Requires enrollment in hardware 2 Minimal bootloader (gummiboot/systemd-boot)

ANSSI CLIP OS: Building a defense-in-depth OS 28/27 I DM-Verity root hash set in kernel command line I Forward error correction support (FEC) I Read only uncompressed SquashFS root ﬁlesystem

4 DM-Verity partition:

Boot chain and root partition integrity protection

ANSSI CLIP OS: Building a defense-in-depth OS 28/27 Boot chain and root partition integrity protection

1 UEFI Secure Boot support: I Custom keys (i.e. not signed by Microsoft) I Requires enrollment in hardware 2 Minimal bootloader (gummiboot/systemd-boot) 3 EFI bundle: I Linux kernel I initramfs I kernel command line 4 DM-Verity partition: I DM-Verity root hash set in kernel command line I Forward error correction support (FEC) I Read only uncompressed SquashFS root ﬁlesystem

ANSSI CLIP OS: Building a defense-in-depth OS 28/27