OS Security Virtualization
Radboud University, Nijmegen, The Netherlands
Winter 2016/2017 I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
I Popular smartphone platforms affected
I Intrusion detection system
I Intrusion prevention system
A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
OS Security – Virtualization2 I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
I Intrusion detection system
I Intrusion prevention system
A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
I Popular smartphone platforms affected
I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
OS Security – Virtualization2 I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
I Intrusion prevention system
A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
I Popular smartphone platforms affected
I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
I Intrusion detection system
OS Security – Virtualization2 I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
I Intrusion prevention system
A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
I Popular smartphone platforms affected
I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
I Intrusion detection system
I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
OS Security – Virtualization2 I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
I Popular smartphone platforms affected
I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
I Intrusion detection system
I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
I Intrusion prevention system
OS Security – Virtualization2 A short recap - I
I Last 2 lectures: Malware & Mandatory Access Control
I Malware evolution from PC to smartphone
I Early days: malware targeting Symbian OS users (Cabir, Pbstealer)
I Popular smartphone platforms affected
I Android: first proof-of-concept malware released in 2008
I iOS: WireLurker
I Windows Phone: FinSpy Mobile
I Blackberry: Trojans using the ‘Backstab’ technique
I Intrusion detection system
I NIDS, HIDS
I NIDS: (i) string, (ii) port and (iii) header condition signatures
I HIDS: signature- and behaviour-based
I Intrusion prevention system
I NIPS, HIPS
I (i) signature-based detection, (ii) anomaly-based detection and (iii) protocol state analysis detection
OS Security – Virtualization2 I Top secret
I Secret
I Confidential
I Unclassified
I Crucial
I Very important
I Important
I Objects are assigned security levels, namely:
I Users are assigned clearance levels; and processes are assigned security levels
I Biba model, for integrity protection I Objects and users are assigned integrity levels
A short recap - II
I Mandatory Access Control (MAC)
I A system implements MAC if the protection state can only be modified by trusted administrators via trusted software
I Bell-LaPadula model
OS Security – Virtualization3 I Crucial
I Very important
I Important
I Top secret
I Secret
I Confidential
I Unclassified
I Users are assigned clearance levels; and processes are assigned security levels
I Biba model, for integrity protection I Objects and users are assigned integrity levels
A short recap - II
I Mandatory Access Control (MAC)
I A system implements MAC if the protection state can only be modified by trusted administrators via trusted software
I Bell-LaPadula model I Objects are assigned security levels, namely:
OS Security – Virtualization3 I Crucial
I Very important
I Important
I Biba model, for integrity protection I Objects and users are assigned integrity levels
A short recap - II
I Mandatory Access Control (MAC)
I A system implements MAC if the protection state can only be modified by trusted administrators via trusted software
I Bell-LaPadula model I Objects are assigned security levels, namely:
I Top secret
I Secret
I Confidential
I Unclassified
I Users are assigned clearance levels; and processes are assigned security levels
OS Security – Virtualization3 I Crucial
I Very important
I Important
A short recap - II
I Mandatory Access Control (MAC)
I A system implements MAC if the protection state can only be modified by trusted administrators via trusted software
I Bell-LaPadula model I Objects are assigned security levels, namely:
I Top secret
I Secret
I Confidential
I Unclassified
I Users are assigned clearance levels; and processes are assigned security levels
I Biba model, for integrity protection I Objects and users are assigned integrity levels
OS Security – Virtualization3 A short recap - II
I Mandatory Access Control (MAC)
I A system implements MAC if the protection state can only be modified by trusted administrators via trusted software
I Bell-LaPadula model I Objects are assigned security levels, namely:
I Top secret
I Secret
I Confidential
I Unclassified
I Users are assigned clearance levels; and processes are assigned security levels
I Biba model, for integrity protection I Objects and users are assigned integrity levels
I Crucial
I Very important
I Important
OS Security – Virtualization3 I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
I Overrunning a memory buffer in the kernel can give a non-root process root privileges
I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
I Prevent malicious (or buggy) programs from:
I Prevent different users, groups, etc. from:
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I How does the OS enforce protection boundaries?
Role of the OS
I A major job of the OS is to enforce protection
OS Security – Virtualization4 I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
I Overrunning a memory buffer in the kernel can give a non-root process root privileges
I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
I Prevent different users, groups, etc. from:
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I How does the OS enforce protection boundaries?
Role of the OS
I A major job of the OS is to enforce protection I Prevent malicious (or buggy) programs from:
I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
OS Security – Virtualization4 I Overrunning a memory buffer in the kernel can give a non-root process root privileges
I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I How does the OS enforce protection boundaries?
Role of the OS
I A major job of the OS is to enforce protection I Prevent malicious (or buggy) programs from:
I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
I Prevent different users, groups, etc. from:
I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
OS Security – Virtualization4 I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
I How does the OS enforce protection boundaries?
Role of the OS
I A major job of the OS is to enforce protection I Prevent malicious (or buggy) programs from:
I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
I Prevent different users, groups, etc. from:
I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I Overrunning a memory buffer in the kernel can give a non-root process root privileges
OS Security – Virtualization4 I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
Role of the OS
I A major job of the OS is to enforce protection I Prevent malicious (or buggy) programs from:
I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
I Prevent different users, groups, etc. from:
I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I Overrunning a memory buffer in the kernel can give a non-root process root privileges
I How does the OS enforce protection boundaries?
OS Security – Virtualization4 Role of the OS
I A major job of the OS is to enforce protection I Prevent malicious (or buggy) programs from:
I Allocating too many resources (denial of service)
I Corrupting or overwriting shared resources (files, shared memory,...)
I Prevent different users, groups, etc. from:
I Accessing or modifying private state (files, shared memory,...)
I Killing each other’s processes
I Prevent viruses, worms, etc. from exploiting security holes in the OS
I Overrunning a memory buffer in the kernel can give a non-root process root privileges
I How does the OS enforce protection boundaries?
I 2-level protection: kernel and user mode
I Multilevel protection: Ring 0-3
OS Security – Virtualization4 I Access to I/O devices
I Manipulate memory management: set up page tables, load/flush the CPU cache, etc
I Call halt instruction: put CPU into low-power or idle state until next interrupt
I Kernel can execute special privileged instructions
I Examples of privileged instructions are:
Kernel and User mode
I What makes the kernel different from user mode?
OS Security – Virtualization5 I Access to I/O devices
I Manipulate memory management: set up page tables, load/flush the CPU cache, etc
I Call halt instruction: put CPU into low-power or idle state until next interrupt
I Examples of privileged instructions are:
Kernel and User mode
I What makes the kernel different from user mode?
I Kernel can execute special privileged instructions
OS Security – Virtualization5 I Manipulate memory management: set up page tables, load/flush the CPU cache, etc
I Call halt instruction: put CPU into low-power or idle state until next interrupt
Kernel and User mode
I What makes the kernel different from user mode?
I Kernel can execute special privileged instructions
I Examples of privileged instructions are:
I Access to I/O devices
OS Security – Virtualization5 I Call halt instruction: put CPU into low-power or idle state until next interrupt
Kernel and User mode
I What makes the kernel different from user mode?
I Kernel can execute special privileged instructions
I Examples of privileged instructions are:
I Access to I/O devices
I Manipulate memory management: set up page tables, load/flush the CPU cache, etc
OS Security – Virtualization5 Kernel and User mode
I What makes the kernel different from user mode?
I Kernel can execute special privileged instructions
I Examples of privileged instructions are:
I Access to I/O devices
I Manipulate memory management: set up page tables, load/flush the CPU cache, etc
I Call halt instruction: put CPU into low-power or idle state until next interrupt
OS Security – Virtualization5 Multilevel Protection: Ring 0-3
I Ring 0: kernel
I Rings 1-2: third-party drivers (less privileged OS code)
I Ring 3: application code
OS Security – Virtualization6 More on Protection Rings - I
- Each memory segment has an associated privilege level (0 through 3) - The CPU has a Current Protection Level (CPL) -> Usually the privilege level of the segment where the program’s instructions are being read from
OS Security – Virtualization7 More on Protection Rings - I
- Each memory segment has an associated privilege level (0 through 3) - The CPU has a Current Protection Level (CPL) -> Usually the privilege level of the segment where the program’s instructions are being read from - Program can read/write data in segments of lower privilege than CPL -> e.g. Kernel can read/write user memory
OS Security – Virtualization8 More on Protection Rings - I
- Each memory segment has an associated privilege level (0 through 3) - The CPU has a Current Protection Level (CPL) -> Usually the privilege level of the segment where the program’s instructions are being read from - Program can read/write data in segments of lower privilege than CPL -> e.g. Kernel can read/write user memory -> But user cannot read/write kernel memory.... Why?
OS Security – Virtualization9 More on Protection Rings - II
- Each memory segment has an associated privilege level (0 through 3) - The CPU has a Current Protection Level (CPL) -> Usually the privilege level of the segment where the program’s instructions are being read from - Program cannot (directly) call code in higher privilege segments -> Why?
OS Security – Virtualization 10 More on Protection Rings - II
- Each memory segment has an associated privilege level (0 through 3) - The CPU has a Current Protection Level (CPL) -> Usually the privilege level of the segment where the program’s instructions are being read from - Program cannot (directly) call code in higher privilege segments -> Why? - Program cannot (directly) call code in lower privilege segments -> Why?
OS Security – Virtualization 11 Types of Virtualization
1. OS-level virtualization 2. Application level virtualization 3. Full/native virtualization 4. Paravirtualization 5. Emulation
OS Security – Virtualization 12 1. OS-level virtualization
I OS allows multiple secure virtual servers to be run
I Makes the subsystem thinks it is running in its own operating system
I Abstracts the services and kernel from an application
I Guest OS is the same as the host OS, but appears isolated; apps see an isolated OS
I For example: Solaris Containers, FreeBSD Jails, Linux Vserver
OS Security – Virtualization 13 2. Application level virtualization
I Application behaves at runtime in a similar way when directly interfacing with the original OS
I Application gives its own copy of components that are not shared
I For instance: own registry files, global objects
I Application virtualization layer replaces part of the runtime environment normally provided by the OS
I Example: Java Virtual Machine (JVM)
OS Security – Virtualization 14 2. Application level virtualization
OS Security – Virtualization 15 I Challenge: Interception and simulation of privileged operations (I/O operations)
I Every operation performed within a given virtual machine must be kept within that virtual machine; virtual operations cannot be allowed to alter the state of any other virtual machine, control program or hardware.
3. Full/native virtualization
I VM simulates “enough" hardware to allow an unmodified guest OS to be run in isolation
I Any software capable of execution on the hardware can be run in the virtual machine
I Example: VMWare Workstation/Server, Mac-on-Linux etc.
OS Security – Virtualization 16 3. Full/native virtualization
I VM simulates “enough" hardware to allow an unmodified guest OS to be run in isolation
I Any software capable of execution on the hardware can be run in the virtual machine
I Example: VMWare Workstation/Server, Mac-on-Linux etc.
I Challenge: Interception and simulation of privileged operations (I/O operations)
I Every operation performed within a given virtual machine must be kept within that virtual machine; virtual operations cannot be allowed to alter the state of any other virtual machine, control program or hardware.
OS Security – Virtualization 16 I Use special API (para-API) that a modified guest OS must use
I Hypercalls trapped by the Hypervisor and serviced
I Provides specially defined ‘hooks’ to allow the guest(s) and host to request and acknowledge operations, which would otherwise be executed in the virtual domain
I Hence, reduces the portion of the guest’s execution time spent performing operations which are substantially more difficult to run in a virtual environment compared to a non-virtualized environment
I For example: Xen, VMWare ESX Server
4. Paravirtualization
I VM does not simulate hardware
I Is a technique that presents a software interface to VMs that is similar but not identical to that of the underlying hardware
OS Security – Virtualization 17 I Provides specially defined ‘hooks’ to allow the guest(s) and host to request and acknowledge operations, which would otherwise be executed in the virtual domain
I Hence, reduces the portion of the guest’s execution time spent performing operations which are substantially more difficult to run in a virtual environment compared to a non-virtualized environment
I For example: Xen, VMWare ESX Server
4. Paravirtualization
I VM does not simulate hardware
I Is a technique that presents a software interface to VMs that is similar but not identical to that of the underlying hardware
I Use special API (para-API) that a modified guest OS must use
I Hypercalls trapped by the Hypervisor and serviced
OS Security – Virtualization 17 4. Paravirtualization
I VM does not simulate hardware
I Is a technique that presents a software interface to VMs that is similar but not identical to that of the underlying hardware
I Use special API (para-API) that a modified guest OS must use
I Hypercalls trapped by the Hypervisor and serviced
I Provides specially defined ‘hooks’ to allow the guest(s) and host to request and acknowledge operations, which would otherwise be executed in the virtual domain
I Hence, reduces the portion of the guest’s execution time spent performing operations which are substantially more difficult to run in a virtual environment compared to a non-virtualized environment
I For example: Xen, VMWare ESX Server
OS Security – Virtualization 17 5. Emulation
I VM emulates complete hardware and software
I Emulator is a hardware/software enabling a system (i.e. host) to behave like another system (i.e. guest)
I Unmodified guest OS for a different system can be run
I Useful for reverse engineering, malware analysis, forensics (taint tracking)
I For example: QEMU, VirtualPC for Mac, Android
OS Security – Virtualization 18 Qubes OS
I Based on a secure bare-metal hypervisor (Xen)
I Networking code sandboxed in an unprivileged VM (using IOMMU/VT-d)
I USB stacks and drivers sandboxed in an unprivileged VM
I No networking code in the privileged domain (dom0)
OS Security – Virtualization 19 Qubes OS
I All user applications run in “AppVMs,” lightweight VMs based on Linux
I Centralized updates of all AppVMs based on the same template I Qubes GUI virtualization presents applications as if they were running locally
I Qubes GUI provides isolation between apps sharing the same desktop I Secure system boot OS Security – Virtualization 20 Compartmentalization in Qubes OS
OS Security – Virtualization 21 Qubes OS Live
OS Security – Virtualization 22 TUDOS - TU Dresden OS
I Demo
I Can be downloaded from: http://demo.tudos.org/eng_download.html
OS Security – Virtualization 23 VM Vulnerabilities
I Hardware-oriented attacks
I Management interface exploits
I Break out of jail attacks (VM escape)
I Virtual-machine based rootkits (Blue Pill)
I Application privilege escalation
I Just-In-Time (JIT) spraying - circumvents the protection of ASLR by exploiting the behaviour of JIT compilation. Has been used to exploit PDF format and Adobe Flash
I Untrusted native code execution
OS Security – Virtualization 24