OS Security EECS 195 Spring 2019 Zhou Li

5/1/2019 Zhou Li 1 Objectives

• Basic security functions provided by operating systems • System resources that require protection • Operating system design principles • How operating systems control access to resources • Contemporary OS protections on memory • The history of trusted • Characteristics of operating system • Formally verified kernel: seL4

5/1/2019 Zhou Li 2 Operating Systems (OS)

• OS: An executive or supervisor for a piece of computing machinery.

5/1/2019 Zhou Li 3 Operating System Functions Users

Security-relevant features: • Enforced sharing • Inter- communication and Synchronization, Concurrency synchronization Control, Deadlock Operating Services Management, • Protection of critical data System Communication, • Guaranteed fair service Accounting • User authentication • Resource Allocation

• File and I/O device access control Data • Allocation and access control to general objects CPU Memory

Program Libraries I/O Devices 5/1/2019 Zhou Li 4 A Brief History of OS

• Single users, no OS, aka executives • Program entered in binary by “switches” physically • User schedules blocks of time for running the machine exclusively • Security issues: physical protection of computer, programs and data • Multiprogramming and shared use, aka monitors • Waste of resources if only for one user’s tasks • Concepts like , sharing and concurrent use are developed • Executives provide service, monitors control the resources • Security issues: protecting one user’s program & data from others • Personal computers, changeover from multiuser mainframes • Many security features like controlled sharing are forsaken

5/1/2019 Zhou Li 5 Multitasking

• Process • Created when running an application • Assigned with system resources (called domain): files, access to devices & communications, memory and execution time. • OS switches control between processes, allocating, deallocating and reallocating resources for processes. • • A process consists of one or more threads, separate streams of execution. • A thread executes in the same domain as other threads. • Much less OS cost during thread switching.

5/1/2019 Zhou Li 6 Protected Objects

• Memory • Sharable I/O devices, such as disks • Serially reusable I/O devices, such as printers • Sharable programs and subprocedures • Networks • Sharable data

• Granularity for access control • Larger the level, easier to implement, but could lead to over-privilege

7 OS Layered Design to Protect Objects

OS layers Subprocesses of User Processes • Security kernel: enforce security • OS kernel: allocate primitive User Processes resources such as time or Compilers, Data base Managers access to hardware devices Utility Functions • Other OS functions: implement File Systems, Device Allocation Operating user’s interface to hardware System Scheduling, Sharing,

Synchronization, Allocation Operating System Security Functions Kernel Security Kernel Hardware

5/1/2019 Zhou Li 8 Protection Rings (Hardware)

• Hardware-enforced by some CPU architectures that provide different CPU modes at the hardware or microcode level. • Process and resources are in different rings (privilege levels). • When a lesser privileged process tries to access a higher privileged process / resource, a exception is reported by the OS. • implementations (Ring 0-3) • Windows, , macOS uses Ring 0 (kernel / privileged mode) and Ring 3 (user mode) • DOS runs everything at Ring 3 Privilege rings for the x86 available https://en.wikipedia.org/wiki/Protection_ring in 5/1/2019 Zhou Li 9 Modular OS Design Users Users Users Users Users User Mode Modules come from different Privileged Mode sources User Interface

• Hardware and drivers from device Sec File Object A/V Net Backup manufacturers • Anti-virus from vendors System Services Interface “hooks” into OS • Cannot trust one another by default I/O Time Synch Memory Comm Sec

Primitive Services

Microkernel Kernel Mode Drivers

Hardware Interface and Abstraction

Hardware

10 Virtualization

• With virtualization, the OS presents each user with just the resources that user should see • The user has access to a virtual machine (VM), which contains those resources • The user cannot directly access resources that are outside the VM (sandbox) • A hypervisor, or VM monitor, is the software that hosts VM • Translates access requests between the VM and the OS • Can support multiple OSs in VMs simultaneously

5/1/2019 Zhou Li 11 Memory management

5/1/2019 Zhou Li 12 Hardware Protection of Memory

Addresses Memory • Goal: program can share selected 0 part of memory with other Hardware programs and even other users; Address Operating System OS and user can coexist in Limitation memory without interference n • Approach: fence n + 1 • OS and user reside in different sides of memory (fixed) Addressing User Program Space • Too restrictive Range • Excess space is wasted

High

5/1/2019 Zhou Li 13 Fence Registers

Address Address • Containing the address of the Limit Limit Register Register end of the OS n + 1 p + 1 • Boundary can be changed Addresses Memory Addresses Memory • Program’s instruction can be 0 0 executed if the data’s address Operating Operating is greater than fence address System Version 1 System Version 2 • Cannot protect one user from n p another n + 1 p + 1 User Program User Program Space Space Addressing Addressing Range Range

High High

5/1/2019 Zhou Li 14 Base/Bounds Registers

Addresses Memory • Base register: variable fence 0

register about lower-bound Operating • Bounds register: upper-bound for a Base Register System n n + 1 user program n + 1 User A • How to protect program address Program Space from modification by another user? Bounds Register p p • switch p + 1 User Program User B Space • OS changes content of base Program Space and bounds registers to reflect q true of another q + 1 User C Program Space user High

5/1/2019 Zhou Li 15 Two Pairs of Base/Bounds Registers

Operating • Erroneous addresses inside a user System space can impact program User A • Subscript is out of range Data Base Program Space • Undefined variable reference an User B Data Bounds Data Space address inside program space • Solution: two pairs of registers User A Data Space User Program and Data • Data registers Space • Program registers User C Program Space

Program Base User C Data Space

Program Bounds User B Program Space

5/1/2019 Zhou Li 16 Tagged Architecture Tag Memory Word

R 0001 • What if we want to protect some data value RW 0137 but not all? R 0099 • Solution: tagged architecture X • Each word of machine memory has one or more X extra to identify its access rights • Example: X • Burroughs B6500-7500 three tag bits to X separate data words, descriptors (pointers) and X control words (stack pointers & addressing X control words) • Problem: compatibility with OS R 4091 RW 0002

Code: R = Read-only RW = Read/Write X = Execute-only 5/1/2019 Zhou Li 17

• Two more approaches for memory protection • Segmentation • Paging • Can be implemented on top of a conventional machine structure • Designed between 1965 and 1975

5/1/2019 Zhou Li 18