MINIX3: a Reliable and Secure Operating System

MINIX3: A Reliable and Secure Operating System

Andrew S. Tanenbaum

and a team of students and programmers who actually did all the work

Vrije Universiteit Amsterdam, The Netherlands

1 GOAL OF OUR WORK: BUILD A RELIABLE OS

Tanenbaum’s deﬁnition of a reliable OS:

“An operating system is said to be reliable when a typical user has never experienced even a single failure in his or her lifetime and does not know anybody who has ever experienced a failure.”

In engineering terms, this is probably mean time to failure > 50 years

I don’t think we are there yet

2 THE TELEVISION MODEL

1. You buy the television 2. You plug it in 3. It works perfectly for the next 10 years

3 THE COMPUTER MODEL (WINDOWS EDITION)

1. You buy the computer 2. You plug it in 3. You install service packs 1 through 9f 4. You install 18 new emergency security patches 5. You ﬁnd and install 7 new device drivers 6. You install antivirus software 7. You install antispyware software 8. You install antihacker software (ﬁrewall) 9. You install antispam software 10. You reboot the computer

4 THE COMPUTER MODEL (2)

11. It doesn’t work 12. You call the helpdesk 13. You wait on hold for 30 minutes 14. They tell you to reinstall Windows

5 TYPICAL USER REACTION

The New York Times recently reported that 25% of computer users have gotten so angry at their computer that they physically hit it.

6 IS RELIABILITY SO IMPORTANT?

• Annoying • Lost work • But also think about – Industrial control systems in factories – Power grids – Hospital operating rooms – Banking and e-commerce servers – Emergency phone centers – Control software in cars, airplanes, etc.

7 IS THIS FEASIBLE?

• We won’t ﬁnd out if we don’t try • Dutch Royal Academy gave me €2 million to try • European Union gave me €2.5 million to give it a shot • So, we’re trying

8 IS RELIABILITY ACHIEVABLE AT ALL?

• Systems can survive hardware failures! – RAIDs can survive failed disks – ECC memory can survive parity errors in memory – TCP/IP can survive lost packets – CD-ROM drives can correct many simultaneous errors • We need to be able to survive software failures, too

9 A NEED TO RETHINK OPERATING SYSTEMS

• Operating systems research need to be refocused – We have nearly inﬁnite hardware on PC-class machines – Plenty of CPU cycles, RAM, bandwidth – Current software has tons of (useless) features – Consequently, the software is slow, bloated, and buggy

• To achieve the TV model, future OSes, must be – Small – Simple – Modular – Reliable – Secure – Self-healing 10 BRIEF HISTORY OF OUR WORK

• (1976) John Lions wrote a book on UNIX V6 • (1979) AT&T released V7 and forbade books on it L • (1985) I started to write a UNIX-like OS from scratch • (1987) MINIX 1 + book for teaching OS classes released • (1997) MINIX 2 (POSIX) & 2nd edition of book released • (2000) MINIX 2 license changed to BSD • (2004) MINIX 3: start of work making a reliable OS • (2006) 3rd edition of book • (2008) European grant • (2010) Focus moved towards embedded systems • (2013) MINIX 3.3.0 moves to NetBSD “compatibility”

11 THREE EDITIONS OF THE BOOK

1 2 3

12 INTELLIGENT DESIGN AS APPLIED TO OPERATING SYSTEMS

• Microkernel (15,000 LoC vs. > 15 million for Linux) – Bugs per 1000 LoC: Most S/W (1-10) – MINIX 3 at least 15 kernel bugs; Linux has > 15,000 – Drivers have 3-7x more bugs than rest of kernel – About 70% of the code is drivers • Highly modular • OS runs as multiple user-mode server processes

13 STEP 1: ISOLATE COMPONENTS

• Move all loadable modules out of the kernel – includes all device drivers and ﬁle systems • Run each module as a separate process with POLA (Principle Of Least Authority)

14 STEP 2: ISOLATE I/O

• Isolate I/O devices • Limit access to I/O ports • Constrain DMA (needs hardware assistance)

15 STEP 3: ISOLATE COMMUNICATION

• Limit interprocess communication • Restrict kernel calls on a per component basis • Restrict IPC on a ‘need-to-communicate’ basis • Make sure faulty receiver cannot hang sender

16 ARCHITECTURE OF MINIX 3

Process

Shell Make ... User

User mode FS 1 FS 2 Proc. ... Other Servers

Disk TTY Net Print ... Other Drivers

Kernel Microkernel handles interrupts, mode processes, scheduling, IPC

17 USER-MODE DEVICE DRIVERS

• Each driver runs as a user-mode process • No superuser privileges • Protected by the MMU • Do not have access to I/O ports, privileged instrs

18 USER-MODE SERVERS

• Each server runs as a separate process • Some key servers – Virtual ﬁle server – Actual ﬁle servers – Process manager – Memory manager – Network server – Reincarnation server

19 A SIMPLIFIED EXAMPLE: DOING A READ

User Users 1 User 4 mode FS Servers

Disk 2 3 Drivers

Kernel

File access when the block is in the FS cache

20 FILE SERVER (2)

User Users 1 User 9 mode FS Servers 2 6 7,8 5 Disk Drivers 3 4

Notiﬁcation Kernel

File access when the block is NOT in the FS cache

21 REINCARNATION SERVER

• Parent of all the drivers and servers • When a driver or server dies, RS collects it • RS checks a table for action to take e.g., restart it • RS also pings drivers and servers frequently

22 DISK DRIVER RECOVERY

RS 1 User Users

User 5 4 FS Servers mode 2

New Disk driver driverX 3. Crash! Drivers

Kernel

System is self healing—this is how we hope to make it reliable

23 KERNEL RELIABILITY/SECURITY

• Fewer LoC means fewer kernel bugs • Small kernel (15,000 LoC) means reduced TCB • NO foreign code (e.g., drivers) in the kernel • Static data structures (no malloc in kernel) • Moving bugs to user space reduces their power

24 IPC RELIABILITY/SECURITY

• Fixed-length messages (no buffer overruns) • Rendezvous system was simple – No lost messages – No buffer management – We had to add asynchronous messages • Interrupts and messages are uniﬁed

25 DRIVER RELIABILITY/SECURITY

• Untrusted code: heavily isolated • Bugs, viruses cannot spread to other modules • Cannot touch kernel data structures • Bad pointers crash only one driver; recoverable • Inﬁnite loops detected and driver restarted • Restricted power to do damage (not superuser)

26 OTHER ADVANTAGES OF USER DRIVERS

• Short development cycle • Normal programming model • No down time for crash and reboot • Easy debugging • Good ﬂexibility

27 FAULT INJECTION EXPERIMENT

• We injected 800,000 faults into each of 3 drivers • Done on the binary drivers • Examples, change src addr, dest addr, loop condition • 100 faults were injected on each experiment • Waited 1 sec to see if the driver crashed • If no crash, inject another 100 faults and repeat • The driver crashed in 18,038 trials • The operating system NEVER crashed

28 PORT OF MINIX 3 TO ARM

• Restructured source tree for multiple architectures • Changed booting to support uboot for ARM • Rewrote the low-level code dealing with hardware • Changed code for context switching, paging, etc. • Removed x86 segmentation code • Imported NetBSD ARM headers and libraries • Ported build.sh for cross-toolchain support • Wrote drivers for SD card and other Beagle devices

29 EMBEDDED SYSTEMS

BeagleBone Black 5 cm

9 cm

30 CHARACTERISTICS

Item Beaglebone Black

31 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7

32 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz

33 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB

34 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB

35 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p

36 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92

37 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps

38 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1

39 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes

40 CHARACTERISTICS

Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

41 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

42 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

43 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

44 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

45 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

46 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

47 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

48 CHARACTERISTICS

Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45

49 CHARACTERISTICS

50 CHARACTERISTICS

51 CHARACTERISTICS

52 I ADMIT I WAS WRONG

• On 29 Jan 1992 I posted to comp.os.minix this:

• “Don`t get me wrong, I am not unhappy with LINUX. It will get all the people who want to turn MINIX in BSD UNIX off my back.”

• I Apologize. Now I do want to turn MINIX into BSD. It just took me 20 years to realize this.

53 MINIX 3 MEETS BSD

+ =

55 WHY BSD?

• MINIX 3 didn’t have enough application software • BSD is a proven, portable, quality product • BSD has better code quality than Linux • Pkgsrc handles packages better than what we had • Thousands of excellent packages available • Active community • License compatibility • Why NetBSD? • Mostly due to its emphasis on portability

56 NETBSD FEATURES IN MINIX 3.3.0

• Clang/LLVM compiler • NetBSD build system • ELF ﬁle format • Source code tree modeled on NetBSD • Headers and libraries are from NetBSD • X11 • Pkgsrc works and builds 5040 NetBSD packages • Nevertheless, it is built on MINIX 3 kernel & servers

57 NETBSD FEATURES MISSING IN MINIX 3.3.0

• Kernel threads (we do have userland pthreads) • Some system calls: – All _LWP*, MSG*, SEM* calls – CLONE – Some GET, IOCTL calls – KQUEUE, KTRACE – VFORK – Job control – Some other minor calls • Nevertheless, we can build over 5000 packages

58 KYUA TESTS

Conclusion: 2139 out of 2651 passed (81%)

59 SYSTEM ARCHITECTURE

Users User- Land Clang Pkgsrc (libc) Pkg 1 Pkg n … (NetBSD) Servers VFS FS MM Rein carnat … OS (MINIX) Drivers Disk Net TTY… USB …

Microkernel (this is the only part running in kernel mode)

60 MINIX 3 ON THE THREE BEAGLE BOARDS

61 YOUR ROLE

• MINIX 3 is an open-source project • I hope some of you will join and help us • Things to do – Add crucial missing system calls – Port more packages (Java, a browser, etc.) – Write the missing drivers for Beagle series – Get it running on Raspberry Pi & other platforms – Port Rump – Port required libraries and then port a GUI

62 MINIX 3 IN A NUTSHELL

• Microkernel reimplementation of NetBSD • Fully open source with BSD license • Highly compatible with NetBSD • Supports both LLVM and gcc • Uses NetBSD pkgsrc • Over 5000 packages build • Go get it at www.minix3.org and try it

63 POSITIONING OF MINIX

• Show that multiserver systems are reliable • Demonstrate that drivers belong in user mode • High-reliability and fault-tolerant applications • $50 single-chip, small-RAM laptops for 3rd world • Embedded systems

64 FUTURE FEATURE: LIVE UPDATE

• Software is updated to: – Fix bugs – Improve performance – Add new features • Goal is to update OS to a new version w/o reboot • Running processes must NOT be restarted • New version of OS may have new data structures • Lots of state in there: open ﬁles, timers,etc.

65 EXAMPLE OF HOW WOULD THIS WORK

User A Apache running A Apache still running

Kernel FreeBSD 10.1 FreeBSD 10.2

• Replace the OS while user processes are running • Very difﬁcult to do with BSD, Linux, Windows, etc.

66 LIVE UPDATE IN MINIX

User A Apache running A Apache still running

User MM Dri- FS MM Dri- FS ver 6.0 ver 7.0

Kernel Microkernel Microkernel

67 HOW DO WE DO THE UPDATE?

• Manager tells some process (e.g. Old-FS) to get ready • Old-FS ﬁnishes its work and queues new work • Manager creates New-FS process with new code • LLVM puts tables inside New-FS listing its data objects • New-FS contacts Old-FS and asks for state it needs • The state is transferred one object at a time • When all state is transferred, Third-FS is created • It talks to New-FS and tries to recreate Old-FS • If they agree New-FS becomes FS, else revert to Old-FS • Like translating English to Dutch, then Dutch to English 68 HOW THE UPDATE WORKS

A Apache running

FS Old FS 6.0

Microkernel

69 HOW THE UPDATE WORKS

A Get Apache running ready

FS 6.0

Microkernel

70 HOW THE UPDATE WORKS

A Apache running

FS FS 6.0 7.0

Microkernel

71 HOW THE UPDATE WORKS

A Apache running

I need variable x FS FS G 6.0 7.0

Microkernel

72 HOW THE UPDATE WORKS

A Apache running

FS Here is variable x FS 6.0 7.0

Microkernel

73 HOW THE UPDATE WORKS

A Apache running

FS FS FS 6.0 7.0 ?

Microkernel

74 HOW THE UPDATE WORKS

A Apache running

FS FS I need variable x FS 6.0 7.0 ?

Microkernel

75 HOW THE UPDATE WORKS

A Apache running

FS FS Here is variable x FS 6.0 7.0 ?

Microkernel

76 HOW THE UPDATE WORKS

A Apache running

FS FS FS 6.0 7.0 ?

Are these the same? Microkernel

77 HOW THE UPDATE WORKS

A Apache running

FS 7.0

Microkernel

78 MUCH BETTER THAN KSPLICE

• KSPLICE can handle only small security patches • KSPLICE patches the running process • Over time, crud accumulates in the process • If the update fails, there is no recovery

79 OTHER USES OF LIVE UPDATE

• Enhanced security: – Update the OS at a high rate to foil return-to-libc attacks – Stop any attack that uses knowledge of memory layout – Reduce exposure to information leakage attacks

• Garbage collection in C (!) – Only live data is copied over to the new version – This can “ﬁx” memory leaks (malloc but no free)

80 RESEARCH: FAULT INJECTION

Inject fault?

Original Basic block unmodiﬁed with fault basic block injected

This structure is created automatically by the LLVM compiler

81 NEW PROGRAM STRUCTURE

This can be optimized by patching the original binary to get any test without recompilation

Overhead is 8% 82 MINIX 3 LOGO

• Why a raccoon? – Small – Cute – Clever – Agile – Eats bugs – More likely to visit your house than a penguin

83 WEBSITE: www.minix3.org

84 DOCUMENTATION IS IN A WIKI

• Wiki.minix3.org • You can help document the system

85 TRAFFIC TO WWW.MINIX3.ORG

Total visits to the main page since 2004: 3.1 million

Actual downloads since 2007: 650,000 (from the log)

86 MINIX 3 GOOGLE NEWSGROUP

87 CONCLUSION

• Current OSes are bloated and unreliable • MINIX 3 is an attempt at a reliable, secure OS • Kernel is very small (15,000 LoC) • OS runs as a collection of user processes • Each driver is a separate process • Each OS component has restricted privileges • Faulty drivers can be replaced automatically • Live update is possible (not in current release)

88 SURVEY

• Please download MINIX 3 from www.minix3.org • Give it a try • Fill out the survey on the main page • We have had 650,000 downloads but we don’t know who they are or what they are doing • We are trying to build a community

89 THE END

90 WEBSITE: www.minix3.org

91 92 MASTERS DEGREE AT THE VU

• If you are interested in computer systems • Look at our masters in parallel & distributed syst. • Google me • Look at my home page • See video linked there or check out

pdcs.vu.nl

93 DISK PERFORMANCE

94 THE COST OF DRIVER RECOVERY • We killed the Ethernet driver every Δt sec to simulate repeated driver crashes

Driver recovery takes about 360 msec 95 RESEARCH: MULTICORE CHIPS

Multicore chip • Network stack has components • Chips may be heterogeneous TCP IP • Where to put each component?

Ether Kernel • Experiments scaling frequencies • Sometimes slower is faster! Core • Sleep/wakeup is expensive

96 RESEARCH: NEW FILE SYSTEM--LORIS

VFS • Better reliabilty • Better ﬂexibility Naming • Handles heterogeneity better • File rather than block oriented Cache • Uses checksums to detect corruption

Logical Introduces concept of a logical ﬁle (1 or more phys ﬁles spread or striped over possibly heterogeneous devices) Physical

Driver