Linux Kernel Programming
Total Page:16
File Type:pdf, Size:1020Kb
1 Linux Kernel Programming Changwoo Min 2 About me • Changwoo Min • Assistant Professor at ECE @ VT • Email: [email protected] • Office: 455 @ Durham • Homepage: https://multics69.github.io/ • Research group: https://cosmoss-vt.github.io/ 3 My research interests • Many-core performance scalability of operating system • What happen if we run Linux on 448-core machine? • Will your application run 448x faster? • Non-volatile memory systems • What happen if storage performance is becoming closer to DRAM performance? • Will your application achieve DRAM-like IO performance? • System security • How operating system should be designed to avoid security holes? 4 What is the Linux Kernel? • One of operating system kernel • e.g., Windows, FreeBSD, OSX, etc. • What does an OS do for you? • Abstract the hardware for convenience and portability • Multiplex the hardware among multiple applications • Isolate applications to contain bugs • Allow sharing among applications 5 View: layered organization • User: applications (e.g., vi and gcc) • Kernel: file system, process, etc. • Hardware: CPU, mem, disk, etc. → Interface between layers 6 View: core services • Processes • Memory • File contents • Directories and file names • Security • Many others: users, IPC, network, time, terminals, etc. → Abstraction for applications 7 Example: system calls • Interface : applications talk to an OS via system calls • Abstraction : process and file descriptor fd = open("out", 1); write(fd, "hello\n", 6); pid = fork(); 8 Why is Linux kernel interesting? • OS design deals with conflicting goals and trade-offs • Efficient yet portable • Powerful yet simple • Isolated yet interactable • General yet performant • Open problems: multi-core and security • How does a state-of-the-art OS deal with above issues? • Hack the Linux kernel! 9 Why is Linux kernel interesting? • Extremely large software project • more than 25 million lines of code • 7,500 lines of code are added every day! 10 Why is Linux kernel interesting? • Very fast development cycles • release about every 70 days • 13,000 patches / release • 273 250 companies / release (or 1,600 developers / release) • One of the most well-written/designed/maintained C code • Ref: Linux Foundation Kernel Report 2017 11 Linux is eating the World • 85.1% of smartphones and tables run Linux (Android) • iOS: 14.9% • 98% of top 1 million web servers run Linux • 99% of super computers run Linux • SpaceX: From Earth to orbit with Linux and SpaceX • Ref: Usage share of OS 12 It is good for your job search • Contributions from unpaid developers had been in slow decline • 14.6% (2012) → 13.6% (2013) → 11.8% (2014) → 7.7% (2015) • Why? • “There are many possible reasons for this decline, but, arguably, the most plausible of those is quite simple: Kernel developers are in short supply, so anybody who demonstrates an ability to get code into the mainline tends not to have trouble finding job offers.” • Ref: Linux Foundation Kernel Report 2017 13 Who should take this course? • Anyone wants to work on the above problems • Anyone cares about what’s going on under the hood • Anyone has to build high-performance systems • Anyone needs to diagnose bugs or security problems 14 About this course • ECE 4414/5414G, CS 4224/5264G: (Advanced) Linux Kernel Programming • Goals • Understand core subsystems of the Linux kernel in depth • Design, implement, and modify Linux kernel code and modules for these subsystems • Test, debug, and evaluate the performance of systems software in kernel or user space, using debugging, monitoring and tracing tools 15 Prerequisite • Undergraduate and graduate students • C programming (strict) • Linux command line (strict) • Computer architecture and operating system (recommended) • Undergraduate students • ECE 3574 (Applied Software Design) or CS 3114 (Data Structures). 16 Text book • Robert Love, Linux Kernel Development, Addison-Wesley 17 Other useful sources • Understanding the Linux Kernel, O’Reilly Media • Professional Linux Kernel Architecture, Wrox • Linux Device Drivers, O’Reilly Media • Understanding Linux Network Internals, O’Reilly Media • Operating Systems: Three Easy Pieces • Intel 64 and IA-32 Architectures Software Developer Manuals 18 Communication • Q&A Protocol • Post your question to Zoom chat • Canvas • https://canvas.vt.edu/courses/115884 • Syllabus, lecture slides, schedule, notes, etc. • Primary way materials are distributed. • Exercise, project submission • Grades posted 19 Communication • Piazza • https://piazza.com/class/ke58vzg3buj4z0 • Use it to ask (and answer) questions • Forum/wiki like software for QA, polls, and announcements 20 Course TA & Office hours • Course TA • Sumit Kumar Monga ([email protected]) • Online Office Hours • Instructor and TA office hour will be provided. • Will announce how to make an appointment through Google Calendar and Zoom. 21 Grading policy • Exercise (10%) • 2% x 5 exercises • Paper reading (15%) • 3% x 5 papers • Project (65%) • 2 small projects: 5% + 10% • 1 medium projects: 20% • 1 final project: 30% • Final exam (10%): Online exam via Canvas 22 About projects (subject to change) • Small projects • P1: Adding new system calls • P2: Kernel module - data structure handling • Medium project • P3: TBD (kernel programming project) • Final project • P4 for 4xxx: TBD (kernel programming project) • P4 for 5xxx: TBD (semester long research project related to kernel) • No late project work will be assigned a grade. 23 Today’s agenda • The history of Linux • Linux open source model and community • High level overview of the Linux kernel 24 History of UNIX (Wikipedia) 1969 Unnamed PDP-7 operating system 1969 Open Source 1971 to 1973 Unix Version 1 to 4 Mixed/Shared Source 1971 to 1973 1974 to 1975 Unix PWB/Unix Version 5 to 6 Closed Source 1974 to 1975 1978 BSD 1978 1.0 to 2.0 Unix 1979 Version 7 1979 Unix/32V 1980 BSD 1980 3.0 to 4.1 1981 Xenix System III 1.0 to 2.3 1981 1982 Xenix 1982 1983 BSD 4.2 3.0 Sun OS System V 1983 1 to 1.1 R1 to R2 1984 SCO Xenix 1984 Unix 1985 Version 8 SCO Xenix 1985 AIX V/286 System V 1986 BSD 4.3 1.0 R3 HP-UX Sun OS 1.0 to 1.2 1986 1.2 to 3.0 SCO Xenix 1987 Unix V/386 9 and 10 1987 (last versions System V HP-UX 1988 BSD 4.3 2.0 to 3.0 1988 from Tahoe R4 1989 Bell Labs) SCO Xenix V/386 1989 BSD 4.3 1990 Reno 1990 BSD NET/2 1991 Linux 0.0.1 1991 Sun OS Minix 4 1.x NEXTSTEP/ 386BSD OPENSTEP 1992 1992 1.0 to 4.0 HP-UX NetBSD 6 to 11 Linux BSD 0.8 to 1.0 1993 1993 0.95 to 1.2.x SCO Unix 4.4 to 3.2.4 UnixWare FreeBSD 4.4 lite2 1.x to 2.x 1994 1994 1.0 to 1995 2.2.x NetBSD OpenBSD 1995 1.1 to 1.2 OpenServer 1.0 to 2.2 5.0 to 5.04 Solaris 1996 2.1 to 9 1996 1997 1997 NetBSD 1.3 1998 FreeBSD AIX 1998 3.0 to 3.2 3.x to 7.1 Minix OpenServer 1999 2.x Mac OS X 5.0.5 to 5.0.7 Server 2000 1999 UnixWare 2000 Linux 7.x 2001 to 2004 2.0 to 2.6.x 2001 to 2004 2005 OpenBSD 2005 NetBSD 2.3 to 5.x 2006 to 2007 Minix FreeBSD 2006 to 2007 3.x 3.3 to 9.x 1.3 to 6.x OpenServer Mac OS X 6.0 HP-UX 2008 10.0 to 10.9.x Solaris 11i to 11i v3 2008 (Darwin) 10 2009 2009 OpenSolaris 2010 and 2010 derivatives 2011 2011 Linux Solaris 3.x 11 2012 to 2013 2012 to 2013 25 Beginning of Linux From: [email protected] (Linus Benedict Torvalds) Newsgroups: comp.os.minix Subject: What would you like to see most in minix? Summary: small poll for my new operating system Message-ID: <[email protected]> Date: 25 Aug 91 20:57:08 GMT Organization: University of Helsinki Hello everybody out there using minix – I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things). I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and Id like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them Linus ([email protected]) PS. Yes – it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(. 26 Linux History • 1991: First apparition, author: Linus Torvalds • 1992: GPL License, first Linux distributions • 1994: v1.0 - Single CPU for i386, then ported to Alpha, Sparc, MIPS • 1996: v2.0 - Symmetric multiprocessing (SMP) support • 1999: v2.2 - Big Kernel Lock removed • 2001: v2.4 - USB, RAID, Bluetooth, etc. • 2003: v2.6 - Physical Address Expansion (PAE), new architectures, etc. • 2011: v3.0 - Incremental release of v2.6 • 2015: v4.0 - Livepatch → today’s latest version: http://www.kernel.org 27 Linux open source model • Linux is licensed under GPLv2 You may copy, distribute and modify the software as long as you track changes/dates in source files. Any modifications to or software including (via compiler) GPL-licensed code must also be made available under the GPL “ along with build & install instructions.