ENABLING USER-MODE PROCESSES IN THE TARGET OS FOR CSC 159 OPERATING SYSTEM PRAGMATICS A Project Presented to the faculty of the Department of Computer Science California State University, Sacramento Submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in Computer Science by Michael Bradley Colson SPRING 2018 © 2018 Michael Bradley Colson ALL RIGHTS RESERVED ii ENABLING USER-MODE PROCESSES IN THE TARGET OS FOR CSC 159 OPERATING SYSTEM PRAGMATICS A Project by Michael Bradley Colson Approved by: __________________________________, Committee Chair Dr. Weide Chang __________________________________, Second Reader Dr. Yuan Cheng ____________________________ Date iii Student: Michael Bradley Colson I certify that this student has met the requirements for format contained in the University format manual, and that this project is suitable for shelving in the Library and credit is to be awarded for the project. __________________________, Graduate Coordinator ___________________ Dr. Jinsong Ouyang Date Department of Computer Science iv Abstract of ENABLING USER-MODE PROCESSES IN THE TARGET OS FOR CSC 159 OPERATING SYSTEM PRAGMATICS by Michael Bradley Colson One of the main responsibilities of an Operating System (OS) is to ensure system stability and security. Most OSes utilize mechanisms built into the CPU hardware to prevent user-mode processes from altering the state of the system or accessing protected spaces and hardware I/O ports. A process should only read from or write to memory addresses in its address space. In order to accomplish the above, the CPU is typically designed to operate in two or more modes with different privilege levels. The OS typically runs in the mode with highest privilege level, which enables it to execute any instruction and gives it access to all the memory in the system. To improve system reliability, security and stability, applications usually run in the mode with the lowest privilege level. At Sacramento State University, CSC 159 (Operating System Pragmatics) students learn to develop an OS using the SPEDE (System Programmer’s Educational Development Environment) framework. SPEDE provides an environment to build an OS executable image and download it to a target system where it is executed as the local OS. While SPEDE provides an excellent development environment for students to learn how v to develop an OS, one limitation is it does not currently support running processes in user-mode (only kernel-mode is supported). In order to give students a better understanding of how an OS can provide security and protection, it is necessary to enable the support for user-mode processes. The goals of this project are to establish a supportive software component for the creation of user-mode processes, enabling and enforcing memory protection for the process runtime and handling of runtime exceptions in a virtual memory paging system, such as page faults and general protection faults in the OS kernel. In order to achieve these goals, additional kernel data structures must be implemented (e.g. paging table structures, a Task State Segment, modified process trap frame, etc.) along with handlers for page faults and general protection faults. _______________________, Committee Chair Dr. Weide Chang _______________________ Date vi ACKNOWLEDGEMENTS I would like to thank the project sponsor, Dr. Weide Chang for his guidance and support on this project. Dr. Chang took time out of his busy schedule to provide me with valuable advice and feedback on all phases of the project. His knowledge of Operating Systems and Computer Architecture played a significant role in the successful completion of this project. I would also like to thank Dr. Yuan Cheng for being the second reader for this project. His guidance and feedback were extremely helpful in completing all the requirements for the project in a timely manner. I would also like to thank the entire CSUS Computer Science department and faculty for supporting me to become a successful student. vii TABLE OF CONTENTS Page Acknowledgements .................................................................................................... vii List of Tables .............................................................................................................. xi List of Figures ............................................................................................................ xii Chapter 1. INTRODUCTION .................................................................................................. 1 1.1 Project Overview ........................................................................................ 1 1.2 Project Objectives ....................................................................................... 2 1.3 Related Work .............................................................................................. 3 1.4 Expected Results ......................................................................................... 4 1.5 Background ................................................................................................. 5 2. OVERVIEW OF THE X86 CPU ARCHITECTURE ............................................ 9 2.1 Intel/AMD x86 Architecture Overview ...................................................... 9 2.2 Protected Mode Structures ........................................................................ 12 2.3 Task Switching .......................................................................................... 14 2.4 Virtual Memory and Address Translation ................................................ 18 2.5 Exceptions and Interrupts ......................................................................... 21 3. OVERVIEW OF THE SPEDE X86 DEVELOPMENT ENVIRONMENT ........ 25 3.1 Overview of SPEDE ................................................................................. 25 3.2 FLASH – Flames Shell ............................................................................. 30 3.3 Flint – Flames Interface ............................................................................ 31 viii 3.4 GNU Debugger ......................................................................................... 32 3.5 Virtual SPEDE .......................................................................................... 33 4. METHODOLOGY ............................................................................................... 36 4.1 Creating User-Mode and Kernel-Mode Processes .................................... 36 4.2 Dispatching User-Mode and Kernel-Mode Processes .............................. 38 4.3 Memory Protection for User-Mode Processes .......................................... 39 4.4 Handling Exceptions and Faults Generated by User-Mode Processes ..... 39 5. IMPLEMENTATION ........................................................................................... 41 5.1 Organization of the Target OS Source Code ............................................ 41 5.2 Kernel Data Structures Overview ............................................................. 43 5.3 Kernel Data Structures to Enable User-Mode Processes .......................... 44 5.4 Kernel Task State Segment Implementation ............................................. 46 5.5 API for Creating and Dispatching User-Mode Processes ......................... 47 5.6 API for Creating and Dispatching Kernel-Mode Processes ..................... 49 5.7 Handling Software Interrupts Generated by User-Mode Processes ......... 50 5.8 Page-Fault Handler Implementation ......................................................... 50 5.9 General-Protection Fault Handler Implementation ................................... 53 6. TESTING .............................................................................................................. 54 6.1 Testing Approach ...................................................................................... 54 6.2 Test Cases ................................................................................................. 56 7. RESULTS AND EVALUATION......................................................................... 60 8. CONCLUSION AND FUTURE WORK ............................................................. 63 ix Appendix A. SOURCE CODE ................................................................................. 65 References .................................................................................................................. 76 x LIST OF TABLES Tables Page 1. Segment Descriptors Added to the GDT in the Target OS ............................. 27 2. SPEDE Library Files........................................................................................ 28 3. Target OS Source Files .................................................................................... 42 4. Target OS Kernel Data Structures ................................................................... 43 5. Test Cases to Verify the Software Requirements ............................................ 56 xi LIST OF FIGURES Figures Page 1. General-Purpose and Segment Registers in an x86 CPU ................................ 12 2. Protection Rings in an x86 CPU ...................................................................... 14 3. 32-bit x86 Task State Segment ........................................................................ 15 4. Paging Structures in an x86 Computer ............................................................ 20 5. Stack Contents after an Exception or Interrupt ...............................................