OPERATING SYSTEM CALL INTEGRITY OF THE LINUX OPERATING SYSTEM by DAYLE GLENN MAJORS ATHESIS Presented to the Faculty of the Graduate School of the UNIVERSITY OF MISSOURI–ROLLA in Partial Fulfillment of the Requirements for the Degree MASTER OF SCIENCE IN COMPUTER SCIENCE 2003 Approved by Dr. Ann Miller, Advisor Dr. Bruce M. McMillin Dr. Paul D. Stigall c 2003 Dayle Glenn Majors All Rights Reserved iii ABSTRACT ”Operating System Call Integrity of the Linux Operating System” examines the security exposures of the Linux Operating System arising from functions used in its construction. The ITS4 commercial package was used to identify suspect calls. While the package highlights a large number of different calls, almost all vulnerable calls in the operating systems were string related operations including printk and printf related functions. The results showed that the drivers directory of the Linux Operating System distribution was the most vulnerable. iv ACKNOWLEDGMENT Several individuals have been very supportive of my thesis. I extend my sincere thanks for their assistance and guidance throughout my master’s program. First, I wish thank Dr. Ann Miller for her guidance and support as my thesis advisor. Her perceptive suggestions were always beneficial. I wish to thank Dr. Bruce McMillin for his discussions of related research and his advice as a committee member. I also appreciate Dr. Paul Stigall for all his advice and input. In addition, Mr. William Siever’s participation in our many discussions of Linux kernel internals has provided valuable insights into its structure. The support provided by the United States Department of Education through the Graduate Assistantship in Areas of National Need (GAANN) has been helpful in advancing my education. I appreciate the proof reading help provided by Clara, my wife, and my daugh- ters Kimberly McKinney and Lori Majors. However, any errors that remain are mine. Finally, I give heartfelt thanks to my wife. Her love, support and encouragement throughout my master’s program have been invaluable. v TABLE OF CONTENTS Page ABSTRACT.................................... iii ACKNOWLEDGMENT.............................. iv LISTOFILLUSTRATIONS............................ vi LISTOFTABLES................................. vii SECTION 1.INTRODUCTION.............................. 1 2.OPERATINGSYSTEMENVIRONMENT................ 8 3.LINUXEVALUATION........................... 14 4.RESULTSANDCONCLUSIONS...................... 19 APPENDIX StringOperationsStatistics........................... 24 BIBLIOGRAPHY................................. 33 VITA........................................ 35 vi LIST OF ILLUSTRATIONS Figure Page 1.1 OperatingSystem.............................. 1 2.1 OSIModel.................................. 12 4.1 DistributionofITS4Flags......................... 23 vii LIST OF TABLES Table Page A.1 StringOperationsinLinuxKernel..................... 24 1. INTRODUCTION The general architectural diagram exhibited in the Unix documentation has become a standard for viewing the structure of modern operating systems. In the diagram, Figure 1.1, software is viewed as a series of concentric circles with the hardware at the center. This representation started with the Multics system’s rings of protections. (See Operating System Concepts [1] pages 402–404 for a discussion Multics rings of protection.) The first ring of programs around the hardware is usually the operating system kernel. The interface it exhibits is thought of as a virtual machine implemented by the operating system kernel. The other rings of complex code, which are program data, complete the operating system. Application programs operate using the interfaces provided by those rings of code. Applications Interpreter Interrupt Handlers Files Hardware Resource Allocation Scheduler Paging Serialization Task Management Compilers Figure 1.1. Operating System The operating system kernel contains several major pieces of software that make up the operating system. First, there will be software to provide allocation and de- allocation of the resources provided by the operating system. Second, there will be software to schedule the use of the processor. Third, there will be software to provide a consistent view of the large variety of unique hardware that provides persistent storage of data and external sources and destinations for data that the system processes. 2 There will be software to provide communication between processes active in the system and its environment. In addition there may be other software, which does not fit into these four groups. An operating system provides software that supports communication with hu- man users, compilers, linkers, file directories, script interpreters, text editors, browsers, et cetera. The software is implemented using the interfaces exported by the kernel. The software that supports these interfaces, in systems designed to run on multiple hardware platforms, are written when possible to be independent of the hardware in programming languages that could be compiled to run on the various hardware platforms. Looking closely at the kernel, there is more than one ring of software involved. The inner one, the interest of this paper, is composed of several pieces of software just as the outer ring, mentioned previously. There is a collection of device drivers and a collection of interrupt handlers. The synchronization and serialization protocols form another component of this inner ring. Finally there is software that activates each task or process when it starts the next time slice. In this paper, this software shall be referred to as the dispatcher since it implements, but does not produce or modify, the schedule. Much of the software in the inner ring was hardware dependent and must be written specifically for a given hardware platform. For example, the inter-process communication should take advantage of hardware support for updating memory so that all updates occurred at one time as seen by all other processor elements. This was not a large concern with single central processor unit (CPU) systems but as additional CPUs were added it became critical. However, even with single CPU systems using paging, if the page were being changed while it was being backed up, incorrect results could occur. An operating system for today’s computers is a large and complex collection of software. This means that the probability of software problems remaining in produc- tion programs is certainly not zero. It is virtually impossible to remove all program- ming errors and omissions from a small program. The number of paths through the program grows by a power of two; that power is the number of branch or decision points. For example, for four branch points there are 16 paths (2 to the 4th power, 3 2 x 2 x 2 x 2) through the code. A small program may have a dozen or more deci- sion points yielding thousands of paths through the program. An operating system typically consists of hundreds of such simple programs, increasing the probability of errors. In addition, possible branch points are introduced by the possibility of hardware interrupts, events the hardware detects, which require the attention of the software. It is possible to instruct the hardware to hold these interrupts, to postpone their processing. Often portions of an operating system execute sequences of instructions with the interrupts held. In this mode the branches to the code, which provide the attention that the hardware needs, do not occur. This mode of processing is called processing with interrupts disabled. If this processing mode is used for too long or too frequently, the system will appear unresponsive and sluggish. Alternatively, it is necessary that this mode be used to save information associ- ated with the hardware events and to retain that information so it can be processed later. Also some events are so critical that the processing must occur immediately or be processed completely without the possibility of anything else happening. For example, complex structures must be kept consistent. If a change requires more than a single hardware instruction’s execution, then the set of instructions necessary to transition from one consistent state to another needs to occur without interruption. This restriction can be relaxed by use of mutual exclusion protocols. Mutual exclu- sion protocols are any procedure used to preclude two or more processes updating a set of data concurrently. Included in these protocols are semaphores, sometimes called locks, critical sections including nonpreemptable critical sections and related program structures whose purpose is to prevent data being corrupted by concurrent updates. As more and more systems have added network processing, the sources for interrupts have become more numerous. However, these additional interrupts are only a symptom of a much larger problem. Frequently, there are requests received from remote users, perhaps malicious users. These requests must be validated before they are processed, which requires the requestor to be identified as having the proper authority. 4 If a malicious user can access a system, then that user may be able to attack it. This user may be able to cause the system to execute some code that has been introduced. If that code executes in a privileged state, then that code can cause significant damage to the system or its files. In systems programmed in the C language there are several service routines that can cause buffer overflow that will overlay data stored immediately following the