Final Year Project Report Bugs in OpenBSD Niall O’Higgins A thesis submitted in part fulfilment of the degree of BA/BSc (hons) in Computer Science Supervisor: Dr. Joseph Kiniry Moderator: Dr. Pavel Gladyshev UCD School of Computer Science and Informatics College of Engineering Mathematical and Physical Sciences University College Dublin March 23, 2006 Table of Contents Abstract . 2 1 Introduction . 4 1.1 Project Specification . 4 1.2 Overview . 5 2 Background Research . 6 2.1 Bugs and API Guidelines . 6 2.2 Bug database . 9 2.3 Automatically finding bugs . 10 2.4 Development Tools . 13 3 Determining How to Find and Store Bugs . 15 3.1 The OpenCVS Case Study . 15 3.2 The Bug Database . 17 4 Design of the Automatic Tool . 19 4.1 Manual Regression Tests . 19 4.2 A Different Approach to Automatic Testing . 20 5 Implementation of the Automatic Tool and Bug Database . 22 5.1 Data Structures . 22 5.2 Algorithms . 23 5.3 GNATS 4 Port . 26 6 Results and Analysis . 28 6.1 Revision Number Zero . 28 6.2 Revision Number Overflow . 29 6.3 Exit Code Bug . 30 7 Conclusions and Future Work . 31 7.1 Conclusions . 31 7.2 Future Work . 31 Page 1 of 33 Abstract This report presents a method for automatically identifying bugs in OpenBSD command- line applications. As OpenBSD is a UNIX-like operating system, this method also applies to general UNIX command-line applications. It is possible to view A UNIX command- line application as a function which transforms various inputs (standard input, some files, command line arguments) into some output (standard output/error, some files, exit code). “Interesting” boundary values are generated and fed to the program, quickly exploring the possible error paths and thus exposing bugs. Additionally, a tool implementing this bug detection is demonstrated and made to automatically populate a flexible Internet-accessible bug database. Furthermore, analysis of a number of bugs found during the course of the project case study is presented. Finally, guidelines are put forward for the API designer, API implementor and API user to avoid bugs in the future. Page 2 of 33 Acknowledgments My supervisor, Dr. Joseph Kiniry, for his input and guidance. Page 3 of 33 Chapter 1: Introduction 1.1 Project Specification OpenBSD [1] is widely regarded as the world’s most secure operating system. It represents the combined effort of hundreds of dedicated individuals, most of whom work in their spare time for free, over nearly a decade. OpenBSD developers try very hard to write code that is correct, a goal which has the con- venient side effect of producing code which is also well-designed, maintainable, clean and secure. The OpenBSD source code has been and continues to be relentlessly trawled for bugs. The cause of a bug could be, for example, due to a misunderstanding of an API, an incorrect assumption on the part of the programmer, or a simple typo. From time to time, a bug is discovered which really represents a whole new class of mistake. Once such a class is discovered, the entire source tree is scoured for further instances of this error — thus one bug fix may lead to many more, often solving problems in other unrelated places before they are even noticed. Finding, analysing and fixing bugs has been central to the success of the OpenBSD project. However, no formal database is maintained of the bugs encountered during development. Furthermore, no official record is made of the tools and methodologies which help in the process of identifying and fixing them. This project focuses on (a) identifying and fixing bugs in the OpenBSD source code (b) analysing and classifying these bugs, and making the resulting information made available in a searchable database (c) exploring the development of software engineering tools and processes to aid in this endeavour. Mandatory • Evolve a structured and systematic methodology to find bugs in the OpenBSD source tree. • Design and implement a flexible Internet-accessible database for the storage of bug-data. • Develop a concise process for analysing and categorising bugs. • Find, fix, analyse and categorise a quantity of bugs in the OpenBSD source tree. Discretionary • Propose guidelines for both the API designer, API implementor and API user to avoid pitfalls in the future. • Design software tools to aid in the discovery of bug instances. Exceptional Page 4 of 33 • Implement software tools which automatically locate instances of a bug class in a source tree. 1.2 Overview This project documents a journey through a real software engineering landscape. Early on, it was decided to focus on a specific area of the OpenBSD source tree and to use this as a case study for research into bugs and their nature. This case study would provide insight into and real-world examples of how bugs are introduced, how they are found and how they can best be avoided. The chosen component of OpenBSD was OpenCVS [2], with a strong focus on its RCS [3] implementation. The author has for the duration of the final project worked as an integral part of a small and dedicated team within the official OpenBSD project which has produced thousands of lines of correct high-quality source code. This work is freely available and is a part of the main OpenBSD source tree. Further discussion of the case study, its motivations and its results, can be found in later chapters. In parallel to working on this case study, the author has strived to relate and use those experiences gained from its development to attain the goals of the project. This report attempts to show how he has extracted from his analysis plausible software engineering guidelines aimed at limiting future pitfalls. The ultimate product of this final project is a working software implementation of a method devised as part of the project for finding bugs in UNIX command-line applications. Furthermore the tool integrates with a bug database such that it reports its finding in a human-readable format. Additionally, this tool has found a significant number of real-world bugs in the case study software. These bugs have been categorised along the lines proposed in this report and a selection of them are presented for analysis. Page 5 of 33 Chapter 2: Background Research The project can be broken down into three distinct areas; the nature of bugs in software and useful API guidelines to help prevent them, the development of a flexible database in which to store information pertaining to these bugs, and finally the creation of an automatic tool to locate instances of bugs. This chapter details existing research into these areas including tools and approaches. 2.1 Bugs and API Guidelines According to Wikipedia [4], “A software bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from working as intended, or produces an incorrect result.”. At the core of this project is the realisation that bugs are largely the result of programmer error. However, often the programmer is given APIs which are complicated and ambiguous, thus difficult to fully understand. So the programmer is in a position where it is easy for her to introduce potentially serious bugs in her program, unless she understands fully all the nuanced and complicated aspects of an API that may in fact seem simple on the surface. The OpenBSD project has been very aggressive in identifying and documenting those C language APIs which are complicated, ambiguous and commonly lead to mistakes. In some cases, they developed their own replacement APIs which implement effectively the same functionality as those they replaced but with much less ambiguous and error-prone interfaces. These OpenBSD APIs have been so successful that they have been adopted by other projects, becoming de-facto standards for secure and correct programs. In order to meet the goals of this project to provide guidelines for the API designer, implementor and user, it was decided to examine this work done by the OpenBSD project. From this work on proposing entirely new APIs it is possible to extract details of what exactly it is that makes a given C API bad or dangerous. It is possible to gain insight into questions such as, what are the qualities of an API which lead to frequent misuse? 2.1.1 strlcpy() and strlcat() Perhaps the most well-known C language API replacement developed by the OpenBSD team have been the strlcpy() and strlcat() functions. These functions are designed to replace the extremely widely used ANSI C string functions strcpy(), strcat() and strncpy(), strncat(). These functions are used to copy and concatenate strings. A comparison of these functions with each other and with their Java equivalent is provided for illustration below. String copy comparison String copy in Java Page 6 of 33 String myString = ‘‘hello world’’; Assume “buffer” variable had been declared earlier as char buffer[10]; String copy in C using ANSI C strcpy() strcpy(buffer, ‘‘hello world’’); String copy in C using ANSI C strncpy() strncpy(buffer, ‘‘hello world’’, sizeof(buffer)); // strncpy() does not guarantee NUL-termination of string, so it must // be done manually. buffer[sizeof(buffer) - 1] = ’\0’; String copy in C using OpenBSD’s strlcpy() strlcpy(buffer, ‘‘hello world’’, sizeof(buffer)); String concatenation comparison String concatenation in Java String mySecondString = myString + ‘‘foo bar’’; String concatenation in C using ANSI C strcat() strcat(buffer, ‘‘foo bar’’); String concatenation in C using ANSI C strncat() strncat(buffer, ‘‘foo bar’’, sizeof(buffer)); // strncat() does not guarantee NUL-termination of string, so it must // be done manually. buffer[sizeof(buffer) - 1] = ’\0’; String copy in C using OpenBSD’s strlcat() strlcat(buffer, ‘‘hello world’’, sizeof(buffer)); The OpenBSD replacement functions address two key issues with the existing ANSI C API.