Automatic Removal of Unused Code for Software Testing Jing Lin BSc Computer Science Submission date: 19th January 2015 Supervisor: Jens Krinke This report is submitted as part requirement for the BSc Degree in Com- puter Science at UCL. It is substantially the result of my own work except where explicitly indicated in the text. The report may be freely copied and distributed provided the source is explicitly acknowledged. Acknowledgement This project is completed with the great help of many people and I would like to give my most sincere thanks to all of them. I would like to express my sincere gratitude and thanks to my supervisor Jens Krinke to spend his time helping me understand and giving me sugges- tions to carry out this project. I would also take this chance to thank my family for their continuous support throughout my studies, my friends for their encouragement and my home university for providing the Erasmus programme to make this happen. Abstract The traditional testing system will report for the whole system when this fails in the test cases. However, this report could be large and specially if one needs to fix the errors of the system. This project aims to solve this problem by removing unused code automatically so that the reduced source code is focused on single test case and it will be easier to analyse and faster to test. Contents 1 Introduction 6 1.1 Motivation . .6 1.2 Aim and goal . .6 1.3 Achievement process . .7 1.4 Report overview . .8 2 Background 9 2.1 Delta debugging . .9 2.1.1 Simplifying and isolating failure-inducing . .9 2.1.2 Locating causes of program failures . 12 2.2 Code coverage . 14 2.2.1 JaCoCo . 15 2.2.2 GCOV . 15 2.3 Delta tool . 16 2.4 Project approach vs. Dead code elimination . 17 3 Requirements and analysis 19 3.1 Problem statement . 19 3.2 Requirements . 19 3.3 Use cases . 20 3.4 Code coverage analysis . 20 3.4.1 LCOV . 20 3.4.2 MELD . 21 1 4 Design and implementation 22 4.1 Delta Coverage tool design . 22 4.2 Structure of Delta Coverage . 22 4.3 Implementation of Delta Coverage . 24 4.3.1 Setup.sh . 24 4.3.2 Test.sh . 28 4.3.3 Run.sh . 29 5 Testing and result evaluation 31 5.1 Testing strategy . 31 5.2 Specific test cases . 32 5.3 Result evaluation based on different C programs . 32 5.3.1 Tower of Hanoi Algorithm . 32 5.3.2 Sudoku Algorithm . 38 5.3.3 Travelling Salesman Problem . 42 6 Conclusions 45 6.1 Summary . 45 6.2 Critical evaluation . 45 6.3 Future work . 46 Appendices 47 Appendix A Evaluation data and results 47 A.1 Tower of Hanoi . 47 A.1.1 Test Cases . 47 A.2 Sudoku . 48 A.2.1 Test Cases . 48 A.3 Travelling Salesman Problem . 48 A.3.1 Test Cases . 48 Appendix B Project Plan 51 Appendix C Code Listing 62 C.1 Delta Coverage tool implementation source code . 62 2 C.1.1 setup.sh . 62 C.1.2 test.sh . 63 C.1.3 run.sh . 63 C.2 The addition function for Sudoku program . 64 3 List of Figures 2.1 Minimizing Delta Debugging algorithm[1]. 11 2.2 Delta Debugging algorithm[1] (Where C3means there are not test case changes). 11 2.3 Search in space[2]. 13 2.4 Search in time: cause transition[2]. 14 2.5 Dead code[9]. 17 2.6 Dead code elimination[9]. 18 4.1 Delta Coverage architecture design. 23 4.2 Delta Coverage workflow. 24 4.3 Implementation Structure : setup.sh .............. 25 4.4 The output messages of GCOV. 26 4.5 A program with coverage information: toh.c.gcov ....... 27 4.6 Implementation Structure: test.sh ................ 29 4.7 Implementation Structure: run.sh ................ 29 5.1 Tower of Hanoi: test1.txt. 33 5.2 Tower of Hanoi: test2.txt. 33 5.3 Coverage estimation: toh test1:info ............... 34 5.4 Coverage estimation : toh test2:info .............. 34 5.5 Differences between removed code and original code from code coverage report toh test1:info.................. 35 5.6 Differences between removed code and original code from code coverage report toh test2:info.................. 36 5.7 Differences between removed code and original code from code coverage report toh test2:info.................. 36 4 5.8 Differences between the reduced source codes: minmial test1:txt toh:c (left hand) and minmial test2:txt toh:c (right hand). 38 5.9 An example of Wikipedia Sudoku: test1:txt........... 39 5.10 World's hardest Sudoku: test2:txt................ 39 5.11 Different coverage estimations with test1:txt (the first row) and test2:txt (the second row). 40 5.12 Differences between removed code with the original source code. 41 5.13 Differences between reduced source codes. 42 5.14 Different coverage estimations with test1:txt(the first row), test2:txt (the second row) and test3:txt (the third row). 43 5.15 Differences between removed code with the original source code. 43 5.16 Differences between reduced source codes for the three test cases: test1:txt(left), test2:txt(centre), test3:txt(right). 44 5 Chapter 1 Introduction 1.1 Motivation During the testing of a system, it is possible to check which codes have been executed by measuring the code coverage information. There are tools that help to find out problems inside the system if a test fails. Notwithstanding, most of these tools are static analysis based, and report for the complete source code. The approach of this project intends to improve the process of code execution analysis and return a reduced source code instead of the complete one. The reduced source code can be used for much more effective fault local- isation during a test. Our proposal not only reduces the program size but also makes the testing process much easier and faster than the whole source code. The reduced system will provide the minimal source code that still produces the same results as the complete system. 1.2 Aim and goal The aim of this project is to develop a tool called Delta Coverage which can automatically remove unused code1 for software testing. An approach 1The unused code is the code that never has been executed during the testing. 6 called Delta Debugging, developed by Andreas Zeller[1][2], can be used for this improvement. This approach can remove parts of the source code and checks whether the reduced source code still produces the same results as the original one. For a better measurement, I focus on Delta Debugging and code coverage. And the goal of this project consists of development of Delta Coverage for C language programs. 1.3 Achievement process To achieve all the aims and goals of this project, my contributions are the following: 1. Studying Delta Debugging's approach[1][2]. 2. Studying about code coverage: • JaCoCo2 is a code cover tool for Java program. • GCOV is a code cover tool for C program and only works on GNU Compiler Collection (GCC). 3. Studying about Delta[3], this tool uses Delta Debugging algorithm in GNU system. 4. Implementation of Delta Coverage tool with the following features: • Use of GCOV to cover a program code. • Use of Delta tool to remove as many unused code as possible. 5. Evaluation and analysis of different C programs with Delta Coverage tool. 2Java Code Coverage 7 1.4 Report overview This project is organized in the following chapters: Chapter 1 deals with an overview of the project, its aims, goals and achievement. Chapter 2 details the information about Delta Debugging and its re- lated works. It also explains, briefly, different test coverage programs such as JaCoCo and GCOV. In addition, it introduces an overview of Delta tool system information. And finally, it compares the proposed approach with dead code3 elimination. Chapter 3 describes thoroughly the problem statement, requirements, use cases and analysis for this approach. Chapter 4 introduces the design of Delta Coverage tool. Also, it ex- plains the structure and process of Delta Coverage tool implementation. Chapter 5 describes the evaluation and analysis of the results in differ- ent test cases using Delta Coverage tool. Chapter 6 details the conclusion of this project and possible future de- velopments. 3http://en.wikipedia.org/wiki/Dead_code 8 Chapter 2 Background 2.1 Delta debugging Delta Debugging is an algorithm that is based on hypothesis to find out bugs. It is one of the methods that can automate the scientific method of debugging. And also, this algorithm simplifies and prevents program bugs. In 1997, while Andreas Zeller was finishing his PhD thesis “Configuration Management with Version Sets1" at the same time he and his student started to work on the visual debugger to visualise the structures data in execution time. Then, they developed the Delta Debugging algorithm which could make debugging programs easier. 2.1.1 Simplifying and isolating failure-inducing The research of Holger Cleve and Andreas Zeller[1] explains the utilities of Delta Debugging algorithm. They use it to simplify failing test case and isolate the difference between passing and failing test cases automatically. This algorithm classifies the outcomes of testing in three difference cases after running the algorithm that has to be debugged: 1. The result is passing (3). 2. The result is failing (7). 1https://www.st.cs.uni-saarland.de/publications/files/zeller-thesis-1997.pdf 9 3. The result is unsolved (?). Delta Debugging would have been useful for one real case in the past decade (Bugzilla, the Mozilla bug database).