Rochester Institute of Technology RIT Scholar Works Theses 5-19-2016 Experimental Design in Game Testing Bhargava Rohit Sagi [email protected] Follow this and additional works at: https://scholarworks.rit.edu/theses Recommended Citation Sagi, Bhargava Rohit, "Experimental Design in Game Testing" (2016). Thesis. Rochester Institute of Technology. Accessed from This Thesis is brought to you for free and open access by RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. Rochester Institute of Technology Experimental Design in Game Testing A Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Industrial and Systems Engineering in the Department of Industrial & Systems Engineering Kate Gleason College of Engineering by Bhargava Rohit Sagi May 19, 2016 DEPARTMENT OF INDUSTRIAL AND SYSTEMS ENGINEERING KATE GLEASON COLLEGE OF ENGINEERING ROCHESTER INSTITUTE OF TECHNOLOGY ROCHESTER, NEW YORK CERTIFICATE OF APPROVAL M.S. DEGREE THESIS The M.S. Degree Thesis of Bhargava Rohit Sagi has been examined and approved by the thesis committee as satisfactory for the thesis requirement for the Master of Science degree Approved by: ____________________________________ Dr. Rachel Silvestrini(Thesis Advisor), Associate Professor, Industrial and Systems Engineering ____________________________________ Dr. Brian K. Thorn, Associate Professor, Industrial and Systems Engineering ____________________________________ Dr. Jessica Bayliss, Associate Professor, Interactive Games and Media ____________________________________ Dr. David Schwartz, Associate Professor, Interactive Games and Media 2 Abstract The gaming industry has been on constant rise over the last few years. Companies invest huge amounts of money for the release of their games. A part of this money is invested in testing the games. Current game testing methods include manual execution of pre-written test cases in the game. Each test case may or may not result in a bug. In a game, a bug is said to occur when the game does not behave according to its intended design. The process of writing the test cases to test games requires standardization. We believe that this standardization can be achieved by implementing experimental design to video game testing. In this thesis, we discuss the implementation of combinatorial testing to test games. Combinatorial testing is a method of experimental design that is used to generate test cases and is primarily used for commercial software testing. In addition to the discussion of the implementation of combinatorial testing techniques in video game testing, we present a method for finding combinations resulting in video game bugs. 3 Contents Abstract ......................................................................................................................................................... 5 1 Introduction ................................................................................................................................................ 5 2 Literature Review ....................................................................................................................................... 7 2.1 Game testing techniques ..................................................................................................................... 7 2.2 Combinatorial testing .......................................................................................................................... 8 2.2.1 Overview of combinatorial testing ............................................................................................... 8 2.2.2 Tools for generating combinatorial tests: ................................................................................... 10 2.2.3 Identifying failure inducing combinations ................................................................................. 11 3. Methodology ........................................................................................................................................... 13 3.1 Combinatorial testing for video game debugging ............................................................................. 13 3.2 Classification of factors .................................................................................................................... 15 3.3 Methodology for finding bugs: ......................................................................................................... 17 3.3.1 Sorting test cases to find Failure Inducing Combination ........................................................... 17 4. Results: .................................................................................................................................................... 21 4.1 Combinatorial testing in video games ............................................................................................... 21 4.1.1 Previously tested games ............................................................................................................. 21 4.1.2 Games not previously tested: ..................................................................................................... 25 4.2 Identifying failure inducing combinations ........................................................................................ 29 4.2.1 Notional example with Grand Prix............................................................................................. 29 4.2.2 Example from the game Erator .................................................................................................. 30 5. Discussion ............................................................................................................................................... 33 5.1 Conclusion ........................................................................................................................................ 33 5.2 Future work ....................................................................................................................................... 34 Bibliography ............................................................................................................................................... 35 Appendix I: ................................................................................................................................................. 35 4 Tables: Table 1 Coverage 1 design ............................................................................................................................ 9 Table 2 Coverage 2 design ............................................................................................................................ 9 Table 3 Tools meeting our requirements .................................................................................................... 11 Table 4 Factors and levels of Grand Prix .................................................................................................... 22 Table 5 Number of test cases per coverage for Grand Prix ......................................................................... 22 Table 6 Coverage 1 test set for Grand Prix ................................................................................................. 23 Table 7 Breakdown of bugs into factor for Grand Prix ............................................................................... 24 Table 8 Grand Prix results .......................................................................................................................... 25 Table 9 Factors and levels of game Erator .................................................................................................. 26 Table 10 Number of test case per coverage for Erator ................................................................................ 26 Table 11 Coverage Vs Bugs in Erator ......................................................................................................... 27 Table 12 Factors and levels of Utopia ......................................................................................................... 27 Table 13 Number of test cases per coverage for Utopia ............................................................................. 28 Table 14 Coverage Vs Bugs in Utopia ........................................................................................................ 28 Table 15 Reference test case ....................................................................................................................... 29 Table 16 Test case with different factor levels ........................................................................................... 29 Table 17 Test case with same Kart ............................................................................................................. 30 Table 18 Test case with same Racer ........................................................................................................... 30 Table 19 Test case with same Race ............................................................................................................. 30 Table 20 Test case with same World .......................................................................................................... 30 Table 21 Test case with same Racer and Kart ............................................................................................ 30 Table 22 Test case with same Race and Kart .............................................................................................