Automated Testing and Validation of Computer Graphics
Total Page:16
File Type:pdf, Size:1020Kb
Automated Testing and Validation of Computer Graphics Implementations for Cross-Platform Game Development A thesis submitted to Auckland University of Technology in fulfilment of the requirements for the degree of Master of Philosophy (MPhil) Supervisors Dr. Stefan Marks Anne Philpott 2017 By Steffan Derek Hooper Colab: Creative Technologies Abstract Commercially released cross-platform video games often feature graphical defects, which negatively impact on the reputations of developers and publishers, as well as on the experience of the players. Game industry testing practices often rely on human testers to assure the quality of a game prior to release. This thesis investigates the question of whether the testing and validation of computer graphics implementations for cross-platform game development can be automated to reduce the burden on hu- man testers, accelerate the testing phase, and improve the quality of games. Using Design Science Research methods and patterns, iterative development and evaluation is undertaken to construct ar- tefacts, drawing upon prior research and industrial works in related fields such as film and television, as well as proprietary game development insight. Elements of existing automated testing systems and image comparison techniques are combined with industry standard cross-platform development tools and methods to create a reusable and generalisable model of an automated test system, featuring image comparison methods, and record and playback techniques for cross-platform game implementations. In conclusion, this thesis shows that it is possible to create a novel system that is able to detect graphical defects in output from a cross-platform game implementation. i Contents Abstract i Attestation of Authorship viii Acknowledgements ix 1 Introduction 1 1.1 Overview . .1 1.2 Research Motivation . .5 1.2.1 Graphical Bugs . .6 1.3 Research Questions . .8 1.4 Research Method . .9 1.5 A Brief History of Computer Graphics and Video Game Technology . .9 1.5.1 Computer Graphics APIs . 10 1.5.2 Middleware and Game Engines . 13 1.6 Summary . 15 2 Literature Review 16 2.1 Structure of the Literature Review . 16 2.2 Video Game Graphical Defects . 17 2.2.1 Examples of Graphical Defects in Commercial Games . 17 2.3 Automated Testing in Games . 33 2.3.1 Industry Insight: Unity Technologies . 34 2.3.2 Industry Insight: Electronic Arts . 41 2.3.3 Games-related Testing Automation Research . 44 2.4 Automated Testing in Related Fields . 47 2.5 Findings of Literature Review . 56 3 Methodological Approach 58 3.1 Introduction . 58 3.2 Methodology: Design Science Research . 59 3.2.1 Patterns . 63 3.3 Method: Research Project Design . 63 3.3.1 Awareness of Problem . 63 3.3.2 Suggestion . 64 3.3.3 Development . 65 3.3.4 Evaluation . 66 3.3.5 Conclusion . 68 3.4 Tools: Artefact Construction . 68 3.4.1 Industry Tools and Techniques for Cross-Platform Game Development . 68 3.4.2 Cross-Platform Development: The Layers of a Target Platform . 71 3.4.3 Image Comparison . 71 3.5 Limitations . 71 3.6 Summary . 72 ii 4 Artefact Development and Evaluation 74 4.1 Introduction . 74 4.2 Awareness of Problem . 75 4.3 Suggestion . 75 4.4 Development . 76 4.4.1 Iteration 1 . 78 4.4.2 Iteration 2 . 83 4.4.3 Iteration 3 . 87 4.4.4 Iteration 4 . 93 4.4.5 Iteration 5 . 95 4.4.6 Iteration 6 . 101 4.4.7 Iteration 7 . 104 4.4.8 Iteration 8 . 113 4.5 Summary of Artefact Development and Evaluation . 115 5 Discussion 116 5.1 Overview . 116 5.2 Summary of Results . 116 5.3 Limitations . 119 6 Conclusion 120 6.1 Future Work . 121 References 122 Glossary 132 Appendices 135 A Game Industry Testing 136 A.1 Types of Testing . 137 A.2 Compliance Testing for Certification . 138 A.3 Industry Insight: Testing . 139 B Iteration 6: RobotWorld Assets 141 B.1 Graphical Assets . 141 C Software Engineering Research Laboratory (SERL) Seminar 2014 142 C.1 Seminar Presentation Slides . 142 D PikPok Developers' Conference 2015 154 D.1 Conference Presentation Slides . 154 iii List of Figures 1.1 Target platform block diagram. Adapted from \The History of Visual Magic in Com- puters: How Beautiful Images are Made in CAD, 3D, VR, and AR" by J. Peddie, Copyright 2013. .3 1.2 Assassin's Creed Unity (Ubisoft Montreal, 2014a) screen capture featuring a graphical defect where the character's face is incorrectly rendered ([Assassin's Creed Unity image screen capture], n.d.). .6 2.1 NBA 2K13 (Visual Concepts, 2012) screen capture featuring a clipping issue where one character's forearm penetrates the face of another character ([NBA 2K13 image screen capture], n.d.). 19 2.2 Garry's Mod (Facepunch Studios, 2004) screen capture of a video demonstrating a z- fighting issue, where the wall in the centre of the frame flickers from the dark grey texture to a different lighter grey texture as the player moves (Exho, 2014). 20 2.3 Battlefield 4 (EA Digital Illusions CE, 2013) screen capture featuring a rendering error where red spots are rendered ([Battlefield 4 image screen capture], n.d.). 21 2.4 Batman: Arkham Knight (Rocksteady Studios, 2015) screen capture featuring a rende- ring error where the in-game character has spherical items rendered around its eyes, as well as artifacts across the lower third of the frame ([Batman: Arkham Knight image screen capture], n.d.). 22 2.5 Mafia II (2K Czech, 2010) screen capture featuring a rendering error where shadowing is incorrectly rendered across half the frame ([Mafia II image screen capture], n.d.). 23 2.6 Grand Theft Auto: San Andreas (Rockstar North, 2004) screen capture featuring a rendering error where the in-game character on the right is missing its head ([Grand Theft Auto: San Andreas image screen capture], n.d.). 24 2.7 Call of Duty: Modern Warfare 3 (Infinity Ward, 2011) screen capture featuring a ren- dering error where the in-game character is missing its head ([Call of Duty: Modern Warfare 3 image screen capture], n.d.). 25 2.8 Rocky (Rage Software, 2002) screen captures featuring graphical errors. 25 2.9 Battlefield 3 (EA Digital Illusions CE, 2011) screen capture featuring a rendering error where the in-game character features an elongated neck ([Battlefield 3 image screen capture], n.d.). 26 2.10 Fallout 4 (Bethesda Game Studios, 2015) screen capture featuring a rendering error where the player can see through the building on the left of the frame ([Fallout 4 image screen capture], n.d.). 27 2.11 Age of Wonders III (Triumph Studios, 2014) screen capture featuring a rendering error where red tooltip text is incorrectly rendered at the top-centre of the frame, revealing the secret location of an enemy army obscured behind clouds ([Age of Wonders III image screen capture], n.d.). 28 2.12 Overwatch (Blizzard Entertainment, 2016) screenshot comparison of the same scene rendered on two different graphics settings, showing that a player using a lower graphics setting can see more of the game environment than a player using a higher graphics setting, due to missing foliage. 28 2.13 Rage (id Software, 2011) screen capture featuring a texturing issue where the game world exhibits a checkerboard pattern ([Rage image screen capture], n.d.). 29 iv 2.14 Assassin's Creed IV: Black Flag (Ubisoft Montreal, 2013) screen capture featuring blur- red text in the heads-up-display of the user interface ([Assassin's Creed IV: Black Flag image screen capture], n.d.). 30 2.15 Unity graphics test result example ([Unity reference image, test result image, difference image], n.d.). 39 2.16 Unity Graphics Tests Tool screenshot ([Unity Graphics Tests Tool image], n.d.). 40 3.1 Activity framework for Design Science Research. Adapted from \On theory development in design science research: Anatomy of a research project" by B. Kuechler and V. Vaishnavi, European Journal of Information Systems, Volume 17, Issue 5, pp. 489-504. Copyright 2008 by Palgrave Macmillan. ..