First Steps in Retrofitting a Versatile Software Testing Infrastructure to Android by Carol Anne Oliver A dissertation submitted to the College of Engineering and Science of Florida Institute of Technology in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science Melbourne, Florida December 2018 © Copyright 2018 by Carol Anne Oliver. All rights reserved. Pyramids in Figures 1, 2, and 4 from Hands-On Mobile App Testing by Daniel Knott © Copyright 2015 Pearson Education, Inc. “First Steps in Retrofitting a Versatile Software Testing Infrastructure to Android,” a dissertation by Carol Anne Oliver. Approved as to style and content. _________________________________________________ Heather Crawford, Ph.D. Assistant Professor, Computer Engineering and Sciences College of Engineering and Science Committee Chair _________________________________________________ John Deaton, Ph.D. Professor College of Aeronautics External Committee Member _________________________________________________ Cem Kaner, J.D., Ph.D. Professor Emeritus, Computer Engineering and Sciences College of Engineering and Science Internal Committee Member _________________________________________________ Keith Gallagher, Ph.D. Associate Professor, Computer Engineering and Sciences College of Engineering and Science Internal Committee Member _________________________________________________ William Shoaff, Ph.D. Associate Professor, Computer Engineering and Sciences College of Engineering and Science Internal Committee Member _________________________________________________ Philip J. Bernhard, Ph.D. Department Head, Computer Engineering and Sciences College of Engineering and Science Abstract First Steps in Retrofitting a Versatile Software Testing Infrastructure to Android Author: Carol Anne Oliver Advisor: Heather Crawford, Ph.D. Are existing software testing technologies satisfying the needs of mobile software testing practitioners? This dissertation provides evidence that the existing technologies fall short of practitioner’s actual needs and analyzes some reasons why this is so. The first contribution presents a framework for analyzing the software testing needs of practitioners, which is then used to assess the existing software testing tools and body of research, revealing significant gaps. The second contribution proposes a vision for a new software testing technology for mobile apps, which is then applied to a real-world problem causing problems for software testing practitioners (namely, the verification of GUI layouts on diverse mobile devices). The third contribution presents a detailed, ecosystem-independent approach for implementing this new solution. The fourth contribution applies this generic implementation to Android, exposes missing infrastructure within Android, and presents first steps toward a solution. Finally, this dissertation discusses the ramifications of lessons learned during this work and a variety of paths for further research. iii Table of Contents Table of Contents ................................................................................................... iv List of Figures ....................................................................................................... viii List of Tables .......................................................................................................... ix Acknowledgement .................................................................................................... x Introduction .............................................................................................................. 1 Vocabulary Note ........................................................................................................... 3 Generality Note ............................................................................................................. 4 Chapter 1 The Existing State of Mobile App Testing ........................................... 5 1.1 The Pyramids View of Mobile Software Testing .................................................... 5 1.2 Major Historical Phases of Computing and Software Testing .............................. 7 1.3 Software Testing Technologies ............................................................................... 10 1.3.1 Physical Devices ................................................................................................ 11 1.3.1.1 Accessing Sufficient Diversity ................................................................................ 12 1.3.1.2 Technology Strengths and Limitations .................................................................... 15 1.3.1.3 Implementation Limitations .................................................................................... 16 1.3.1.4 Physical Devices Technology Summary ................................................................. 17 1.3.2 Virtual Devices .................................................................................................. 18 1.3.2.1 Technology Strengths and Limitations .................................................................... 20 1.3.2.2 Implementation Limitations .................................................................................... 21 1.3.2.3 Virtual Devices Technology Summary ................................................................... 22 1.3.3 Simulation Environments ................................................................................... 23 1.3.3.1 Technology Strengths and Limitations .................................................................... 24 1.3.3.2 Implementation Limitations .................................................................................... 24 1.3.3.3 Simulation Environments Summary ........................................................................ 25 1.3.4 Visual GUI Tree ................................................................................................. 25 1.3.4.1 Technology Strengths and Limitations .................................................................... 26 1.3.4.2 Implementation Limitations .................................................................................... 28 1.3.4.3 Visual GUI Tree Technology Summary .................................................................. 30 1.3.5 Image Comparison ............................................................................................. 31 1.3.5.1 Technology Strengths and Limitations .................................................................... 33 1.3.5.2 Implementation Limitations .................................................................................... 34 1.3.5.3 Image Comparison Technology Summary .............................................................. 36 1.3.6 Code Manipulation ............................................................................................. 36 1.3.6.1 Technology Strengths and Limitations .................................................................... 41 1.3.6.2 Implementation Limitations .................................................................................... 42 1.3.6.3 Code Manipulation Technology Summary .............................................................. 43 1.3.7 System Monitoring ............................................................................................. 43 1.3.7.1 Technology Strengths and Limitations .................................................................... 44 1.3.7.2 Implementation Limitations .................................................................................... 45 1.3.7.3 Built-in Program Diagnostics .................................................................................. 45 iv 1.3.7.4 System Monitoring Technology Summary .............................................................. 45 1.4 Mobile Software Testing Technologies Recap ...................................................... 46 Chapter 2: A Release-Readiness View of Mobile App Testing .......................... 49 2.1 Release-Readiness Levels Framework ................................................................... 50 2.1.1 Multi-Device Scenarios Exist at Each Level ..................................................... 52 2.2 Level 0: Dev Testing ................................................................................................ 53 2.3 Level 1: Ready to Test ............................................................................................. 54 2.4 Level 2: Closely Controlled .................................................................................... 56 2.5 Level 3: Predictable Variations .............................................................................. 59 2.6 Level 4: Discovers Unexpected ............................................................................... 62 2.7 Level 5: Exhausted Capacities ................................................................................ 63 2.8 Release-Readiness Levels Model Recap ................................................................ 65 Chapter 3: Vision of a New Technology Solution to These Gaps ...................... 66 3.1 Recapping the Problem Space ................................................................................ 66 3.1.1 Requirements for New Testing Tools ................................................................ 67 3.1.1.1 Target Audience ...................................................................................................... 6 7 3.2 Addressing the New Testing Tool