Improvements of and Extensions to Fsmweb: Testing Mobile Apps
Total Page:16
File Type:pdf, Size:1020Kb
University of Denver Digital Commons @ DU Electronic Theses and Dissertations Graduate Studies 1-1-2019 Improvements of and Extensions to FSMWeb: Testing Mobile Apps Ahmed Fawzi Al haddad University of Denver Follow this and additional works at: https://digitalcommons.du.edu/etd Part of the Digital Communications and Networking Commons Recommended Citation Al haddad, Ahmed Fawzi, "Improvements of and Extensions to FSMWeb: Testing Mobile Apps" (2019). Electronic Theses and Dissertations. 1639. https://digitalcommons.du.edu/etd/1639 This Dissertation is brought to you for free and open access by the Graduate Studies at Digital Commons @ DU. It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of Digital Commons @ DU. For more information, please contact [email protected],[email protected]. Improvements of and Extensions to FSMWeb: Testing Mobile Apps A Dissertation Presented to the Faculty of the Daniel Felix Ritchie School of Engineering and Computer Science University of Denver In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy by Ahmed Fawzi ALhaddad August 2019 Advisor: Prof. Anneliese Andrews c Copyright by Ahmed Fawzi ALhaddad, 2019 All Rights Reserved Author: Ahmed Fawzi ALhaddad Title: Improvements of and Extensions to FSMWeb: Testing Mobile Apps Advisor: Prof. Anneliese Andrews Degree Date: August 2019 Abstract A mobile application is a software program that runs on mobile device. In 2017, 178.1 billion mobile apps downloaded and the number is expected to grow to 258.2 billion app downloads in 2022 [19]. The number of app downloads poses a challenge for mobile application testers to find the right approach to test apps. This dissertation extends the FSMWeb approach for testing web applications [50] to test mobile applications (FSMApp). During the process of analyzing FSMWeb how it could be extended to test Mobile Apps, a number of shortcomings were detected which we improved upon. We discuss these first. We present an approach to generate black-box tests to test fail-safe behavior for web applications. We apply the approach to a large commercial web application. The approach uses a functional (behavioral) model to generate tests. It then determines at which states in the execution of behavioral test failures can occur and what mitigation requirements need to be tested. Mitigation requirements are used to build mitigation models for each failure type. From those mitigation models failure mitigation tests are generated. Next, this dissertation provides an approach for selective black-box model-based fail-safe regression testing for web applications. It classifies existing tests and test requirements as reusable, retestable, and obsolete. Removing reusable test require- ments reduces test requirements between 49% to 65% in the case study. The ap- proach also uses partial regeneration for new tests wherever possible. Third, we present the new FSMApp approach to test mobile applications and compare the ii approach with several other approaches [88, 37]. A number of case studies ex- plore applicability, scalability, effectiveness, and efficiency of FSMApp with other approaches. Future work makes suggestion on how to improve test generation and execution efficiency with FSMApp. iii Acknowledgements I would like to express my deepest appreciation to my advisor Professor Anneliese Andrews for her tremendous support during my Ph.D. adventure. I cannot thank you enough for your patience, encouragement, leadership, and extensive knowledge. Your continuous assistance and guidance aided me to expand and challenge my research dissertation. I am grateful for having you as my advisor for my Ph.D. research. In addition, I would like to thank my committee members, Professor Gareth Eaton, Professor Scott Leutenegger and Associate Professor Christopher GauthierDickey, for believing in me by being members in my oral defense committee. I truly appreci- ate your time and assistance. My special thanks to my parents, siblings, grandpar- ents, friends, and colleagues for their fantastic support during my academic journey. iv Contents 1 Introduction .................................. 1 1.1 Problem . 1 1.2 Existing Body of Work Related to FSMWeb . 3 1.3 Research Agenda . 4 2 Background .................................. 7 2.1 Black-Box Model-Based Testing . 7 2.2 Testing Mobile Apps . 9 2.2.1 MBT and App Testing . 9 2.2.2 General App Testing Tools . 26 2.3 Regression Testing (RT) . 30 2.4 Fail-Safe Testing . 32 3 Original Approach .............................. 36 3.1 Testing Fail-Safe Behavior w/FSM Web . 36 3.1.1 Process . 36 3.2 FSMWeb Approach . 37 3.2.1 Example . 41 3.3 External Failures (F), State-Event matrix (SE) . 42 3.4 Potential Failure Scenarios . 44 3.5 Generate Test Requirements . 45 3.6 Mitigation Requirements and Models . 45 3.7 Weaving Rules and Test Generation . 48 3.8 Generate Tests, Execute and Validate . 49 4 Approach Improvements .......................... 50 4.1 Mortgage System Case Study: Fail-Safe Testing . 50 4.1.1 Case Study Research Questions . 50 4.1.2 Mortgage System . 51 4.1.3 FSMWeb Behavioral Model . 54 4.1.4 Failure Applicability and Mitigation Requirements . 62 4.1.5 Test Requirements and their Effectiveness . 64 4.1.6 Comparison of Effort: GA vs. Exhaustive Search . 71 v 4.1.7 Findings . 72 4.1.8 Threats to Validity . 73 4.2 Regression Testing Approach . 75 4.2.1 Process . 76 4.2.2 Detailed Approach . 80 4.2.3 Discussion . 100 4.2.4 Comparison to Earlier Work . 103 4.3 Mortgage System Case Study: Regression Test . 106 4.3.1 Case Study Objectives . 106 4.3.2 Case Study Research Questions . 107 4.3.3 General Description . 107 4.3.4 RQ1: Applicability . 108 4.3.5 RQ2: Efficiency of Selective Regression Testing . 112 4.3.6 RQ3: Genetic Algorithm vs. Coverage Criteria . 113 4.3.7 RQ4: Reusable Failure Scenarios . 113 4.3.8 Threats to Validity . 114 4.3.9 Practical Considerations . 115 5 Extensions: Testing Mobile Apps .................... 118 5.1 Testing Process for Mobile Apps . 118 5.2 Example Used to Illustrate Approach . 120 5.3 Phase 1: Build Model . 120 5.3.1 Partition the Mobile App into Clusters (Cs). 120 5.3.2 Define Logical App Pages (LAPs) and Input-Action Constraints for Each . 128 5.4 Phase 2: Generate Test Sequences . 146 5.4.1 Paths through FSMs/AFSM . 146 5.4.2 Path Aggregation . 149 5.4.3 Input Selection . 156 5.5 Phase 3: Execute and Validate Tests . 157 6 Comparing FSMApp and other Approaches .............. 163 6.1 Event Sequence Graph (ESG) Method . 163 6.2 A GUI Crawling-Based Technique for Android Mobile Application Testing . 175 6.3 Comparison of Results . 183 7 FSMApp Validation via Case Studies .................. 189 7.1 Rationale . 189 7.2 Case Study Objectives . 189 7.3 Preparation for Case Study . 189 7.4 Case Study Research Questions . 190 7.5 Units of Analysis . 192 vi 7.6 Case Study General Descriptions & Rationale . 192 7.7 Case Study Results & Discussion . 196 7.7.1 RQ1: Applicability . 196 7.7.2 RQ2: Scalability . 197 7.7.3 RQ3: Efficiency . 198 7.7.4 RQ4: Effectiveness . 203 7.7.5 Compare FSMApp with Other Approaches . 204 7.8 Threats to Validity . 212 8 Future Work ................................. 224 8.1 Regression Testing . 224 8.2 New system domains . 224 8.3 Building Tools . 224 8.4 Effectiveness . 225 8.5 Improving Efficiency of Test Execution . 225 8.5.1 Efficiency Improvements During Execution . 225 8.5.2 Implementation . 229 9 Conclusions .................................. 233 Appendix A Android Terms ......................... 237 Appendix B Family Medicines List App Screens ............ 238 Appendix C BNF Grammar for Input Selection Constraints ..... 239 Appendix D Family Medicines List App .................. 242 D.1 Clusters and Nodes . 242 D.2 Reduced Test Sequences and Test Values . 245 D.3 Selenium Code of Test Paths . 252 Bibliography ................................... 262 vii List of Tables 1.1 Publication . 5 2.1 Classification Framework [193] . 11 2.2 Studies Under Each Topic (Subcategory) [233] . 15 2.3 Manual and Automation Papers on Model building and Test Generation 16 2.4 Categories for Tools . 29 3.1 Constraints on Inputs . 39 3.2 FSMWeb Constraint of Typical Input Types . 39 3.3 Test Paths Through AFSM,FSM1,FSM2 . 42 3.4 Test Paths for BM ............................ 43 3.5 State-Event Matrix SE . 43 3.6 Potential Failure Scenarios SP ...................... 45 3.7 Mitigation Requirements . 47 3.8 Mitigation Patterns and Weaving Rules . 48 3.9 Selected (i; p; e) Triplets and Resulting FMT . 49 4.1 Software Characteristics of the Mortgage System [70] . 52 4.2 Decomposition of Mortgage System into Partitions . 55 4.3 Nodes for CD Form-FSM . 55 4.4 Input Constraints of Aggregate FSM for Mortgage System . 57 4.5 Input Constraints for LPD Cluster . 58 4.6 Input Constraint for Closing Documents (CD) FSM . 60 4.7 Test Sequence for Closing Documents (CD) FSM . 60 4.8 Test Paths of Aggregate FSM for the Mortgage System . 61 4.9 Test Paths of FSM for LPD Cluster . 61 4.10 Statistics of Tests Size . 62 4.11 Paths Generated by Substitution of T1 . 62 4.12 Paths Generated by Substitution of T3 . 63 4.13 Failure Types in Cluster Closing Documents (CD) . 64 4.14 Mitigation Requirement for CD Cluster . 65 4.15 State-Event Matrix for CD Cluster . 66 4.16 Abstract Test Paths (CD Cluster) . 66 viii 4.17 Initial FMT for First Generation . ..