tApp: testing the future by Karsten Westra University of Groningen March 17, 2014 1 Change log Version Date Author Comment 0.1 01-02-2013 Karsten Westra - Initial version/plan. 0.2 16-07-2013 Karsten Westra - Added layout for application chapter (6). - Illustrated basic layout using example. Minor detailed description of general workflow. 0.8 29-08-2013 Karsten Westra - Changed title "Specification: tApp" to "tApp: Testing the future". - Added change log. - Added acknowledgment. - Expanded applications chapter (6). 0.8.1 30-08-2013 Karsten Westra Chapter6: - explained device package - explained device browser. 0.8.2 05-09-2013 Karsten Westra - Explained and implemented report browser and graphs. - Stakeholder summary. - Added references chapter 0.8.3 07-09-2013 Karsten Westra Expanded chapter 7.7 with detailed cases: t/m Monkeytalk demo app, Fit for free, LG Klimaat. 09-09-2013 0.9 10-09-2013 Karsten Westra Written discussion. 1.0 11-09-2013 Karsten Westra Written conclusion 1.0.1 02-10-2013 Karsten Westra Thorough review. until 05-10-2013 1.1 07-10-2013 Karsten Westra Added chapter with related work. 1.2 09-10-2013 Karsten Westra Thorough review of references to chapters, goals, until figures and tables. 11-10-2013 1.3 14-10-2013 Karsten Westra * Added specification, distribution, reporting layout to chapter5 (previously known as implementation). * Reviewed chapter6 + added references to relevant requirements, figures and sections. 1.4 15-10-2013 Karsten Westra * Reviewed size and placements of figures in chapter6. * Reviewed and added references to figures 1.5 16-10-2013 Karsten Westra Merged Chapter 5.4 with 6. Rewritten implementation chapter and called it application. Rewritten introduction and reviewed references. 2 Version Date Author Comment 1.6 17-10-2013 Karsten Westra Reviewed old applications chapter and renamed it ’experiment’. 1.7 17-10-2013 Karsten Westra Review and rewrite of chapters7 and8. 2.0 18-10-2013 Karsten Westra Reviewed and rewritten discussion and conclusion. 2.1 22-11-2013 Karsten Westra * Changed structure of chapter 2.7 into Specification, Distribution and Reporting. * Added 7.1.2, 7.1.3 and 7.1.5 to discussion. * Renamed reference labels. * Changed al usages of you by a user, a developer and so on. 2.2 19-01-2014 Karsten Westra Refined chapter2: until * Added general comments about automated testing 12-02-2014 and how and why to tackle fragmentation with it. * Elaborated on how other researchers tackle fragmentation. * added section on the "perfect solution" in conclusion subsection. 3 Contents 1 Introduction 9 1.1 Scope............................................. 10 1.2 Goal.............................................. 10 1.3 Thesis structure........................................ 11 2 Related Work 12 2.1 Automated testing...................................... 12 2.2 Tackle fragmentation..................................... 12 2.2.1 Proposed solutions.................................. 12 2.2.2 Useful theories.................................... 13 2.3 Existing tools......................................... 13 2.3.1 Monkeyrunner.................................... 14 2.3.2 UIAutomator..................................... 14 2.3.3 Robotium....................................... 15 2.3.4 Seetest......................................... 15 2.3.5 Telerik Test Studio.................................. 16 2.3.6 Monkeytalk...................................... 16 2.4 Conclusion.......................................... 17 2.4.1 UI element recognition................................ 18 2.4.2 Presentation of issues................................ 18 2.4.3 Cover representative group of device types..................... 18 2.4.4 Nice to haves..................................... 18 3 Concept 19 3.1 Description of stakeholders................................. 20 3.2 Stakeholder requirements.................................. 20 3.3 Summary........................................... 21 4 Design 23 4.1 General data flow....................................... 24 4.1.1 Project and versioning................................ 24 4.1.2 Test and device package............................... 24 4.2 Detailed data flow...................................... 24 4.3 Refined design........................................ 26 4.3.1 Create a project................................... 26 4.3.2 Specification perspective.............................. 27 4.3.3 Distribution perspective............................... 28 4.3.4 Reporting perspective................................ 29 4.4 Summary........................................... 29 5 Application 31 5.1 Specification.......................................... 31 5.1.1 Test package..................................... 31 5.2 Distribution.......................................... 32 5.2.1 Test execution.................................... 32 5.2.2 Device package.................................... 32 4 5.2.3 Collected device information............................ 33 5.3 Reporting........................................... 33 5.3.1 Stable behavior.................................... 33 5.3.2 Verify expected values of components....................... 34 5.3.3 Execute native code................................. 34 5.3.4 Boundaries...................................... 34 5.4 Equivalence class....................................... 35 5.4.1 Predict behavior................................... 37 5.5 Summary........................................... 37 6 Use cases 38 6.1 General usage......................................... 38 6.1.1 Dashboard...................................... 38 6.1.2 From nothing to test report............................ 39 6.2 Device package........................................ 42 6.2.1 Device data...................................... 43 6.2.2 Preparing an existing project............................ 44 6.2.3 Script/Suite exports................................. 44 6.2.4 Device package internals............................... 46 6.3 Device browser........................................ 46 6.4 Report browser........................................ 47 6.5 More complex examples................................... 50 6.5.1 Experiment...................................... 51 6.5.2 Explanation of cases: ’Monkeytalk demo’, ’Fit For Free’ and ’LG Klimaat’... 51 6.5.3 Monkeytalk demo.................................. 52 6.5.4 Fit for Free...................................... 56 6.5.5 LG Klimaat...................................... 60 7 Discussion 63 7.1 Challenges........................................... 63 7.1.1 Specification..................................... 63 7.1.2 Validation of specification.............................. 64 7.1.3 Portability of specification.............................. 64 7.1.4 Reporting...................................... 64 7.1.5 Beyond reporting................................... 65 7.1.6 Running from background............................. 65 7.1.7 Test execution and app preparation........................ 65 7.1.8 Settings........................................ 66 7.1.9 Record from device.................................. 66 7.1.10 Equivalence classes and predict behavior...................... 67 7.2 Improvements......................................... 67 7.2.1 Ease of test result inspection (G1)......................... 67 7.2.2 Level of detail (G2) and Queries (G3)....................... 67 7.2.3 Ease of test execution (G4)............................. 68 7.2.4 Scalability (G5).................................... 68 7.3 Summary........................................... 68 5 8 Conclusion 69 8.1 Specification.......................................... 69 8.2 Distribution.......................................... 69 8.2.1 Settings........................................ 69 8.3 Reporting........................................... 70 8.4 Summary........................................... 70 A List of requirements 73 A.1 Specification (FRS)...................................... 73 A.1.1 Distribution (FRDI)................................. 74 A.1.2 Reporting (FRRE).................................. 74 A.1.3 Device package (FRDP)............................... 74 A.1.4 Test package (FRTP)................................ 75 A.1.5 Non-functional requirements (NFG)........................ 75 A.1.6 Evolution requirements (ERG)........................... 75 B Scripts/suites used 76 B.1 Monkeytalk demo....................................... 76 B.2 Fit for free.......................................... 77 B.3 LG Klimaat.......................................... 78 B.3.1 Scenario part 1.................................... 78 B.3.2 Scenario part 2.................................... 78 6 Acknowledgements First of all I would like to thank my main supervisor (Alex Telea) for giving the large amounts of feedback and suggestions. He really helped me with the process from idea to working prototype. Thank Peperzaken, the app development company who I work for. Thanks a lot for having the patience to let me finish my study next to work. And thanks for lending me a workspace setup with test devices and apps. It really helped me to try everything on a range of devices with existing apps. Thank to my family who always supported my choices and wishes. You motivated me whenever possible when things got really challenging. Thanks to you all for helping me get to where I am now! 7 Abstract Mobile phones are a fast growing technology on the market. Nearly everybody has a smart phone nowadays. There is an astonishing amount of different device types to choose from.