ASSIGNMENT OF BACHELOR’S THESIS Title: Automated Acceptance Testing on macOS – A survey study focused on Avast Passwords for mac Student: David Mokoš Supervisor: MSc Felix Javier Acero Salazar Study Programme: Informatics Study Branch: Web and Software Engineering Department: Department of Software Engineering Validity: Until the end of summer semester 2018/19 Instructions 1] Survey the different technologies available for writing acceptance tests on macOS. 2] Select an appropriate test stack: GUI driving technology, testing framework, etc. 3] Prepare application for acceptance testing based on the selected test stack. 4] Implement a suite of acceptance tests to cover the most relevant requirements of Avast Passwords for mac. References Will be provided by the supervisor. Ing. Michal Valenta, Ph.D. doc. RNDr. Ing. Marcel Jiřina, Ph.D. Head of Department Dean Prague November 28, 2017 Czech Technical University in Prague Faculty of Information Technology Department of Software Engineering Bachelor’s thesis Automated Acceptance Testing on macOS – A survey study focused on Avast Passwords for Mac David Mokoˇs Supervisor: MSc. Felix Javier Acero Salazar May 14, 2018 Acknowledgements First and foremost, I would like to thank my thesis supervisor, MSc. Felix Javier Acero Salazar, for his patient guidance, advice and encouragement he has provided throughout the whole time I have been working in Avast. I have been lucky to meet such a hardworking, inspirative and helpful person who cares so much about my work and who always keeps his word. His curiosity and positive mindset inspired me and helped me particularly when exploring new ideas. His careful revising contributed hugely to the creation of the thesis. I take this opportunity to express gratitude to all of the Department faculty members for their help and support throughout my studies. I also thank my parents for the endless encouragement, support and attention. Last but not least, I am grateful to my partner who supported me and accepted my limited free time during this venture. Declaration I hereby declare that the presented thesis is my own work and that I have cited all sources of information in accordance with the Guideline for adhering to ethical principles when elaborating an academic final thesis. I acknowledge that my thesis is subject to the rights and obligations stip- ulated by the Act No. 121/2000 Coll., the Copyright Act, as amended, in particular that the Czech Technical University in Prague has the right to con- clude a license agreement on the utilization of this thesis as school work under the provisions of Article 60(1) of the Act. In Prague on May 14, 2018 . Czech Technical University in Prague Faculty of Information Technology © 2018 David Mokoˇs.All rights reserved. This thesis is school work as defined by Copyright Act of the Czech Republic. It has been submitted at Czech Technical University in Prague, Faculty of Information Technology. The thesis is protected by the Copyright Act and its usage without author’s permission is prohibited (with exceptions defined by the Copyright Act). Citation of this thesis Mokoˇs,David. Automated Acceptance Testing on macOS – A survey study focused on Avast Passwords for Mac. Bachelor’s thesis. Czech Technical University in Prague, Faculty of Information Technology, 2018. Abstract Software development requires adequate methods for quality assurance. One of those methods is acceptance testing. This thesis introduces the automated acceptance testing and its need in the software development process along with its common pitfalls. It summarizes the existing tools and technologies that could be used for automating acceptance tests on macOS as well as best practices for writing maintainable automated acceptance test suites. The gathered knowledge from the literature review is put into practice through the implementation of an automated acceptance test suite for Avast Passwords for Mac, a macOS application developed by Avast. Keywords acceptance testing, test automation, GUI (Graphical User In- terface) driving on macOS, automated tests maintainability vii Abstrakt Ve v´yvoji softwaru je pro zajiˇstˇen´ıkvality nezbytn´epouˇzit´ıvhodn´ych metod. Mezi tyto metody patˇr´ıakceptaˇcn´ıtestov´an´ı. Tato pr´acepopisuje v´yhody akceptaˇcn´ıhotestov´an´ıv procesu v´yvoje softwaru a z´aroveˇnupozorˇnujena jeho ˇcast´eprobl´emy. Pr´acetak´eshrnuje dostupn´en´astroje a technologie, kter´ejsou vhodn´epro automatizaci akceptaˇcn´ıch test˚una macOS, a z´aroveˇn popisuje osvˇedˇcen´ezp˚usoby pro v´yvoj snadno udrˇzovateln´ych automatick´ych akceptaˇcn´ıch test˚u. Znalosti z´ıskan´ez liter´arn´ıreˇserˇsejsou pˇrevedeny do praxe skrze implementaci sady automatick´ych akceptaˇcn´ıch test˚upro Avast Pass- words for Mac, aplikaci pro macOS od spoleˇcnostiAvast. Kl´ıˇcov´a slova akceptaˇcn´ıtestov´an´ı,automatizace test˚u,ovl´adan´ıGUI (Graph- ical User Interface) na macOS, udrˇzovatelnost automatick´ych test˚u ix Contents Introduction 1 Goals of the Thesis . 2 1 State of the Art 3 1.1 Software Testing Fundamentals . 3 1.1.1 Business Facing Tests that Support the Development Process . 3 Functional Acceptance tests . 3 1.1.2 Technology Facing Tests that Support the Development Process . 3 Unit Tests . 4 Component Tests . 4 Deployment Tests . 4 1.1.3 Business Facing Tests that Critique the Project . 5 Exploratory tests . 5 Showcases . 5 Usability Tests . 5 1.1.4 Technology Facing Tests that Critique the Project . 6 Nonfunctional Acceptance tests . 6 1.1.5 Regression Testing . 6 1.2 Acceptance Testing . 6 1.2.1 Overview . 6 1.2.2 Automation . 7 1.2.3 Implementation Patterns . 8 Behavior driven development . 8 Application Driver Layer . 10 Page Object Pattern . 11 Layered Architecture . 11 1.3 Acceptance Testing on macOS . 11 xi 1.3.1 The macOS operating system . 11 Accessibility . 11 1.3.2 GUI Driving Tools . 12 Xcode User Interface Testing . 12 AppleScript . 13 Javascript for Automation . 14 Appium For Mac . 14 2 Analysis 17 2.1 The Application Under Test – Avast Passwords for Mac . 17 2.1.1 Introduction . 17 2.1.2 Features . 17 2.1.3 Architecture . 19 2.2 Automated Acceptance Test Suite . 21 2.2.1 Requirements . 21 2.2.2 Architecture . 22 2.3 Tools Selection . 23 2.3.1 Acceptance Criteria . 23 2.3.2 Test Implementation Layer . 24 2.3.3 Application Driver Layer . 24 Xcode UI Testing . 24 AppleScript . 25 JavaScript for Automation . 27 Appium for Mac . 28 Tool Selection . 31 3 Design & Realization 33 3.1 Test Suite Implementation . 33 3.1.1 Acceptance Criteria . 33 3.1.2 Test Implementation Layer . 35 3.1.3 Application Driver Layer . 37 Interacting with the UI Elements . 37 Page Object Pattern . 40 3.2 Test Suite Execution . 43 3.3 Challenges . 45 3.3.1 Locating UI Elements . 45 3.3.2 Interacting with the Environment . 45 3.3.3 Performing Privileged Operations . 46 3.3.4 Integration with the Continuous Integration Pipeline . 47 3.4 Implementation State . 48 Conclusion 51 Further work . 51 xii Bibliography 53 A Acronyms 59 B Contents of enclosed SD card 61 C Diagrams 63 xiii List of Figures 1.1 Brian Marick’s four quadrant diagram describing types of tests used in software projects . 4 1.2 The three layers of maintainable automated acceptance tests . 12 2.1 The credit card list screen of Avast Passwords for Mac . 18 2.2 The different components of Avast Passwords for Mac as well as the main interactions that occur between them . 20 2.3 The three layers of maintainable automated acceptance tests along with the tools that can be used . 23 2.4 Usage of the Accessibility Inspector tool to determine the accessi- bility identifiers of the UI elements . 30 3.1 Folder structure of the Gherkin .feature files and corresponding steps implementation . 36 3.2 The domain model of the Application Driver Layer . 38 3.3 Class diagram of the Element class and one of its subclasses– TextFieldElement ........................... 39 3.4 Class diagram of the login list screen and login detail screen (page objects) of the application . 42 3.5 The login list screen and login detail screen of Avast Passwords for Mac ................................... 42 3.6 The integration of the test suite into the Continuous Integration pipeline . 48 C.1 Class diagram of Element class and all of its subclasses . 63 C.2 All implemented screens (Page Objects) of the Avast Passwords for Mac application . 64 xv List of Tables 2.1 Selection of the tool for the application driver layer of Avast Pass- words for Mac ............................. 31 xvii Introduction A key challenge faced by software industry is quality. To provide the highest quality software, testing is an essential part of any development process [1]. Acceptance testing, an essential part of software testing, is typically conducted at the end of the release cycle to guarantee that all functional requirements of the software are met [2]. Acceptance tests are usually conducted manually by QA engineers each time the software is released. This process is very tedious, prone to human error, time consuming and thus expensive for the developer team. This does not fit with the paradigm of agile development, that relies on quick feedback and short development cycles [3]. To ease and speed up this process, the automation of acceptance tests may be seen as a promising initiative. It can significantly reduce the time QA engineers spend on repetitive tasks and also makes the feedback loop shorter, which helps to find the defects sooner, when they are less expensive to fix [2, 4]. However even acceptance testing automation has its pitfalls. First, unless carefully designed, automated acceptance test suites can be very expensive to develop and maintain. Tests may become too brittle and overly coupled with the implementation details of the application under test, or in the worst scenario, tests may become unreliable and report false positive results [5].