An Exploratory Assessment Study on an Open-Source Web Application

POLITECNICO DI TORINO Master’s degree course in Computer Engineering Master’s Degree Thesis Test Fragility: An exploratory assessment study on an Open-Source Web Application Supervisors Candidate Assistant Prof. Luca Ardito Huang Shijie Research Assistant. Riccardo Coppola Student Number: s233098 Prof. Morisio Maurizio Academic Year 2019-2020 This work is subject to the Creative Commons Licence Abstract Context: With the explosive growth of web applications in the last two decades, web application testing is an integral part of the web application development process. Frequent web application testing can minimize the chance of bugs ruining the customer experience and it also gives you a bet- ter overall idea about how your app performs, what its strengths are, and where the weak points are hidden. Automating web application testing is a highly automated process for testing web applications, and manual testing is not suitable for critical and complex applications in terms of both human resources and time, so automation testing has been introduced to overcome manual testing problems. Automation is a must in the interests of effective- ness and efficiency. Goal: The objective of this thesis is to understand the automated web application testing techniques for web applications, and analyze the main fragility (i.e., need for maintenance of existing cases) causes of web application tests. This evaluation has been performed by means of an exploratory experiment. The test fragility results are evaluated for two different automated testing technologies. Method: This article makes an exploratory assessment of web applications and their related vulnerability causes. Firstly a small test suite (five test cases) has been analyzed - four automated web application testing tools (SeleniumIDE-is open-source without writing code and allows for record and replay; Selenium-is an open-source framework and allows for writing code or scripts; TestComplete-automation testing tool with a hybrid object and visual recognition engine, and has script or scriptless flexibility; EyeAutomate- visual GUI Testing with intelligent image recognition technique) are applied - for Polito web application. Then introduced the concept of vulnerability in the computer field. Next, Selenium WebDriver and eyeAutomate do testing on the Dolibarr web application - The iterative process was used under 19 test cases in a test suite, then the cause of all test corruption is recorded. Moreover, the experimental results record (compute the percentage of modified tests under a test suite of one version and compute the percentage of each modified test case among the 8 versions ) of Dolibarr under the Sele- nium/EyeAutomate test are summarized. Results: Throughout the exploratory research, by comparing the modified LOC/instruction, the advantages and disadvantages of the two test tools for Dolibarr were compared and analyzed. The data gathered on the application Dolibarr in the test report largely show that applying the Selenium i automated testing tool is much more stable than EyeAutomate. Moreover, the summary of the main fragility cause of Dolibarr under Selenium testing gives that the most frequent causes of fragility were changes in the DOM Tree. Conclusion: According to the results of these experiments, it is evident that fragility happens frequently on a typical open-source web application project, and that the changes in the DOM structure or graphic changes may invalidate many test cases during the evolution of a web app. These results can be used by developers and practitioners to understand which modifica- tions can lead to much-required maintenance effort in their test cases. ii Acknowledgements I would like to thank Assistant Prof. Luca Ardito Research Assistant. Ric- cardo Coppola and Prof. Morisio Maurizio, who advised the project. It makes me discover the universe of web application testing research and dedi- cates me all the time I needed and answers all of my questions and concerns. I am extremely thankful to Research Assistant. Riccardo Coppola, who man- aged and administrated my research activities. I am extremely thankful to Assistant Prof. Luca Ardito and Prof. Morisio Maurizio, for supporting me well throughout my research work. I would like to thank them for guiding me throughout this journey, and I am very grateful to have received this opportunity. Their advice, discussion, and effective comments were always a source of motivation. This work would not have been possible without them. I would like to thank my parents for encouraging me, motivating me and believe in me, this thesis would not have been possible without them. I would not have reached this stage without the support, love, and prayers of my family and friends. iii Abbreviations CSRF Cross-Site Request Forgery API Application Program Interface GUI Graphical User Interface BDD Behavior-Driven Development AI Artificial Intelligence HTML Hypertext Markup Language CSS Cascading Style Sheets QA Quality Assurance URL Uniform Resource Locator DLL Dynamic-Link Library XML Extensible Markup Language ERP Enterprise Resource Planning CRM Customer Relationship Management DOM Document Object Model LOC Lines Of Code iv Contents Abstract.................................i Acknowledgements........................... iii Abbreviations.............................. iv List of Tables vii List of Figures viii 1 Introduction 1 2 Background 5 2.1 Web applications.........................5 2.2 Testing...............................6 2.2.1 Test plan.........................6 2.2.2 Test case..........................7 2.2.3 Test script.........................7 2.2.4 Test coverage.......................7 2.2.5 Black-Box Testing....................8 2.2.6 White-Box Testing....................8 2.2.7 Grey-Box Testing.....................8 2.2.8 Manual testing......................9 2.2.9 Automation testing....................9 2.3 Exploration of web-based testing techniques.......... 10 2.4 Selenium IDE........................... 13 2.4.1 SetUp........................... 13 2.4.2 How it works....................... 13 2.4.3 Example test suite with SeleniumIDE.......... 14 2.5 Selenium.............................. 17 2.5.1 SetUp........................... 18 2.5.2 How it works....................... 19 v 2.5.3 Example test suite with Selenium............ 20 2.6 TestComplete........................... 24 2.6.1 SetUp........................... 25 2.6.2 How it works....................... 26 2.6.3 Example test suite with TestComplete......... 29 2.7 EyeAutomate........................... 33 2.7.1 SetUp........................... 33 2.7.2 How it works....................... 34 2.7.3 Example test suite with EyeAutomate......... 35 3 Empirical Evaluation 43 3.1 Fragility concept......................... 43 3.2 Object of study: Dolibarr.................... 46 3.3 Experiment............................ 47 3.4 Test case definition........................ 48 4 Results 51 4.1 Experiment results........................ 51 4.1.1 Result Under Selenium WebDriver........... 52 4.1.2 Summary of Dolibarr Change Types under Selenium. 52 4.1.3 Report on Dolibarr Test Cases under Selenium..... 59 4.1.4 Result Under EyeAutomate............... 64 4.1.5 Summary of Dolibarr Change Types under EyeAutomate 64 4.1.6 Report on Dolibarr Test Cases under EyeAutomate.. 71 4.1.7 Comparison between Selenium and EyeAutomate... 76 4.2 Fragility of Dolibarr........................ 79 5 Conclusion and Future Work 83 Bibliography 85 vi List of Tables 2.1 Description of Usage Scenarios.................. 12 3.1 Description of Dolibarr Usage Scenarios............. 49 4.1 Report on V3 of Dolibarr, with Selenium........... 59 4.2 Report on V4 of Dolibarr, with Selenium........... 60 4.3 Report on V5 of Dolibarr, with Selenium........... 60 4.4 Report on V6 of Dolibarr, with Selenium........... 61 4.5 Report on V7 of Dolibarr, with Selenium........... 61 4.6 Report on V8 of Dolibarr, with Selenium........... 62 4.7 Report on V9 of Dolibarr, with Selenium........... 63 4.8 Report on V10 of Dolibarr, with Selenium........... 63 4.9 Report on V3 of Dolibarr, with EyeAutomate......... 71 4.10 Report on V4 of Dolibarr, with EyeAutomate......... 72 4.11 Report on V5 of Dolibarr, with EyeAutomate......... 73 4.12 Report on V6 of Dolibarr, with EyeAutomate......... 73 4.13 Report on V7 of Dolibarr, with EyeAutomate......... 74 4.14 Report on V8 of Dolibarr, with EyeAutomate......... 74 4.15 Report on V9 of Dolibarr, with EyeAutomate......... 75 4.16 Report on V10 of Dolibarr, with EyeAutomate........ 76 4.17 Count changed LOCs with Selenium for Dolibarr....... 78 4.18 Count changed instructions with EyeAutomate for Dolibarr.. 79 4.19 Main causes of fragilities in broken test cases.......... 81 vii List of Figures 2.1 Login................................ 11 2.2 Drop-Down List.......................... 11 2.3 Operate Video........................... 11 2.4 Print Certificate.......................... 12 2.5 Write Email............................ 12 2.6 Login with Selenium IDE..................... 14 2.7 Drop-down List with Selenium IDE............... 15 2.8 Operate Video with Selenium IDE................ 15 2.9 Print Certificate with Selenium IDE............... 16 2.10 Write Email1 with Selenium IDE................ 16 2.11 Write Email2 with Selenium IDE................ 17 2.12 Login with Selenium....................... 21 2.13 Drop-down List with Selenium.................. 21 2.14 Operate Video with Selenium.................. 22 2.15 Print Certificate with Selenium................

Load more