MASARYK UNIVERSITY FACULTY OF INFORMATICS Corner cases and possible improvements of automated testing MASTER'S THESIS Petra Mikova Brno, Spring 2020 MASARYK UNIVERSITY FACULTY OF INFORMATICS Corner cases and possible improvements of automated testing MASTER'S THESIS Petra Mikova Brno, Spring 2020 This is where a copy of the official signed thesis assignment and a copy of the Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Petra Mikova Advisor: RNDr. Adam Rambousek, Ph.D. i Acknowledgements I would like to thank my advisors Mgr. Jiri Vanek and RNDr. Adam Rambousek, Ph.D. for their help, valuable advice and support, and the members of OpenJDK QE team in Red Hat for their expertise and input. ii Abstract The aim of this thesis is analysis of current state of OpenJDK QE in Red Hat, identification of biggest issues and their resolution. That is achieved both by outlining new and reusing existing tools and software. Most of the problematic corner cases mentioned in this thesis are not product specific, making the solutions applicable for other QE teams. The last part of the thesis is dedicated to improving automated results processing and reporting. iii Keywords OpenJDK, Jenkins, quality engineering, test automation, Java iv Contents Introduction 1 1 Automation in testing 3 2 OpenJDK 4 2.1 What is OpenJDK 4 2.2 OpenJDK and Red Hat 4 2.3 OpenJDK QE 5 3 Test automation tooling 6 3.1 Jenkins 6 3.1.1 Architecture 6 3.2 Beater 7 3.2.1 Beaker usage 8 3.2.2 Beaker machine pool 9 3.2.3 Downsides 10 4 Major issues of OpenJDK QE 11 4.1 Mixing results of various projects 11 4.1.1 Sorting and comparing results 12 4.1.2 Separating jobs 14 4.2 Incomplete matrix of current jobs 14 4.2.1 Problems caused by manual cloning 15 4.3 Template job generation 16 5 O-tool 18 5.1 The O-tool idea 18 5.2 Build and test matrix 19 5.2.1 Matrix declaration 20 5.3 Job generation 25 5.4 Implementation 29 6 Common issues and handling 30 6.1 Hanging Jenkins jobs 30 6.1.1 Aborting of long-running jobs 32 6.1.2 Implied consequences 34 v 6.2 Product download issues 34 6.2.1 Inaccessible remote server 34 6.2.2 Insufficient executor storage 35 6.3 Known test failures 41 6.3.1 Old tests with compatibility issues 41 6.3.2 Testcase timeouts 42 6.3.3 Dependency issues 44 6.3.4 False negatives of unmaintained test suites ... 44 7 Test result evaluation and processing 48 7.1 New available result reporting 48 7.2 Result comparison 49 7.2.1 Narrowing the case 49 7.2.2 Results comparing 50 7.2.3 Jenkins report plugin 52 7.3 Test fail processing possible improvements 55 7.3.1 Type of the fail recording 55 7.3.2 Broader history discovery 55 7.3.3 Automatic reproducer extraction 55 7.3.4 Test author search 56 7.3.5 Blame a commit 56 8 Conclusion 60 Bibliography 61 vi List of Tables vii List of Figures 3.1 Master node with several additional machines, that are used as slave nodes, split into several executor slots. 7 3.2 Artificial node that gathers all Jenkins jobs that are running in Beaker. 9 4.1 Different builds originating from different forests mixed in one test job. 13 5.1 JDK project representation and configuration. 27 5.2 JDK test project representation and configuration. 28 6.1 A stable test suite build trend 31 6.2 Unstable test suite with manually shut down hanging jobs. 32 6.3 Current build step that starts the executor and runs the test suite. 33 6.4 Build Timeout plugin version of execute shell. 33 6.5 Example configuration for node monitoring 37 7.1 Results of jtreg test suite displayed in the Jenkins Report JCK plugin view 53 7.2 Comparison of previous and current run of jtreg test suite shows a new failing test. 54 Introduction These days, information technology and software are an integral part of our lives. It is in the cars we drive, the planes that fly above our heads, the phones in our hand and in the banks that control flow of the money Technology is everywhere and it is so intertwined with our lives, that the costs of a single failure can be unimaginable. Failures don't have to always cost us only money; it can cost us lives, knowledge or it can cause instability of our society. That makes technology reliability a key asset. Quality engineering is an inseparable part of software develop• ment. Each day, thousands of quality engineers and testers find mis• takes and errors in the software that could cause, if released, serious issues to people, property and break customer trust. They find new ways, how the software can be misused, broken or exploited, but also check that already found issues do not appear again. Programming languages and their libraries are like any other soft• ware product. However, their impact can be much wider than most single purpose applications, because the developers use it to create their software. A single bug can affect hundreds of applications and expose unexpected vulnerabilities. The Java platform has a special place in the software world; it is one of the most favored programming languages, it is heavily used in financial services, data centres and other important industries, and it is the official language for Android development, that currently dominates the world of mobile operating systems. That is also because of the fact that Java is considered a secure and reliable platform. The expectations are high from the community and the developers working on the implementations are working hard to make sure they are fulfilled. There are many implementations of Java platform. The open source implementation is called OpenJDK and it has a wide range of contributors from many companies like Oracle, Red Hat, IBM, Amazon, Azul and more. The goal of this thesis is to analyze the current state of automated testing of OpenJDK in the Red Hat OpenJDK QE team, to identify the biggest issues that are being struggled with and to outline possible solutions to these problems. The problems are not usually product or 1 company specific, so the designed solutions are applicable to other software and can be used by other engineers for similar purposes. First, we will put the thesis into context - explain why it is important to introduce automated solutions and briefly acquaint the reader with OpenJDK. Then we will introduce the most important tools currently used by the team, that are either directly related to the issues or a key part of the testing process. Then we will analyze the major problems in the testing process and test suites and either try to find an existing solution for these problems, or propose some. At last, we will outline possible changes in the results processing that could have significantly benefiting impact on the workflow, as it seems to be one of the bigger bottlenecks in the testing process. Those changes could be both implemented as a result processing tool, or be used to extend existing programs created by the members of the team. 2 1 Automation in testing "A major problem of the software industry is its inability to develop error- free software." [1] The quote above pretty much sums up the current situation in software development. Today's software products are becoming increasingly complex, but the need for reliability, stability, performance and functionality persists. That makes quality engineer• ing and software testing an integral part of software development, because with growing code base and functionality, it gets harder to ensure the quality of the product. The increasing requirements for testing are then reflected in higher need of manpower in quality assurance and engineering. Even though manual testing has its perks and is useful especially for cases that are difficult to automate, the automatic testing is usually more cost- effective, less error prone and more accurate, especially in the regres• sion testing 1, because these test suites tend to grow with every bug found. These days, automation is a key factor in quality engineering. Once created, the automated tests are easy to repeat and reproduce, fast in execution and if configured correctly, much more reliable. It enables us to maximize usage of resources and in general, it increases produc• tivity, especially in the time, when manpower is much more expensive than machine resources. However, the test automation is not just about writing and main• taining the tests. It is also about the scripts that execute them, mecha• nisms that record and collect the results and also the interpretation of the results. In general we can say, the more tests and results we have, the more we want and need to automate the process. 1. Regression testing is repetitive re-rerunning of tests, that ensures that the devel• oped and tested software performs the same way after a change. 3 2 OpenJDK 2.1 What is OpenJDK OpenJDK is an open source implementation of Java Standard Edition. It is licensed under the GNU General Public License v2 with classpath exception[2].