Regression Testing of Transcompiled Cross-Platform Applications
Total Page:16
File Type:pdf, Size:1020Kb
Regression Testing of Transcompiled Cross-Platform Applications Dissertation der Mathematisch- und Naturwissenschaftlichen Fakult¨at der Eberhard Karls Universit¨atT¨ubingen zur Erlangung des Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) vorgelegt von Dipl.-Inform. Matthias Hirzel aus M¨unsingen T¨ubingen 2018 Gedruckt mit Genehmigung der Mathematisch-Naturwissenschaftlichen Fakult¨at der Eberhard Karls Universit¨atT¨ubingen. Tag der m¨undlichen Qualifikation: 20.09.2018 Dekan: Prof. Dr. Wolfgang Rosenstiel 1. Berichterstatter: Prof. Dr. Herbert Klaeren 2. Berichterstatter: Prof. Dr. Torsten Grust Abstract Regression testing is a well-accepted and widely used way to ensure that already existing functionality still works as expected when code has been modified in order to realize new features or to resolve known bugs. To this end, devel- opers re-execute previously created tests and check whether these tests still pass. However, depending on the application, regression testing might be very time consuming. This already applies when testing small units, but plays a more and more decisive role when integrated functions and especially end-to- end tests (such as UI or web tests) have to be re-executed. In the worst case, executing all the tests takes several days or even weeks. Consequently, it takes far too much time to get feedback about test failures. In order to solve this problem, researches have presented many different techniques. Originally, these techniques have been tailor-made for standard desktop applications. With the dawn of web applications, it has turned out that the approaches available so far are not directly applicable to web applications. So more and more researches have attended to the special challenges to realize regression testing techniques for this new kind of application. In both standard desktop and web applica- tions, especially regression test selection and test prioritization have shown to be a good strategy to reduce the testing effort. Today, another kind of applications becomes more and more important: Transcompiled cross-platform applications. The basic idea is to create an ap- plication in a source programming language A and to compile it automatically with a special transcompiler into another target programming language B. This way of developing new software has several advantages in different ar- eas: In web applications for example, it is easier to write the software in a specific source programming language that has sophisticated debug support. In mobile applications, it is a means of cost reduction. The developers write an application only once. By transcompiling the code, they can deliver many different platforms such as Android, iOS, and Windows Phone. Apart from that, with transcompilation, it is basically possible to create both a desktop application and a web application without writing the code twice. Neverthe- less, this new way of developing code creates several challenges in regression testing. Once more, existing approaches for desktop or web applications are not directly applicable. In this thesis, we investigate three main problems that affect regression test- ing in the special context of transcompiled cross-platform applications. First, i ii we focus on the test effort reduction problem that comprises three questions: (a) Which tests should run at all, (b) how to execute tests in an efficient, memory- and time-saving way, and (c) which execution order would be suitable if many tests should be re-executed. A main challenge is the nature of UI/web tests as they execute many different functions in different application states rather than isolated, small functions as unit tests do. Even small code changes might affect lots of tests that are all intended to check different functionality. In addi- tion, the distinction between source and target programming language as well as the dependencies in between makes it difficult to determine a reduced set of UI/web test cases that should be re-executed due to code changes. The second problem addresses fault localization in transcompiled cross- platform applications. UI/web tests execute code in the target programming language. If a test fails, the decisive question is which parts in the code of the source programming language are responsible for the failure. Due to the automatic transcompilation and additional code optimization and obfuscation, it is difficult to identify the reasons. As a solution to these problems, we present an approach that improves and extends an already existing regression test selection technique. Our approach consists of several steps. First of all, it analyzes which code changes have been made in the source programming language. Based on this analysis, we select all the UI/web tests that run corresponding pieces of code in the target programming language. As a prerequisite, we have to close the gap between the source and the target programming language. To overcome this issue, we introduce and investigate two special code instrumentation techniques. They are universally applicable in different transcompilers and address the special challenges in transcompiled cross-platform applications. Both instrumentation techniques also take into account the problem to identify precisely which code change(s) in the source programming language might be responsible for a failure when running a test on the code of the target language. In order to make our approach more effective, we explore different settings. Our aim is to reduce the overhead for the analysis and to keep the memory consumption as low as possible. To this end, we propose and evaluate several heuristics that decide individually how fine-grained the analysis of the source code should be. Besides, we apply lookaheads and two search algorithms to detect more possible bugs in a single analysis. In order to decide which execution order is suitable if many tests have to be re-executed, we combine our regression test selection technique with one of six novel prioritization techniques. The overhead of the extra test case prioritization is very low. At the same time, the prioritization gives a clear test execution order. Thus, the most important tests that have the biggest chance to reveal faults run first. This even enables developers to reduce the set of tests on their own if the test selection chooses many tests for re-execution and if there are constraints (e.g. execution time) that prevent to re-execute all the tests. Moreover, the combination of test selection and prioritization can be used to realize continuous integration even for UI/web tests. iii Finally, the third challenge arises from the coverage identification problem. In order to ensure that the software application works as expected, it is very important to have many tests that cover all possible use cases. Usually, this is checked by applying different code coverage metrics. However, existing code coverage tools cannot be applied in transcompiled cross-platform applications. They fail to determine which parts of the code in the source programming language are covered by UI/web tests. We introduce an extension of our in- strumentation technique that is applicable for both unit tests, integration tests, and for UI/web tests in particular. It supports standard desktop applications, but more importantly, it calculates multiple coverage measures for transcom- piled cross-platform applications. Zusammenfassung Regressionstests sind eine anerkannte und weit verbreitete Methode um die kor- rekte Funktionsweise bereits existierender Features in Software-Anwendungen sicherzustellen, nachdem neue Funktionalit¨athinzugef¨ugtoder bekannte Fehler behoben wurden. Zu diesem Zweck f¨uhrenEntwickler bereits existierende Tests erneut aus und pr¨ufen,ob sie noch immer fehlerfrei durchlaufen. Je nach Art der Software-Anwendung k¨onnenRegressionstests jedoch sehr zeitintensiv sein. Dies kann bereits auf den Test kleiner Komponenten (englisch: Units) zutreffen, spielt aber eine immer gr¨oßereRolle wenn zusammengesetzte Funktionen oder End-to-end Tests { wie (Benutzer-) Oberfl¨achentests oder Webtests { erneut ausgef¨uhrtwerden m¨ussen. Im schlimmsten Fall dauert die Ausf¨uhrungaller Tests mehrere Tage oder sogar Wochen. Folglich dauert es viel zu lange bis man Feedback zu fehlgeschlagen Tests erh¨alt. Um dieses Problem zu l¨osen haben Forscher viele verschiedene Techniken vorgestellt. Urspr¨unglich waren diese auf gew¨ohnliche Desktop-Anwendungen zugeschnitten. Mit dem Aufkom- men von Web-Anwendungen hat sich dann herausgestellt, dass sich die bislang verf¨ugbarenAns¨atzedarauf nicht direkt ¨ubertragen lassen. Aus diesem Grund haben sich immer mehr Forscher den speziellen Herausforderungen gewidmet, die sich aus der Realisierung von Regressionstesttechniken f¨urdiese neue Art von Anwendung ergeben. Sowohl in Desktop- als auch in Web-Anwendungen haben sich speziell Regressionstestselektion und Testpriorisierung als eine gute Strategie erwiesen, um den Testaufwand zu reduzieren. Heutzutage gewinnt eine andere Art von Anwendung immer gr¨oßereBedeu- tung: Transkompilierte plattform¨ubergreifende Anwendungen. Der Grundge- danke dabei ist, eine Anwendung in einer Ausgangsprogrammiersprache A zu erstellen und mit Hilfe eines speziellen Transkompilers automatisch in eine an- dere Zielprogrammiersprache B zu kompilieren. Diese Art der Entwicklung neuer Software hat mehrere Vorteile in unterschiedlichen Bereichen: Beispiels- weise ist es in Web-Anwendungen einfacher, die Software in einer speziellen Ausgangsprogrammiersprache zu schreiben die