CORE Metadata, citation and similar papers at core.ac.uk Provided by DigitalCommons@USU Utah State University DigitalCommons@USU All Graduate Theses and Dissertations Graduate Studies 5-2012 Test Case Generation Using Combinatorial Based Coverage for Rich Web Applications Chad Maughan Utah State University Follow this and additional works at: https://digitalcommons.usu.edu/etd Part of the Computer Sciences Commons Recommended Citation Maughan, Chad, "Test Case Generation Using Combinatorial Based Coverage for Rich Web Applications" (2012). All Graduate Theses and Dissertations. 1405. https://digitalcommons.usu.edu/etd/1405 This Thesis is brought to you for free and open access by the Graduate Studies at DigitalCommons@USU. It has been accepted for inclusion in All Graduate Theses and Dissertations by an authorized administrator of DigitalCommons@USU. For more information, please contact [email protected]. TEST CASE GENERATION USING COMBINATORIAL BASED COVERAGE FOR RICH WEB APPLICATIONS by Chad Maughan A thesis submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in Computer Science Approved: Dr. Renee Bryce Dr. Daniel Bryce Major Professor Committee Member Dr. Daniel Watson Dr. Mark R. McLellan Committee Member Vice President for Research and Dean of the School of Graduate Studies UTAH STATE UNIVERSITY Logan, Utah 2012 ii Copyright c Chad Maughan 2012 All Rights Reserved iii ABSTRACT Test Case Generation Using Combinatorial Based Coverage for Rich Web Applications by Chad Maughan, Master of Science Utah State University, 2012 Major Professor: Dr. Renee Bryce Department: Computer Science Web applications are increasingly moving business and processing logic from the server to the browser. Traditional, multiple-page request/response applications are quickly being replaced by single-page applications where complex application logic is downloaded on the initial page load and data is then subsequently fetched asynchronously via the browser's native XMLHttpRequest (XHR) object. These new generation web applications are called Rich Web Applications (RWA). Frameworks such as the Google Web Toolkit (GWT), and JavaScript model-view-controller (MVC) frameworks such as Backbone.js are facilitating this move. With this migration, testing frameworks need to follow the logic by moving analysis and test generation from the server to the client. One problem hindering the movement of testing in this domain is the adoption of semantic URLs. This paper introduces a novel approach to systematically identify variables in semantic URLs and use them as part of the test generation process. Using a sample RWA seeded with various JavaScript faults, I demonstrate in this thesis, as an empirical study, that combinatorial testing algorithms and reduction strategies also apply to new RWAs. (40 pages) iv PUBLIC ABSTRACT CHAD M. MAUGHAN Rich Web Applications (RWA) that are data driven and feature responsive user inter- faces are rapidly growing in popularity. Popular sites such as Twitter, Pandora, and Angry Birds (browser version) are all examples of popular RWAs. These RWAs are more complex and are developed differently than many previous web sites. More of the processing power needed to run these applications is performed on the client machine, not the server. Due to this development strategy, testing tools need new techniques to identify web application variables, capture errors, and identify problems. In this thesis, I introduce novel techniques to identify variables in RWA semantic URLs and automatically generate tests for RWAs using a form of testing called combinatorial testing. v CONTENTS Page ABSTRACT :::::::::::::::::::::::::::::::::::::::::::::::::::::: iii PUBLIC ABSTRACT ::::::::::::::::::::::::::::::::::::::::::::::: v LIST OF TABLES ::::::::::::::::::::::::::::::::::::::::::::::::: vii LIST OF FIGURES :::::::::::::::::::::::::::::::::::::::::::::::: viii CHAPTER 1 INTRODUCTION ::::::::::::::::::::::::::::::::::::::::::::::: 1 2 BACKGROUND :::::::::::::::::::::::::::::::::::::::::::::::: 5 2.1 JavaScript Background..............................5 2.2 Traditional URL Formats............................6 2.3 Semantic URL Formats.............................6 2.4 Variable Detection Algorithm for Semantic URLs...............7 2.5 Algorithm Description..............................8 3 EXPERIMENTS :::::::::::::::::::::::::::::::::::::::::::::::: 11 3.1 Research Questions................................ 11 3.2 Sample Application................................ 11 3.3 JavaScript Faults................................. 12 3.4 Testing Technologies............................... 16 3.5 Navigation Graph................................ 17 3.6 Abstract URLs.................................. 18 3.7 Capturing Errors................................. 20 3.8 Combinatorial Testing.............................. 21 4 RESULTS ::::::::::::::::::::::::::::::::::::::::::::::::::::: 23 4.1 Size Impact of Abstract URL Test Suite.................... 23 4.2 Effectiveness of Abstract URL Test Suite................... 24 4.3 Size Impact of Combinatorial Coverage..................... 24 4.4 Effectiveness of Combinatorial Coverage.................... 26 5 CONCLUSIONS :::::::::::::::::::::::::::::::::::::::::::::::: 27 REFERENCES ::::::::::::::::::::::::::::::::::::::::::::::::::: 30 vi LIST OF TABLES Table Page 3.1 Sample Rich Web Application Information................... 12 3.2 Sample Rich Web Application Seeded Errors.................. 17 4.1 Size of Test Suites by Strategy.......................... 24 4.2 Effectiveness of Single Coverage With Abstract URL Reduction....... 24 4.3 Size of Test Suites by Strategy.......................... 24 4.4 Sample Combinatorial Tests........................... 25 4.5 Effectiveness of Combinatorial Coverage..................... 26 5.1 Environment Variables.............................. 28 vii LIST OF FIGURES Figure Page 2.1 Algorithm for variable identification of semantic URLs............9 2.2 A visual representation of branching complexity in an application structure. 10 3.1 Sample rich web application screenshot..................... 13 3.2 Depth first crawling of a rich web application................. 19 3.3 An example JavaScript error in the Firebug console.............. 21 3.4 Browser exception catching........................... 22 4.1 Hierarchical variable HTML form example................... 25 CHAPTER 1 INTRODUCTION After the introduction of HTML5 in 2011, and its compelling set of new APIs [13], Rich Web Applications (RWA) have rapidly gained in popularity. RWAs differ from the more recent Rich Internet Applications (RIA) in that they don't require proprietary browser plug-ins. They also differ from traditional request/response web applications in that they typically use a single HTML page and update the view on the client-side instead of building many HTML pages on the server-side. The concept of a \rich" in-browser application experience was introduced years earlier with technologies such as Adobe's Flex, Microsoft's Silverlight, and Oracle's JavaFX [17]. Each of these technologies uses proprietary browser add-ons to provide a responsive user interaction and are categorized as Rich Internet Applications (RIA). These proprietary add- ons provide additional functionalities to the web that were missing before HTML5, but also decreased its openness [23]. Rich Web Applications (RWA) differ from RIAs in that RWAs focus on creating a specific browser independent application without the reliance on any proprietary, client- side browser plug-ins. Indeed, Ian Hickson, one of the W3Cs HTML5 editors, stated \one of our goals is to move the Web away from proprietary technologies" [23]. Additionally, there are key differences between traditional web applications and RWAs [20]. First, a traditional web application consists of many static HTML pages or scripts (e.g., JSP or PHP) to render different portions of the site [8, 25]. A user of a traditional web application is required to retrieve a full page for each interaction with the application. Conversely, a RWA typically consists of a single HTML page that acts as an entry point. This single page entry point contains complex application logic implemented in bundled JavaScript or a collection of JavaScript script files. This application logic is downloaded 2 on the first HTML entry point page load and is then run on the browser. A RWA then updates its views by manipulating the Document Object Model (DOM) using HTML com- posites downloaded with that first page load. It also retrieves data associated with these components by asynchronously making Ajax requests (i.e., XmlHttpRequest or XHR) to the server. The server then responds with data in a JSON or XML format. Along with this architecture change, RWAs have adopted a new URL structure that relies heavily on sometimes complex URL fragments to maintain history and state within the application. These URLs are sometimes called semantic or clean URLs. These semantic URLs are much easier for the end-user to understand, describe, and infer site application structure. While easier for user understanding, semantic URLs pose a significantly more difficult challenge in systematically identifying variables for subsequent testing. As RWAs adopt the HTML5 strategy of moving away from proprietary technologies, there is also a movement to follow the HTTP protocol as it was designed| a \generic, stateless protocol" [2]. This leads to many RWAs being stateless. A stateless web applica- tion has many benefits, including performance,