Using Property-Based Testing, Weighted Grammar-Based Generators and a Consensus Oracle to Test Browser Rendering Engines and To
Total Page:16
File Type:pdf, Size:1020Kb
USING PROPERTY-BASED TESTING, WEIGHTED GRAMMAR-BASED GENERATORS AND A CONSENSUS ORACLE TO TEST BROWSER RENDERING ENGINES AND TO REPRODUCE MINIMIZED VERSIONS OF EXISTING TEST CASES by JOEL DAVID MARTIN Presented to the Faculty of the Graduate School of The University of Texas at Arlington in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY THE UNIVERSITY OF TEXAS AT ARLINGTON December 2019 Supervising Committee: Farhad Kamangar, Supervising Professor David Levine, Supervising Professor Manfred Huber Gergley Zaruba Copyright by Joel David Martin 2019 The copyright of Chapter2 was transferred to the IEEE as part of publication in COMPSAC 2018 (\Property-Based Testing of Browser Rendering Engines with a Consensus Oracle") [1]. Please refer to the IEEE for information about licensing the content from Chapter2. Chapter3 is a manuscript that is intended for separated publication. All copyrights on Chapter3 are currently reserved to Joel Martin. The remainder of this work, excluding Chapter2 and Chapter3, is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. ACKNOWLEDGMENTS Thank you to the members of my Supervising Committee for their support and guidance. In particular, I would like to thank David Levine who spent many long hours with me discussing the research and helping with the numerous large and small details that were necessary to complete my thesis. Many thanks to my employer, Viasat, for the financial investment toward my education. Thank you to my supervisors at Viasat, Ingolf Krueger and Markus Kaltenbach, for their support and encouragement throughout my graduate education. I would like to apologize for the harm done to numerous figurative trees during the completion of this research. As a thank you to the reader(s) of this dissertation, I offer you a gift. Contact me with proof that you have read this work and I will send you a prize (limited quantities available). Thank you to Aaron Brooks for being a sounding board for many of the ideas that were explored in this research. Thank you to Matt Oquist, Ingolf Krueger, and Rebecca Martin for their detailed reviews and the valuable feedback that they provided. DEDICATION I dedicate this work to my brilliant and beautiful wife, Dr. Rebecca Martin. She worked hard to ensure that I would be successful in this endeavor. Let it be noted, for posterity, that she received her doctorate 8.5 years prior to me receiving mine. ABSTRACT Using Property-Based Testing, Weighted Grammar-Based Generators, and a Consensus Oracle to Test Browser Rendering Engines and to Reproduce Minimized Versions of Existing Test Cases Joel David Martin, Ph.D. The University of Texas at Arlington, 2019 Supervising Professors: Farhad Kamangar and David Levine Verifying that a web browser rendering engine correctly renders all valid web pages is challenging due to the size of the input space (valid web pages), the difficulty of determining correct rendering for any given web page (the test oracle problem), and the degree to which normal variation in browser rendering behavior can obscure other differences (fonts, bor- ders, input controls, etc). These challenges lead to manual human involvement during the testing process. We propose a new Property-Based Testing (PBT) approach that addresses these challenges in order to enable automated web browser render testing. Our approach is composed of the following modules: a system for translating HyperText Markup Lan- guage (HTML) and Cascading Style Sheets (CSS) specification data into grammar-based generators; a grammar weighting system that controls test case generation along with mul- tiple methods for automatically adjusting those weights; and a consensus oracle of multiple rendering engines to identify failing test cases. Our approach increases the practicality of real-world testing. It has the ability to re- produce existing test cases from external sources, is able to shrink test cases to assist with root-cause analysis, and supports generic test specifications. We validated and characterized the effectiveness of our approach with a constrained markup grammar developed for this purpose. Applying our approach while testing Mozilla Firefox and Google Chrome revealed 1695 unique test cases that yield rendering differences. Testing of Mozilla's new Servo web vi browser revealed multiple bugs including eleven crashes and resource leaks. We reported these issues to Mozilla developers and are working together to develop and verify solutions. vii Contents 1 Introduction1 1.1 Motivation and Objectives............................1 1.1.1 Browser Testing Challenges.......................1 1.1.2 Solving the Limitations of PBT with Instacheck............3 1.1.3 Use case: a New Web Browser Engine.................4 1.1.4 Objectives.................................5 1.2 Modules and Document Structure........................7 1.2.1 Modules..................................7 1.2.2 Document Structure...........................9 2 \Property-Based Testing of Browser Rendering Engines with a Consensus Oracle" (COMPSAC, July 2018) 11 2.1 Abstract...................................... 12 2.2 Introduction.................................... 12 2.3 Property-Based Testing (PBT)......................... 13 2.3.1 Test Generation using Clojure test.check................ 14 2.3.2 Test Shrinking............................... 15 2.4 Test System Architecture............................. 16 2.5 Grammar-based Input Generators........................ 18 2.5.1 HTML5 Grammar............................ 18 vi 2.5.2 CSS3 Grammar.............................. 20 2.6 Generator Tuning................................. 21 2.7 Consensus..................................... 22 2.7.1 Challenges................................. 22 2.7.2 Calculating Consensus.......................... 23 2.8 Results....................................... 24 2.9 Related Work................................... 27 2.10 Conclusion and Future Work........................... 27 3 \Instacheck: An Automated Property-based Testing Model to Increase Coverage and Enable Directed Testing using Weighted Grammars" 28 3.1 Abstract...................................... 29 3.2 Introduction.................................... 29 3.3 Property-based Testing (PBT).......................... 30 3.3.1 PBT Input Properties.......................... 31 3.3.2 PBT Test Shrinking........................... 32 3.3.3 Limitations of PBT............................ 33 3.4 Instacheck..................................... 34 3.4.1 Instacheck Grammars........................... 34 3.4.2 Grammar Weights............................ 36 3.4.3 Grammar Paths, and Treks....................... 38 3.4.4 Grammar Weight Reduction and Propagation............. 41 3.4.5 Reproducing and Shrinking Existing Test Cases............ 44 3.5 Results....................................... 45 3.5.1 Test Grammars.............................. 45 3.5.2 Controlling Coverage........................... 48 3.5.3 Increasing Coverage with Weight Reduction.............. 52 3.5.4 Shrinking................................. 54 vii 3.5.5 Reproducing Existing Test Cases.................... 55 3.5.6 Case-study: Detecting Browser Rendering Differences......... 59 3.6 Related Work................................... 61 3.7 Conclusion..................................... 62 3.7.1 Future Work................................ 62 4 Instacheck In Greater Depth 63 4.1 Clojure test.check PBT Example........................ 63 4.2 Clojure test.check Generators.......................... 67 4.3 Extended Backus-Naur form (EBNF) Generators and Weight Tuning.... 71 4.3.1 MG4 and MG8 Grammars........................ 76 4.4 Translating EBNF Grammars to Clojure Generators.............. 82 4.5 Instacheck Command Line and Library Usage................. 85 4.5.1 Generate Random Test Cases...................... 86 4.5.2 Test and Shrink.............................. 89 4.5.3 Reproduce and Shrink an existing Test Case.............. 94 4.5.4 Translate EBNF Specification into Clojure Generators........ 98 5 HTML and CSS Grammars (html5-css3-ebnf) 101 5.1 Backus-Naur form (BNF) and EBNF Background............... 101 5.2 Hypertext Markup Language version 5 (HTML5) Grammar.......... 103 5.2.1 HTML5 Background........................... 103 5.2.2 Parsing and Translating HTML5.................... 103 5.3 Cascading Style Sheet Level 3 (CSS3) Grammar................ 107 5.3.1 CSS3 Background............................. 107 5.3.2 The CSS3 Value Definition Syntax (VDS) Grammar.......... 110 5.3.3 Parsing and Translating CSS3...................... 113 5.4 Parsing Challenges................................ 117 viii 5.5 html5-css3-ebnf Usage.............................. 119 5.5.1 Generate EBNF Parsing Grammars................... 119 5.5.2 Parse Web Pages............................. 121 6 Bartender: Browser Automated Render TestiNg DrivER 125 6.1 The State of Browser Render Testing...................... 125 6.2 Bartender System Architecture......................... 128 6.2.1 Optimizing and Translating EBNF Grammars to Clojure Generators 130 6.2.2 Bartender Testing Process........................ 133 6.3 Consensus..................................... 135 6.3.1 Consensus Algorithms.......................... 136 6.3.2 Consensus Challenges and Solutions................... 139 6.3.3 Consensus Example............................ 145 6.4 WebDriver Browser Automation......................... 157 6.5 Monitoring and Reporting Applications..................... 159 6.5.1 Runtime Monitoring Application...................