Applying Model-based Testing to HTML Rendering Engines – A Case Study? Jens R. Calamé1 and Jaco van de Pol2 1 Centrum voor Wiskunde en Informatica, P.O.Box 94079, 1090 GB Amsterdam, The Netherlands 2 University of Twente, Faculty EEMCS P.O.Box 217, 7500 AE Enschede, The Netherlands [email protected],[email protected] Abstract. Conformance testing is a widely used approach to validate a system correct w.r.t. its specification. This approach is mainly used for behavior-oriented systems. BAiT (Behavior Adaptation in Testing) is a conformance testing approach for data-intensive reactive systems. In this paper, we validate the applicability of BAiT to systems, which are not behavior-oriented (reactive) but document-centered. In particular, we apply BAiT to the test of the HTML rendering engine Gecko, which is used by Mozilla Firefox. In order to do so, we formally specify a part of the CSS box model in the specification language µCRL and implement a wrapper for the Gecko renderer. Then, we automatically generate test cases and run tests with BAiT in a controlled experiment in order to demonstrate our approach on the relevant part of Gecko. 1 Introduction Testing as a dynamic approach to software quality assurance is widely accepted in industry and is a well-studied field in academia. State-of-the-art testing ap- proaches are model-based [4], i.e. tests are not generated in an ad-hoc manner, but founded on the specification of the software product under consideration. One of the most rigorous model-based test approaches is conformance testing, which tests whether an implementation conforms its specification focusing on functional requirements. These notions are made precise in the formal methods community [12]. In this paper, we want to apply conformance testing to rendering engines of web browsers. In state-of-the-art webdesign, content and design are kept separate from each other. Content is defined in the Hypertext Markup Language (HTML), while the design is specified in a Cascading Style Sheet (CSS). When a web page is rendered, the information from the CSS is used to position elements of content on the rendered page. If a web document has a complex structure, rendering algorithms can turn out to be erroneous, leading to “broken” web pages with mispositioned elements. Rendering a modern web application, whose ? Part of this research has been funded by the Dutch BSIK/BRICKS project. appearance is dynamically changed on the client side using script languages, like web applications based on Asynchronous JavaScript and XML (AJAX, [16]), is even more demanding for rendering engines. Performing a sufficient conformance test in this context is tedious, so that an automated solution is preferable. In this paper, we present a feasibility study for automated testing of rendering engines using the test tool BAiT. Here, we validate the applicability of BAiT [5, 6], a blackbox test execution tool for non-deterministic, data-oriented reactive systems, to test the rendering engine of a web browser w.r.t. the positioning of boxes in the CSS box model. Boxes in HTML are entities like for instance a complete HTML document (body element) or a paragraph (elements p or div), which contain content or other boxes and which are positioned either absolutely or relative to each other. The box model is part of the W3C CSS Specification [15, Sect. 8]. We present a feasibility study, applying conformance testing using BAiT to the rendering engine Gecko3, which is used by the open source web browser Mozilla Firefox 4. In order to perform the tests, we formalize the CSS specifica- tion and design test purposes. Furthermore, we implement a wrapper component between the tester and Gecko in order to achieve a mapping between an action- oriented specification and the document-oriented rendering engine. For several setups of web pages, we then automatically generate parameterizable test cases. Those test cases can be instantiated with varying data settings for the posi- tions of boxes, so that they are reusable for different page layouts. Then, the test cases are executed automatically against the test wrapper and the results retrieved from Gecko are interpreted in order to automatically assign verdicts. 1.1 Related Work There exists a number of static test suites for the rendering capabilities of web browsers. Each of those test suites consists of a set of HTML and CSS documents with different page layouts. The most well-known one is probably the ACID 2 Test 5. It tests web browsers for their full compliance to the actual version of CSS by rendering a web page with a vast amount of CSS features enabled. Another set of test suites for the standard compliance of web browsers are the W3C Cascading Style Sheets Test Suites6. Here, again, we have a set of static documents, which test rendering capabilities for distinct features of CSS. Finally, Mozilla Firefox itself provides a set of static layout regression tests7, which can be run in debug builds of the software. Most of the named test suites are, however, not automated. In fact, the files in the test suites, i.e. the test cases, have to be loaded into the browser and then the result of rendering the page has to be visually assessed. This process is not 3 http://www.mozilla.org/newlayout/ 4 http://www.mozilla.com/en-US/firefox 5 http://acid2.acidtests.org/ 6 http://www.w3.org/Style/CSS/Test/ 7 http://www.mozilla.org/newlayout/doc/regression_tests.html automatic at all, neither on the level of test case generation, nor on that of test execution. This means, that a certain amount of test cases has to be designed and executed manually and the results have to be visually evaluated. This process is time-consuming compared to an automated test process, where test cases – or at least test data – is generated for a number of standard and critical cases. In this case, the number of test cases to be generated and executed can be optimized in order to reduce the absolute number of test cases. The regression tests of Mozilla Firefox are at least automated on the level of test execution, however, they are still founded on a static set of test cases. The approach, which we propose in this paper, provides not only an auto- mated test execution and evaluation of rendering results for a fragment of the CSS features, the box model, but also an automatic generation and variation of tested web page layouts. We chose this fragment, because rendering results, i.e. the position of a box, can be objectively measured (in pixels) rather than having to be visually assessed. The approach of a fully automatic test case generation and execution has the advantage regarding the other named approaches, that the executed test cases can cover more variation w.r.t. data parameters (i.e. the position of boxes), but also regarding the structure of the rendered web pages. The first issue enables us to reuse equally structured test cases (i.e. web pages) and by that to reduce the number of generated test cases. The latter allows us to test rendering web pages with a different interrelation of elements and by that cover a larger variety of possible failures in the IUT. We are aware of several case study reports of model-based testing, concerning topics like the Conference Protocol [2], the Storm Surge Barrier in the Nether- lands [9], smart card applications [8], the telecommunication sector [3] or – ver- tical to our work – the generation of test purposes for the Session Initiation Protocol [1]. To the best of our knowledge, however, this is the first application of model-based automatic test generation and testing techniques to document- centered applications, esp. to HTML rendering engines. 2 The Test Environment The test environment for our case study consists of two main components. On the one hand, we have the tester, which controls the run of the experiment. The tester will be discussed in Sect. 2.1. On the other hand, we have the implementation under test (IUT), which we will discuss in Sect. 2.3. This is the object under consideration, which we will actually be testing throughout the case study. In Sect. 2.2, we will give an introduction to the CSS box model. 2.1 Firefox and Cascading Style Sheets Firefox. Mozilla Firefox is a stand-alone web browser, which has its roots in the Netscape Communicator from the 1990s. Most of its code was put under an open-source license in 1998 and founded the basis for the Mozilla Suite, from which Firefox arose as a stand-alone browser in 2004. Web page content (HTML): Web page layout (CSS): ... div.warning { <div class="warning"> border-style: solid; ... border-color: red; <div class="warning" color:red; id="warning1">...</div> } ... div.warning#warning1 { </div> font-style: italic; ... } Fig. 1. Two differently formatted boxes A web browser reads and interprets HTML structured data to display web pages. The task of actually displaying is carried out by a rendering component. While loading a web page, this component incrementally builds up a Document Object Model (DOM) tree from the HTML document to be displayed together with declarative layout information. Mozilla Firefox uses the renderer Mozilla Gecko, whose version 1.8.1 we consider in this paper as the IUT. Cascading Style Sheets (CSS). In the 1990s, a declarative stylesheet language was developed for structured documents in order to properly divide the content of a web page from its design. Currently, the de-facto standard for CSS is in version 2.1 [15]. This version is currently not fully supported by all web browsers, including Mozilla Firefox.