Automated Cross-Browser Compatibility Testing ∗ Ali Mesbah Mukul R. Prasad Electrical and Computer Engineering Trusted Systems Innovation Group University of British Columbia Fujitsu Laboratories of America Vancouver, BC, Canada Sunnyvale, CA, USA [email protected] [email protected] ABSTRACT web browsers render web content somewhat differently [18, With the advent of Web 2.0 applications and new browsers, 24, 25, 26]. However, the scope and impact of this problem the cross-browser compatibility issue is becoming increas- has been rapidly growing due to two, fairly recent trends. ingly important. Although the problem is widely recognized First, modern, rich-content web applications have a heavy among web developers, no systematic approach to tackle client-side behavioral footprint, i.e., they are designed to ex- it exists today. None of the current tools, which provide ecute significant elements of their behavior exclusively on the screenshots or emulation environments, specifies any notion client-side, typically within the web browser. Further, tech- of cross-browser compatibility, much less check it automat- nologies such as Ajax [12], Flash, and event-handling for ically. In this paper, we pose the problem of cross-browser dynamic HTML, which support this thick-client behavior, compatibility testing of modern web applications as a `func- are the very aspects in which web browsers differ. tional consistency' check of web application behavior across Second, recent years have seen an explosion in the num- different web browsers and present an automated solution ber of available web browsers. There are nearly 100 different for it. Our approach consists of (1) automatically analyzing web browsers available today [31]. Coupled with the differ- the given web application under different browser environ- ent kinds and versions of operating systems in which these ments and capturing the behavior as a finite-state machine; operate, this yields several hundred different client-side en- (2) formally comparing the generated models for equivalence vironments in which a web application can be used, each on a pairwise-basis and exposing any observed discrepan- with a slightly different client-side rendering of the applica- cies. We validate our approach on several open-source and tion [7, 9]. In the context of the above problem, the key industrial case studies to demonstrate its effectiveness and contributions of this paper are: real-world relevance. • We define the cross-browser compatibility problem and present an in-depth discussion of the issues surround- Categories and Subject Descriptors ing possible solutions to it. D.2.5 [Software Engineering]: Testing and Debugging • We present a systematic, fully-automated solution for cross-browser compatibility testing that can expose a substantial fraction of the cross-browser issues in mod- General Terms ern dynamic web applications. Reliability, Verification • We validate our approach on several open-source as well as industrial case studies to demonstrate its effi- Keywords cacy and real-world relevance. Dynamic analysis, web testing, cross-browser compatibility The rest of the paper is organized as follows. In the next section we discuss the genesis of the cross-browser compati- bility problem and how it plays out in modern web applica- 1. INTRODUCTION tions. We present a formal characterization of the problem Web applications pervade all aspects of human activity to- and put the solutions offered by this paper in that context. day. The web browser continues to be the primary gateway Section 3 reviews related work on cross-browser testing. In channeling the end-user's interaction with the web applica- Section 4 we present our proposed approach for performing tion. It has been well known for some time that different cross-browser compatibility testing, followed by a descrip- tion of the salient implementation aspects of that approach, ∗This author was a visiting researcher at Fujitsu Laborato- ries of America, when this work was done. in Section 5. Section 6 presents an empirical evaluation of our approach on some open-source as well as industrial case studies. In Section 7 we discuss the strengths and weak- nesses of our solution and lessons learnt from our experience. We conclude the paper in Section 8. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies 2. CROSS-BROWSER COMPATIBILITY bear this notice and the full citation on the first page. To copy otherwise, to Problem Genesis. The cross-browser compatibility republish, to post on servers or to redistribute to lists, requires prior specific (CBC) problem is almost as old as the web browser it- permission and/or a fee. ICSE ’11, May 21–28, 2011, Waikiki, Honolulu, HI, USA self. There are several reasons for its genesis and growth Copyright 2011 ACM 978-1-4503-0445-0/11/05 ...$10.00. in recent years. The earliest instances of this problem had A: DOM and/or Trace‐level B: Browser‐level Observable Differences Differences (a) Rendering in IE 8.0. D: Differences detected C: Ideal set of by ppproposed method differences to detect Figure 2: Browser incompatibility landscape. from the client-tier to the server tiers, the response received (b) Rendering in Firefox 3.5. and subsequent content served on the client, thus influenc- Figure 1: CBC issues in an industrial Web 2.0 ap- ing the overall evolution of the web application on the client plication: missing widgets in Firefox. side. Figure 1 shows screen-shots taken from a real en- terprise (proprietary) Web 2.0 application, illustrating this scenario. While the application renders correctly in Inter- its roots in the parallel evolution of different browsers and net Explorer, the Firefox rendering of it is missing the New the lack of standards for web content. Currently, standards User and Delete User widgets, marked with a red eclipse in do exist for basic web content (e.g., HTML 4.0) and for IE. Thus, the human user would be unable to exercise these scripting languages such as JavaScript, JScript, and Ac- features from this screen, thereby completely removing a tionScript (e.g., ECMA-262). However, in their bid to be very important set of behaviors, namely the application ad- inclusive and render websites that do not adhere to these ministrator being able to add and remove users, from the standards (e.g., legacy websites) web browsers typically im- evolution of the web application under Firefox.1 plement their own extensions to the web standards and in Problem Definition. Our working definition of the doing so differ in their behavior. The explosive growth in CBC problem, for the purposes of this paper, is based on the the number of browsers and client-side environments has Venn diagram of Figure 2. The set B depicts those cross- only exacerbated this issue. In addition, modern, dynamic browser differences that can be observed by the human user web applications execute an increasingly bigger fraction of on the web browser, while the set A represents those cross- their functionality in the client-tier i.e., within the end-user's browser differences that are reflected as differences in the web browser, thus further amplifying differences in observed client-side state of one or more screens (e.g., differences in behavior. Cross-browser compatibility is widely recognized the DOM representation or CSS properties of various DOM as an important issue among web developers [26] but hardly elements) or differences in the set of possible traces, i.e., ever addressed directly during the software development pro- alternating sequences of screens and user actions, on the cess. Typically web applications are developed with a single client-side. Further, there is the set of differences B − A target client-side configuration in view and manually tested which can be visually observed by the user but is not re- for a few more, as an after-thought. flected as a difference in the client-side state or in the evolu- Look-and-feel versus Functionality. The common tion of that state. These differences are typically, though not perception is that the CBC problem is confined to purely necessarily, stylistic and stem from different browsers' ren- `look-and-feel' differences on individual screens. While that dering of identical web content (e.g., CSS files). Although was indeed the case for the previous generation of web appli- such differences are important in their own right, they are cations, the problem plays out in a more systemic, functional not the focus of this paper. There is also the set A − B way in modern Web 2.0 applications. Thus, we believe that of differences, which exist in the DOM representation but the issue should be addressed as a generalized problem of do not result in any user-observable differences in behavior. checking functional consistency of web application behav- We argue that these differences are not relevant since the ior across web browsers, rather than purely a `look-and-feel' final metric of incompatibility is that which can be observed issue. This view is also held by several web developers [10]. by the end user. The set C = A \ B, which comprises the The following scenario illustrates how the problem plays differences reflected in both the user-observable behavior as out in modern web applications. The human end-user inter- well as the underlying client-side state, represents the most acts with the web application, through the browser, by view- interesting target for our technique. However, as explained ing dynamically served content and responding with user- below in Section 4, typically the set of differences detected events (e.g., clicks, mouse-overs, drag-and-drop actions) by our technique is a set D, where C ⊆ D ⊆ A.