End-User Record and Replay for the Web by Shaon Kumar Barman A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Rastislav Bodík, Chair Professor Koushik Sen, Co-chair Professor Dawn Song Professor Susan Stone Fall 2015 End-User Record and Replay for the Web Copyright 2015 by Shaon Kumar Barman 1 Abstract End-User Record and Replay for the Web by Shaon Kumar Barman Doctor of Philosophy in Computer Science University of California, Berkeley Professor Rastislav Bodík, Chair Professor Koushik Sen, Co-chair The usefulness of today’s websites is limited by their form and ease of access. Even though the web contains an ever-expanding wealth of information, much of it exists in a form that is not directly useful. How can end-users access the web in a way that meets their needs? We present record and replay (R+R) as a way to bridge the gap between a website’s func- tionality and the end-user’s goal. R+R leverages an interface the user knows and is stable — that is, the webpage — in order to automate repetitive tasks. A R+R system observes a user interacting with a website and produces a script which, when executed, repeats the original interaction. End-users can use R+R to automate a sequence of actions and programmers can use these recordings as an API to execute more complicated tasks. Unfortunately, as websites become more complex, R+R becomes increasingly difficult. The challenge with modern websites is that a single demonstration of the interaction has limited information, making scripts fragile to changes in the website. For past R+R systems, this was less of an issue because of the static nature of websites. But as the web becomes more dynamic, it becomes difficult to produce a robust script that mimics the interactivity of the user and can adapt to changes on the page. To solve this problem, we developed Ringer, a R+R system for the web. Ringer is built on three key abstractions — actions, triggers, and elements. Ringer takes a user demonstration as input and synthesize a script that interacts with the page as a user would. To make Ringer scripts robust, we develop novel methods for web R+R. In particular, Ringer uses the following features: • Inferring triggers automatically which synchronize the script with the state of the webpage • Monitoring the replay execution to ensure actions faithfully mimic the user • Identifying elements on the replay-time page using a similarity metric 2 To evaluate our work, we run Ringer on a suite of real-world benchmarks by replaying interactions on Alexa-ranked websites. We compare Ringer against a current state-of-the-art replay tool and find that Ringer is able to replay all 29 benchmark interactions, compared to only 5 benchmarks for the previous approach. Additionally, our benchmarks show that a replayer needs to synchronize with the state of a webpage in order to replay correctly, motivating Ringer’s use of triggers. We show that our trigger inference algorithm can synthesize sufficient synchronization, while also having the added benefit of speeding up the replay execution. Finally, we show that R+R is useful as a building block for end-user applications by building two such tools using Ringer. One allows end-users to scrape structured data from a website simply through demonstration. The other allows end-users to aggregate real-time data from various websites in the form of live tiles, by specifying the data they want on a website through demonstration. i Contents Contents i List of Code Listings iii List of Figures iv List of Tables vi 1 Introduction 1 1.1 Record and Replay................................... 2 1.2 Ringer.......................................... 5 2 Previous Work in Browser Automation6 2.1 Applications and Challenges............................. 6 2.2 Audience........................................ 8 2.3 Required Skills..................................... 9 2.4 Limitations of Current Approaches ......................... 10 2.5 Conclusion....................................... 12 3 Overview 13 3.1 Properties of Record and Replay........................... 14 3.2 Motivating Example.................................. 15 3.3 Approach........................................ 17 3.4 Limitations....................................... 22 3.5 Conclusion....................................... 23 4 Level of Abstraction 24 4.1 User Program ..................................... 24 4.2 Correctness Requirements............................... 25 4.3 Design Choices..................................... 27 4.4 Evaluation ....................................... 31 4.5 Conclusion....................................... 36 ii 5 Trigger Inference 37 5.1 Alternative Methods.................................. 37 5.2 Ringer Approach.................................... 41 5.3 Problem Formalism .................................. 43 5.4 Assigning Triggers................................... 45 5.5 Matching and Identifying HTTP Responses..................... 48 5.6 Evaluation ....................................... 50 5.7 Conclusion....................................... 54 6 Faithful Actions 55 6.1 Challenges with the DOM Event Architecture.................... 55 6.2 Ringer Approach.................................... 57 6.3 Record and Replay Algorithm ............................ 61 6.4 Evaluation ....................................... 66 6.5 Conclusion....................................... 70 7 Element Identification 71 7.1 Problem Statement................................... 71 7.2 Past Approaches.................................... 72 7.3 Ringer Approach.................................... 76 7.4 Algorithm ....................................... 76 7.5 Evaluation ....................................... 80 7.6 Conclusion....................................... 82 8 Applications 83 8.1 API for Developers................................... 84 8.2 Scraping Relational Data by Demonstration..................... 85 8.3 Real-time Updates by Demonstration ........................ 88 8.4 Conclusion....................................... 90 9 Related Work 91 10 Conclusion 94 Bibliography 95 iii List of Code Listings 6.3.1 Record algorithm: Capturing event information................... 61 6.3.2 Replay algorithm: Outer loop and dispatching events............... 63 6.3.3 Record algorithm: Capturing event deltas...................... 64 6.3.4 Replay algorithm: Capturing event deltas...................... 64 6.3.5 Replay algorithm: Compensating for deltas..................... 65 6.3.6 Record algorithm: Identifying atomic cascading events.............. 65 iv List of Figures 1.1.1 Minimized JavaScript code ................................ 3 1.1.2 Page changes to Southwest.com across multiple accesses ............... 4 1.1.3 Visual cue on Amazon.com................................ 4 2.1.1 Model of browser ..................................... 7 3.1.1 Model of browser ..................................... 14 3.2.1 Screenshot of Amazon.com page............................. 15 3.2.2 User program for Amazon.com scenario......................... 16 3.2.3 Screenshots of Amazon.com page during motivating scenario............. 17 3.3.1 Comparison of how Ringer interacts with the browser versus a user......... 18 3.3.2 Ringer program with triggers for Amazon.com..................... 20 3.3.3 Recorded program for Amazon.com........................... 21 3.3.4 Two execution traces from the Amazon.com scenario ................. 21 4.3.1 Screenshots of Southwest.com scenario ......................... 27 5.1.1 Sequence of events during Amazon.com scenario ................... 38 5.1.2 Selenium code which waits for silver camera price to load............... 39 5.1.3 Expressions contained in Selenium’s Expected Conditions library .......... 40 5.2.1 Simplified version of Amazon.com script ........................ 42 5.2.2 Two execution traces from the Amazon.com scenario ................. 42 5.4.1 Algorithm to assign triggers to actions.......................... 47 5.5.1 Language of HTTP response trigger expressions.................... 48 5.5.2 Algorithm to compute trigger expressions based upon HTTP responses . 49 5.6.1 Accuracy vs. speedup on the suite of benchmark websites............... 53 6.2.1 Screenshots of Southwest.com scenario ......................... 58 6.2.2 Event traces comparing atomic events vs. naive replay................. 59 6.3.1 Trace of events that occur during Southwest.com example............... 62 7.2.1 Screenshot of Amazon.com page............................. 73 7.2.2 CoScripter recording of user’s interaction with Amazon.com............. 73 v 7.2.3 Selenium IDE recording of user’s interaction with Amazon.com ........... 74 7.4.1 Algorithm to construct training data used to assign weights to attributes . 77 7.4.2 Example element from Southwest.com.......................... 78 7.4.3 Algorithm to identify matching element on replay page................ 79 7.4.4 Algorithm to calculate similarity score.......................... 80 8.2.1 Overview of user interactions with WebCombine to scrape data off of Google Scholar 86 8.3.1 Mockup of live tile interface................................ 88 8.3.2 Screenshot of calendar showing when campus pools are open at UC Berkeley . 89 vi List of Tables 2.2.1 Browser automation tools categorized by application and audience ......... 8 2.3.1 Current browser automation tools vs. skills required to use