Empirical Studies of Javascript-‐ Based Web Applicaxon
Total Page:16
File Type:pdf, Size:1020Kb
Empirical Studies of JavaScript- based Web Applica8on Reliability Karthik Pa@abiraman1 Frolin Ocariza1 Kar.k Bajaj1 Ali Mesbah1 Benjamin Zorn2 1 University of Bri.sh Columbia (UBC), 2MicrosoE Research (MSR) Web 2.0 Applicaons Copyright: Karthik Paabiraman, 2014 Web 2.0 Applicaon: Amazon.com Menu Amazon’s Third party Search bar Web 2.0 applicaons allow rich UI funconality within a single web page own ad gadget ad Func8on Copyright: Karthik Paabiraman, 2014 Modern Web Applicaons: JavaScript • JavaScript: Implementaon of ECMAScript standard – Client-Side JavaScript: used to develop web apps • Executes in client’s browser – send AJAX messages • Responsible for web applicaon’s core func.onality • Not easy to write code in – has many “evil” features 4 Copyright: Karthik Paabiraman, 2014 JavaScript: History Brief History of JavaScript (Source: TomBarker.com) JavaScript (JS) had to “look like Java” only less so, be Java’s dumb kid brother or boy-hostage sidekick. Plus, I had to be done in ten days or something worse than JS would have happened – Brendan Eich (Inventor of JavaScript) Copyright: Karthik Paabiraman, 2014 • • 10000 20000 30000 40000 50000 60000 70000 80000 90000 Thousands of lines of code, oEen > 10,000 97 of the 0 Google YouTube Yahoo Baidu QQ MSN Amazon JavaScript: Prevalence Sina WordPress Alexa Ebay LinkedIn Bing MicrosoE Yandex 163 top 100 websites use JavaScript Copyright: Karthik Paabiraman, 2014 mail.ru PayPal FC2 Flickr IMDb Lines of code Apple Livedoor BBC Sohu go.com Soso Youku AOL CNN MediaFire ESPN MySpace MegaUpload Mozilla 4shared Adobe About LiveJournal Tumblr goDaddy CNET YieldManager Sogou Zedo Ifeng Pirate Bay ImageShack Weather NY Times Nelix JavaScript: “good” or “Evil” ? Vs Eval Calls (Source: Richards et al. [PLDI-2010]) 100000 E 10000 v 1000 a 100 l 10 Real web applica8ons do not s8ck to the “good” parts 1 s 0.1 Copyright: Karthik Paabiraman, 2014 Studies of JavaScript Web Applicaons Performance and parallelism: Reliability Security and Privacy: JSMeter [Ratanaworabhan-2010], [Yue-2009], [Richards-2009], [Fortuna-2011] ? gatekeeper[guarnieri-2009], [Jang-2010] Goal: Study and improve the reliability of JavaScript web applicaons performance reliability security Copyright: Karthik Paabiraman, 2014 Does Reliability Maer ? • Snapshot of iFeng.com: Leading media website in China an error occurred when processing this direc8ve Copyright: Karthik Paabiraman, 2014 This Talk • Mo.vaon and Approach • Three approaches for measuring JS Reliability – Error Messages [ISSRE 2011] – With F. Ocariza and B.G. Zorn – Bug Reports [ESEM 2013] – With F. Ocariza, K. Bajaj, A. Mesbah – Stack Overflow Reports [MSR 2014] – With F. Ocariza, A. Mesbah • Conclusion and Next Steps Copyright: Karthik Paabiraman, 2014 JSER: JavaScript Error Messages • All excep.ons thrown are logged to JS console Mul8ple excepons Copyright: Karthik Paabiraman, 2014 JSER: Error Messages Vs. Stac Analysis • No false posi.ves unlike stac analysis • Capture interac.ons with third-party code (adver.sements) • Capture interac.ons with the DOM JSER Vs. Copyright: Karthik Paabiraman, 2014 JSER: Tools • Chose 50 web applicaons from Alexa top 100 • Created Selenium tests for normal interac.ons • Capture JavaScript Errors printed to Firebug Copyright: Karthik Paabiraman, 2014 JSER: Research Ques.ons Do errors occur in web apps How do errors correlate and if so, what categories do with stac and dynamic they fall in ? characteris.cs of the app? How do errors vary by speed of tes.ng ? Are they all determinis.c ? Copyright: Karthik Paabiraman, 2014 JSER: Method 1. Descrip.on of error message 2. Line of code corresponding to error 3. Domain number and line number Two errors are different if any aribute is different Copyright: Karthik Paabiraman, 2014 JSER: Error Frequencies Results • Average of 4 dis8nct error messages for each app – Standard dev: 3 – Max: 16 (Cnet) – Min: 0 (Google) Total Disnct Errors 18 16 14 12 10 8 6 4 2 0 Copyright: Karthik Paabiraman, 2014 JSER: Error Classificaon Results • 94 % of errors fall into four predominant categories Distribu8on of Error Messages 4% # of Permission Denied Errors 6% # of Null Excep.on Errors 27% # of Undefined Symbol Errors 54% # of Syntax Errors 9% # of Miscellaneous Errors Copyright: Karthik Paabiraman, 2014 JSER: Research Ques.ons Errors occur in web applicaons How do errors correlate (4 per applicaon on average) with stac and dynamic and fall into four categories characteris.cs of the app? How do errors vary by speed of tes8ng ? Are they all determinisc ? Copyright: Karthik Paabiraman, 2014 JSER: Effect of Tes.ng Speed • Varied tes.ng speed for replaying events in Selenium • Performed three execu.ons in each tes.ng speed Fast Medium Slow 0 ms 500 ms 1000 ms Copyright: Karthik Paabiraman, 2014 JSER: Tes.ng Speed Results (CNN) Error Message (shortened) F F F M M M S S S 1 2 3 1 2 3 1 2 3 Permission Denied for 4 4 4 1 3 3 2 2 3 view.atdmt.com to call <fname> on marquee.blogs.cnn.com targetWindow.cnnad showAd is not 0 2 5 0 0 0 0 0 0 a function window.parent.CSIManager is un- 0 0 0 0 0 0 1 1 0 defined Copyright: Karthik Paabiraman, 2014 JSER: Effect of Tes.ng Speed • All three tes.ng modes expose different errors Total disnct errors Fast Mode Medium Mode Slow Mode Copyright: Karthik Paabiraman, 2014 JSER: Non-Determinism • More than 70% of errors: non-determinis.c Total non-determinis.c errors Copyright: Karthik Paabiraman, 2014 JSER: Research Ques.ons Errors occur in web applicaons How do errors correlate (4 per applicaon on average) with stac and dynamic and fall into four categories characteriscs of the app? Error occurrences vary with speed of tes.ng. About 70% of errors are non-determinis.c. Copyright: Karthik Paabiraman, 2014 JSER: Stac/Dynamic Characteris.cs Stac Characteriscs Dynamic Characteriscs Measured using Phoenix & Firebug plugins From Richards et al. [PLDI – 2010] • Alexa Rank • Number of called func.ons • Bytes of JavaScript code • Number of eval calls • Number of domains • Proper.es deleted • Domains containing JS • Object inheritance overridings Copyright: Karthik Paabiraman, 2014 JSER: Correlaons Summary Stac Characteriscs Dynamic Characteriscs • Alexa Rank • Number of called func.ons • Bytes of JavaScript code • Number of eval calls • Number of domains • Proper.es deleted • Domains containing JS • Object inheritance overridings Copyright: Karthik Paabiraman, 2014 JSER: Research Ques.ons Errors occur in web applicaons Errors correlate with Alexa (4 per applicaon on average) rank, no of domains but and fall into four categories not with loc or eval calls Error occurrences vary with speed of tes.ng. About 70% of errors are non-determinis.c. Copyright: Karthik Paabiraman, 2014 JSER: Implicaons of the Results • Programmers – Need to make code robust against other code/scripts – Make sure interac.ons with DOM are checked • Testers – Perform integraon tes.ng to see effects of ads – Need to test at mul.ple tes.ng speeds, mul.ple .mes • Stac analysis tool developers – Target most common classes of errors – Need to model the DOM in the analysis Copyright: Karthik Paabiraman, 2014 This Talk • Mo.vaon and Approach • Three approaches for measuring JS Reliability – Error Messages [ISSRE 2011] – With F. Ocariza and B.G. Zorn – Bug Reports [ESEM 2013] – With F. Ocariza, K. Bajaj, A. Mesbah – Stack Overflow Reports [MSR 2014] – With F. Ocariza, A. Mesbah • Conclusion and Next Steps Copyright: Karthik Paabiraman, 2014 Bug Report Study: goals • What errors/mistakes cause JavaScript faults? • What impact do JavaScript faults have? Bug Report Study of twelve popular, Open Source JavaScript Applicaons Copyright: Karthik Paabiraman, 2014 Bug Report Study: Objects Eight JavaScript Web Applica8ons Four JavaScript Libraries Copyright: Karthik Paabiraman, 2014 Bug Report Study : Method • Collect bug reports from bug repositories – Focus on bugs that are marked fixed to avoid spurious bugs – Organized into a uniform format (XML file) Filter out Pick the first Search for all reports that 30 reports and bug reports are not analyze them that have the marked “fixed” manually to word OR the fault determine “JavaScript” does not cause/impact involve JS Copyright: Karthik Paabiraman, 2014 Bug Report Study: Ques.ons • RQ1: What types of JavaScript faults occur in web apps? • RQ2: What is the impact of JavaScript faults ? • RQ3: How long does it take to fix a JavaScript fault? • RQ4: Are JavaScript faults browser-specific ? Copyright: Karthik Paabiraman, 2014 Bug Report Study: Categories Incorrect Method Parameter Fault: Unexpected or invalid value passed to JS method or assigned to JS property DOM-Related Fault: The method is a DOM API method - Account for around two-thirds of JavaScript Faults Copyright: Karthik Paabiraman, 2014 Bug Report Study: DOM html body head table div p script Text: tr p “Hello world” Want to retrieve this element 34 Copyright: Karthik Paabiraman, 2014 Bug Report Study: DOM-Related Faults JavaScript code: var x = document.getElementById(“elem”);document.getElementById(“elme”); Will return null Inexistent ID div DOM: id: elem 35 Copyright: Karthik Paabiraman, 2014 DOM-Related Fault: Example var elem, retrievedStr = [Retrieved via XHR]; var dotsInStr = retrievedStr.split(".").length; if (dotsInStr == 0) { var prefix = "id_"; elem = $("#" + prefix + retrievedStr); } else { elem = $(retrievedStr); } elem[0].focus() Copyright: Karthik Paabiraman, 2014 DOM-Related Fault: Example var elem, retrievedStr = [Retrieved via XHR]; var dotsInStr = retrievedStr.split(".").length; if (dotsInStr == 0) { var prefix = "id_"; elem