Characterizing Callbacks in Javascript

Don’t Call Us, We’ll Call You: Characterizing Callbacks in JavaScript

Keheliya Gallaba Ali Mesbah Ivan Beschastnikh University of British Columbia University of British Columbia University of British Columbia Vancouver, BC, Canada Vancouver, BC, Canada Vancouver, BC, Canada [email protected] [email protected] [email protected]

Abstract—JavaScript is a popular language for developing 1 var db = require(’somedatabaseprovider’); web applications and is increasingly used for both client-side 2 http.get(’/recentposts’, function(req, res){ and server-side application logic. The JavaScript runtime is 3 db.openConnection(’host’, creds, function(err, conn){ inherently event-driven and callbacks are a key language feature. 4 res.param[’posts’]. forEach(function(post) { 5 conn.query(’select * from users where id=’ + post Unfortunately, callbacks induce a non-linear control flow and can [’user’], function(err, results){ be deferred to execute asynchronously, declared anonymously, 6 conn.close(); and may be nested to arbitrary levels. All of these features make 7 res.send(results[0]); callbacks difficult to understand and maintain. 8 }); 9 }); We perform an empirical study to characterize JavaScript 10 }); callback usage across a representative corpus of 138 JavaScript 11 }); programs, with over 5 million lines of JavaScript code. We Listing 1. A representative JavaScript snippet illustrating the comprehension find that on average, every 10th function definition takes a and maintainability challenges associated with nested, anonymous callbacks callback argument, and that over 43% of all callback-accepting and asynchronous callback scheduling. function callsites are anonymous. Furthermore, the majority of callbacks are nested, more than half of all callbacks are asynchronous, and asynchronous callbacks, on average, appear more frequently in client-side code (72%) than server-side (55%). Listing 1 illustrates three common challenges with callbacks. We also study three well-known solutions designed to help with the complexities associated with callbacks, including the error- First, the three callbacks in this example (on lines 2, 3, and 5) first callback convention, Async.js library, and Promises. Our are anonymous. This makes the callbacks difficult to reuse and results inform the design of future JavaScript analysis and code to understand as they are not descriptive. Second, the callbacks comprehension tools. in the example are nested to three levels, making it challenging Index Terms—Empirical Study, JavaScript, Callback, Asyn- to reason about the flow of control in the code. Finally, in line chrony, Event-driven Programming 5 there is a call to conn.query, which invokes the second asynchronously I. INTRODUCTION callback parameter . That is, the execution of the inner-most anonymous function (lines 6–7) is deferred until Callbacks, or higher-order functions, are available in many some later time. As a result, most of the complexity in this programming languages, for instance, as function-valued argu- small example rests on extensive use of callbacks; in particular, ments (e.g., Python), function pointers (e.g., C++), and lambda the use of anonymous, nested, and asynchronous callbacks. expressions (e.g., Lisp). In this paper we study JavaScript Though the issues outlined above have not been studied callbacks since JavaScript is the dominant language for build- in detail, they are well-known to developers. Searching for ing web applications. For example, a recent survey of more “callback hell” on the web brings up many articles with best than 26K developers conducted by Stack Overflow found that practices on callback usage in JavaScript. For example, a JavaScript is the most-used programming language [31]. The prominent problem with asynchronous callbacks in JavaScript callback language feature in JavaScript is an important factor is error-handling. The built-in try/catch mechanism does not to its success. For instance, JavaScript callbacks are used work with asynchronous callbacks. In response, the JavaScript to responsively handle events on the client-side by execut- developer community has, over time, arrived at a best practice ing functions asynchronously. And, in Node.js1, a popular known as “error-first protocol”. The intent is to reserve the first JavaScript-based framework, callbacks are used on the server- argument in a callback for communicating errors. However, as side to service multiple concurrent client requests. this is a best practice and adhering to this protocol is optional, Although callbacks are a key JavaScript feature, they have it is not clear to what extent developers use it in practice. not received much attention in the research community. We think that this is a critical omission as the usage of callbacks In this paper, we report on an empirical study to characterize is an important factor in developer comprehension and main- JavaScript callback usage across 138 large JavaScript projects. tenance of JavaScript code. These include 86 Node.js modules from the NPM public package registry used in server-side code and 62 subject sys- 1 https://nodejs.org tems from a broad spectrum of categories, such as JavaScript MVC frameworks, games, and data visualization libraries. Analyzing JavaScript code statically to identify callbacks and to characterize their properties for such a study presents a number of challenges. For example, JavaScript is a loosely typed language and its functions are variadic, i.e., they accept a variable number of arguments. We developed new JavaScript analysis techniques, building on prior techniques and tools, to identify callbacks and to measure their various features. The focus of our study is on gaining an understanding of callback usage in practice. We study questions such as, how often are callbacks used, how deep are callbacks nested, are Fig. 1. The JavaScript event loop model. anonymous callbacks more common than named callbacks, are callbacks used differently on the client-side as compared to the invoked synchronously before f returns, or it can be deferred server-side, and so on. Finally we measure the extent to which to execute asynchronously some time after f returns. developers rely on the “error-first protocol” best practice, and JavaScript uses an event-driven model with a single thread the adoption of two recent proposals to mitigate callback- of execution. Programming with callbacks is especially useful related challenges, the Async.js library [21] and Promises [6], when a caller does not want to wait until the callee completes. a new JavaScript language feature. To this end, the desired non-blocking operation is scheduled The results of our study show that (1) callbacks are passed as a callback and the main thread continues its synchronous as arguments more than twice as often in server-side code than execution. When the operation completes, a message is en- in client-side code, i.e., 24% of all server-side call-sites use a queued into a task queue along with the provided callback. callback in server-side code, while only 9% use a callback in The event loop in JavaScript prioritizes the single thread to client-side code; (2) anonymous callbacks are used in 42% of execute the call stack first; when the stack is empty, the event all callback-accepting function call-sites; (3) there is extensive loop dequeues a message from the task queue and executes callback nesting, namely, most callbacks nest 2 levels, and the corresponding callback function. Figure 1 illustrates the some nest as deep as 8 levels, and (4) there is an extensive event loop model of JavaScript for a non-blocking HTTP get use of asynchrony associated with callbacks — 75% of all call with a callback named cb. client-side callbacks were used in conjunction with built-in Named and anonymous callbacks. JavaScript callbacks can asynchronous JavaScript APIs. be named functions or anonymous functions (e.g., lines 2, 3, These results indicate that existing JavaScript analyses and or 5 of Listing 1). Each approach has its tradeoffs. Named tools [23], [19] often make simplifying assumptions about callbacks can be reused and are easily identified in stack JavaScript callbacks that might not be true in practice. For traces or breakpoints during debugging activities. However example, some of them ignore anonymous callbacks, asyn- naming causes the callback function to persist in memory and chrony, and callback nesting altogether. Our work stresses the prevents it from being garbage collected. On the other hand, importance of empirically validating assumptions made in the anonymous callbacks are more resource-friendly because they designs of JavaScript analysis tools. are marked for garbage collection immediately after being We believe that our characterization of the real-world use executed. However, anonymous callbacks are not reusable and of callbacks in different types of JavaScript programs will be may be difficult to maintain, test, or debug. useful to tool builders who employ static and dynamic analyses (e.g., which language corner-cases to analyze). Our results will Nested callbacks. Developers often need to combine several also make language designers more aware of how developers callback-accepting functions together to achieve a certain task. use callback-related language features in practice. For example, two callbacks have to be nested if the result of the first callback needs to be passed into the second callback Next, we overview the key features of JavaScript necessary in a non-blocking way (see lines 2-8 in Listing 3). This to understand our work. structure becomes more complex when the callbacks need to be conditionally nested. Control-flow composition with nested II. CALLBACKS IN JAVA SCRIPT callbacks increases the complexity of the code. The term A callback is a function that is passed as an argument ‘callback hell’ [26] has been coined by developers to voice to another function, which is expected to invoke it either their common frustration with this complexity. immediately or at some point in the future. Callbacks can be Error-first callbacks. In synchronous JavaScript code the seen as a form of the continuation-passing style (CPS) [32], in throw keyword can be used to signal an error and try/catch which control is passed explicitly in the form of a continuation; can be used to handle the error. When there is asynchrony, in this case the callback passed as an argument represents a however, it may not be possible to handle an error in the continuation. context it is thrown. Instead, the error must be propagated asyn- Synchronous and asynchronous callbacks. There are two chronously to an error handler in a different context. Callbacks types of callbacks. A callback passed to a function f can be are the basic mechanism for delivering errors asynchronously in JavaScript. Because there is no explicit language support for 1 // Before: nested callbacks asynchronous error signaling, the developer community has 2 $("#button").click(function() { 3 promptUserForTwitterHandle(function(handle) { proposed a convention — dedicate the first argument in the 4 twitter.getTweetsFor(handle , function(tweets) { callback to be a permanent place-holder for error signaling. 5 ui.show(tweets); 6 }); More exactly, the error-first callback protocol specifies that 7 }); during a callback invocation, either the first error argument is 8 }); non-null (indicating an error), or the error argument is null 10 // After: Using Async.js waterfall method and the other arguments contain data (indicating success), but 11 $("#button").click(function() { 12 async.waterfall([ not both [1]. Listing 2 shows an example of this protocol. If 13 promptUserForTwitterHandle , an error occurs while reading the file (Line 4), the anonymous 14 twitter.getTweetsFor , 15 ui.show function will be called with error as the first argument. For 16 ] this error to be handled, it will be propagated by passing it 17 , handleError); as the first argument of the callback (in line 5). The error can 18 }); then be handled at a more appropriate location (lines 17–21). 20 // After: sequential join of callbacks with Promises 21 $("#button").clickPromise() When there is no error, the program continues (line 8) and 22 .then(promptUserForTwitterHandle) invokes the callback with the first (error) argument set to null. 23 .then(twitter.getTweetsFor) 24 .then(ui.show); Listing 3. Rewriting nested callbacks using Async.js or Promises. 1 var fs = require(’fs’); 2 // read a file 3 function read_the_file(filename , callback) { 4 fs.readFile(filename , function (err, contents) { 5 if (err) return callback(err); Promises are a JavaScript language extension. A Promise- based function takes some input and returns a promise ob- 7 // if no error, continue 8 read_data_from_db(null, contents , callback); ject representing the result of an asynchronous operation. A 9 }); promise object can be queried by the developer to answer 10 } questions like “were there any errors while executing the async 12 function read_data_from_db(err, contents , callback) { call?” or “has the data from the async call arrived yet?” A 13 //some long running task 14 } promise object, once fulfilled, can notify any function that depends on its data. Listing 3 illustrates how nested callbacks 16 read_the_file(’/some/file’, function (err, result) { 17 if (err) { in vanilla JavaScript (lines 1–8) can be re-expressed using 18 //handle the error Promises (20–24). 19 console.log(err); 20 return; Next, we describe the methodology underlying our study of 21 } 22 // do something with the result JavaScript callbacks. 23 }); III. METHODOLOGY Listing 2. Error-first callback protocol. To characterize callback usage in JavaScript applications, The error-first callback protocol is intended to simplify we focus on the following three research questions. exception handling for developers; if there is an error, it will RQ1: How prevalent are callbacks? always propagate as the first argument through the API, and RQ2: How are callbacks programmed and used? API clients can always check the first argument for errors. RQ3: Do developers use external APIs to handle callbacks? But, there is no automatic checking of the error-first protocol Our analyses are open source [4] and all of our empirical in JavaScript. It is therefore unclear how frequently developers data is available for download [5]. adhere to this protocol in practice. A. Subject Systems Handling callbacks. A quick search in the NPM repository2 reveals that there are over 4,500 modules to help developers We study 138 popular open source JavaScript subject sys- with asynchronous JavaScript programming. Two solutions tems from six distinct categories: NPM modules (86), web that are gaining traction in helping developers handle call- applications (16), game engines (16), client-side frameworks backs are libraries, such as Async.js [21] and new language (8), visualization libraries (6), and games (6). NPM modules features such as Promises [6]. We now briefly detail these are used only on the server-side. Client-side frameworks, two approaches. visualization libraries, and games include only client-side The Async.js library exposes an API to help developers code. Web applications and game engines include both client- manage callbacks. For example, Listing 3 shows how nested side and server-side code. Table I presents these categories, callbacks in vanilla JavaScript (lines 1–8) can be expressed whether or not the category includes client-side and server- using the waterfall method available in Async.js (11–18). side application code, and the aggregate number of JavaScript files and lines of JavaScript code that we analyze for each 2 https://www.npmjs.com/ applications category. TABLE I setTimeout) then p is a callback; (3) if p is used as an JAVA SCRIPT SUBJECT SYSTEMS IN OUR STUDY argument to an unknown function f , then recursively check Category Subject Client Server Total Total if p is a callback parameter in f . systems side side files LOC Note that for f to be a callback-accepting function, it is NPM Modules 86 4,983 1,228,271 insufficient to find an invocation of f with a function argument Web Apps. 16 1,779 494,621 Game Engines 16 1,740 1,726,122 p. To be a callback-accepting function, the argument p must Frameworks 8 2,374 711,172 be invoked inside f ,orp must be passed down to some other DataViz Libs. 6 3,731 958,983 function where it is invoked. Games 6 347 119,279 Total 138 14,954 5,238,448 We do not analyze a subject’s dependencies, such as libraries, to find callbacks5. The only time we analyze external libraries is when a subject system calls into a library and passes The 86 NPM modules we study are the most depended- a function as an argument. In this case our analysis check on modules in the NPM repository [3]. The other subject whether the passed function is invoked as a callback in the systems were selected from GitHub Showcases [2], where library code. popular and trending open source repositories are organized We also study whether callback usage patterns are different around different topics. The subject systems we consider are between server-side and client-side JavaScript code. Categoriz- JavaScript-only systems. Those systems that contain server- ing the projects known as purely client-side (MVC frameworks side components are written in Node.js3, a popular server-side like Ember, Angular) or purely server-side (NPM modules) is JavaScript framework. Overall, we study callbacks in over 5 easy. But, some projects contain both server-side and client- million lines of JavaScript code. side JavaScript code. To distinguish client-side code from server-side code, we use the project directory structure. We B. Analysis assume that client-side code is stored in a directory named To address the three research questions, we have developed www, public, static,orclient. We also identify client-side a static analyzer to search for different patterns of callbacks code through developer code annotations (e.g., /* jshint in JavaScript code4. browser:true, jquery:true */). Our static analyzer builds on prior JavaScript analysis 1 function getRecord(id, callback) { 2 http.get(’http://foo/’ +id,function (err, doc) { tools, such as Esprima [15] to parse and build an AST, 3 if (err) { Estraverse [33] to traverse the AST, and TernJS [14], a type 4 return callback(err); 5 } inference technique by Hackett and Guo [13], to query for 6 return callback(null, doc); function type arguments. We also developed a custom set of 7 }); 8 } analyses to identify callbacks and to measure their various properties of interest. 10 var logStuff = function () { ... } 11 getRecord(’007’, logStuff); Duplicate code is an issue for any source code analysis tool that measures the prevalence of some language feature. Our Listing 4. An example of an asynchronous anonymous callback. analysis maintains a set of dependencies for a subject system For example, consider the code in Listing 4. To check and guarantees that each dependency is analyzed exactly once. if the getRecord() function invocation in line 11 takes To resolve dependencies expressed with the require keyword, a callback, we analyze its definition in line 1. We find we use TernJS and its Node plugin. that there is a path from the start of getRecord() to the In the rest of this section we detail our analysis for each of callback invocation (which happens to be the logStuff the three research questions. argument provided to the getRecord() invocation). The path Prevalence of callbacks (RQ1). To investigate the prevalence is: getRecord() −→ htt p.get() →− Anonymous1() →− callback(). of callbacks in JavaScript code we consider all function defi- Therefore, logStuff() is a callback function because it is nitions and function callsites in each subject system. For each passed as a function argument and it is invoked in that function. subject, we compute (1) the percentage of function definitions This makes getRecord() a callback-accepting function. that accept callbacks as arguments, and (2) the percentage of We label a callsite as callback-accepting if it corresponds function callsites that accept callbacks as arguments. to (1) a function that we determine to be callback-accepting, We say that f is a callback-accepting function if we find as described above, or (2) a function known a-priori to that at least one argument to f is used as a callback. be callback-accepting (e.g., setTimeout()). The getRecord To determine whether a parameter p of a function f defini- callsite in line 11 is callback-accepting because the function tion is a callback argument, we use a three-step process: (1) if getRecord was determined to be callback-accepting. p is invoked as a function in the body of f then p is a callback; Callback usage in practice (RQ2). Developers use call- (2) if p is passed to a known callback-accepting function (e.g., backs in different ways. Some of these are known to be problematic [26] for comprehension and maintenance (e.g., 3 https://nodejs.org 4Our static analysis approach considers most program paths, but it does not 5For instance, JavaScript files under the node_modules directory are ex- handle cases like eval that require dynamic analysis. cluded. see section II). We characterize callback usage in three ways: 1 define(’admin/general/dashboard’, ’semver’, function(semver we compute (1) the percentage of callbacks that are anony- ){ mous versus named, (2) the percentage of callbacks that are 2 var Admin = {}; asynchronous, and (3) the callback nesting depth. 4 $(’#logout -link’).on(’click’, function() { 5 $.post(RELATIVE_PATH + ’/logout’, function() { Anonymous versus named callbacks. If a function callsite is 6 window.location.href = RELATIVE_PATH + ’/’; identified as callback-accepting, and an anonymous function 7 }); expression is used as an argument, we call it an instance of 8 }); an anonymous callback. 10 ...

Asynchronous callbacks. A callback passed into a function 12 $(’.restart’).on(’click’, function() { can be deferred and invoked at a later time. This deferral 13 bootbox.confirm(’Are you sure you wish to restart NodeBB?’, function(confirm) { happens through known system APIs, which deal with the 14 if (confirm) { task queue in common browsers and Node.js environments. 15 $(window).one(’action:reconnected’, function() { 16 app.alert({ alert_id: ’instance_restart’,}); Our analysis detects a variety of APIs, including DOM events, 17 }); network calls, timers, and I/O. Table II lists examples of these 19 socket.emit(’admin.restart’); asynchronous APIs. 20 } TABLE II 21 }); 22 }); EXAMPLE ASYNCHRONOUS APIS AVA I L A B L E TO JAVA SCRIPT PROGRAMS 23 return Admin; 24 }); Category Examples Availability Listing 5. Example of multiple nested callbacks in one function. DOM events addEventListener, onclick Browser Network calls XMLHTTPRequest.open Browser Error-first callbacks. To detect error-first callbacks (see section II) we use a heuristic. We check if the first parameter p Timers setImmediate(), Browser, of a function f definition has the name ‘error’or‘err’. Then, (macro-Task) setTimeout(), Node.js we check if f ’s callsites also contain ‘error’or‘err’ as their setInterval() first argument. Thus, our analysis counts the percentage of Timers process.nextTick() Node.js function definitions that accept an error as the first argument, (micro-task) as well as the percentage of function callsites that are invoked with an error passed as the first argument. I/O APIs of fs, net Node.js Async.js. If a subject system has a dependency on Async.js (e.g., in their package.json), we count all invocations of the For each function definition, if a callback argument is passed Async.js APIs in the subject’s code. into a known deferring function call, we label this callback as Promises. We count instances of Promise creation and asynchronous. consumption for each subject system. If a new expression Callback nesting. The number of callbacks nested inside one returns a Promise object (e.g., new Promise()), it is counted after the other is defined as the callback depth. According to as a Promise creation. Invocations of the then() method this definition, Listing 1 has a callback depth of three because on a Promise object are counted as Promise consumption — of callbacks at lines 2, 3, and 5. this method is used to attach callbacks to a promise. These In cases where there are multiple instances of nested callbacks are then invoked when a promise changes its state callbacks in a function, we count the maximum depth of (i.e., evaluates to success or an error). nesting, including nesting in conditional branches. This under- approximates callback nesting. For example, Listing 5 shows IV. RESULTS an example code snippet from an open source application6. There are two sets of nested callbacks in this example, namely, A. Prevalence of Callbacks (RQ1) at Lines 1, 4, 5 (depth of three) and a second set at Lines 1, 12, In the subject systems that we studied, on average, 10% 13, 15 (depth of four). Our analysis computes the maximum of all function definitions and 19% of all function callsites nesting depth for this function, which is four. were callback-accepting. Figure 2 depicts the percentage of Handling callbacks (RQ3). There are a number of solutions callback-accepting function definitions and callsites, per cate- to help with the complexities associated with callbacks. We gory of systems (Table I), and in aggregate. The figure also consider three well-known solutions: (1) the prevalence of the shows how these are distributed across client-side and server- error-first callback convention, (2) the prevalence and usage side, indicating that server-side code generally contains more of Async.js [21], a popular control flow library, and (3) the functions that take callbacks than client-side code. prevalence of Promises [6], a recent language extension, which Finding 1: On average, every 10th function definition takes a provides an alternative to using callbacks. For each solution, callback argument. Callback-accepting function definitions we characterize its usage — are developers using the solution are more prevalent in server-side code (10%) than in client- and to what extent. side code (4.5%). 6https://github.com/NodeBB/NodeBB ebleetemr xesv sg fclbcsi server- in callbacks of usage extensive more the believe We esuididctsta rga nlsstcnqe that techniques analyses program that indicates studied we goetepeec faycrn r nplcbeadmay and inapplicable are asynchrony of presence the ignore hti vnulypse oa snhoosAIcl like call API asynchronous an to passed eventually is that (RQ2) Callbacks Asynchronous — Usage Callback B. community Node.js inception. continuation-passing-style the its in the from advocated to was that attributed programming be of can code side others. than code fact server-side the more to contain ascribed categories be some can that differences these however, inter- that large, believe The not we were and functions. usage such func- callback in of callback-accepting differences invocations category both and of definitions fraction tion category higher applications web a the contained example, For degree higher usage. a callback contain of categories Some considered. we gories Implications. Implications. alie e aeoy coscin/evr n ntotal. in and and definitions client/server, function across callback-accepting category, of per percentage callsites for Boxplots 2. Fig. setTimeout() evrsd,aycrnu alak sg s5%o average. on the 55% is On usage asynchronous. callbacks were with all asynchronous 72% Of callsites server-side, code, code. client-side client-side of usage in in that higher 56% callsites find is We callbacks of asynchronous categories. mean of server-side into and a data client-side the partitions and the also figure 56% The callbacks. was of asynchronous there median applications all a Across callsites. function accepting idn 3 Finding 2 Finding snhoosclbcs o callbacks, Asynchronous oepeaeti evrsd oe(4)ta nclient-side in than (9%). (24%) code code server-side in prevalent more ncin-iecd 7% hnsre-iecd (55%). code server-side than (72%) code client-side in alakagmn.Clbc-cetn function Callback-accepting argument. callback sarmne,a snhoosclbc sacallback a is callback asynchronous an reminder, a As iue3sostepeaec faycrnu callback asynchronous of prevalence the shows 3 Figure Percentage 20 40 60 80 0 oeta afo l alak r asynchronous. are callbacks all of half than More : naverage, On : DataViz Definitions Callsites h xesv s faycrn ntesubjects the in asynchrony of use extensive The alak r tlzdars l ujc cate- subject all across utilized are Callbacks sesbeto I- o oedetails). more for III-B subsection (see Engines

Frameworks vr 5 every average, n Games

WebApps th ucinclst ae a takes callsite function

permr frequently more appear NPM

Client

Server callsites

Total are ihti.W hn httepolmo epn eeoesrea- developers helping of problem the that think We this. with i.3 oposfrpretg faycrnu alakacpigfunction callback-accepting asynchronous of percentage for callsites. Boxplots 3. Fig. evrsd oecnan lgtyhge ecnaeof percentage higher code. slightly client-side that than a indicates callbacks contains anonymous partitioned and data code, code same server-side server-side the and shows client-side also understand between 4 to con- Figure difficult high maintain. are fairly and callbacks is anonymous which 48%, that to sidering 23% the from across ranges percentage categories median The callsites. function accepting (RQ2) Callbacks Anonymous — Usage Callback C. community. research deserves the server-side, from the attention and more client-side the on asynchronous both containing help callbacks, bases code to JavaScript large tools about few son are there impedes Yet and [8]. flow comprehension control program style program programming complicates asynchronous significantly The us. surprised callbacks for account must JavaScript of Analyses asynchrony. results. poor to lead function total. callback-accepting in anonymous and of client/server, percentage across category, for per Boxplots callsites 4. Fig. iue4sostepeaec faoyoscallback- anonymous of prevalence the shows 4 Figure with paired scheduling asynchronous of use extensive The Percentage Percentage 100 100 20 40 60 80 20 40 60 80 0 0

DataViz DataViz

Engines Engines

Frameworks Frameworks

Games Games

WebApps WebApps

NPM NPM

Client Client

Server Server

Total Total Finding 4: Over 43% of all callback-accepting function 1 $(document).ready(function () { callsites are invoked with at least one anonymous callback. 2 $(’.star’).click(function (e) { There is little difference between client-side and server-side 3 ... 4 }) code in the extent to which they use anonymous callbacks. 5 }) Listing 6. Example of a nested callback introduced by the wrapping Implications. This finding indicates that in a large fraction of $(document).ready() function from the jQuery library. cases, a callback is used once (anonymously) and is never reused again. It seems that developers find anonymous callbacks useful in spite of the associated comprehension, debugging, for example, for module exporting, loading, or to wait for the and testing challenges. We think that this topic deserves further DOM to be fully loaded on the client-side. Due to the extra study — it is important to understand why developers use callback surrounding the project code, in these type of projects, anonymous callbacks and prefer them over named callbacks. callbacks begin nesting at level 2. Listing 6 lists an example of Possible reasons for using anonymous callbacks could be code this kind of nesting with the $(document).ready() function brevity, or creating temporary local scopes (e.g., in closures). (line 1) from the popular jQuery library. This function waits We also think that the high fraction of anonymous callbacks for the DOM to load. It increases the callback nesting in the indicates that this popular language feature is here to stay. rest of the code by 1 (e.g., the callback on line 2 has a nesting Therefore, it is worthwhile for the research community to level of 2). invest time in developing tools that will support developers in handling anonymous callbacks. E. Solutions — Error-first Protocol (RQ3) We found that 20% of all function definitions follow the D. Callback Usage — Nested Callbacks (RQ2) error-first protocol. The median percentage across the cat- Figure 5 presents our results for the total number of in- egories ranges from 4% to 50%. The fraction of function stances of nested callbacks at each observed nesting level. definitions that adhere to the error-first protocol is almost twice We found that the majority of callbacks nest two levels deep. as high in the server-side code (30%) than in the client-side Figure 5 shows this unusual peak at nesting level of 2. We code (16%). In addition, the error-first protocol was the most also found that callbacks are nested up to a depth of 8 common solution among the three solutions we considered. (there were 29 instances of nesting at this level). In these For example, 73% (63 out of 86) NPM modules and 93% (15 extreme cases developers compose sequences of asynchronous out of 16) web applications had instances of the error-first callbacks with result values that flow from one callback into protocol. the next. These extreme nesting examples are available as part th of our dataset [5]. Finding 6: Overall, every 5 function definition adheres to the error-first protocol. The error-first protocol is used twice as often in server-side code than in client-side code

Implications. Although we found that a non-trivial fraction of JavaScript functions rely on the error-ﬁrst protocol, it remains an ad-hoc solution that is loosely applied. The relatively low

No. of instances No. and highly variable use of the error-first protocol means that developers must check adherence manually and cannot depend 0 8,000 16,00012345678 24,000 Nesting Level on APIs and libraries to enforce it. Such idiom-based strategies for handling exceptions are known to be error-prone in other Fig. 5. Instances of nested callbacks for a particular nesting level. languages, such as C [10]. It would be interesting to study if this is also the case for JavaScript functions that follow the Finding 5: Callbacks are nested up to a depth of 8. There error-first callback idiom. is a peak at nesting level of 2. F. Solutions — Async.js (RQ3) Implications. As with anonymous and asynchronous call- To study Async.js, we considered subject systems that use backs, callback nesting taxes code readability and compre- this library. We found that only systems in the web applications hension. We find that nesting is widely used in practice and and NPM modules categories used this library. Our results note that developers lack tools to manage callback nesting. show that 9 of 16 (56%) web applications and just 9 of We believe that there is ample opportunity in this area for 85 (11%) NPM modules use the Async.js library to manage tool builders and software analysis experts. The number of asynchronous control flow. Table III shows the top 10 used instances decreases from level 1 to 8, except at level 2. Based functions from the Async.js API (by number of callsites) for on our investigation of numerous level 2 callback nesting ex- these two categories of subject systems. amples, we believe that the peak at level 2 is due to a common This table indicates that the sets of functions used in the JavaScript practice in which project code is surrounded with top 10 list are similar. But, there are notable differences: for an anonymous function from an external library. This is used, example, nextTick was the most used Async.js method in TABLE III Finding 7: More than half of the web applications (56%) TOP 10 ASYNC.JS INVOKED METHODS IN JAVA SCRIPT WEB APPLICATIONS ∗ use the Async.js library to manage asynchronous control (LEFT) AND NPM MODULES (RIGHT). THE SYMBOL DENOTES CALLS flow. The usage is much lower (11%) in the NPM modules. THAT DO NOT APPEAR IN BOTH TABLES. In addition, the Async.js library is used differently (rank Rank Method Count Rank Method Count variance of 23.8) in these two categories of subject systems. 1 nextTick 18 1 parallel 189 2 queue∗ 16 2 apply 81 3 each 14 3 waterfall 72 Implications. Libraries, such as Async.js, provide one means 3 setImmediate∗ 14 4 series 61 of coping with the complexity of callbacks. However, because 3 series 14 5 each 48 library solutions are not provided natively by the language 6auto∗ 11 6map 37 6 waterfall 11 7 eachSeries∗ 20 runtime, the developer community can be divided on which 6 parallel 11 8 eachLimit∗ 12 library to use, especially as there are many alternatives (see ∗ 9map 10 9 whilst 10 section II). We think that the difference in Async.js API usage 9 apply 10 9 nextTick 10 by developers of web applications (that include both client- side code and server-side code) and NPM modules (exclu- sively server-side code) indicates different underlying concerns around callbacks and their management. We think that this th web applications and just the 9 most used method in NPM usage deserves further study and can inform the design of modules. The nextTick method in Async.js is used to delay future libraries and proposals for language extensions [18], the invocation of the callback until a later tick of the event [22], [7]. loop, which allows other events to precede the execution of the callback. In Node.js code the nextTick is implemented using G. Solutions — Promises (RQ3) the process.nextTick() method in the runtime. In browsers Table IV shows the percentage of subjects that create this call is implemented using setImmediate(callback) or promises and the percentage of subjects that use promises. setTimeout(callback, 0). In this case Async.js provides a single interface for developers to achieve the same functional- TABLE IV ity in both client-side and server-side code. PERCENTAGE OF SUBJECT SYSTEMS CREATING AND USING PROMISES In NPM modules parallel is the most widely used Category Subjects creating Subjects using Async.js method (it is the 6th most popular among web Promises (%) Promises (%) applications). This call is used to run an array of independent DataViz libraries 6 31 functions in parallel. Game Engines 0 25 Frameworks 50 75 As a final example, the second-most used call in web- Games 0 17 applications, queue, does not appear in the top ten calls used Web Applications 13 50 by NPM modules. The queue method functionality is similar NPM Modules 3 12 to that of parallel, except that tasks can be added to the Total 8 26 queue at a later time and the progress of tasks in the queue Figure 6 shows box plots for the number of Promise creating can be monitored. instances using new Promise() constructs and Promise usage, We should note that because JavaScript is single-threaded, e.g., then(), in the different subject categories, partitioned both the parallel and queue Async.js calls do not expose across client/server, and in total. It should be noted that not true parallelism. Here, parallel execution means that there may all application access a Promise through the new Promise() be points in time when two or more tasks have started, but have statement; some invoke library functions that return Promises. not yet completed. In aggregate, we found that 37 of 138 (27%) applications use Promises. They were predominantly used by client-side There are significant differences between web applications frameworks (75%), with a maximum of 513 usage instances and NPM modules in terms of the Async.js API usage. To (across all frameworks) and a standard deviation of 343 usage characterize this difference, we first ranked the API func- instances. In all the other subject systems, usage of Promises tions according to the number of times they were used in was rare, with a mean close to zero. There was one outlier, web applications and NPM modules. Then we analyzed the the bluebird NPM module7, that had 2,032 Promise usage difference between the ranks of each function in the two instances. This module implements the Promises specification categories. For example, the rank of nextTick is 9 and 1 in as a library. We therefore omit it from our results. the NPM modules and web applications, respectively, making the absolute difference 8. Overall, the rank differences had a Finding 8: 27% of subject systems use Promises. This mean of 6.2, a median of 5.5, and a variance of 23.8. This usage is concentrated in client-side code, particularly in indicates that the Async.js library is used differently in these JavaScript frameworks. two categories of subject systems. 7 https://www.npmjs.com/package/bluebird as o xml,w on alaknsigb aigthe taking by nesting callback count we example, For ways. esuid hscudb eas rmssi eaieynew relatively a is Promises because be could This studied. we eair htw a nlz.Freape ed o handle not do we example, in For code analyze. can we that behaviors of instances can of This number callbacks. function. the nested of a approximation for under nesting an provide callback of depth maximum o xml,w ontcnie rjcsta s JavaScript use that source projects mono-lingual. open consider primarily not from are do we comes and example, JavaScript consider it For we use Second, First, maturity. that and representative, ways. projects size particular be of a not of number projects might a sample our in code, JavaScript of eeoest irt xsiglrepoet ouePromises. use to projects large existing help migrate would to Tools Promises developers into code. callbacks JavaScript refactor maintainable automatically to to that more lead interesting and Promises be whether quality and would higher evolves It adoption this Node.js. how and study browsers to systems addition the not among have Promises we of handling, related uptake error challenges significant and a the nesting observed as of such many callbacks, resolving to to approach based by Implications. instances creation and usage Promise total total. in of and distribution client/server, across The category, 6. Fig. shnst dniycin-ieadsre-iecode. server-side and client-side also identify We annotations to it. code follows hints use the properly as and that conventions it naming mean that directory on or necessarily rely protocol, not the adherence, does over-counting uses name be code also argument argument may an different we since a And, but used. followed is is name missing protocol by the adherence where protocol as cases under-counting function be name may a will we of protocol that argument error-first naming first the a to the uses adhering analysis code callback heuristic: error-first The conventions. u td.W vriwteetrasi hssection. this in threats these overview We study. our u nlssaesai.Ti iisteknso JavaScript of kinds the limits This static. are analyses Our edcddt on etrso alakuaei particular in usage callback of features count to decided We hr r ohitra n xenltrast aiiyfor validity threats. to Internal threats external and internal both are There xenlthreats. External No. of instances 0 100 200 300 400 500

eval DataViz

ttmnsi u study. our in statements Engines lhuhPoie sapoiiglanguage- promising a is Promises Although .T V.

Frameworks u nlssrl nsm development some on rely analyses Our lhuhw td vr5mlinlines million 5 over study we Although RAST VALIDITY TO HREATS

Games

WebApps Creation Usage

err NPM or

error Client

Server hetis threat A .

Total a tde yFruae l [12]. al. et Fortuna by studied was snhoosprogramming. Asynchronous ol oe iia td a odce yMrisnet Martinsen by conducted was study similar A code. world hyfound They hyfudta omnymd supin about assumptions made commonly that found They iltn nomto os[7.Prleimi aacitcode JavaScript in Parallelism [17]. flows information violating aacitapplications. JavaScript aacitapiain a tde yRcad ta.[30]. al. et Richards by studied was applications JavaScript ic h ujc ytm r l pnsuc,orsuyshould study our source, open repeatable. all be are systems other subject systems. to the since subject lead of will variety it broader that a hope consider we and that kind, studies worthwhile, its is usage of practice callback JavaScript study in of first study the characterization represent our as JavaScript result, believe study and of our a categories in types different As systems other five server-side. subject to the the generalize However, not on projects. may Java findings and our client-side the on in 2] 3] rs-iesrpig(S)[4,adprivacy- and [34], (XSS) scripting cross-site inclu- JavaScript [35], remote [24], on sions studies include Examples studied. code. both JavaScript in server-side traction usage and callback gained client- considers has study Node.js, Our developers. study, of among this top Since on bugs. JavaScript, JavaScript server-side client-side of and causes equivalent root ize with replaced extensions. be language or could code safer it scenarios, usage oee,t h eto u nweg,teehv enno been practice. have in there usage knowledge, callback [26]. our of developers of studies among empirical best topic is- the discussion to callback-related recurrent However, and a are applications sues real-world in language xml,w hrceieteuaeo alaknsigand nesting callback For of deeper. usage delves the and characterize code we JavaScript example, considers However, work such programming. features our asynchronous language similarly enhance new to study and Promises Our callbacks as of constructs. usage language (mis)use the developers new covers how these studied of have some They practice. in used euneo us hc sasmlrvrino Promises. of a version into simpler a continuations, is of which converting imbrication Dues, for of an compiler sequence a or propose callbacks, [9] al. nested et to Brodu according context. function its anonymous detected the any parses De- names It and de-anonymizer. code. nested code function client-side JavaScript included a in is they callbacks [11] on smells, cofun focus JavaScript only of but list callbacks, their In code. snhoosiim r evl sd u e dosthat idioms the new of but advantage used, take heavily can callback-based are pro- that asynchronous idioms found of They asynchronous usage applications. C# the in on gramming study large-scale a ducted l 2] ihrse l 2]suidtepeaec of prevalence the studied real- [29] some al. et least Richards at [20]. in al. violated are JavaScript in dynamism l u miia aaadtostaepbil available; publicly are toolset and data empirical our All euiyvleaiiisi aacitcd aeas been also have code JavaScript in vulnerabilities Security character- to study empirical an conducted [25] al. et Ocariza eerhr aesuidvrosapcso h JavaScript the of aspects various studied have Researchers iaiFr ta.[3 tde oesel nJavaScript in smells code studied [23] al. et Fard Milani eval ob evsv,adage hti most in that argued and pervasive, be to I R VI. ELATED async/await h yai eaiu of behaviour dynamic The kre l 2]rcnl con- recently [27] al. et Okur W ORK ewrsaerarely are keywords eval . anonymous callbacks, which are known to cause maintenance [11] D. M. Clements. Decofun. problems. https://github.com/davidmarkclements/decofun. [12] E. Fortuna, O. Anderson, L. Ceze, and S. Eggers. A limit study of Concurrency bug detection. EventRacer [28] detects data JavaScript parallelism. In Proc. of Intl. Symposium on Workload races in JavaScript applications. Zheng et al. [36] propose Characterization (IISWC), pages 1–10, 2010. [13] B. Hackett and S.-y. Guo. Fast and Precise Hybrid Type Inference for a static analysis method for detecting concurrency bugs JavaScript. In Proc. of the Conf. on Programming Language Design caused by asynchronous calls in web applications. Similarly, and Implementation (PLDI), pages 239–250. ACM, 2012. WAVE [16] is a tool for identifying concurrency bugs by [14] M. Haverbeke. Tern. https://github.com/marijnh/tern, 2015. [15] A. Hidayat. Esprima. https://github.com/jquery/esprima, 2015. looking for the same sequence of user events leading to [16] S. Hong, Y. Park, and M. Kim. Detecting Concurrency Errors in different final DOM-trees of the application. Client-Side JavaScript Web Applications. In Proc. of Intl. Conf. on Software Testing, Verification and Validation (ICST), pages 61–70. Program comprehension. Clematis [8] is a technique for IEEE, 2014. helping developers understand complex event-based and asyn- [17] D. Jang, R. Jhala, S. Lerner, and H. Shacham. An empirical study of chronous interactions in JavaScript code by capturing low- privacy-violating information flows in JavaScript web applications. In Proc. of Conf. on Comp. and Communications Security, pages level interactions and presenting those as higher-level behav- 270–283. ACM, 2010. ioral models. Theseus [19] is an IDE extension that helps [18] Y. P. Khoo, M. Hicks, J. S. Foster, and V. Sazawal. Directing developers to navigate asynchronous and dynamic JavaScript JavaScript with arrows. ACM Sigplan Notices, 44(12):49–58, 2009. [19] T. Lieber. Theseus: understanding asynchronous code. In Proc. of the execution. Theseus has some limitations; for example, it does Conf. on Human Factors in Computing Systems, pages 2731–2736. not support named callbacks. Our study demonstrates that ACM, 2013. over half of all callbacks are named, indicating that many [20] J. Martinsen, H. Grahn, and A. Isberg. A comparative evaluation of JavaScript execution behavior. In Proc. of Intl. Conf. on Web applications will be negatively impacted by similar limitations Engineering (ICWE), pages 399–402. Springer, 2011. in existing tools. [21] C. McMahon. Async.js. https://github.com/caolan/async. [22] L. A. Meyerovich, A. Guha, J. Baskin, G. H. Cooper, M. Greenberg, VII. CONCLUSIONS A. Bromfield, and S. Krishnamurthi. Flapjax: A Programming All modern JavaScript applications that handle and re- Language for Ajax Applications. In Proc. of the Conf. on Object Oriented Programming Systems Languages and Applications spond to events use callbacks. However, developers are fre- (OOPSLA), pages 1–20, New York, NY, USA, 2009. ACM. quently frustrated by “callback hell” — the comprehension and [23] A. Milani Fard and A. Mesbah. JSNose: Detecting JavaScript Code maintainability challenges associated with nested, anonymous Smells. In Proc. of the Intl. Conf. on Source Code Analysis and Manipulation (SCAM), pages 116–125. IEEE, 2013. callbacks and asynchronous callback scheduling. This paper [24] N. Nikiforakis, L. Invernizzi, A. Kapravelos, S. Van Acker, W. Joosen, presents an empirical study of callbacks usage in practice. We C. Kruegel, F. Piessens, and G. Vigna. You are what you include: study over 5 million lines of JavaScript code in 138 subject Large-scale evaluation of remote JavaScript inclusions. In Proc. of the Conf. on Computer and Comm. Security, pages 736–747. ACM, 2012. systems that span a variety of categories. We report on the [25] F. Ocariza, K. Bajaj, K. Pattabiraman, and A. Mesbah. An Empirical prevalence of callbacks, their usage, and the prevalence of Study of Client-Side JavaScript Bugs. In Proc. of the ACM/IEEE Intl. solutions that help to manage the complexity associated with Symposium on Empirical Software Engineering and Measurement (ESEM), pages 55–64. IEEE, 2013. callbacks. We hope that our study will inform the design of [26] M. Ogden. Callback Hell. http://callbackhell.com, 2015. future JavaScript analysis and code comprehension tools. Our [27] S. Okur, D. L. Hartveld, D. Dig, and A. v. Deursen. A Study and analysis [4] and empirical results [5] are available online. Toolkit for Asynchronous Programming in C#. In Proc. of the Intl. Conf. on Software Engineering (ICSE), pages 1117–1127. ACM, 2014. ACKNOWLEDGMENTS [28] V. Raychev, M. Vechev, and M. Sridharan. Effective race detection for event-driven programs. In Proc. of the Intl. Conf. on Object Oriented This work was supported in part by an NSERC Strategic Programming Systems Languages & Applications (OOPSLA), pages Grant and the Institute for Computing, Information and Cog- 151–166. ACM, 2013. [29] G. Richards, C. Hammer, B. Burg, and J. Vitek. The eval that men do. nitive Systems (ICICS) at UBC. In Proc. of the European Conf. on Object-Oriented Programming (ECOOP), pages 52–78. Springer, 2011. REFERENCES [30] G. Richards, S. Lebresne, B. Burg, and J. Vitek. An Analysis of the [1] Error Handling in Node.js. Dynamic Behavior of JavaScript Programs. In Proc. of the Conf. on https://www.joyent.com/developers/node/design/errors, 2014. Programming Language Design and Implementation (PLDI), pages [2] Github Showcases. https://github.com/showcases, 2014. 1–12. ACM, 2010. [3] Most depended-upon NMP packages. [31] Stack Overflow. Developer Survey. https://www.npmjs.com/browse/depended, 2014. http://stackoverflow.com/research/developer-survey-2015, 2015. [4] CallMeBack. https://github.com/saltlab/callmeback, 2015. [32] G. J. Sussman and G. L. Steele Jr. Scheme: A interpreter for extended [5] Don’t Call Us, We’ll Call You: Characterizing Callbacks in JavaScript. lambda calculus. Higher-Order and Symbolic Computation, Dataset release. http://salt.ece.ubc.ca/callback-study/, 2015. 11(4):405–439, 1998. [6] Promises/A+ Promise Specification. https://promisesaplus.com, 2015. [33] Y. Suzuki. Estraverse. https://github.com/estools/estraverse, 2015. [7] TypeScript. http://www.typescriptlang.org, 2015. [34] J. Weinberger, P. Saxena, D. Akhawe, M. Finifter, R. Shin, and [8] S. Alimadadi, S. Sequeira, A. Mesbah, and K. Pattabiraman. D. Song. An Empirical Analysis of XSS Sanitization in Web Understanding JavaScript Event-based Interactions. In Proc. of the Intl. Application Frameworks. Technical Report EECS-2011-11, UC Conf. on Software Engineering (ICSE), pages 367–377. ACM, 2014. Berkeley, 2011. [9] E. Brodu, S. Frénot, and F. Oblé. Toward Automatic Update from [35] C. Yue and H. Wang. Characterizing insecure JavaScript practices on Callbacks to Promises. In Proc. of the Workshop on All-Web Real-Time the web. In Proc. of Intl. Conf. on World Wide Web (WWW), pages Systems (AWeS), pages 1:1–1:8, New York, NY, USA, 2015. ACM. 961–970. ACM, 2009. [10] M. Bruntink, A. van Deursen, and T. Tourwé. Discovering Faults in [36] Y. Zheng, T. Bao, and X. Zhang. Statically locating web application Idiom-based Exception Handling. In Proc. of the Intl. Conf. on bugs caused by asynchronous calls. In Proc. of the Intl. Conf. on Software Engineering (ICSE), pages 242–251. ACM, 2006. World Wide Web (WWW), pages 805–814. ACM, 2011.