Poisoning the Event-Driven Architecture: A Full-Stack Look at Node.js and Friends Anonymous Author(s) Abstract induced by each additional thread artificially limits the number of In recent years, both industry and the open-source community have clients each server can handle, requiring the service provider to op- turned to the Event-Driven Architecture (EDA) to provide more erate more servers than is strictly necessary. The EDA significantly scalable web services. Though the Event-Driven Architecture can reduces the overhead paid for each additional client, multiplying offer better scalability than the One Thread Per Client Architecture, the number of clients each server can handle. its use can leave service providers vulnerable to a Denial of Service Figure 1 gives some intuition about how the EDA achieves its attack that we call Event Handler Poisoning (EHP). low per-client overhead. Rather than creating a new thread for each In this work we define EHP attacks and examine the ways in request, which introduces overheads in time (context-switching) which EDA frameworks expose applications to this risk. We also and space (per-thread stack), the EDA simply stores the request as touch on application-level EHP vulnerabilities. We focus on three an Event in a queue to be handled by a fixed set of Event Handlers. In popular EDA frameworks: Node.js (JavaScript), Twisted (Python), other words, the EDA achieves scalability using thread multiplexing. and libuv (C/C++). By multiplexing these unrelated computations onto the same Event Our analysis studies both CPU-bound and I/O-bound approaches Handlers, the EDA amortizes the space and time overhead of each to Event Handler Poisoning. For CPU-bound attacks we focus on additional client. Since each client costs less in the EDA, EDA-based Regular Expression Denial of Service (ReDoS). For I/O-bound ap- services can handle a larger number of clients concurrently. proaches we examine the weaknesses of asynchronous I/O imple- mentations in these frameworks. Using our Constant Worst-Case Execution Time pattern, EDA- based services can become invulnerable to EHP attacks. Guided by this pattern, in our system node.cure we implement two framework- level defenses for Node.js that serve as effective antidotes to the concerns we raise. CCS Concepts • Security and privacy → Denial-of-service attacks; Software security engineering; Web application security; Keywords Figure 1: This is the Event-Driven Architecture. Incoming Event-Driven Architecture; Denial of Service; Node.js; Algorithmic events are handled sequentially by the event loop. The Event Complexity; ReDoS Loop may generate new events in the form of tasks it of- floads to the Worker Pool. We refer to the event loopand the workers in the worker pool as Event Handlers. 1 Introduction Web servers are the lifeblood of the modern Internet. To handle Alas, the scalability promised by the EDA comes at a price. Mul- the demands of millions of clients, service providers replicate their tiplexing unrelated work onto the same thread(s) may reduce over- services across many servers. Since each additional server costs head, but it also moves the burden of time sharing out of the thread time and money, service providers want to maximize the number of library or operating system and into the application itself. Where clients each server can handle. Over the past decade, this goal has OTPCA-based services can rely on the preemptive multitasking led the software community to seriously consider a paradigm shift promised by the underlying system to ensure that resources are in their software architecture — from the One Thread Per Client shared fairly, using the EDA requires the service to enforce its own Architecture (OTPCA) to the Event-Driven Architecture (EDA) (also cooperative multitasking [71]. In other words, thread multiplex- known as the reactor pattern). Though using the EDA may increase ing destroys isolation. For example, if a client’s request takes a the number of clients each server can handle, it also exposes the long time to handle in an OTPCA-based service, preemptive mul- service to a class of Denial of Service (DoS) attack unique to the titasking will ensure that other requests are handled fairly. But if EDA: Event Handler Poisoning (EHP). this same lengthy request is submitted to an EDA-based service, Traditionally, web services have been implemented using the and if the service incorrectly implements cooperative multitasking, One Thread Per Client Architecture (OTPCA), and support more then this request will deny service to all other clients because they clients by increasing the number of threads. In practice the overhead cannot use the Event Handler that is processing the request. We call this state of affairs Event Handler Poisoning. Anonymous Submission to ACM CCS 2017, Due 19 May 2017, Dallas, Texas 2017. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 We present a general principle, borrowed from the real-time https://doi.org/10.1145/nnnnnnn.nnnnnnn systems community, that EDA-based services can use to defend 1 Anonymous submission #23 to ACM CCS 2017 against EHP. As EHP attacks are possible when one request to an Although EDA approaches have been discussed and implemented EDA-based server doesn’t cooperatively yield to other requests, in the academic and professional communities for decades (e.g. [30, our antidote is simple: the server must ensure that every stage 42, 48, 69, 74, 77]), historically EDA has only seen wide adoption in of the request handling takes no more than constant time before user interface settings like web browsers. How things have changed. it yields to other requests. In applying this principle, EDA-based The EDA is increasingly relevant in systems programming today servers extend input sanitization from the realm of data into the thanks to the explosion of interest in Node.js [12], an event-driven realm of the algorithms used to process it. We evaluate several server-side JavaScript framework. Although the OTPCA and EDA popular EDA frameworks in light of this principle, identifying EHP architectures have the same expressive power [55], in practice EDA vulnerabilities in each. Our node.cure prototype extends Node.js to implementations have been found to offer greater scaling [67]. defend against the vulnerabilities we identified there. Several variations on the server-side EDA have been proposed: This paper is an extension of a workshop paper [28]. There we single-process event-driven (SPED), asymmetric multi-process event- introduced the idea of EHP attacks, argued that some previously- driven (AMPED), and scalable event-driven (SEDA). The AMPED reported security vulnerabilities were actually EHP vulnerabilities, architecture is depicted in Figure 1, while in SPED there is no worker and gave a sketch of our antidote. Here we offer the complete pool [66]. In the SEDA architecture, applications are designed as a package: a refined definition of EHP, a general principle for defense, pipeline composed of stages, each of which resembles Figure 1 [77]. examples of vulnerabilities in the wild, and effective framework- Figure 1 illustrates the widely used AMPED EDA formulation, level defenses. which is common to all mainstream general-purpose EDA frame- In summary, this paper makes the following contributions: works today. In the AMPED EDA (hereafter simply “the EDA”) the operating system or a framework places events in a queue (e.g. (1) We are the first to define Event Handler Poisoning, an attack that through Linux’s poll system call1), and pending events are handled exploits a general vulnerability in the EDA (§3). Our definition by an event loop. The event loop may offload expensive activity to of EHP is general, and captures attacks targeting either the a small, fixed-size worker pool, which in turn generates an event event loop or the worker pool. to inform the event loop when an offloaded task completes. The (2) Using concepts borrowed from the real-time systems commu- worker pool is implemented using threads or separate processes, nity, we describe a general antidote for Event Handler Poisoning and offers services like “asynchronous” (blocking but concurrent) (§4). Our antidote gives developers precise guidance, replacing access to the file system or DNS. the dangerous heuristics currently offered in the EDA commu- In applications built using EDA, there is a set of possible Events nity. (e.g. “new network traffic”) for which the developer supplies asetof (3) We identify potential EHP vulnerabilities in software built using Callbacks [46]. During the execution of an EDA-based application, the popular Node.js EDA framework (§5). In doing so we extend events are generated by external activity or internal processing and the state of the art in ReDoS detection. routed to the appropriate Callbacks. Each Callback is executed on (4) Informed by our EHP antidote, we identify weaknesses in one of the Event Handlers; synchronous Callbacks are executed on Node.js itself that expose its community to EHP. In node.cure the event loop, asynchronous Callbacks on a worker. we implement effective solutions to these problems, defending Node.js applications against two forms of EHP: ReDoS (§5.2) 2.2 Popular EDA Frameworks and Large/Slow I/O (§6.3). An EDA approach can be (and has been) implemented in most pro- gramming languages. Table 1 lists the dominant2 general-purpose 2 Background EDA framework for a range of languages: Node.js (JavaScript) [12], In this section we contrast the EDA with the OTPCA (§2.1). We Twisted (Python) [19], libuv (C/C++) [9], Reactor (Java) [16], Event- then explain our choice of EDA frameworks for study (§2.2), and Machine (Ruby) [5], Zotonic (Erlang) [21], and Microsoft’s P# [41]. finish by explaining formative work on Algorithmic Complexity To capture the ongoing investment in each framework, we surveyed (AC) attacks (§2.3).