CRA: A Common Runtime for Applications MSR Technical Report, MSR-TR-2019-2 Badrish Chandramouli Umar Farooq Minhas Ibrahim Sabek∗ Microsoft Research Microsoft Research University of Minnesota [email protected] [email protected] [email protected] ABSTRACT graph. Dataflows can operate on both historical (bounded) Today, a modern data center hosts a wide variety of applica- and real-time (unbounded) datasets. In this paper, we start tions comprising batch, interactive, machine learning, and with the premise that such distributed stateful dataflows are streaming applications. A large majority of these applications a very general primitive, and encompass a wide range of can be abstracted as a “distributed dataflow” graph. Even applications deployed over the cloud and edge today. For though this commonality exists and can be exploited in the example, a scan-based analytics system reads data from stor- applications’ design and implementation, each application is age or in-memory caches (input vertices) and runs an ana- typically written from scratch, resulting in significant ineffi- lytics query using a dataflow of relational operator vertices. ciencies in developer productivity. In this paper, we factor out A streaming pipeline directly maps to an acyclic dataflow, the commonalities for many types of big data applications, whereas machine learning computations map to dataflows into a generic dataflow layer called Common Runtime for with iterative cycles in the dataflow graph. Consider an ap- Applications (CRA). In parallel, another trend, with container- plication in a completely different domain of microservices. ization technologies, has taken a serious hold on cloud-scale These systems consist of a dataflow of compute instances data centers, with direct implications on the design, imple- (actors) that communicate with one another using remote mentation, and deployment of next-generation of data-center procedure calls that directly map to a dataflow abstraction. application. Container engines (e.g., Docker and CoreOS) Similarly, Web and cloud-edge [24] applications consist of and cloud-scale container orchestrators (e.g., Kubernetes and client vertices that communicate with Web frameworks such Docker Swarm) are two important technologies that enable as PHP and NodeJS running on vertices in the cloud, which this trend. Container orchestrators have made deployment a communicates result requests back to the clients – another lot easy, and they solve many infrastructure level problems, instance of a distributed dataflow. e.g., service discovery, auto-restart, and replication. For best in class performance, there is a need to marry the next genera- 1.1 Problem and Today’s Solutions tion applications with containerization technologies. To that Given the importance of (distributed) dataflows across di- end, CRA leverages and builds upon containerization and verse application domains, there is a strong need to make resource orchestration capabilities of Kubernetes/Docker, it easy for application developers to author dataflows that and makes it easy to build a wide range of cloud-edge appli- are distributed, efficient, scalable, resilient, and potentially cations on top. To the best of our knowledge, we are the first long-running. Several frameworks and software layers have to present a cloud native runtime for building data center ap- been proposed by the data processing and systems communi- plications. To show the practicality of our approach, we built ties, in order to help create and deploy such applications. At a distributed analytics engine on top of CRA, namely Quill. the lowest level of the stack, we have raw virtual machines We show through in-depth micro- and macro-benchmark (e.g., in Infrastructure-as-a-Service cloud offerings) which are results, that CRA provides significant performance improve- hard to deploy, manage, and program in a distributed setting. ment over an unoptimized implementation on modern cloud Most applications instead use a layer above the machine: data platforms. CRA is available as open source, and can be down- intensive applications such as map-reduce have historically loaded at https://github.com/Microsoft/CRA. used YARN [3] as a resource manager, whereas microservices are deployed on Kubernetes [14] and Docker [12]. There is 1 INTRODUCTION increasing interest in running data-intensive applications on With the growth in data volumes acquired by businesses Kubernetes (often abbreviated as k8s) and Docker as well. today, there is a need to deploy rich dataflows over the data. Unfortunately, both YARN and Kubernetes offer bare- A dataflow consists of a graph of computation vertices that bones compute abstraction with no support for customized each hold state and read/write data to other vertices in the dataflows that are: (a) easy to deploy, (b) usable for offline and real-time dataflows or a mix of both, and (c) resilient to ∗Work started during internship at Microsoft Research. failures. Another option is to instead use specific dataflow systems such as Storm [8] to implement other applications. CRA has two layers of abstraction. First, users define a However, these layers do too much: they make too many logical dataflow topology as a directed graph of named ver- assumptions and choices that limit the general applicability tices. Vertices are associates with endpoints that may be of the framework across a broad range of applications. For connected via edges (or connections). Vertices may either be example, they prescribe a particular data format, query model, single (one logical copy of the computation) or sharded (with specific resiliency strategy (such as none, checkpoint-replay, N copies of the computation). Applications create the logical or active-active), scale out scheme, data delivery semantics dataflow programmatically, and specify code for each vertex (e.g., exactly once or at most once), data processing seman- and endpoint in the system. Endpoints communicate with tics (e.g., offline or real-time queries), network topology, and other endpoints via standard network stream connections distribution. As a result, these solutions are not usable across that are provided by the runtime, or using shared memory if the broad range of applications identified above. For instance, possible. we would not consider running an offline job on Storm if Second, users specify a physical deployment topology for efficiency were a concern. Finally, these solutions arenot the dataflow. The physical topology consists of a set ofCRA optimized for containerized environments that are becoming workers, each with a unique name. A worker is an OS pro- ubiquitous today. cess that can host one or more vertex instances. Multiple As a result, today, there is severe fragmentation in the different vertices may be instantiated on the same worker application ecosystem, where each system has created its instance, and vertices on the same worker may communicate own abstractions and implementations of their own building via shared memory for performance (e.g., when reading from blocks to achieve their dataflow requirements at high perfor- a vertex hosting cached input datasets). A sharded vertex mance. Examples include Storm [8], Spark [7], and Flink [2], is mapped to a set of worker instances. Further, users may which share no code commonalities today (apart from the map a vertex to more than one worker. In this case, only one lowest layer of YARN or Kubernetes). This fragmentation copy is designated as “active”, while the other instances are has resulted in repeated “re-invention of the wheel” with used as active or passive standby copies. redundant re-implementation of significant parts of the stack Interestingly, CRA’s physical deployment is one step re- that could have been shared. Further, this made it very hard moved from actual execution on a cluster as follows. CRA to run such diverse applications in a shared environment. workers are packaged into Docker containers, and we use a container orchestration framework such as Kubernetes [14] 1.2 Our Solution: CRA to deploy the worker instances. The orchestration frame- work handles hard problems such as code deployment and We propose a new runtime layer called Common Runtime worker instantiation, liveness and heartbeats for resiliency, for Applications (CRA)1. CRA provides a generic distributed and monitoring tools. CRA leverages these features of the or- dataflow abstraction and deployment functionality without chestrator and provides the additional functionality to make making choices that applications would like to control. At the it possible to build resilient long-running dataflows with cus- same time, CRA offers significant functionality that makes tomized resiliency strategies, data delivery semantics, data it easier to build such applications. For example, CRA does processing and query semantics, and scale out schemes. not interfere with the data plane of the distributed dataflow, We have used CRA to implement several concrete dataflow exposing raw network streams between (virtual) endpoints, applications. Examples include (1) a data-intensive analyt- instead of a specific data format or protocol. This allows ics application (Quill [25]) that enables offline and real-time applications to choose their own data format and applica- analytics over temporal data using Trill [26] (a real-time tion protocol between dataflow vertices. CRA exposes the streaming library); and (2) a resilient distributed actor [22] capability of running multiple copies of a vertex and switch- framework based on reliable message delivery between ac- ing over between them on failure, allowing applications to tors. build dataflow graphs with active-active or active-standby This paper focuses on the design and features of CRA, resiliency models (in addition to the usual checkpoint-replay and describes how we built support for data-intensive ap- capability). CRA supports sharding primitives and rich com- plications in CRA. With CRA and its data layer, we were munication patterns that enable complex and potentially able to re-build a previous system (Quill) for Kubernetes/- long-running dataflows to be constructed, deployed, and Docker in less than 200 lines of code. We use CRA/Quill to maintained. Figure 1 shows where CRA fits in the full appli- perform a detailed analysis of performance of data-intensive cation stack. applications in the Kuberbetes/Docker environment, in com- parison to naive deployments of today’s big data analytics 1CRA is available as open source, and can be downloaded at solutions such as Spark. We believe the latter contribution https://github.com/Microsoft/CRA. is interesting in its own right. It provides new insights into 2 mnlti”sfwr rhtcue u oterrgddesign, rigid their to due architectures software “monolithic” hc sarsl famr ope otaesak htneeds that stack, software complex more a of result a is which provide We 2. Sec. in containerization on background with hr a enagoigidsr rn omv wyfrom away move to trend industry growing a been has There orno oecmlxadhtrgnoshrwr stack. hardware heterogeneous and complex more a on run to the with deal to key also are environments. Containers different across architecture, microservices in applications a individual run more and much deploy, package, it to make simpler which when tools accompanying the format, with packaging combined astandard defines tainer [ CoreOS and and flexible more service agile. larger the making modified deployed, independently and be to applications al- well-defined, individual architecture lows using this Further, other REST). each (e.g., APIs to light-weight talk that smaller – coupled, loosely archi- applications many an comprises such service a In tecture, architecture. “microservices” a towards BACKGROUND CONTAINERIZATION 2 respectively. 9, and 8 Sec. work in related paper discuss the we Finally, conclude ex- 7. and other Sec. as in well CRA as to results tensions these We of 6. implication Sec. the discuss in environments data-intensive containerized of in evaluation in computation extensive Quill an implement provide to We it 5. used Sec. data we our how describe We and layer 4. description processing Sec. a in by architecture followed system is CRA’s applica- This of to 3. exposes Sec. it in interface developers the tion and CRA of overview an leveraged optimally performance. be for may infrastructures these how in explores features and and applications, as Kubernetes data-intensive such for are infrastructures Docker new fit a good how otieiaintcnlg,pplrzdb okr[ Docker by popularized technology, Containerization start We follows. as organized is paper the of rest The
Dataflow Processing Stack Analytics Scan iue1 R plcto Stack Application CRA 1: Figure CommonRuntime Applications for (CRA) 11 ,i h e nbe fmcoevcs con- A microservices. of enabler key the is ], Kubernetes/Docker/YARN/VM Streaming Pipelines HDFS User Applications Iterative Iterative ML services Micro arxo hell of matrix - … [ 18 12 ], ] 3 eatc itiue irsrie” n uoae h deploy- the automates and microservices”, distributed “elastic, egtvrulzto” ncnrs otehaywih hy- heavy-weight the to contrast in virtualization”, weight hc saaoost h yevsri h rdtoa VM traditional the in hypervisor the to analogous is which mznAS irsf zr) ti omnpatc to practice common a is it Azure), Microsoft AWS, Amazon n sspotdb l h ao lu edr nldn Mi- including vendors cloud major the all by supported is and [ Docker Docker 2.1 machines. virtual of top on containers (e.g., run setting cloud real a in that note we However, faster deploy. are and containers to binaries result, a needed As the possible. and whenever image, libraries, containers OS hand, host other single the a On share (OS) appli- VM. the system the run inside to operating hosted needed full cations libraries a and binaries of the copy and image, a carries is that VM difference every a main by The replaced format. is container format light-weight (VM) Docker machine (e.g., virtual engine The containerization Engine). a by replaced virtualization is traditional setting a in Xen) or VMWare, (e.g., pervisor [ technologies virtualization based pervisor rvsoig ceuig uorsat n auto-scaling. and auto-restart, resource scheduling, for provisioning, mechanisms provides it containers Further, cluster. of a across scaling and scheduling, maintenance, ment, [ Google [ at open-sourced scale, at deployed and orchestra- invented container tor, popular most the arguably is Kubernetes Kubernetes 2.2 present we next. which Kubernetes, by orchestrated containers [ Mesosphere achieved typically is by This and clusters. deployment, large build, across their monitoring manage efficiently to need the Engine. Docker the by that host for image container a is host, application target a on deploy application to Finally, an images. Docker built called previously repository download central a to Registry pushed on Docker be built then images can host the local sharing, containers a enable as To images, host(s). those target run the and on instantiate to as well Docker as build to allows Engine in a Docker specifications container Given settings. to[ reader the refer we details terminology more Docker For workflow. the the of and overview an Cloud. present IBM we and Below Cloud, Google AWS, Amazon Azure, crosoft otie ehooisaesmtmsrfre oa “light- as to referred sometimes are technologies Container n fcalne fdaigwt otiesa cl is scale at containers with dealing of challenges of One is ecosystem Docker the of heart the At otie orchestrators container 12 sagal h otpplrcnanrformat, container popular most the arguably is ] 16 pulled 14 ,adKbree [ Kubernetes and ], .Kbree scmol sdt deploy to used commonly is Kubernetes ]. hr te eeoescnsac and search can developers other where , rmaDce eityadrno the on run and Registry Docker a from uha okrSam[ Swarm Docker as such 14 .I hswr,w use we work, this In ]. otie images container 19 okrEngine Docker Dockerfile , 32 21 n later and ] .Tehy- The ]. 13 29 a , ], ]. , , The main unit of deployment in a Kubernetes cluster is order to implement failover, and the ability to pack multiple a Pod, which is simply an abstract resource that bundles components within the same OS process or container. one or more Docker containers together. An important point In CRA, we leverage (1) Kubernetes and Docker for re- to note is that all the containers hosted inside a single Pod, source orchestration; and (2) Resilient Cloud Tables and Stor- share the resources such as IP address and port space, and age (e.g., Azure Tables and Blobs in Microsoft Azure) for storage. Further, all the containers in a Pod are always co- decentralized metadata and persistent state (e.g., binaries, located on a single host, and are always co-scheduled. We application state, logs) management. Kubernetes provides us refer the readers to [27] for more details on Kubernetes. with a resource orchestrator that packs Docker containers within a cluster of physical machines (or VMs). It also pro- 2.3 Data-intensive Computing on vides us with tools to monitor containers, check for failures Containers using heartbeats, and recover failed containers on another machine. The open-source containerization community has mainly focused on “stateless” services. Although, there have been 3.2 Basic CRA Concepts and Design some recent advances in providing support for data-intensive services, which are “stateful”, the industry is lacking a good At the simplest conceptual layer of CRA, we allow users to understanding of what it takes to run data-intensive ser- express their computation as a logical graph of computation vices in these new environments. One of the goals of our units called vertices. Each vertex may be associated with work is to devise new architectures for container-friendly named input and output endpoints, which are used to connect data-intensive computing and to shine a bright light on the pairs of vertex instances. Vertices are themselves packed strengths and weaknesses of running such services on con- into CRA worker instances, a shared-memory concept which tainer platforms. roughly corresponds to an operating system process. CRA worker instances can be spawned off as processes from the 3 CRA OVERVIEW AND INTERFACE OS shell. Alternatively, in Kubernetes, we can host one or more CRA workers in each Pod, and let Kubernetes deploy 3.1 Requirements these Pods on a set of physical (or virtual) machines. Our goal is to create abstractions that one can use to build We describe these concepts with a simple running example and deploy a distributed graph of objects that can send and re- where we have a pair of vertices connecting to and from each ceive data from one another. From an application developer’s other, sending and receiving a fixed sequence of integers. perspective, the key requirements for such abstractions are: 3.2.1 Defining Vertices. The following code fragment shows • Format Independence: Ability to send and receive data the definition of a simple CRA vertex that (a) writes a fixed in an application-specific format and protocol, without sequence of integers to an output endpoint; and (b) reads interference from the runtime. a sequence of integers from an input endpoint and writes • Communication: Ability to communicate between com- them to the console. ponents via either message passing (TCP across nodes, class IntExchangeVertex : IVertex { fast loopback TCP within nodes) or shared memory, de- public IntExchangeVertex() { } pending upon the application’s capability and physical placement. public void Initialize(object arg) { • Location Independence: Ability to write an application int numInts = (int)arg; with multiple distributed components without having to AddEndpoint("in1", new IntReader(numInts)); specify the physical location of each component. AddEndpoint("out1", new IntWriter(numInts)); • Failure Independence: The application should not worry }} about restarting components or establishing network As shown, vertices implement an Initialize function connections in case of failure. which CRA uses to create a vertex instance with user-defined • Sharding: The application must be able to shard its state parameters. The AddEndpoint function is used to define two and computation transparently, and possibly vary the CRA endpoints (an input and an output), passing each of number of shards dynamically during the deployment them the number of integers to send/receive. We next define lifetime. the two endpoints to send/receive the sequence of integers: From the deployer’s perspective, the key requirements are class IntReader : IInputEndpoint { the ability to programmatically control the physical place- int numInts; ment of components on machines, containers within ma- public IntReader(int numInts) { chines, the ability to run multiple copies of computations in this.numInts = numInts; 4 } Here, the DefineVertex function call associates the string “IntExchangeVertex” with the creation of an instance of public void FromStream(Stream s) { type IntExchangeVertex. The second parameter to the func- for(int i=0; i var cl = new ClientLibrary(); 3.3 Sharded Vertices & Endpoints cl.DefineVertex("IntExchangeVertex", Building upon the concept of a graph of vertices, we pro- () => new IntExchangeVertex()); vide a layer that exposes sharded equivalents of vertices and endpoints to users. cl.InstantiateVertex("inst01","vert01", A sharded vertex represents a certain number of copies "IntExchangeVertex", 5); cl.InstantiateVertex("inst02","vert02", (called shards) of a vertex instantiated in the data center, "IntExchangeVertex", 5); but referenced as a single entity. For clarity, we will refer to the normal vertices from Section 3.2 as simple vertices. We cl.Connect("vert01","in1","vert02","out1"); assume that the number of shards is fixed at deployment cl.Connect("vert02","in1","vert01","out1"); time, and revisit this assumption in the context of dynamic topologies in Section 7.3. 5 Analogous to our endpoints from earlier (we will call them int numInts; simple endpoints), we define the notion of sharded endpoints, public ShardedIntWriter(int numInts) { identified by endpoint names as before. A sharded input this.numInts = numInts; (or output) endpoint implements the sharded equivalent of } FromStream (or ToStream), that takes an array of Stream objects as argument. Interestingly, sharded input and output public void ToStream(Stream[] s) { for(int i=0; i Figure 2: CRA Vertices and Endpoints Figure 3: CRA Overall Architecture the source vertex and two physical input endpoints on each • List of worker instances that have been defined. For work- shard of the destination vertex. ers that are active, additional information such as the IP endpoint and port are stored as part of the metadata. 4 SYSTEM ARCHITECTURE • List of vertex instances associated with specific workers. 4.1 Overview A vertex may be instantiated on more than one worker – in this case, at most one instance may be designated CRA is designed as an embedded client library that is linked as “active”. Each instance is also associated with instan- to code for three distinct entities: tiation parameters that are associated with instantiating • Vertices and Endpoints: The user code for vertices and end- that specific instance. For example, this may include the points are given a reference to the client library, which query subplan that the vertex instance is supposed to they can use to create or delete connections, instantiate execute. new vertices, etc. • The list of logical connections between vertex endpoints, • Deployers: Code that is used to create vertices and connec- which represents the edges in the CRA dataflow graph. tions, and instantiate them on specific worker instances • For sharded vertices, the sharded vertex names and list (identified by names) may be written by deployment of instances associated with the shard. code, that lives outside vertices and endpoints. For in- stance, an SQL query compiler may be responsible for 4.3 Worker Lifecycle translating the query into a graph that is deployed and Recall that the user registers a set of CRA worker instances, executed. with a unique name for each worker. When a worker (with a • Worker Bootstrap: The worker instances themselves are unique name) is started by the resource orchestrator on some written as a simple bootstrap program that uses the CRA physical machine, it exposes a listening server on a registered client library to create a worker instance and start it port (whose information is present in the metadata entry in the current process. We offer the CRA worker asa for that worker instance). This allows remote workers and Docker container that can be deployed on a cluster using clients to establish connections and issue various requests resource orchestrators such as Kubernetes, that handle such as connecting to an endpoint, notifying of a newly (re-)deployment and failure detection. added vertex to the worker, and vertex removal. We first describe how and what metadata is handled by When a worker is created, it first queries the metadata CRA. We then describe how workers operate, and interact tables to identify what vertices need to be instantiated on with vertex and endpoint code. We then describe the thread- that worker instance. For each vertex instance, CRA down- ing model and cover how system management is performed loads the application code from the appropriate location in using CRA. Figure 3 depicts the overall architecture of CRA. storage and dynamically instantiates the vertex object. It then downloads the vertex parameter and calls a special 4.2 Metadata Management method Initialize on the vertex object, passing it the pa- rameters registered during logical vertex instantiation and CRA stores metadata in a key-value store. This is a pluggable stored as part of the instance metadata, as described in the module. Our implementation for Azure uses Azure Tables to section on metadata management. Then, for each connec- store such data. CRA stores a variety of metadata: tion originating or ending at this vertex, CRA establishes the • List of logical vertex definitions, along with a pointer TCP connection from this endpoint to its matching remote to files on storage that contain the binary related tothe endpoint. When a connection between two endpoints is es- vertex. tablished, CRA calls the FromStream and ToStream calls on 7 the corresponding endpoints in order to invoke user code the metadata, find the newly activated replica, and establish after a successful connection. This process is repeated for all connection to it. In parallel, the newly activated replica also connections to or from the instance. proactively tries to create its outgoing and incoming connec- Note that even if the user defines a connection from A to tions, so that the distributed vertex graph is restored to the B, CRA may physically establish the connection from B to A. logically correct global state. This is important in cloud-edge scenarios, where the cloud machine may not be able to access the edge device, whereas 4.6 Flexible Threading Model the edge device can access the cloud machine. Similarly, connections from a public machine to a machine inside an The interfaces shown in the previous section are synchronous. organization’s private cloud is made possible in the same In this case, vertices can spawn a set of threads to manage way. their own concurrency within the process. This is useful if the vertex application is multi-threaded, such as Trill [26]. 4.4 Handling Failures CRA also supports asynchronous versions of the interfaces, so that a pool of threads can be shared by all the endpoints on CRA handles both connection and node failures for the ap- the worker process. The latter is a better fit for applications plication. When a CRA worker fails, it depends on the re- such as microservices that share a pool of threads to perform source orchestrator (such as Kubernetes) to detect this and small units of activities. reinstate the container elsewhere on the cluster. On worker re-creation, CRA goes through the process described above to reinstate the vertices and connections on that worker. 5 DATA PROCESSING IN CRA Other workers may be hosting vertices that are reading 5.1 The Dataset Abstraction and writing to the failed vertex. When their network streams To efficiently build dataflow applications, we provide a Dataset break, the endpoint code receives an exception that it can abstraction that allows users to create and process datasets handle appropriately, and transfer up to CRA (which instan- using CRA. By implementing such an abstraction, CRA knows tiated the ToStream or FromStream) code that was inter- how to deploy datasets inside CRA vertices, and processes acting with the network stream. The CRA worker then re- them in an efficient manner. The proposed abstraction is establishes the connection and returns control back to user used to define keyed datasets (e.g., Dataset Actions ToStorage Subscribe 120 30 2 CRA Workers ToBinaryStream 100 4 CRA Workers 8 CRA Workers 25 Subscribe 80 20 60 Table 1: Quill Re-implementation Using CRA Sharded 15 40 Dataset. 10 Throughput (Gbps) 20 Throughput (Gbps) 5 1 Pod N Pods 0 1 2 4 8 Setup of N CRA Workers No. of Failed CRA Workers (c) Effect of Shared Memory (d) Overhead of Failures two tasks; one that ends with a split operation, and another that starts with a merge operation. In our running example, Figure 4: Micro-benchmark Evaluation of CRA we can trigger the deployment as follows: var s4 = s3.Deploy(); dataset operation is used to implement each Quill function- The deployed plan will consist of two tasks. The first task ality. Note that different data movement operations (e.g., executes the create, transform, rekey, and split operations, Reshard and Broadcast) are implemented by providing differ- while the second task executes the merge operation. Note ent split and merge expressions as described in Section 5.2.2. that deploying the plan never triggers the execution. To start Analogously, different action operations (e.g., ToMemory and execution, we should call a subscribe call as follows: ToStorage) are implemented using different subscribe lambda s4.Subscribe(()=>new LogToConsole()); expressions. In total, the implementation of Quill-on-CRA has less than 200 lines of code. This shows the practicality This forwards the lambda expression to the subscribe oper- of CRA in building powerful dataflow applications. We ex- ation of each dataset shard and waits till the whole execution tensively evaluated Quill-on-CRA in Section 6.2 and showed finishes and logs the results. its efficiency and scalability. 5.3 Case Study: Quill-on-CRA 6 EXPERIMENTAL EVALUATION We show a case study on the applicability of CRA in building efficient dataflow applications. We used CRA to re-implement 6.1 Micro-benchmarking Evaluation Quill [25], a high-performance distributed platform for of- Our goal in this section is to (1) measure the overhead of fline and real-time analytics over temporal data. We refer to containers in K8s vs. virtual machines, (2) explore different our re-implementation of Quill as Quill-on-CRA. packing strategies for deploying data intensive applicatins Implementing StreamableDataset. Quill is based on a on K8s, and (3) measure the impact of specific CRA’s opti- sharded temporal dataset, named ShardedStreamable, which mizations on performance and fault-tolerance. is built on top of a real-time streaming library called Trill [26]. Setup: For all the experiments in this section, we use a So, our first step in implementing Quill-on-CRA was pro- sharded broadcast scenario as an example. For a given num- viding a new dataset type, named StreamableDataset, that ber of vertices, we construct a fully connected mesh, i.e., basically implements the Dataset abstraction presented ear- each vertex is connected to every other vertex. Each vertex lier in this section, using the ShardedStreamable constructs. sends (receives) a fixed amount of data to (from) every other Re-implementing Operations. By having the Stream- vertex. We fix this parameter to 100 MB in our experiments. ableDataset implementation, we gain access to the whole set We use sharded broadcast as an example of the basic “data of operations that are provided by the CRA sharded dataset exchange” operator quite commonly used in many analyt- abstraction on top of StreamableDataset. Thus, our second ical queries. Hence, studying its performance is important. step was re-implementing the original Quill API using these For these experiments, our target metric is the measured sharded dataset operations. Table 1 shows which sharded throughput in Gigabits per second (Gbps). Unless otherwise 10 noted, for this section, we run all of experiments on two 1400 1400 Quill-on-CRA Quill-on-CRA Azure VMs. Similarly, for K8s, we use an Azure Kubernetes 1200 Spark 1200 Spark Service (AKS) cluster with two agent nodes. 1000 1000 800 800 6.1.1 Overhead of Containerization. In the first experi- 600 600 ment, we measure the overhead of containerization vs. VMs. Tuples/sec) 400 Tuples/sec) 400 Throughput (Million 200 Throughput (Million 200 We vary the number of CRA workers (or instances) in each 10/10GB 20/20GB 30/30GB 10/10GB 20/20GB 30/30GB VM/Pod and measure the throughput. Figure 4(a) shows the No. of Nodes/Dataset Size No. of Nodes/Dataset Size results of this experiment. Overall, throughput with VMs is (a) Scan Query (6% Selectivity) (b) Scan Query (67% Selectivity) between 1.4x to 3.6x higher as compared to K8s. This experi- ment shows that the networking subsystem in K8s imposes a Figure 5: Big Berkeley Data Benchmark (Scan Query) much higher overhead as compared to VMs. Our experience (results not presented here) shows that this overhead can be overcome by opening more parallel TCP connections when number of CRA workers to 8, running in two Pods (4 work- running in K8s. ers per Pod). We then randomly fail some of CRA workers during the experiment. We show the results in Figure 4(d), 6.1.2 Packing CRA Workers on K8s Pods. As noted earlier, with 0, 1, 2, 4, and 8 failed CRA workers. We see that CRA’s CRA provides full flexibility over deployments – allowing performance gracefully degrades with the number of failures. the developers to pack different number of CRA workers Note that, K8s automatically detects Pod failures and restarts inside a given container (VMs or Pods) to achieve the best the Pods. This experiment shows that CRA effectively builds performance. In this experiment, we vary the number of on top of this automatic detect and restart capability of K8s workers packed in a K8s Pod and measure the impact on to provide fault tolerance for applications built on top of throughput. We show the results in Figure 4(b), where x-axis CRA. is the number of CRA workers per Pod, while the different colored bars represent the total number of CRA workers. 6.2 Berkeley Big Data Benchmark We can see that as we pack more CRA workers in a Pod, we get higher throughput (moving from left to right in the Evaluation figure). This means that for network intensive applications, it Our goal in this section is to evaluate the performance of is desirable to more tightly pack CRA workers into Pods. As Quill-on-CRA (Section 5.3), which is a case study on building we will show later, this not only results in higher throughput efficient dataflow applications using CRA, in comparison to at K8s level, but CRA can exploit specific optimizations to a state-of-the-art big data analytics platform. We use Spark- get even better performance. SQL [20], with Spark v2.2, as our baseline. Deployment. We are interested in the Kubernetes deploy- 6.1.3 Effect of CRA’s Shared Memory Optimization. As ment. Similar to micro-benchmark evaluation, we use Azure described earlier, CRA can detect if two workers are run- Kubernetes Service (AKS) to deploy both systems. As best ning on the same machine (sharing the same IP address), configuration, we assign two workers per pod for each of in which case, we short circuit the communication path to CRA and Spark, where each AKS agent node has only one use shared memory instead of regular TCP sockets. Previous pod. experiment showed that more tightly packing CRA instances Benchmark. We compare the performance of both sys- into Pods improves throughput. In this next experiment, we tems, while running the big data benchmark [30] that was want to directly measure the impact of CRA’s shared memory used to evaluate both original Quill [25] and SparkSQL [20]. communication on throughput. We show the results in Fig- The benchmark contains three types of typical analytical ure 4(c). We explore two packing strategies (1) fully-packed: SQL queries; scan, aggregate and join, which run on two pack all CRA workers into a single pod, labeled as “1 Pod” types of datasets; rankings and uservisits (Schema is omitted (2) fully unpacked: one Pod per CRA worker, labeled as “N due to space constraints). As instructed in the benchmark, Pods”. We run with 2, 4, and 8 CRA workers in total (rep- we fix the size of each dataset per agent node to be 1GB(18M resented as different bars). These results show that CRA’s tuples) and 25GB (152M tuples) for rankings and uservisits shared memory communication can improve throughput by datasets, respectively. However, to evaluate the scalability up to 5x as compared to regular TCP sockets, showing the of both systems, we vary the number of agent nodes from effectiveness of this optimization. 10 to 30 in all experiments, and hence, in total, we have up 6.1.4 Effect of Failures on CRA. In this experiment, we to 750GB data from the two datasets. Since we assign two study the impact of failures, which are quite common in prac- workers per node, this also means that we have from 20 to tice, on CRA’s performance. For this experiment, we fix the 60 running workers for each of CRA and Spark. Our target 11 100 80 1200 400 Quill-on-CRA Quill-on-CRA Quill-on-CRA Quill-on-CRA 90 350 Spark 70 Spark 1000 Spark Spark 80 60 300 70 800 50 250 60 40 600 200 50 150 Tuples/sec) Tuples/sec) 30 Tuples/sec) 400 Tuples/sec) 40 100 30 20 Throughput (Million Throughput (Million 200 50 20 10 Throughput (Thousands Throughput (Thousands 10/240GB 20/480GB 30/720GB 10/240GB 20/480GB 30/720GB 10/250GB 20/500GB 30/750GB 10/250GB 20/500GB 30/750GB No. of Nodes/Dataset Size No. of Nodes/Dataset Size No. of Nodes/Dataset Size No. of Nodes/Dataset Size (a) Aggregate Query (52M Groups) (b) Aggregate Query (231M Groups) (a) Join Query (65K Groups) (b) Join Query (231M Groups) Figure 6: Big Berkeley Data Benchmark (Aggregate Figure 7: Big Berkeley Data Benchmark (Join Query) Query) 6.2.3 Join Query. The last query is a join between rank- ings and uservisits datasets to obtain the sourceIP and asso- metric is the measured throughput by each query in tuples ciated pageRank that gave rise to the highest revenue for 20 per second. years. Figures 7(a) and 7(b) show the results in case of having 65K and 231M groups according to sourceIP, respectively. 6.2.1 Scan Query. We first start with the scan query which Similar to the aggregate query, this experiment confirms basically selects two columns, pageURL and pageRank from our observation about increasing the average improvement rankings dataset where the pageRank is higher than a cer- ratio when the number of groups increases (i.e., increasing tain value X. Figures 5(a) and 5(b) show the throughput re- the amount of data shuffling). The average throughput of sults for the two systems when using a scan selectivity of Quill-on-CRA is 1.9x and 2.7x higher than Spark with 65K 6%(low value) and 67%(high value), respectively. Overall, the and 231M groups, respectively. throughput with Quill-on-CRA is between 2.2x to 5.4x higher than Spark in both selectivity cases. The lower throughput 7 DISCUSSION of Spark comes from the high overhead of scheduling the 7.1 Takeaways on Data Intensive Use of operation during the execution time. In contrast, Quill-on- Kubernetes/Docker CRA never incurs such overhead because of the lightweight runtime management in CRA. As noted earlier, Kubernetes and the accompanying ecosys- tem has been solely focused on “stateless” (web) services. 6.2.2 Aggregate Query. The next query in the benchmark Only recently the community has turned its attention to is an aggregate query over the uservisits dataset which com- supporting data intensive (stateful) applications as a first putes the total revenue originated per sourceIP. Figures 6(a) class citizen (e.g., by proposing Stateful sets [15]). Although and 6(b) report the throughput results for Quill-on-CRA and CRA’s design is independent of a particular resource orches- Spark when the number of groups is 52M and 231M, respec- trator, CRA takes a very important step in demonstrating tively. In general, Quill-on-CRA achieves higher throughput how to build data intensive, stateful applications on top of than Spark for two main reasons. First, the execution time Kubernetes. We summarize our preliminary findings below: of this query is bounded by the amount of data that will be • Networking Overhead: Containers impose additional shuffled among workers, and since CRA utilizes the shared networking overhead. Our experiments and experience memory communication between processes within the same show that, the per-connection throughput achieved from pod, its performance will be better than Spark which em- within a container could be, in many cases, much lower ploys network communication between workers even within than the corresponding throughput with a VM. This is the same pod. Second, the aggregate query plan that is gener- related to how networking layers are designed and imple- ated by Quill is much more efficient than the one generated mented in Kubernetes, with more layers of abstraction, by the Spark optimizer as shown in [25]. Note that the av- and hence a higher overhead. Architects of data intensive erage improvement ratio in case of 231M groups (1.5x) is applications and platforms need to be aware of this limi- higher than the average in case of 52M groups (1.2x). This tation and work around it, for example, by opening more confirms the first reason mentioned earlier as the amount parallel connections when running inside containers. of data shuffled in case of 231M groups is higher than 52M • Flexible Deployment: When it comes to containeriza- groups, and hence the CRA shared memory communication tion platforms, such as Kubernetes, there is a lot of flex- will be utilized much more. ibility in choosing how to pack processes into Docker containers and how to pack those containers inside Pods. 12 CRA exploits this flexibility at its very core. Our experi- dynamic shards is currently being added to CRA; thus, the ments show that packing multiple processes in Pods can details and evaluation of this capability are outside the scope enable CRA to exploit shared memory communication, of this paper. which can be up to 5x more performant, as opposed to communicating over TCP sockets. This is an important 7.4 Detached Mode for CRA design choice. By default, CRA vertices are written in the context of a base • Exploiting Orchestrator Capabilities: Kubernetes solves class that registers the vertex with the system. However, in many infrastructure level problems, including service dis- some cases, we may want existing code to easily connect covery, failure detection, auto-restart, and replication. In to vertices in the CRA vertex graph, without actively par- general, it is important to follow a layering approach and ticipating itself. For example, an edge device may wish to fully exploit these capabilities at the higher layers. In our connect to an ingress vertex in the cloud (e.g., when devices current design, CRA builds on top of the failure detec- are connecting to a sharded server exposing a REST API). tion, and auto-restart capabilities of Kubernetes. Going For such scenarios, CRA supports a detached vertex mode, forward, there is a possibility to deeply integrate CRA’s where it can programmatically connect to an existing vertex elastic sharding with StatefulSets [15] in Kubernetes. and get back a network stream over which it can send and receive data. This facility makes it easy to integrate external 7.2 Other Applications on CRA data sources to the CRA ecosystem of vertices. Beyond SQL, spatial, and temporal analytics, CRA supports the creation of vertices that can handle diverse applications 8 RELATED WORK such as machine learning, real-time stream processing, it- Simiar to CRA, Hyracks [23], Dryad [28], and Nephele [33], erative, graph computation, distributed microservices, and adopt the notion of representing data processing as vertices key-value stores. For instance, machine learning and itera- and edges in a DAG. However, these systems provide a differ- tive computations can create sharded topologies with self- ent level of abstraction and have different goals. As opposed loops to implement the iteration. Real-time stream process- to CRA, which provides a framework to build dataflow style ing using long-running vertices that are made durable with processing engines on top, these systems provide data pro- checkpoint-replay, active-active, and active-standby forms cessing engines of their own, and hence the other aspects of of resiliency. Distributed microservices and three-tier archi- building and deploying applications (e.g., sharding, recover- tectures (clients, Web servers, databases) can also be mapped ability, elastic scaling) are built into the engine. This tight to a multi-level CRA topology across edge and cloud. A dis- coupling makes these systems less flexible. tributed key-value store uses the key-value requests as a Apache Tez [31] is probably the most closely related to stream from clients to servers that host the state, with re- our work. Tez is an open-source framework which, at its sponses sent asynchronously along reverse connections. heart, provides a library for YARN that facilitates building dataflow style processing engines. Similar to CRA, using the 7.3 Dynamic Topologies Tez API, one can define an arbitrary DAG representing a cus- Since CRA applications may be long-running, there is often tom dataflow. Tez promotes component re-use among differ- a need to change the topology, such as adding or removing ent verticals built on top of YARN [3]. For example, Hive [4], vertices in the topology or allowing the number of shards in and Spark [7] can be built using the Tez APIs, but applica- a sharded vertex to grow or shrink over time (e.g., to handle tions still have to take ownership of resiliency strategies workload variations). CRA can support dynamic topologies and use other frameworks to target non-YARN deployments. by explosing the ability to add or remove vertices at any Apache REEF [6] and Apache Twill [10] facilitate building point. Further, it exposes information related to sharded to applications on YARN but provide a lower level interface the CRA vertices and endpoints. Briefly, when a vertex is than Tez, and hence are more general purpose. Apache REEF instantiated, it is provided with sharding information that provides a library for building applications which can be identifies how many vertices are in the shard, and onwhich ported to multiple resource orchestrators such as Apache set of worker instances the shards are instantiated. Similar YARN and Apache Mesos. Apache Twill provides a Java-like information is provided to vertices that connect to a sharded multi-threaded programming API for writing YARN appli- input or output endpoint. By providing and updating the cations. Twill’s main goal is to reduce the complexity of information to all interested parties whenever the vertex developing YARN applications by offering a programming topology changes, an application can support a dynamic framework and common functionalities needed by many topology according to its application-level semantics. While large scale distributed applications. To that end, all these dynamic topologies are supported in CRA, full support for systems have similar goals as CRA. However, unlike CRA, 13 Tez, REEF, and Twill are tied to the YARN ecosystem and applications to be first-class citizens on these emerging plat- have been exclusively developed to avoid “re-inventing the forms. Our experimental results on Quill-on-CRA showed wheel” in the YARN community. CRA is much more general a significant performance advantage of CRA dataflows vs. purpose and can be widely used inside and outside of the unoptimized data flows. Overall, we believe our research and YARN ecosystem, as we show in this paper. Further, CRA system present interesting insights into how data-intensive exposes raw network streams to applications and gives users applications can exploit containerization technologies. Fur- full control over the data plane. CRA also exposes a capa- ther, we believe these insights are quite valuable for the bility to run highly resilient long-running workflows with larger community. Last, we welcome the community to build support for different application-defined resiliency strate- on CRA, which is now available as open-source software. gies such as active-active and checkpoint-replay, as well as first-class support for making it easy to write applications REFERENCES with sharding and dynamic topologies. Apache Spark [7] and Apache Flink [2] provide an API and [1] Apache Drill. https://drill.apache.org/, 2018. [2] Apache Flink. https://flink.apache.org/, 2018. a general purpose data processing engine. Similar to CRA, [3] Apache Hadoop YARN. https://hadoop.apache.org/docs/current/ they also have a notion of a DAG used to represent data hadoop-yarn/hadoop-yarn-site/YARN.html, 2018. processing. However, CRA provides a lower-level API which [4] Apache Hive. https://hive.apache.org/, 2018. gives much more flexibility to the application programmer to [5] Apache Impala. https://impala.apache.org/, 2018. build and deploy custom dataflow graphs on top. It is possible [6] Apache REEF. http://reef.apache.org/, 2018. [7] Apache Spark. https://spark.apache.org/, 2018. to implement the Spark and Flink processing engines on top [8] Apache Storm. http://storm.apache.org/, 2018. of the common runtime provided by CRA. As an example, in [9] Apache Tajo. http://tajo.apache.org/, 2018. this paper, we showed how to build a general purpose data [10] Apache Twill. http://twill.apache.org/, 2018. processing layer, namely Quill [25], on top of CRA, with only [11] CoreOS. https://coreos.com/, 2018. 200 lines of code. We showed the generality of the CRA API, [12] Docker. https://www.docker.com/, 2018. [13] Docker Swarm. https://docs.docker.com/engine/swarm/, 2018. the value of reusing existing components, and the significant [14] Kubernetes. https://kubernetes.io/, 2018. impact it has on improving developer productivity. [15] Kubernetes StatefulSets. https://kubernetes.io/docs/concepts/ DBMSs expose SQL for computations, but extending them workloads/controllers/statefulset/, 2018. with custom functionality, such as implementing new shuffle [16] MESOSPHERE. https://mesosphere.com/, 2018. or join strategies, is difficult. A majority of the new gen- [17] Presto. https://prestodb.io/, 2018. [18] The Matrix of Hell. https://blog.docker.com/2013/08/ eration of big data processing systems, such as Drill [1], paas-present-and-future/, 2018. Impala [5], Tajo [9], and Presto [17] also have similar limita- [19] VMWare Hypervisor. https://my.vmware.com/en/web/vmware/ tions. They are custom data processing engines, built from evalcenter?p=free-esxi6, 2018. scratch, and perform well for specific workloads. On the [20] M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, other hand, in CRA, a developer does not need to start from T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia. Spark sql: Rela- tional data processing in spark. In SIGMOD, 2015. scratch, has full control over data processing, and is free to [21] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neuge- define and execute arbitrary applications, as long asthey bauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In can be expressed as a distributed, possibly cyclic, sharded SOSP, 2003. dataflow graph. [22] P. Bernstein, S. Bykov, A. Geller, G. Kliot, and J. Thelin. Orleans: Dis- tributed virtual actors for programmability and scalability. Technical report, 2014. [23] V. Borkar, M. Carey, R. Grover, N. Onose, and R. Vernica. Hyracks: A 9 CONCLUSIONS flexible and extensible foundation for data-intensive computing. In ICDE, 2011. In this paper, we started with the goal of significantly chang- [24] B. Chandramouli, J. Claessens, S. Nath, I. Santos, and W. Zhou. Race: ing the way big data applications are designed and imple- Real-time applications over cloud-edge. In SIGMOD, 2012. mented, with a particular focus on boosting developer pro- [25] B. Chandramouli, R. C. Fernandez, J. Goldstein, A. Eldawy, and A. Qua- ductivity. We presented the design and implementation of mar. Quill: Efficient, transferable, and rich analytics at scale. VLDB Common Runtime for Applications (CRA), a cloud-native Endow., 2016. [26] B. Chandramouli, J. Goldstein, M. Barnett, R. DeLine, D. Fisher, J. C. runtime with a common interface to build a wide variety of Platt, J. F. Terwilliger, and J. Wernsing. Trill: A high-performance data center applications. We showed how system and appli- incremental query processor for diverse analytics. VLDB Endow., 2014. cation designers can exploit next-generation virtualization [27] K. Hightower, B. Burns, and J. Beda. Kubernetes: Up and Running: Dive technologies such as containers, which have become the into the Future of Infrastructure. O’Reilly Media, Inc., 2017. de-facto building blocks for packaging and deploying cloud- [28] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys, scale applications. The CRA architecture enables data-centric 2007. 14 [29] K. Matthias and S. P. Kane. Docker: Up & Running: Shipping Reliable processing applications. In SIGMOD, 2015. Containers in Production. O’Reilly Media, Inc., 2015. [32] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and [30] A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, J. Wilkes. Large-scale cluster management at google with borg. In and M. Stonebraker. A comparison of approaches to large-scale data EuroSys, 2015. analysis. In SIGMOD, 2009. [33] D. Warneke and O. Kao. Nephele: Efficient parallel data processing in [31] B. Saha, H. Shah, S. Seth, G. Vijayaraghavan, A. Murthy, and C. Curino. the cloud. In MTAGS, 2009. Apache tez: A unifying framework for modeling and building data 15