Geo-Distributed Stream Processing Caleb Stanford February 2020 Abstract Distributed stream processing systems (DSPS) are specialized software platforms for processing large quantities of data in real time. While such systems (e.g. Apache Storm, Apache Flink) have achieved widespread use, they generally rely on sending data to a central computing cluster or data center to be processed, which means there are fundamental limits to (1) the end-to-end latency and (2) the network bandwidth used to communicate with the data center, particularly when the data to be processed is highly geo-distributed. In this report, we will survey some recent methods to extend DSPS to the geo- distributed setting by executing stream processing jobs over a geo-distributed network of nodes. We will present representative papers from three different approaches: for- mal optimization techniques [1], extensions to the programming model [2], and using approximation to greatly reduce the size or frequency of data that needs to be sent between nodes [3]. We will discuss how these approaches differ, how they compare in terms of performance, and areas where future research is needed. 1 Introduction In the last 15 years, distributed stream processing systems (DSPS) have emerged as a way to conveniently express and efficiently deploy computations over streaming data. Notable examples of such systems include early streaming databases like Aurora [5] and Borealis [4], and more recently Apache Storm [20], Apache Flink [17], Apache Samza [18, 28], Apache Spark Streaming [19, 45], Twitter (now Apache) Heron [40], Google MillWheel [7], and Naiad [27]. The primary feature of stream processing systems as contrasted with batch processing systems is that, besides high throughput and fault tolerance, they additionally offer continuous processing of data with low latency. All mainstream distributed stream processing systems, however, share a limiting assump- tion: they require that all data is sent to a central distributed data center or computing cluster to be processed. This implies that, first, although they can achieve latency in the milliseconds, this includes only the time from when an event enters the data center to when a response is produced; and the end-to-end latency of data being sent over the network may be much higher. Second, in cases where data is extremely high volume, such as in processing geo-distributed video streams, the bandwidth used to send all the data to a cen- tral location can be prohibitive. This is made worse because available bandwidth between geo-distributed sites can vary over time, and periods of low availability cause spikes in la- tency ([3], Figure 1). These limitations motivate the need for a new generation of stream processing systems which embrace the geo-distributed nature of data: they should enable processing data without sending it to a central cluster. 1 Figure 1: Available bandwidth over time between Amazon EC2 instances in Ireland and California. Chart is taken from [3]. Note that these limitations are not unique to distributed stream processing; the idea that systems should be aware of geo-distributed network conditions goes back to the long line of work in geo-distributed, WAN-aware, and more recently edge and fog computing systems. Another way of saying the problem is that in the DSPS domain, currently available solutions adopt the cloud computing paradigm. The vast amount of data that are being generated in the near future, particularly from video streams and IoT devices, are too high to be processed by today's networks. We seek to bring the ideas of edge computing [37] to the DSPS domain because it is a promising software platform for the emerging applications over this future data. In this report, we investigate three recent proposed solutions to the geo-distributed stream processing problem. We have identified three categories of work in this domain. First, some papers have focused on extending the system with optimization strategies to account for geo-distributed data and nodes. This includes more sophisticated scheduling and resource placement. We discuss one promising such approach called Optimal Operator Placement [1], based on formalizing the placement problem as a mixed-integer linear programming (MILP) problem. Second, some papers have focused on providing new programming frameworks which allow better control over the placement and allocation of geo-distributed resources. We look at the system SpanEdge [2], which can be seen as a less automated, but much more lightweight version of the first approach. Finally, some papers have focused on the promising idea that, for many geo-distributed applications, a good way to deal with the vast amount of data and bandwidth constraints is to rely on approximation, by decreasing the quality or quantity of data sent, but in a way that retains a similar output. AWStream [2] is one recent such system, which also extends the programming model in an interesting way to express possible degradation functions, and compares them for which is the most accurate while also reducing bandwidth the most. In summary, we discuss the following three papers: • Optimal operator placement for distributed stream processing applications. Valeria Cardellini, Vincenzo Grassi, Francesco Lo Presti, and Matteo Nardelli. Distributed Event-Based Systems (DEBS) 2016. [1] (Section 3) • SpanEdge: Towards unifying stream processing over central and near-the-edge data centers. Hooman Peiro Sajjad, Ken Danniswara, Ahmad Al-Shishtawy, and Vladimir Vlassov. Symposium on Edge Computing (SEC) 2016. [2] (Section 4) • AWStream: Adaptive wide-area streaming analytics. Ben Zhang, Xin Jin, Sylvia Ratnasamy, John Wawrzynek, and Edward A. Lee. SIGCOMM 2018. [3] (Section 5) 2 Before discussing these papers, we provide some background in Section 2. Specifically, we explain what existing DSPS are in wide use and what they have in common, particularly with respect to the programming model. We use this to set up a common problem framework in which the geo-distributed stream processing problem can be studied, and we consider all three approaches as instances of this framework in Sections 3 through 5. Next, we survey other papers along the three categories in Section 6. Finally, we provide an overall comparison of the different directions, as well as promising directions for future work, in Section 7. 2 Background and Problem Framework 2.1 Background Distributed Stream Processing Systems. There are a vast number of DSPS, including many research prototypes as well as actively developed software in widespread use. Some of the major systems are listed in Table 2. Different systems often differ in some details, partic- ularly with regard to implementation choices (e.g. scheduling and optimization strategies, fault tolerance mechanisms) and sometimes with regard to the abstractions that are offered to the programmer. We focus in this report on the features that they share in common, rather than what distinguishes them, in order to provide a common framework to consider the geo-distributed streaming problem. All DSPS in current use share the dataflow programming model. This means that the programmer writes, in some form or another, a dataflow graph. In some systems, e.g. Apache Storm, the dataflow graph is written out explicitly, whereas in others, such as Apache Flink, the graph is implicit. Additionally, systems may offer high-level libraries for creating or composing dataflow graphs; in particular, these include libraries for complex windowing operations and for SQL- and CQL-based streaming queries. The dataflow programming model exposes task and pipeline parallelism; to expose additional data parallelism, DSPS use operator replication. Similar to how a MapReduce [15] job is implicitly parallelized, all operators in a dataflow grpah (unless configured otherwise) may be split into several copies; this is part of the programming model as well, and affects the semantics. The Dataflow Programming Model. We introduce the programming model through a simplified example based on a geo-distributed real-time video analytics use case. Imagine a large-scale system of video cameras, perhaps located in several cities throughout a country. Each video camera produces a stream of video data, at a certain frame rate and image resolution. Suppose that we want to identify pedestrians and report to a central location the summary of all pedestrian activity in the last 10 minutes, i.e., where pedestrians are most active. To do so, we want to classify each image from each camera using an out-of- the-box classifier; then, to prevent noise and to summarize the total activity, we want to aggregate the data from all classifiers in the last 10 minutes in a particular location (e.g., one intersection or group of intersections). In the end, we report for each location the total amount of pedestrian activity. (One could imagine taking further steps, such as adding a smoothing filter which removes reports of pedestrian activity that last only for a single frame, assuming that these must be erroneous.) A dataflow pipeline for this is given in Figure 3. The input data consists of raw video data. Notice that the pipeline contains only one operator at each stage; that is, we treat 3 Questions on Stable System Year Active? StackOverflow Release (as of 2020-02-04) Aurora 2003 [5] 2003 [29] No{ Borealis 2005 [4] 2008 [30] No{ 2011 [32] 2019 [20] Yes 2458 2011 2019 Yes 3667 [17, 10] Google MillWheel 2013 [7] { No{ 2013 [45] 2019 [19] Yes 4744 (Apache Spark Streaming) 2013 2019 Yes 79 [18, 28] Naiad 2013 [27] { No{ 2015 2019 Yes 43 [40, 24] Figure 2: A selection of major distributed stream processing systems. 4 sliding window input keyBy(deviceID) classify keyBy(location) output (10 min) Figure 3: Example DSP dataflow graph for a program to classify video streams and report total pedestrian activity. all input data at all cameras as a single input stream; and we write transformations over that stream.