On the Streaming Model Augmented with a Sorting Primitive

† ‡ § Gagan Aggarwal∗ Mayur Datar Sridhar Rajagopalan Matthias Ruhl Stanford University Google IBM Almaden Google

Abstract ciently computable”; in fact, any problem that requires sig- nificantly super-linear computation time is practically im- The need to deal with massive data sets in many practi- possible to solve on these inputs. In this paper, we address cal applications has led to a growing interest in computa- the question of “what is efficiently computable on massive tional models appropriate for large inputs. The most impor- data sets”. tant quality of a realistic model is that it can be efficiently The main bottleneck for massive data set computations implemented across a wide range of platforms and operat- on modern computing hardware is the cost for I/O opera- ing systems. tions, and usually not the cost for in-memory computations. In this paper, we study the computational model that re- It is well-known that modern computing hardware is opti- sults if the streaming model is augmented with a sorting mized for sequential access to data, and there are substantial primitive. We argue that this model is highly practical, and penalties for non-sequential data access, manifested for ex- that a wide range of important problems can be efficiently ample as seek times, cache misses, and pipeline stalls. This solved in this (relatively weak) model. Examples are undi- has, in the past, led to an interest in the streaming model of rected connectivity, minimum spanning trees, and red-blue computation [20, 5, 16], and practical applications built on line segment intersection, among others. This suggests that the streaming primitive [18, 10]. using more powerful, harder to implement models may not It is really hard to write code that accesses massive always be justified. data sets non-sequentially without a substantial degrada- Our main technical contribution is to show a hard- tion in performance compared to sequential access. Usually, ness result for the “streaming and sorting” model, which this code would have to be platform and operating system demonstrates that the main limitation of this model is that specific. Consequently, there are very few general-purpose it can only access one data stream at a time. Since our primitives available that access data in a non-local fashion, model is strong enough to solve “pointer chasing” prob- and still obtain maximal throughput from the I/O subsys- lems, the communication complexity based techniques com- tem. Of these, “sorting” is the most readily available and monly used in showing lower bounds for the streaming most researched primitive [3, 6, 26, 11]. model cannot be adapted to our model. We therefore have Consequently, streaming computations with an added to employ new techniques to obtain these results. sorting primitive are a natural and efficiently implementable Finally, we compare our model to a popular restriction class of massive data set computations. In this paper, we of external memory algorithms that access their data mostly study this computational model and present results of two sequentially. kinds. First, we establish that many natural problems solv- able for massive data sets can already be solved in our model. This indicates that it is not necessary to consider more powerful models, as these do not seem to enable the 1. Introduction solution of more problems, but run the risk of being not ef- ficiently implementable. Second, we show hardness results Recently, massive data sets have appeared in an increas- for our model, i.e. demonstrate that there are clear limita- ing number of application areas. The sheer size of this tions to the computational power of this class. data, often in the order of terabytes, means that “polyno- mially computable” is no longer synonymous with “effi- 1.1. Our Contributions Supported in part by a Stanford Graduate Fellowship and NSF Grant ∗ EIA-0137761. E-Mail: [email protected] First, we formally define the “streaming and sorting” † E-Mail: [email protected] model in section 2. We then demonstrate in section 3 that ‡ E-Mail: [email protected] “streaming and sorting” admits efficient solutions to a num- § E-Mail: [email protected] ber of natural problems, such as undirected connectivity, minimum spanning trees, suffix array construction, and 1.2. Related Work even some geometric problems. Moreover, all problems solvable in NC with a linear number of processors can also The “streaming model” was defined implicitly in the be solved in our model. These problems are all known to work of Munro and Paterson [20], and even earlier, in the be hard in the streaming model, suggesting that the addi- context of algorithms for systems with tape-based storage tion of a sorting operation extends that model in a meaning- and little memory. The growing interest in massive data set ful way. computations has led to numerous publications on this topic The fact that most of the problems studied under more in recent years; a comprehensive survey of this area is be- powerful models (e.g. [1]) can be solved in our model yond the scope of this introduction. demonstrates that the additional power of those models is The streaming primitive has been studied in the past few not strictly necessary, and that by studying the “weaker” years in the graphics community (see [10, 22, 17, 18], which class of “streaming and sorting”, we do not lose any real also list further references), and this application area re- computational power, while gaining a simpler and provably cently also received some interest in the theory community efficient model. [15, 2]. In section 4 we turn to hardness results for the “stream- Borodin, Nielsen and Rackoff [9] study a computational ing and sorting” model, asking the question what cannot model in which the data is accessed sequentially, but the be computed in our computational model. As far as we as-yet-unread part of the input can be re-sorted depending are aware, there has been no past work on hardness results on the already read input. This way of combining stream- on computational models that are strict extensions of the ing and sorting is much more powerful than our approach, streaming model. Our results might, therefore, be the first e.g. the problem ALTERNATING SEQUENCE from section of their kind. 4.1 that is hard in our model is easy in theirs. However, their Intuitively, a problem is solvable in our model if access model was not meant for computations on massive data sets, to data can be split into several linear passes over the data, and in fact does not seem practical in that setting. with a known pattern of data reordering (sorting) between The other computational models for massive data set successive passes. A natural candidate for a problem that computations studied in the literature are mostly restrictions is hard for our model is therefore a problem that requires of the “external memory algorithms” (EMA) model intro- a non-predictable order of data access. One such candidate duced by Aggarwal and Vitter [4, 24]. Many of these mod- is “pointer chasing”, in particular since it has a high com- els (such as the one by Abello et al [1]) inherit the potential munication complexity (see e.g. [21]). Surprisingly, the k- to access data in a non-sequential fashion, making them po- round pointer chasing problem can be solved in O(logk) tentially harder to implement efficiently in practice. passes in the “streaming and sorting” model, and is there- The “mostly-sequential” EMA model studied in section fore not a “hard” problem for this model. This demonstrates 5 has been considered by several researchers [13, 14, 8], that communication complexity techniques used to show who emphasized that it is a particularly efficient subclass of lower bounds in the streaming model (see e.g. [7]) cannot EMAs. be adapted to show hardness results for our model. In the main technical result of this paper, we introduce 2. Streaming and Sorting a new problem called “alternating sequence” (a variant of pointer chasing), and show its hardness for the “streaming We will first define the notion of a “streaming and sort- and sorting” model. To this end, we develop a new lower- ing” algorithm. A stream is a sequence S = x1x2 ...xn of bounding technique based on purely combinatorial argu- items xi Σ, where Σ is some problem-specific universe. A ments. memory∈m streaming pass is a function computed by a Tur- In section 5, we discuss the relationship between the ing machine M with a local memory of m bits that reads an streaming and sorting model and another model previously input stream S and writes an output stream StrM(S). The discussed in the literature, which we name “linear exter- machine M is allowed to move only left-to-right on both nal memory algorithms” (LEMA). Intuitively, this model is streams, i.e. it can only read S in a single linear pass, and it equivalent to a streaming and sorting model enhanced with can only append to the output string, never erase what it has the ability to concurrently access multiple streams and not already written. A memory m sorting pass is a function de- just one. It turns out that the LEMA model is strictly more fined using a Turing machine M with memory m that com- powerful than streaming and sorting (without simultaneous putes a partial order on Σ, i.e. given two items of Σ, it re- access to multiple streams); in particular, it can efficiently turns which one is greater, or whether they are equal or in- solve the “alternating sequence” problem. comparable. If S is the input of a sorting pass, then the out- We conclude the paper in section 6 by discussing further put SortM(S) is S reordered according to the partial order- research directions and open problems. ing defined by M. Note that the result is not uniquely de- fined if there are incomparable items in S; in practice, this Proof: Our algorithm is very similar to the one commonly problem is easy to avoid. Also note that M’s computation used in RNC to solve this problem. The algorithm proceeds in the sorting pass is side-effect-free, i.e. no state is main- as follows. We assign a random number, a 3logn-bit inte- tained between comparisons. ger, to each vertex of the graph. Then each vertex gets la- beled by the smallest number among the ones assigned to Definition 1 (StrSort) its neighbors and itself. We then merge all vertices that re- We let StrSort(pStr, pSort,m) be the class of functions com- ceive the same label. In expectation, this reduces the number putable by the composition of up to pStr streaming passes of nodes in the graph by a constant factor. Thus, by repeat- and pSort sorting passes, each with memory m, where we ing this process a logarithmic number of times, we obtain assume that a graph without any edges, where each vertex represents a connected component of the original graph. By keeping the local memory is maintained between streaming track of which intermediate vertices s and t get merged into, • passes, and we can answer the connectivity query by simply checking streams produced at intermediate stages are of length whether they end up in the same component. • O(n), where n is the length of the input stream. The remainder of the proof consists of proving that the  number of nodes decreases by a constant factor in each We set StrSort(p,m) := p +p pStrSort(p0, p00,m). ∪ 0 00≤ contraction phase, and that each such phase can be imple- Note that StrSort(p,0,m) is not streaming as it is usu- mented by an algorithm in StrSort(O(1),O(logn)). ally defined, since customarily the p passes are performed Correctness. We have to show that in expectation the num- on the input stream, without ever writing any intermediate ber of nodes decreases by a constant factor in each contrac- streams. However, one can easily see that by constructing tion phase. the streams only implicitly, StrSort(p,0,m) can be simu- Suppose the number of nodes before a relabeling phase lated by a p-pass streaming algorithm with memory m p · is n. Since with high probability all randomly assigned num- that does not produce any intermediate streams. bers are distinct, we can assume without loss of generality that the assigned numbers are actually 1,2,...,n , since 3. Algorithms only their relative order matters for the relabeling.{ }Further, we can assume that there are no nodes with out-degree 0 As stated in the introduction, adding the sorting primitive in the graph, because if either s or t are mapped to such a greatly enhances the computational power of the streaming node, we can decide the connectivity question immediately, model, and many natural problems are efficiently solvable and otherwise these nodes can simply be ignored. in the StrSort-model. By “efficiently”, we mean “using lit- A node v can get assigned a label greater than n/2 only if tle memory and few passes”. In particular we are interested both itself and its neighbors get assigned random numbers in algorithms that use poly-logarithmic memory and a poly- greater than n/2. Since this only happens with probability logarithmic number of passes. For this purpose, we define 1/2 per node, and v has at least one neighbor, v gets a la- k k PL-StrSort := kStrSort(O(log n),O(log n)). bel greater than n/2 with probability at most 1/4. Thus, the ∪ Clearly, tasks like computing the median, or computing expected number of nodes that receive a label greater than frequency moments, while hard for the traditional stream- n/2 is only n/4. The expected total number of distinct la- 3 ing model, are trivial in PL-StrSort, requiring only a sin- bels in the relabeled graph is therefore at most 4 n (assum- gle sorting pass to order the input elements. We will now ing the worst case that all labels less than n/2 are actually discuss some non-trivial examples of problems solvable in used). This shows the desired reduction in size. PL-StrSort, but not in the traditional streaming model. Implementation. Let us now sketch the implementation of a contraction phase. We assume that our stream is the list 3.1. Undirected Connectivity V of the vertices v1,v2,...,vn, followed by the list E of the vertex-pairs representing the edges. Since the graph is undi- First, we give a PL-StrSort-algorithm for undirected rected, E contains both pairs (u,v) and (v,u) for an edge s-t-connectivity. For this problem we assume that the in- between u and v. put stream consists of edges (u,v) of an undirected graph While both V and E are part of one stream, we will and two distinguished vertices s and t, and the question is sometimes state operations on the two halves independently, whether there is a path from s to t in the graph. assuming that in such a pass, the other half of the stream is passed through unchanged. Lemma 2 Undirected s-t-connectivity can be solved in randomized 1. First, in a pass over V , we produce a stream R that StrSort(O(logn),O(logn)). contains pairs of node names vi and random distinct numbers ri, e.g. 3logn-bit random numbers: 3. Compute the connected components of (V,E0). 4. We now recursively compute minimum spanning trees (v1,r1)(v2,r2)(v3,r3)...(vn,rn) for

2. Now we determine the new label for each node, i.e. the (a) each connected component in (V,E0), and lowest value of ri among its neighbors and itself. (b) the graph (V 0,E0) where V 0 is the set of connected (a) First, we sort the pair of streams and , so that R E components in (V,E0), and E0 contains the edges the pairs (vi,ri) are directly followed by all edges in E E0 connecting pairs of components. whose first component is v : \ i It is not hard to see that the union of the edges in the in- (v1,r1)(v1, )(v1, )...(v2,r2)(v2, )... dividual spanning trees yields the answer to the minimum spanning tree problem for G. In one pass on this stream, we can produce a new In the StrSort-implementation, the current stream al- stream of edges E 0, where for the first component ways contains a concatenation of all sub-problems currently of each edge, vi is replaced by the corresponding being considered. During a divide step, the edges corre- ri: sponding to the newly created sub-problems might appear (r1, )(r1, )...(r2, )... in an arbitrary order in the input stream. By applying a sort-

(b) Now we sort R and E 0, so that the pairs (vi,ri) ap- ing pass, we can rearrange these edges so that edges belong- pear right before all edges whose second compo- ing to the same sub-problem appear consecutively. nent is equal to vi: The algorithm is similar to Kruskal’s algorithm, but reorders computations in a way that is compatible with (v1,r1)( ,v1)( ,v1)...(v2,r2)( ,v2)... streaming computations. The sorting passes are used to re- arrange the data so that they appear in the correct order for In this stream, the number r assigned to a node i subsequent streaming passes. vi occurs right before the list of numbers as- signed to the nodes incident to vi. Thus, in one linear pass, we can determine the small- 3.3. Red-Blue Line Intersection est number among them for each v , which i We now consider a geometrical problem, moti- yields a stream L of nodes vi with their new la- vated by geometric range queries, called RED-BLUE- bels `i: INTERSECTION. The input is a list of red and blue line seg-

(v1,`1)(v2,`2)(v3,`3)...(vn,`n) ments in the plane. The red line segments are parallel to the x-axis, the blue segments parallel to the y-axis. The out- 3. Now that we know the correct labels, all that remains put is the number of intersection points between red and is to relabel the edges in E. This can be done by re- blue line segments. (In many applications, one is actu- peating step 2(a) for both the first and second compo- ally interested in a list of intersection points, but that might nent of E, using L instead of R . While producing this require an output greater than the input, which our stream- new stream, we can also eliminate all edges of the form ing model does not allow.) (`,`), yielding a new stream of edges Enew. The new list of vertices can be obtained from L by Lemma 3 outputting only the second component `i of each pair, RED-BLUE-INTERSECTION can be computed in determin- sorting the result, and removing duplicates.  istic StrSort(O(logn),O(logn)). Proof: We assume that no endpoint of a blue line segment 3.2. Minimum Spanning Tree lies on a red line segment. This assumption simplifies the presentation, but can be removed by standard techniques. Using the above algorithm for undirected connectivity as Let the slab of a line segment x y x y (with a subroutine, it is not hard to compute minimum spanning ( 1, 1) ( 2, 2) x x ) be the vertical strip x x R of −the plane. Then trees in the “streaming and sorting” model. 1 < 2 [ 1, 2] the following claim follows by a simple× case analysis. We are going to use a divide and conquer approach to compute a minimum spanning tree of a graph G = (V,E). Claim 4 The algorithm is as follows. The number of intersections of red and blue line segments 1. Sort the edges E = e1,e2,...,em by increas- is equal to L U, where { } − ing weight. L := the number of pairs of red line segments s and • 2. Let E0 = e1,e2,...,em/2 be the “lighter” half of the lower end-points of blue line segments p, such that p edges. { } is in the slab of s and below s, and U := the same for upper end-points of blue line seg- Lemma 5 • ments.  A uniform bounded fan-in circuits with width O(n) and depth d(n) can be evaluated in deterministic StrSort us- The streaming algorithm computes the two values L and U ing d(n) streaming and sorting passes, and O(logn) mem- and outputs their difference. For symmetry reasons, it is suf- ory. ficient to show how to compute L. For this, we create a Proof Sketch: This Lemma can be shown similar to PRAM stream of all red line segments and blue lower endpoints. simulations in the external memory model [12]. To evalu- We use a divide-and-conquer approach, also known as dis- ate a circuit, we inductively generate for each level ` of the tributional sweep [24]. circuit a stream S` that contains a list of the inputs taken by The basic idea of the approach is the following. The the circuit nodes on that level, ordered by node. (Note that R input is divided into m vertical slabs [xi,xi+1] where S1 can easily be computed from the input.) One streaming ∞ x x x ∞, such that each slab× contains = 0 < 1 < < m = + pass on S` can compute the outputs of all nodes on level `. roughly− the same··number· of blue lower endpoints. This di- To go from these outputs to S`+1, all we have to do is rear- vision can be easily accomplished by a sort operation. Now range them according to the input pattern of the next level. we sort all these points by increasing y-coordinate. As we This can be done by labeling the outputs with the numbers process this stream, for each of the m slabs, we keep track of the gates that take them as inputs (and creating duplicates of the number of blue lower endpoints seen so far. When if an output is input to multiple gates). Sorting on these la- we encounter a red line segment, then for each slab that it bels yields the desired order.  crosses completely, we add the number of blue lower end- points seen in that slab to our running sum. The (possibly) 4. Hardness two ends of a segment that do not completely cross a slab are recursively passed down to the subproblem in the slab The “streaming and sorting” model is clearly much more it is in. We then recursively solve the m slab problems. In powerful than the traditional streaming model, as evidenced logm n passes, this finds all possible intersections, proving by the algorithms mentioned above. So it is only natural to  Lemma 3. ask what cannot be computed in this model. Intuitively, problems hard for the streaming and sort- We note that a generalization of the above algo- ing model would require data access in unpredictable pat- rithm can be used to solve the red-blue-intersection terns, i.e. the data accesses cannot be rearranged in a poly- problem for arbitrary orientations of red and blue line seg- logarithmic number of fixed-order passes over the data. At ments in StrSort(O(log2 n),O(logn)). More details will be first glance, a good candidate for a hard problem might in the full version of this paper. therefore be “pointer-chasing”. In this problem, one is given an array of n numbers in the range 1,...,n . If the i-th ar- { } 3.4. Simulation of Circuits ray element is j, then we say that the i-th element points to the j-th element. Given a number k, we start at the first el- Many other problems can be solved in PL-StrSort be- ement of the array, and then repeatedly move from the cur- cause they also admit algorithms that access data in a se- rent element to the element it points to. The output is the el- ries of linear passes, interleaved with a known pattern of ement we reach after k steps. data reordering. Examples are substring matching, suffix ar- However, while the pointer-chasing problem seems to re- ray computations, undirected graph connectivity, comput- quire data access in arbitrary patterns, and is indeed very ing maximal independent sets, finding a minimum cut in a hard from a communication complexity point of view [21], undirected graph and many other problems (see [23] for a it can be solved in PL-StrSort. This is because it is possible more comprehensive overview). to square a graph in the StrSort model in a constant num- Some of these results are consequences of the fact that ber of passes, i.e. we can compute the result of following in our computational model, it is quite straightforward to pointers 2,4,8,... times. This allows us to solve the pointer- evaluate uniform linear width, poly-logarithmic depth cir- chasing problem in O(logk) passes in the StrSort model. cuits. Since problems in NC that require only a linear num- ber of processors can be solved by such circuits, this gives 4.1. ALTERNATING SEQUENCE us a systematic way of constructing streaming algorithms for these problems. Examples of such problems are the We now state a problem that builds on the pointer- undirected connectivity algorithm mentioned above, and the chasing paradigm, but while remaining conceptually sim- computation of a maximal independent set [19]. ple, is provably intractable in the StrSort model. The prob- lem is called ALTERNATING SEQUENCE and is defined as follows. Input: A stream of pairs Since we process m of A-elements for each copy of ( ), ∗ (a1,a10 )(a2,a20 )...(an,an0 )(b1,b10 )(b2,b20 )...(bn,bn0 ). after reading the whole stream, the A-elements have been processed upto index m (n/m)1/2 = √nm. This shows that Output: The sequence ai b j ai b j ai b j ..., satisfying 1 1 2 2 3 3 we will make a progress·of √nm on one of the two indices. (i) i1 = 1 In the second (and later) phases, we repeat the same (ii) ik = min i > ik 1 ai = b0j for k 2 pattern, but the A- and B-subsequences of length √nm { − | k 1 } ≥ − start where we left off in the previous phase. Since one (iii) jk = min j > jk 1 b j = ai0 for k 1, using { − | k } ≥ of the indices advances by √nm in each phase, we will j0 = 0. have constructed the whole output in at most 2n/√nm = The sequence ends as soon as either i = n, j = n or k k O((n/m)1/2) phases, which yields the claimed number of the minima in equations (ii) or (iii) do not exist. passes. This problem is best explained by an example. Con- sider the following sequence, where the a a -pairs are ( i, i0) 4.3. Lower bound in the top stream, and the (b j,b0j)-pairs are in the bot- tom stream. For this example, the output would start with We now show the harder part of the Theorem, the lower 1,7,4,17,2,3,1,11,.... bound of p = Ω(n1/3/m) passes for the StrSort model. We Stream a show this bound by fixing an arbitrary algorithm, and then (1,7) (2,3) (4,17) (5,19) (2,3) (11,1) (1,11) adversarially choosing its input such that it cannot output the correct solution unless the number of passes meets the claimed bound. In this adversarial model we only decide (17,3) (7,4) (4,17) (3,8) (7,3) (17,2) (3,1) on the equality of two items when the algorithm compares Stream b them. Theorem 6 The proof consists of two main parts. First, we show that The problem ALTERNATING SEQUENCE can be solved in in a single pass, we cannot make too much progress towards StrSort(O((n/m)1/2),m). However, it cannot be solved in the solution of the problem. This is because whatever O(n)- deterministic StrSort(p,m) unless pm = Ω(n1/3). We as- size input stream we use in that pass, there will be a roughly n/m-length alternating sequence that we cannot output sume that elements are indivisible, and that only compar- p isons between elements are allowed. based on linearly scanning the stream. In the second part of our proof, we apply this con- 4.2. Upper bound struction to a sequence of passes, which slightly weakens the n/m-bound as the algorithm gains more information p 1/3 Proof: We first give an algorithm to compute ALTER- about the processed data. In the end, it yields the Ω(n /m) NATING SEQUENCE in StrSort(O((n/m)1/2),m). The al- lower bound on the number of passes. gorithm proceeds in 2(n/m)1/2 phases. In the first phase, The key lemma for the first half of the proof is the fol- we construct a stream containing (n/m)1/2 copies of the se- lowing. It shows that a string that contains all possible AL- quence TERNATING SEQUENCE answers as subsequences must be 2 ( ) A1A2 ...A√nmB1B2 ...B√nm, very long (at least n + 1 symbols). The converse of this ∗ statement is what we will need: a string of length O(n) can where Ai and Bi are short for (ai,a ) and (bi,b ), respec- i0 i0 only contain all ALTERNATIVE SEQUENCE answers for in- tively (we will continue to use this notation for the remain- stances of length O(√n). Here and in the following, we will der of this proof). Clearly, the stream has length O(n), and use A and B to stand for (a ,a ) and (b ,b ), respectively. can be constructed by outputting elements multiple times, i j i i0 j 0j indexed by their desired position on the tape, followed by a sorting pass. Lemma 7 We can use this stream to output the beginning of the an- Let S be the set of alternating increasing sequences of the al- swer upto the point where either of the indices ik or jk be- phabet Σ = A1,...,An, B1,...,Bn , i.e. strings of the form { } comes greater than √nm. This is done as follows. When [Ai1 ]B j1 Ai2 B j2 ...Aik [B jk ] where i` < i`+1 and j` < j`+1 for reading the first sequence of A’s, we keep A1,A2,...,Am 1 ` < k, and at most the very last symbol is equal to An or ≤ in memory. This enables us to construct the answer upto Bn. By [...] we mean that the symbol is optional. Let s be ik m, jk √nm on the following sequence of B’s. When a Σ-string that contains all strings in S as subsequences (i.e. we≤exhaust≤the A’s in our memory, we continue to the next the elements of each s S occur in order in s, but not nec- ∈ 2 stretch of A’s, and put Am+1,...,A2m in memory, use that essarily consecutively). Then s has length at least n + 1. with the following stretch of B’s, and so on.  Although we will not need this later, it is interesting Thus, going from i to i + 1 there will be at most one enough to note that the bound in the lemma is actually tight, α(...,k,`), β(...,k,`) pair for which one of the predicates i.e. there are strings of length n2 + 1 of the desired form. becomes false, after both having been true so far. Since at 2 Let ai to be the string AiAi+1 ...An 1 and bi be the string the beginning, all n such α,β-pairs are true, it takes at least − 2 2 BiBi+1 ...Bn 1. Then the following string of length n + 1 n elements of s to hit all these pairs once. And for the last has the desired− properties: pair hit, it takes one more character to satisfy the remain- ing true predicate in it, so s has to contain at least n2 + 1 b1a1b1a2b2a3b3a4b4 ...an 1bn 1AnBn characters.  − − We leave the details to the reader, and concentrate on the Lemma 7 shows that is hard to interleave the two streams lower bound. A1A2 ...An and B1B2 ...Bn into one stream such that all pos- sible answers to ALTERNATING SEQUENCE appear as sub- Proof (Lemma 7): Fix s Σ , and let s be the suffix of s ∗ i sequences in the interleaved stream. For our application of that begins at position i (e.g.∈ s = s). We will now define 1 streaming passes, we are however interested in the mini- sets of strings S such that all strings in S have to appear as i i mum stream length such that a memory m algorithm could subsequences in si. We set S1 := S, and define the other Si produce all possible answers to ALTERNATING SEQUENCE. inductively. If the i-th symbol of s is A , then we set k This is answered by the following corollary.

Si+1 = s “s does not start with Ak” and (s Si Aks Si) . { | ∈ ∨ ∈ } Corollary 9 Here “A s” stands for the string obtained by prepending A k k Let s be a string, such that all alternating sequences of the to s. Replacing A with B gives the definition in the case k k form stated in Lemma 7 can be output by a linear scan of s that the i-th symbol of s is B . k by a machine of memory m. Then the length of s is at least It is not hard to see that for each i the strings in Si nec- n 2  ( m+1 ) + 1. essarily have to be contained as subsequences in si. So to show the desired lower bound on the length of s, it suffices 2 Note that Corollary 9 is not tight, consider e.g. the case to show that Si = 0/ for i n + 1. 6 ≤ m = Ω(n), where the lower bound is just a constant, but Let us define predicates α(i,k,`) and β(i,k,`) as “Si con- clearly each Ai and each Bi has to appear at least once in s, tains a string with the substring AkB ” and “Si contains a ` giving a trivial lower bound of 2n. But the corollary is still string with the substring B Ak”, respectively. If at least one ` strong enough for our purposes. of these predicates is true, it implies that Si = 0/. We will now show that going from i to i + 1, not too man6 y of the pred- Proof (Corollary 9): Group the elements of A ,A ,...,A icates change from true to false. The intuition behind this 1 2 n into n groups of m + 1 consecutive elements each (i.e. is that some predicates imply other predicates. For exam- m+1 A ,...,A form the first group, A ,...,A the sec- ple α(i,k,`) implies β(i,k ,`) for all k > k, since an alter- 1 m+1 m+2 2m+2 0 0 ond, and so on), and do the same for the B’s. By “inter- nating sequence which contains A B can be continued as k ` leaving an A-group with a B-group” we mean that we al- AkB A for any k > k. These implications limit the num- ` k0 0 ternate the m + 1 elements in the A-group in ascending or- ber of predicates that can become false. We will show the der with the m + 1 elements of the B-group, for example following. A1B1A2B2 ...Am+1Bm+1 for the first two groups. Now con- sider only the strings in S that are concatenations of such Claim 8 interleaved groups. Let k,k0,`,`0 be numbers between 1 and n with k = k0 or ` = ` . If α(i,k,`), β(i,k,`), α(i,k ,` ) and β(i,k ,6 ` ) are Any memory m algorithm outputting these strings upon 0 0 0 0 0 s all6 true, then at least three of α(i + 1,k,`), β(i + 1,k,`), reading must for each interleaved group pair read at least one A-element and one B-element from s. This is because α(i + 1,k0,`0) and β(i + 1,k0,`0) are true. the groups have size m + 1 each, so they could not possi- Proof: Let us consider the case where the i-th element of s bly have been entirely in memory before outputting the in- is an A-element, say A j (the “B-case” is similar). Then the terleaved group pair. β-predicates will be unchanged from i to i + 1. And the α- By restricting our view to “representatives of groups”, predicates can only change if j = k or j = k0. In fact, the not distinguishing the individual elements of groups, the only way that both these predicates could become false is if previous observation implies that s must actually be an in- j = k = k0 holds. This implies ` = `0, wlog ` < `0. But the terleaved string in the sense of Lemma 7 on the group rep- β 6 n truth of (i,k,`) then implies that Si and therefore Si+1 con- resentatives. Since there are m+1 A- and B-groups each, α tain strings that contain B`AkB` , and thus (i + 1,k,`0) re- Lemma 7 therefore implies that s has to have length at least 0  n 2  mains true, which proves the claim. ( m+1 ) + 1. For our adversarial argument, we now fix a particular the constant used for the O(n) upper bound on stream streaming algorithm. We allow the algorithm to construct lengths. arbitrary input streams for each of its passes. Even in this In summary, we can accomplish the selection of ti’s and more powerful model, we can still prove the lower bound. s ’s as long as t t = Ω( nm3(i + 1)) for all i. This i i+1 i p To fix the input we will now inductively construct a se- gives the following upper− bound on k: quence of numbers 1 = t1 < t2 < t3 < < tk = n, and al- ··· ternating sequences s1 s2 s3 sk (where by s s0 k k ⊂ ⊂ ⊂ ··· ⊂ ⊂ Ω √ 3 √ 3 we mean that s is a prefix of s0), such that for all 1 i < k: ∑ ( nm i) n = ∑ i = O(qn/m ) ≤ i=1 ≤ ⇒ i=1 (i) sk is the correct output to the ALTERNATING SE- ∑k 3/2 QUENCE problem, Since i=1 √i = O(k ), this means the construction is possible for k up to ( n/m3)2/3 = n1/3/m, which con- (ii) all elements of si not in si 1 are from the set p − cludes the proof of Theorem 6.  Ati ,...,At 1,Bti ,...,Bt 1 , and { i+1− i+1− } (iii) si is chosen so that, even based on the comparisons 4.4. A decision version of ALTERNATING SE- the algorithm performed in the first i 1 passes, it is QUENCE not possible for the algorithm to output− the elements of si si 1 in order during pass i. \ − A polynomial separation analogous to Theorem 6 can The last point implies in particular that the algorithm can- also be shown using the following decision problem: the in- not output the correct solution within k passes. put is just like ALTERNATING SEQUENCE, but now every For the first step of the inductive construction, we choose pair (ai,ai0) and (b j,b0j) has a color which is either red or 2 t2 such that (t2/m + 1) + 1 is greater than the length of blue. The desired answer is whether the last element of the the stream in pass 1. This means that t2 can be chosen output of ALTERNATING SEQUENCE is a red or a blue ele- as O(m√n), where the constant depends only on the con- ment. stant in the O(n) bound we imposed on the maximal stream We briefly sketch how the computation of the ALTER- length. By Corollary 9 this implies that there will be an al- NATING SEQUENCE function can be reduced in StrSort ternating sequence of the Ai and Bi with indices bounded to this decision problem. We will do this with a factor 2 by t2 that cannot be output by a memory m algorithm upon O(log n) increase in the number of required passes. Thus, reading the stream. We let s1 be one such sequence. the hardness of the function implies the hardness of the de- For the inductive step, we have to modify our argument cision problem. to account for the fact that the algorithm already has made So suppose we have an oracle for the decision problem (a some comparisons in the previous passes, and our choice of bit of care shows that each invocation of this oracle can be si must be consistent with them. replaced by a corresponding StrSort algorithm in the fol- In the previous i 1 passes, the algorithm can have per- lowing). First, this allows us to find the last element of the formed a total of O((−i 1)nm) comparisons – each element ALTERNATING SEQUENCE output (and not just its color) in the memory could ha−ve been compared to every element in O(logn) streaming passes. This is easily done using bi- in the first i 1 streams. We are enforcing the policy that nary search: color half the input pairs red and half the pairs whenever a comparison− during pass j involves an element blue, and invoke the oracle. This shows us in which half of with index greater than t j+1, then we will always return “un- the pairs the last element is. We then recurse on this half (color it half red/half blue), and so on. After O logn appli- equal”. Also, we enforce that ai = a j, ai0 = a0j, bi = b j and ( ) b = b for all i = j. 6 6 6 cations of the oracle, we know the identity of the last ele- i0 6 0j 6 For pass i, we therefore want to choose ti+1 such that ment. there is an alternating sequence si+1 si with indices be- The algorithm producing the last element of the AL- \ tween ti and ti+1 not contained in a stream of length O(n), TERNATING SEQUENCE output can be used to give and such that the sequence does not consecutively contain a divide-and-conquer solution to ALTERNATING SE- any of the O((i 1)nm) pairs for which we already an- QUENCE itself. For this, we consider the first half of − swered “unequal”. If one excludes a set of ` (A,B)-pairs the input streams (a1,a10 )(a2,a20 )...(an/2,an0 /2) and from appearing consecutively in the strings of S, then a sim- (b1,b10 )(b2,b20 )...(bn/2,bn0 /2). Given the above algo- ple modification of the proof of Lemma 7 yields a lower rithm, we can compute the last two elements in the AL- 2 bound on s of n ` + 1. TERNATING SEQUENCE output for these half-problems. | | − So we have to choose ti+1 such that ((ti+1 ti)/(m + (The second-to-last element can be computed by first com- − 1))2 nm(i + 1) + 1 > O(n), which can be satisfied by puting the last element, and then deleting it and the part of − 2 3 ti+1 ti = Ω( nm + nm (i + 1)), in particular by ti+1 the stream following it from the input, and again comput- − 3 p − ti = Ω(√nm √i + 1), where the constant only depends on ing the last element.) The knowledge of these elements “in the middle” of the 5.2. Relation to StrSort output allows us to split the input streams in half, and con- struct solutions for both halves independently. Note that in While LEMA is somewhat similar to the StrSort model each divide step, at least one of the streams gets split ex- (in that the data is read and written sequentially), the PL- actly in half, showing that only O(logn) recursions are nec- LEMA model turns out to be strictly more powerful than essary. Thus, ALTERNATING SEQUENCE can be reduced to the PL-StrSort model. Since we can sort in LEMA using the red/blue decision problem, while multiplying the num- a logarithmic number of passes, we can simulate any algo- 2 ber of passes by O(log n). rithm for the PL-StrSort model in the PL-LEMA model while increasing the number of passes by at most a loga- rithmic factor. On the other hand, the ALTERNATING SE- 5. Linear Access External Memory Algo- QUENCE problem which needs at least a polynomial num- rithms ber of passes in StrSort (as shown in Theorem 6) can be solved very efficiently (using only two passes) in PL- 5.1. Definitions LEMA. The picture in section 4.1 suggests how to solve it using concurrent access to two disks. First, one divides the input onto two disks, one containing the a-pairs, and the In this section, we will elucidate the relationship of the other the b-pairs. Then, we begin by reading a , and scan- StrSort model to the well-studied external memory model. 10 ning the second disk until a matching b is found. Then we External memory algorithms (EMAs) study the effect that j scan the first disk to find an a matching b , and so on, al- block-oriented external data storage has on the efficiency of i 0j ternating between the two disks. In this sense, the solution algorithms (see [24] for a recent survey). An external mem- makes maximal use of the fact that it can read the two disks ory algorithm can access the external storage (disk) only independently. in units of blocks (each containing B items), and perfor- mance is measured in terms of the total number of disk ac- cesses (reads and writes). Thus, a good data storage scheme and exploitation of locality are necessary for efficient al- 6. Conclusion gorithms. The parallel disk model introduced by Vitter and Shriver [25] has the following parameters: In this paper, we have studied a model for massive data set computations – which we think can be efficiently imple- N = problem size (in data items), mented – by extending the streaming model with a sorting M = internal memory (in data items), primitive. This model allows for the efficient solution of a variety of natural problems known to be hard for the stream- B = block transfer size (in data items), and ing model, but computable using stronger models which are D = number of independent disk drives. harder to implement in practice. These problems include undirected connectivity, MST, red-blue line intersections, In this paper we are only interested in the single proces- and several others. We have also proved one of the first hard- sor case, so we will ignore the (sometimes studied) addi- ness results for models of massive data set computations tional parameter of the number of processors. that are strict extensions of the streaming model. The com- As mentioned in the introduction, reading data items munication complexity based techniques which were pre- sequentially from a disk achieves a substantially higher viously used in showing lower bounds for the streaming throughput than performing random accesses. Thus, exter- model do not generalize to our model, and consequently, nal memory algorithms that only perform sequential reads new methods had to be developed. or writes are a particularly efficient subclass of EMAs. We Since this computational class is a quite natural exten- therefore define a linear access EMA (LEMA) as an EMA sion of the traditional streaming model, and efficiently im- with the following parameter: plementable in practice, it deserves further study. For ex- P = number of out-of-sequence reads/writes on the disks ample, can a breadth first or depth first traversal of a graph (“passes”) be computed efficiently in this model? Can our lower- bound result be extended to randomized algorithms or the A read or write operation is considered “out-of- bit model, where operations other than comparisons are al- sequence” if it does not act on the data block follow- lowed? And are there more natural problems that are hard ing the one last read or written on the corresponding disk. to solve in this model, yet can be realistically solved on We are interested in algorithms for which P and M are massive data sets? A candidate problem could be SUBSE- small, i.e. poly-logarithmic in N. The class of these prob- QUENCE, in which given two strings, the problem is to de- lems is denoted by PL-LEMA. cide whether one is a subsequence of the other. We have also compared the “streaming and sorting” [11] L. Chavet. FSORT, 2002. Available online at www.fsort. model to a model we called LEMA, which is a restriction of com. external memory algorithms to mostly linear accesses to ex- [12] Y.-J. Chiang, M. T. Goodrich, E. F. Grove, R. Tamassia, D. E. ternal storage. We showed that the ability to read input from Vengroff, and J. S. Vitter. External-Memory Graph Algo- two disks simultaneously makes LEMA strictly more pow- rithms. In Proceedings SODA, pages 139–149, 1995. erful than the “streaming and sorting” model. As mentioned [13] M. Farach, P. Ferragina, and S. Muthukrishnan. Overcom- above, it would be interesting to find more natural prob- ing the Memory Bottleneck in Suffix Tree Construction. In Proceedings FOCS, pages 174–183, Nov. 1998. lems than ALTERNATING SEQUENCE that separate the two classes. It is also an interesting open problem whether in- [14] M. Farach-Colton, P. Ferragina, and S. Muthukrishnan. On the Sorting-Complexity of Suffix Tree Construction. Journal creasing the number of independently accessible disks in of the ACM, 47(6):987–1011, 2000. LEMA always leads to a polynomial increase in computa- [15] S. Guha, S. Krishnan, K. Mungala, and S. Venkatasubrama- tional power. We saw such an increase in the case of one nian. Application of the Two-Sided Depth Test to CSG Ren- versus two disks as demonstrated by ALTERNATING SE- dering. In Proceedings of SIGGRAPH Symposium on Inter- QUENCE. active 3D Graphics, 2003. [16] M. R. Henzinger, P. Raghavan, and S. Rajagopalan. Comput- Acknowledgments ing on data streams. In DIMACS series in Discrete Mathe- matics and Theoretical Computer Science, volume 50, pages We thank Jon Feldman for his help in proving Lemma 107–118, 1999. 7, the anonymous reviewers for their comments, and David [17] G. Humphreys, M. Eldridge, I. Buck, G. Stoll, M. Everett, Karger, T.S. Jayram, Piotr Indyk, Ron Fagin, and the partic- and P. Hanrahan. WireGL: A Scalable Graphics System for ipants of the DIMACS Working Group on Streaming Data Clusters. In Proceedings of SIGGRAPH, pages 129–140, 2001. Analysis for helpful discussions. [18] G. Humphreys, M. Houston, Y.-R. Ng, R. Frank, S. Ahern, P. Kirchner, and J. T. Klosowski. Chromium: A Stream Pro- References cessing Framework for Interactive Graphics on Clusters. In Proceedings of SIGGRAPH, pages 693–702, 2002. [1] J. Abello, A. L. Buchsbaum, and J. R. Westbrook. A Func- [19] M. Luby. A Simple Parallel Algorithm for the Maximal In- tional Approach to External Graph Algorithms. Algorith- dependent Set Problem. In Proceedings STOC, pages 1–10, mica, 32:437–458, 2002. 1985. [2] P. Agarwal, S. Krishnan, N. Mustafa, and S. Venkatasubra- [20] J. I. Munro and M. S. Paterson. Selection and Sorting with manian. Streaming Geometric Optimization Using Graphics Limited Storage. Theoretical Computer Science, 12:315– Hardware. Technical Report TD-5HL2NX, AT&T, 2003. 323, 1980. [3] R. C. Agarwal. A Super Scalar Sort Algorithm for RISC Pro- [21] S. J. Ponzio, J. Radhakrishnan, and S. Venkatesh. The com- cessors. In Proceedings SIGMOD, pages 240–246, 1996. munication complexity of pointer chasing. Journal of Com- [4] A. Aggarwal and J. S. Vitter. The input/output complexity of puter and System Sciences (JCSS), 62(2):323–355, 2001. sorting and related problems. Communications of the ACM, [22] T. J. Purcell, I. Buck, W. R. Mark, and P. Hanrahan. Ray 31(8):1116–1127, 1988. Tracing on Programmable Graphics Hardware. In Proceed- [5] N. Alon, Y. Matias, and M. Szegedy. The Space Complex- ings of SIGGRAPH, pages 703–712, 2002. ity of Approximating the Frequency Moments. Journal of [23] M. Ruhl. Ef®cient Algorithms for New Computational Mod- Computer and System Sciences, 58(1):137–147, 1999. els. PhD thesis, Massachusetts Institute of Technology, Sept. [6] A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, D. E. Culler, 2003. J. M. Hellerstein, and D. A. Patterson. High-Performance [24] J. S. Vitter. External Memory Algorithms and Data Struc- Sorting on Networks of Workstations. In Proceedings SIG- tures: Dealing with Massive Data. ACM Computing Surveys, MOD, pages 243–254, 1997. 33(2):209–271, 2001. [7] Z. Bar-Yossef, T. Jayram, R. Kumar, and D. Sivakumar. An [25] J. S. Vitter and E. A. M. Shriver. Algorithms for parallel information statistics approach to data stream and communi- memory I: Two-level memories. Algorithmica, 12(2-3):110– cation complexity. In Proceedings FOCS, 2002. 147, 1994. [8] M. A. Bender, R. Cole, E. D. Demaine, and M. Farach- [26] J. Wyllie. SPsort: How to Sort a Terabyte Quickly. Technical Colton. Scanning and Traversing: Maintaining Data for report, IBM Almaden Research Center, 1999. Available on- Traversals in a Memory Hierarchy. In Proceedings ESA, line at www.almaden.ibm.com/cs/gpfs-spsort. pages 139–151, Sept. 2002. html. [9] A. Borodin, M. N. Nielsen, and C. Rackoff. (Incremen- tal) Priority Algorithms. Algorithmica, 37(4):295–326, Sept. 2003. [10] I. Buck and P. Hanrahan. Data Parallel Computation on Graphics Hardware. Manuscript, 2003.