Polarized Data Parallel Data Flow
Total Page:16
File Type:pdf, Size:1020Kb
Polarized Data Parallel Data Flow Ben Lippmeierα Fil Mackayβ Amos Robinsonγ α,β Vertigo Technology (Australia) α,γ UNSW (Australia) γ Ambiata (Australia) fbenl,fi[email protected] fbenl,[email protected] [email protected] Abstract • We use continuation passing style (CPS) to provoke the Glas- We present an approach to writing fused data parallel data flow pro- gow Haskell Compiler into applying stream fusion across grams where the library API guarantees that the client programs chunks processed by independent flow operators. (x3.1) run in constant space. Our constant space guarantee is achieved by Our target applications concern medium data, meaning data that observing that binary stream operators can be provided in several is large enough that it does not fit in the main memory of a normal polarity versions. Each polarity version uses a different combina- desktop machine, but not so large that we require a cluster of tion of stream sources and sinks, and some versions allow constant multiple physical machines. For lesser data one could simply load it space execution while others do not. Our approach is embodied in into main memory and use an in-memory array library. For greater the Repa Flow Haskell library, which we are currently using for data one needs to turn to a distributed system such as Hadoop [16] production workloads at Vertigo. or Spark [19] and deal with the unreliable network and lack of Categories and Subject Descriptors D.3.4 [Programming Lan- shared memory. Repa Flow targets the sweet middle ground. guages]: Processors—Compilers; Optimization Keywords Arrays; Fusion; Haskell 2. Streams and Flows A stream is an array of elements where the indexing dimension is 1. Introdution time. As each element is read from the stream it is available only in that moment, and if the consumer wants to re-use the element at a The functional language ecosystem is blessed with a multitude of later time it must save it itself. A flow is a bundle of related streams, libraries for writing streaming data flow programs. Stand out exam- where each stream carries data from a single partition of a larger ples for Haskell include Iteratee [8], Enumerator [14], Conduit [17] data set — we might create a flow consisting of 8 streams where and Pipes [5]. For Scala we have Scalaz-Streams [3] and Akka [9]. each one carries data from a 1GB partition of a 8GB data set. We Libraries like Iteratee and Enumerator guarantee that the client manipulate flow endpoints rather than the flows themselves, using programs run in constant space, and are commonly used to deal the following data types: with data sets that do not fit in main memory. However, the same libraries do not provide a notion of parallelism to help deal with the data Sources i m e implied amount of data. They also lack support for branching data = Sources flows where produced streams are consumed by several consumers { arity :: i without the programmer needing to hand fuse the consumers. We , pull :: i -> (e -> m ()) -> m () -> m () } provide several techniques to increase the scope of such stream processing libraries: data Sinks i m e • Our parallel data flows consist of a bundle of streams, where = Sinks each stream can be processed in a separate thread. (x2) { arity :: i , push :: i -> e -> m () • Our API uses polarized flow endpoints (Sources and Sinks) , eject :: i -> m () } to ensure that programs run in constant space, even when pro- duced flows are consumed by several consumers. (x2.3) Type Sources i m e classifies flow sources which produce • We show how to design the core API in a generic fashion so elements of type e, using some monad m, where the streams in that chunk-at-a-time operators can interoperate smoothly with the bundle are indexed by values of type i. The index type i is element-at-a-time operators. (x3) typically Int for a bundle of many streams, and () for a bundle of a single stream. Likewise Sinks i m e classifies flow sinks which consume elements of type e. In the Sources type, field arity stores the number of individ- ual streams in the bundle. To receive data from a flow producer we apply the function in the pull field, passing the index of type i for the desired stream, an eat function of type (e -> m ()) to con- This is the author’s version of the work. It is posted here for your personal use. Not for sume an element if one is available, and an eject computation of redistribution. The definitive version was published in the following publication: type (m ()) which will be invoked when no more elements will FHPC’16, September 22, 2016, Nara, Japan ever be available for that stream. The pull function will then per- ACM. 978-1-4503-4433-3/16/09... form its (m ()) computation, for example reading a file, before http://dx.doi.org/10.1145/2975991.2975999 calling our eat or eject, depending on whether data is available. 52 In the Sinks type, field arity stores the number of individual sourceFs :: [FilePath] -> IO (Sources Int IO Char) streams as before. To send data to a flow consumer we apply the sourceFs names push function, passing the stream index of type i, and the element = do hs <- mapM (\n -> openFile n ReadMode) names of type e. The push function will perform its (m ()) computation let pulls i ieat ieject to consume the provided element. If no more data is available we = do let h = hs !! i instead call the eject function, passing the stream index of type eof <- hIsEOF h i, and this function will perform an (m ()) computation to shut if eof then hClose h >> ieject down the sink — possibly closing files or disconnecting sockets. else hGetChar h >>= ieat Consuming data from a Sources and producing data to a Sinks return (Sources (length names) pulls) is synchronous, meaning that the computation will block until an element is produced or no more elements are available (when con- sinkFs :: [FilePath] -> IO (Sinks Int IO Char) suming); or an element is consumed or the endpoint is shut down sinkFs names (when producing). The eject functions used in both Sources and = do hs <- mapM (\n -> openFile n WriteMode) names Sinks are associated with a single stream only, so if our flow con- let pushs i e = hPutChar (hs !! i) e sists of 8 streams attached to 8 separate files then ejecting a single let ejects i = hClose (hs !! i) stream will close a single file. return (Sinks (length names) pushs ejects) drainP :: Sources i IO a -> Sinks i IO a -> IO () 2.1 Sourcing, Sinking and Draining drainP (Sources i1 ipull) (Sinks i2 opush oeject) Figure 1 gives the definitions of sourceFs, sinkFs which create = do mvs <- mapM makeDrainer [0 .. min i1 i2] flow sources and sinks based on a list of files (Fs = files), as well mapM_ readMVar mvs as drainP which pulls data from a flow source and pushes it to a where sink in P-arallel. We elide type class constraints to save space. makeDrainer i = do Given the definition of the Sources and Sinks types, writing mv <- newEmptyMVar sourceFs and sinkFs is straightforward. In sourceFs we first forkFinally (newIORef True >>= drainStream i) open all the provided files, yielding file handles for each one, (\_ -> putMVar mv ()) and the pull function for each stream source reads data from the return mv corresponding file handle. When we reach the end of a file we eject drainStream i loop = the corresponding stream. In sinkFs, the push function writes the let eats v = opush i v provided element to the corresponding file, and eject closes it. ejects = oeject i >> writeIORef loop False The drainP function takes a bundle of stream Sources, a bun- in while (readIORef loop) (ipull i eats ejects) dle of stream Sinks, and drains all the data from each source into the corresponding sink. Importantly, forks a separate drainP Figure 1. Sourcing, Sinking and Draining thread to drain each stream, making the system data parallel. We use Haskell MVars for communication between threads. After fork- ing the workers, the main thread waits until they are all finished. Besides map_o and fold_o we also need an operator to branch Now that we have the drainP function, we can write the “hello the flow so that the same data can be passed to our counting world” of data parallel data flow programming: copy a partitioned operators as well as written back to the file system. data set from one set of files to another: The branching operator we use is as follows: copySetP :: [FilePath] -> [FilePath] -> IO () dup_ooo :: (Ord i, Monad m) copySetP srcs dsts => Sinks i m a -> Sinks i m a -> Sinks i m a = do ss <- sourceFs srcs dup_ooo (Sinks n1 push1 eject1) sk <- sinkFs dsts (Sinks n2 push2 eject2) drainP ss sk = let pushs i x = push1 i x >> push2 i x ejects i = eject1 i >> eject2 i 2.2 Stateful Streams, Branching and Linearity in Sinks (min n1 n2) pushs ejects Suppose we wish to copy our files while counting the number of This operator takes two argument sinks and creates a new one. characters copied. To count characters we can map each to the When we push an element to the new sink it will push that element constant integer one, then perform a fold to add up all the integers. to the two argument sinks. Likewise, when we eject a stream in To avoid reading the input data from the file system twice we use a the new sink it will eject the corresponding stream in the two counting process that does not force the evaluation of this input data argument sinks.