Workflow representation and runtime based on lazy functional streams Matthew J. Sottile, Geoffrey C. Hulette, and Allen D. Malony {matt,ghulette,malony}@cs.uoregon.edu Department of Computer and Information Science University of Oregon Eugene, OR 97403 ABSTRACT tions on streams. This representation is both concise and Workflows are a successful model for building both distributed powerful, with well-defined semantics. We get a number of and tightly-coupled programs based on a dataflow-oriented features from Haskell for free, including sophisticated static coordination of computations. Multiple programming lan- analysis and type checking, an efficient binding to C rou- guages have been proposed to represent workflow-based pro- tines, and portability to a wide range of platforms. Further- grams in the past. In this paper, we discuss a representa- more, recent additions to the Haskell language for multi- tion of workflows based on lazy functional streams imple- threaded applications have proven to be useful for managing mented in the strongly typed language Haskell. Our intent side-effects of stream operations while remaining in a purely is to demonstrate that streams are an expressive intermedi- functional setting. ate representation for higher-level workflow languages. By embedding our stream-based workflow representation in a We will first introduce (or refresh) the reader to the concept language such as Haskell, we also gain with minimal effort of lazy functional streams, and describe how computational the strong type system provided by the base language, the activities within workflows can be lifted into the stream- rich library of built-in functional primitives, and most re- ing model. A set of stream-based composition and control cently, rich support for managing concurrency at the lan- primitives will be presented in order to support sophisticated guage level. structures in workflow-based programs. Finally, we will dis- cuss and demonstrate a few example workflows that can be 1. INTRODUCTION concisely represented using streams. Workflows have emerged as a successful programming model for building both tightly-coupled and distributed systems 2. RELATED WORK out of existing software components. There exists a large There are many approaches to encoding workflows as exe- and growing ecosystem of programming languages, runtime cutable programs. systems, and reusable components that support the work- flow model. There are several languages, such as WOOL [7] Kepler [1, 11] is a system for composing scientific workflows. and AGWL [3], that provide a high-level syntax for describ- Its execution model is taken from Ptolemy II [8]. Ptolemy ing workflows as a set of activities whose inputs and outputs II is based on a generalized actor model, where actors rep- are interconnected by channels of data. It is a challenge to resent computations, and have inputs and outputs that are translate this fairly abstract representation down into an connected via uni-directional channels. The semantics of the executable program. Here, we describe a general-purpose workflow are configurable. For example, one workflow may executable representation of workflows, based on lazy func- employ the Process Network model [10], where writes to a tional streams. Our intent is to provide a well defined repre- channel are non-blocking, reads from a channel will block sentation of workflows based on functional streams, allowing if there is no available data, and an actor will \fire" only this representation to be the target of higher level workflow in the event that all inputs are available. Another work- language compilers. flow may choose Synchronous Dataflow semantics, where stronger guarantees of progress are available at the cost of In particular, we leverage the functional programming lan- more specification [9]. guage Haskell to encode workflows as a composition of func- Taverna [15] uses a kind of Lambda calculus as its coordi- Permission to make digital or hard copies of all or part of nation formalism. The semantics detailed in [16] are strict, this work for personal or classroom use is granted without fee such that that activities must process all incoming elements provided that copies are not made or distributed for profit before the next activity sees any output. Though simple to or commercial advantage and that copies bear this notice understand, strictness limits pipeline parallelism and expres- and the full citation on the first page. To copy otherwise, siveness. Streams are the idea that solves these problems. to republish, to post on servers or to redistribute to lists, Taverna is incorporating a notion of streams in their next requires prior specific permission and/or a fee. software version. WORKS 09, November 15, 2009, Portland Oregon, USA Copyright c 2009 ACM 978-1-60558-717-2/09/11... $10.00 Functional reactive programming is similar to our work, in that it also deals with using functional concepts to compose The stream version of this function is similar, but it con- stream-like data types [5]. In FRP, the basic data type is a sumes values from a potentially unbounded list of inputs, \signal," which is a function from some type representing a and produces a potentially unbounded list of outputs. Both continuous value that varies with time to a type representing versions, however, employ exactly the same logic. The stream- discrete events. Signal functions, then, take a signal as input ing version of square (using Haskell's list as a simple kind and produce another signal as output. Programs are writ- of stream data structure) looks like this: ten using a minimal set of signal combinators. These com- binators are equivalent to Hughes' arrow combinators [6], and many FRP systems have adopted these are their set of squarestream :: [Int] -> [Int] primitives. We are currently examining the possibility of squarestream (x:xs) = (x*x):(squarestream xs) a formal connection between arrows and our own workflow representation. In the stream model, the function is defined to take a list of Dataflow languages like Sisal [4], Id [13], and pH [12] are inputs (x followed by the remainder of the list, xs), and pro- related to modern workflow languages. Dataflow languages duce a list containing the value x squared followed by the view the entire program as combinations of dataflow rela- recursive application of the function to the remaining ele- tionships, all the way down to fine grained operations such ments of the input list. Notice the lack of a recursive base as arithmetic. Workflow languages employ many of the same case handling the empty list { this is not a function on fi- dataflow concepts, but are usually concerned with the coor- nite lists, but rather on streams with unbounded length. We dination of more coarse-grained operations, and the move- can improve this function by making the acquisition of the ment of data between them. Workflow languages do not head and tail explicit, allowing us to maintain the stream impose this flow-based programming model on the internal abstraction while giving us the flexibility change the under- implementation of these coarse-grained computations. Of lying stream representation: note, Id originally introduced MVars, which are now found in Haskell, and which we use extensively in our representa- squarestream :: [Int] -> [Int] tion of effectful streams. squarestream s = (x*x):(squarestream xs) where x = head s In this paper, we give details on how to construct several xs = tail s workflow primitives. Although we do not make the connec- tion explicitly for each primitive, they are represented in the set of workflow patterns described in [17]. Abstracting this pattern to make a higher-order function representing a stream computation based on a singleton func- We use Haskell because it has convenient features for our tion is straightforward: purposes, including simple expression of streams and built-in MVars. Conceptually, however, our representation is language- independent. The stream data structure can be encoded in applyToIntStream :: [Int] -> (Int -> Int) -> [Int] almost any language. MVars are easy to implement in any applyToIntStream s f = (f x): language with concurrency and locking. (applyToIntStream xs f) where x = head s Yet another approach to executable workflow representation xs = tail s is rule-based workflow execution [2]. In this paradigm, ac- tivity firings are triggered by the fulfillment of declarative rules, not unlike the rules in a Makefile. Furthermore, all In this case, any function that takes a single integer and pro- data is stored into relational database tables, making it easy duces an integer can be applied to an unbounded stream. We to examine the provenance of any particular result, and to can abstract the pattern even further by making the argu- re-run partial sections of a workflow based on updated rules ments polymorphic. This function is a general application or activities. of a singleton function to each element of a stream: 3. LAZY FUNCTIONAL STREAMS applyToStream :: [a] -> (a -> b) -> [b] The fundamental primitive in our model is that of a lazy applyToStream s f = (f x):(applyToStream xs f) functional stream. There are two important parts to this where x = head s label { the notion of a functional stream and the concept xs = tail s of lazy evaluation. We will start by discussing functional streams, and will visit the role of laziness in Section 3.4. Functional programmers will note that our applyToStream Stream functions can be viewed as higher-order versions of function is equivalent to the familiar map function over lists. regular functions. Instead of operating on a single value, We create our own version here because, as we show in Sec- they operate on a sequence of values. For example, consider tion 3.1, we want to operate on more complex stream data this simple function to compute the square of an integer: structures. Our applyToStream function is a useful workflow abstraction square :: Int -> Int because a workflow program is a the composition of compu- square x = x*x tations that work with single elements.