1 On the Duality of Streams 2 3 How Can Linear Types Help to Solve the Lazy IO Problem? 4 5 Jean-Philippe Bernardy Josef Svenningsson 6 University of Gothenburg Chalmers University of Technology 7 Gothenburg 41255, Sweden Gothenburg ???, Sweden 8 9 ACM Reference format: transparency (as Kiselyov [17] has shown). For example, one 10 Jean-Philippe Bernardy and Josef Svenningsson. 2016. On the Du- may rightfully expect1 that both following programs have 11 ality of Streams. In Proceedings of ACM Conference, Washington, the same behavior: 12 DC, USA, July 2017 (Conference'17), 12 pages. main = do inFile openFile "foo" ReadMode 13 DOI: 10.1145/nnnnnnn.nnnnnnn contents hGetContents inFile 14 putStr contents 15 treams, Continuations, Linear Types 16 hClose inFile 17 1 Introduction main = do inFile openFile "foo" ReadMode 18 As Hughes [12] famously noted, the strength of functional contents hGetContents inFile 19 programming languages resides in the composition mecha- hClose inFile 20 nisms that they provide. That is, simple components can be putStr contents 21 built and understood in isolation; one does not need to worry Indeed, the putStr and hClose commands act on unrelated 22 about interference effects when composing them. In particu- resources, and thus swapping them should have no observable 23 lar, lazy evaluation affords to construct complex programs by effect. However, while the first program prints the foo file, 24 pipelining simple transformation functions. Indeed, whereas the second one prints nothing. Indeed, because hGetContents 25 strict evaluation forces to fully reify each intermediate result reads the file lazily, the hClose operation has the effect to 26 between each computational step, lazy evaluation allows to truncate the list. In the first program, printing the contents 27 run all the computations concurrently, often without ever forces reading the file. One may argue that hClose should 28 allocating more than a single intermediate element at a time. not be called in the first place | but then, closing the handle 29 Unfortunately, lazy evaluation suffers from two drawbacks. happens only when the contents list can be garbage collected 30 First, it has unpredictable memory behavior. Consider the (in full), and relying on garbage collection for cleaning re- 31 following function composition: 32 sources is brittle; furthermore this effect compounds badly 33 f :: [a ] ! [b ] with the first issue discussed above (unpredictability of buffer- 34 g :: [b ] ! [c ] ing). If one wants to use lazy effectful computations, again, 35 h = g ◦ f the compositionality principle is lost. 36 In this paper, we propose to tackle both of these issues One hopes that, at run-time, the intermediate list ([b]) will 37 by taking advantage of linear types. One way to read this only be allocated element-wise, as outlined above. Unfortu- 38 paper is as an advocacy for linear types support in Haskell. nately, this desired behavior does not always happen. Indeed, 39 Indeed, even though one can tackle the above issues by using a necessary condition is that the production pattern of f 40 Monad-based libraries [16] our linear-logic-based types natu- matches the consumption pattern of g; otherwise buffering 41 rally capture a wealth of useful production and consumption occurs. In practice, this means that a seemingly innocuous 42 patterns. change in either of the function definitions may drastically 43 The first noteworthy type, corresponding to on-demand change the memory behavior of the composition, without 44 production of elements, is called a source (Src). An adap- warning. If one cares about memory behavior, this means 45 tation of the first code example above to use sources would that the compositionality principle touted by Hughes breaks 46 look as follows. down. 47 Second, lazy evaluation does not extend nicely to effect- f :: Src a ( Src b 48 ful processing. That is, if (say) an input list is produced g :: Src b ( Src c 49 by reading a file lazily, one is exposed to losing referential h = g ◦ f 50 Thanks to type-checking, we get the guarantee that the 51 Permission to make digital or hard copies of all or part of this work 52 for personal or classroom use is granted without fee provided that composition does not allocate more memory than the sum copies are not made or distributed for profit or commercial advantage 53 of its components. The second useful type, driving the and that copies bear this notice and the full citation on the first page. 54 Copyrights for components of this work owned by others than ACM consumption of elements, is called a sink (Snk). For example, 55 must be honored. Abstracting with credit is permitted. To copy the standard output is naturally given a sink type: otherwise, or republish, to post on servers or to redistribute to lists, 56 requires prior specific permission and/or a fee. Request permissions stdoutSnk :: Snk String 57 from [email protected]. 1 58 Conference'17, Washington, DC, USA This expectation is expressed in a Stack Overflow question, accessible © 2016 ACM. 978-x-xxxx-xxxx-x/YY/MM. $15.00 at this URL: http://stackoverflow.com/questions/296792/haskell-io- 59 DOI: 10.1145/nnnnnnn.nnnnnnn and-closing-files 60 61 Conference'17, July 2017, Washington, DC, USA Jean-Philippe Bernardy and Josef Svenningsson 1 Using it, we can implement the printing of a file as follows. work and future work are discussed respectively in sections 7 2 and 8. We conclude in Sec. 9. main = fileSrc "foo" `fwd` stdoutSnk 3 4 Beyond frugal memory usage, we have the guarantee of a 2 Preliminary: negation and 5 timely release of resources, even in the presence of exceptions. continuations 6 In the above, fileSrc provides the contents of a file, and fwd In this section we recall the basics of continuation-based pro- 7 forwards data from a source to a sink. The types are as gramming. We introduce our notation, and justify effectful 8 follows: continuations. 9 fileSrc :: FilePath ! Src String We begin by assuming a type of effects Eff , which we keep 10 fwd :: Src a ( Snk a ( IO () abstract for now. We can then define negation as follows: 11 12 Sources provide data on-demand, while sinks decide when type N a = a ( Eff 13 they are ready to consume data. This is an instance of the A shortcut for double negations is also convenient. 14 push/pull duality. In general, push-streams control the flow type NN a = N (N a) 15 of computation, while pull-streams respond to it. We will 16 see that this polarization does not need to match the flow of The basic idea (imported from classical logic) pervading this 17 data: for example we support data sources with push-flavor, paper is that in the presence of effects, producing a result of 18 called co-sources (CoSrc). Co-sources are useful for example type α is equivalent to consuming an argument of type Nα. 19 when a source data stream needs precise control over the Dually, consuming an argument of type α is equivalent to 20 execution of effects it embeds (see Sec. 6). For example, producing a result of type Nα. We call these equivalences 21 sources cannot be demultiplexed, but co-sources can. the duality principle. 22 In a program which uses both sources and co-sources, the In classical logic, negation is involutive; that is: NN α = α. 23 need might arise to compose a function which returns a co- However, because we work within Haskell, we do not have 3 24 source with a function which takes a source as input: this is this equality . We can come close enough though. First, 25 the situation where list-based programs would silently cause double negations can always be introduced, using the shift 26 memory allocation. In our approach, this mismatch is caught operator: by the type system, and the user must explicitly conjure a 27 shift :: a NN a buffer to be able to write the composition: ( 28 shift x k = k x 29 f :: Src a CoSrc b ( Second, it is possible to remove double negations, but only 30 g :: Src b Src c ( if an effect can be outputted. Equivalently, triple negations 31 h = g ◦ buffer ◦ f 32 can be collapsed to a single one: The contributions of this paper are 33 unshift :: N (NN a) ( N a 34 • The formulation of principles for compositional resource- unshift k x = k (shift x) 35 aware programming in Haskell (resources include mem- Aside. The above two functions are the return and join 36 ory and files), based on linearity and polarization. of the double negation monad4; indeed adding a double 37 While these principles are borrowed from linear logic, negation in the type corresponds to sending the return value 38 as far as we know they have not been applied to to its consumer. However, we will not be using this monadic 39 Haskell programming before. structure anywhere in the following, because single negations 40 • An embodiment of the above principles, in the form of 2 play a central role in our approach. The monadic structure 41 a Hask-LL library for streaming IO. Besides support- is a mere diversion. 42 ing compositionality as outlined above, our library 43 features two concrete novel aspects: 2.1 Structure of Effects 44 1. A more lightweight design than state-of-the-art When dealing with purely functional programs, continuations 45 co-routine based libraries. have no effects. In this case, one can let Eff remain abstract, 46 2. Support for explicit buffering and control struc- or define it to be the empty type: Eff = ?. This choice 47 tures, while still respecting compositionality (Sec.