DOCSLIB.ORG

Peach Documentation Release

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Peach Documentation Release peach Documentation Release Alyssa Kwan Nov 11, 2017 Contents 1 Table of Contents 3 1.1 Solutions to Common Problems.....................................3 2 Indices and tables 5 i ii peach Documentation, Release Welcome to peach. peach is a functional ETL framework. peach empowers you to easily perform ETL in a way inherent to the functional programming paradigm. Why peach? Please see the Solutions to Common Problems section of the documentation. peach is the culmination of learnings from over a decade of mistakes made by the principal author. As such, it represents best practices to deal with a wide array of ETL and data lake / data warehouse problems. It also represents a sound theoretical framework for approaching this family of problems, namely “how to deal with side effects amidst concurrency and failure in a tractable way”. Contents 1 peach Documentation, Release 2 Contents CHAPTER 1 Table of Contents 1.1 Solutions to Common Problems 1.1.1 Clean Retry on ETL Job Failure Problem Jobs are side-effecting; that is their point: input data is ingested, and changes are made to the data lake or warehouse state in response. If a job fails partway through execution, it leaves all sort of garbage. This garbage gets in the way of retries, or worse yet, any other tries at all. If all of the job artifacts - both intermediate and final - are hosted upon a transactional data store, and jobs are only using features that are included in transactions (for instance, some RDBMS’s don’t include schema changes in transaction scope, so you can’t create tables and have them automatically cleaned up), then congratulations! There is no problem. The data warehouse is at a scale where this is possible. For the rest of us, the key principle is idempotency. Write jobs so that they clean up the artifacts created by previous runs first. The big problem with this is naming. Intermediate artifacts must use immutable, well-known names, which limits the degree of parallelism to 1 (one), i.e. no parallelism. Or you generate new, non-conflicting names for your shards each job execution, and you are OK with leaving uncollected garbage in the case of process crash. Collecting the garbage later requires having kept careful and correct records about which shards are actually orphans. In either case, idempotency requires careful thought and discipline, which heightens barriers to entry for engineers to join efforts. Solution peach is functional. This means it’s founded on the principles of functional programming, which means that one of the principles it focuses on is referential transparency. This principle simply means that calling the same function with the same inputs more than once always results in the same output. Obviously, the only way that can work is if the function is free of side effects. 3 peach Documentation, Release But ETL is fundamentally side effecting - we’re changing the data lake / data warehouse in response to input. How can we have referential transparency? The key is making time itself an explicit input. Let us take the following: function inputs_between_times returns the set of inputs received from time 1 (one) exclusive to time 2 (two) inclusive function data_lake returns the state of the data lake based on some past state of the data lake with some new set of inputs integrated into it Based on this, the state of the data lake currently (now) is expressed as: data_lake_at_time_current= data_lake( data_lake_at_time_previous, inputs_between_times( time_previous, time_current, ), ) What does this approach imply? 1. ETL outputs are just cache. The fundamental data is the time-ordered set of inputs into the system. peach is really good at maintaining cache. 2. Any prior state of the lake / warehouse can be regenerated at any time. Use it for rollback. Use it for debugging. Load prior states into other warehouses to give some department an unchanging snapshot, perhaps to do end-of- fiscal-period work. More on this <TBD>. (a) Write functions that compare the state of the lake / warehouse at two different points in time. Automate your change analysis and debugging. More on this <TBD>. (b) Write functions that compute the set of change operations given the state of the lake / warehouse and the set of new inputs. Know exactly what is being inserted, updated, and deleted, in what order and why. Deepen your change analysis and debugging. More on this <TBD>. 4 Chapter 1. Table of Contents CHAPTER 2 Indices and tables • genindex • modindex • search 5.
Load more

Recommended publications
  • Exploring Languages with Interpreters and Functional Programming Chapter 8
    Exploring Languages with Interpreters and Functional Programming Chapter 8 H. Conrad Cunningham 24 September 2018 Contents 8 Evaluation Model 2 8.1 Chapter Introduction . .2 8.2 Referential Transparency Revisited . .2 8.3 Substitution Model . .3 8.4 Time and Space Complexity . .6 8.5 Termination . .7 8.6 What Next? . .7 8.7 Exercises . .8 8.8 Acknowledgements . .8 8.9 References . .9 8.10 Terms and Concepts . .9 Copyright (C) 2016, 2017, 2018, H. Conrad Cunningham Professor of Computer and Information Science University of Mississippi 211 Weir Hall P.O. Box 1848 University, MS 38677 (662) 915-5358 Browser Advisory: The HTML version of this textbook requires a browser that supports the display of MathML. A good choice as of September 2018 is a recent version of Firefox from Mozilla. 1 8 Evaluation Model 8.1 Chapter Introduction This chapter introduces an evaluation model applicable to Haskell programs. As in the previous chapters, this chapter focuses on use of first-order functions and primitive data types. A goal of this chapter and the next one is enable students to analyze Haskell functions to determine under what conditions they terminate normally and how efficient they are. This chapter presents the evaluation model and the next chapter informally analyzes simple functions in terms of time and space efficiency and termination. How can we evaluate (i.e. execute) an expression that “calls” a function like the fact1 function from Chapter 4? We do this by rewriting expressions using a substitution model, as we see in this chapter. This process depends upon a property of functional languages called referential transparency.
    [Show full text]
  • Week 3: Scope and Evaluation Order Assignment 1 Has Been Posted!
    Week 3: Scope and Evaluation Order CSC324 Principles of Programming Languages David Liu, Department of Computer Science Assignment 1 has been posted! Closures and static scope static (adjective) determined only by the program source code dynamic (adjective) determined only when the program is run E.g., referential transparency is a static property How are closures implemented? Are they static or dynamic? Or both? Function bodies can be processed statically (e.g., Haskell compiler generates code once per lambda). The closure environment (and therefore the closure itself) can only be generated dynamically. The closure depends only on where the function is evaluated, not where that function is called. (define (make-adder n) (lambda (x) (+ x n))) (define adder (make-adder 1)) (adder 100) So we can determine where each free identifier obtains its values statically, based on where its enclosing function is defined. scope (of an identifier) The parts of program code that may refer to that identifier. static (aka lexical) scope The scope of every identifier is determined by the structure of the source code (e.g., by nesting of lambdas and lets). Every identifier obtains its value from the closest enclosing expression that binds it. (define (make-adder n) (lambda (x) (+ x n))) (define adder (make-adder 1)) ; (0x..., {n: 1}) (let* ([n 100]) (adder 2)) Implementing static scope in an interpreter A simple interpreter (define/match (interpret env expr) [(_ (? number?)) expr] [(_ (? symbol?)) (hash-ref env expr)] [(_ (list '+ l r)) (+ (interpret env l) (interpret env r))]) A simple interpreter (define/match (interpret env expr) [(_ (? number?)) expr] [(_ (? symbol?)) (hash-ref env expr)] [(_ (list '+ l r)) (+ (interpret env l) (interpret env r))]) The environment is passed recursively when interpreting each subexpression.
    [Show full text]
  • SI 413, Unit 3: Advanced Scheme
    SI 413, Unit 3: Advanced Scheme Daniel S. Roche ([email protected]) Fall 2018 1 Pure Functional Programming Readings for this section: PLP, Sections 10.7 and 10.8 Remember there are two important characteristics of a “pure” functional programming language: • Referential Transparency. This fancy term just means that, for any expression in our program, the result of evaluating it will always be the same. In fact, any referentially transparent expression could be replaced with its value (that is, the result of evaluating it) without changing the program whatsoever. Notice that imperative programming is about as far away from this as possible. For example, consider the C++ for loop: for ( int i = 0; i < 10;++i) { /∗ some s t u f f ∗/ } What can we say about the “stuff” in the body of the loop? Well, it had better not be referentially transparent. If it is, then there’s no point in running over it 10 times! • Functions are First Class. Another way of saying this is that functions are data, just like any number or list. Functions are values, in fact! The specific privileges that a function earns by virtue of being first class include: 1) Functions can be given names. This is not a big deal; we can name functions in pretty much every programming language. In Scheme this just means we can do (define (f x) (∗ x 3 ) ) 2) Functions can be arguments to other functions. This is what you started to get into at the end of Lab 2. For starters, there’s the basic predicate procedure?: (procedure? +) ; #t (procedure? 10) ; #f (procedure? procedure?) ; #t 1 And then there are “higher-order functions” like map and apply: (apply max (list 5 3 10 4)) ; 10 (map sqrt (list 16 9 64)) ; '(4 3 8) What makes the functions “higher-order” is that one of their arguments is itself another function.
    [Show full text]
  • Functional Programming Functional Vs. Imperative Referential
    Functional vs. Imperative Referential transparency Imperative programming concerned with “how.” The main (good) property of functional programming is Functional Programming referential transparency. Functional programming concerned with “what.” COMS W4115 Every expression denotes a single value. Based on the mathematics of the lambda calculus Prof. Stephen A. Edwards (Church as opposed to Turing). The value cannot be changed by evaluating an expression Spring 2003 or by sharing it between different parts of the program. “Programming without variables” Columbia University No references to global data; there is no global data. Department of Computer Science It is inherently concise, elegant, and difficult to create subtle bugs in. There are no side-effects, unlike in referentially opaque Original version by Prof. Simon Parsons languages. It’s a cult: once you catch the functional bug, you never escape. The Joy of Pascal Strange behavior Variables program example(output) This prints 5 then 4. At the heart of the “problem” is fact that the global data var flag: boolean; flag affects the value of f. Odd since you expect function f(n:int): int In particular begin f (1) + f (2) = f (2) + f (1) if flag then f := n flag := not flag else f := 2*n; gives the offending behavior flag := not flag Mathematical functions only depend on their inputs end They have no memory Eliminating assignments eliminates such problems. begin In functional languages, variables not names for storage. flag := true; What does this print? Instead, they’re names that refer to particular values. writeln(f(1) + f(2)); writeln(f(2) + f(1)); Think of them as not very variables.
    [Show full text]
  • Perhaps There Is a Much Better Way Goals
    DjangoCon 2014 Tony Morris Perhaps There Is A much Better Way Goals Our existing common goals The goals for today Goals Our common goals to implement software efficiently to arrive at an initial result as quickly as possible over time, to reliably arrive at results as quickly as possible (aka maintenance) a result is valid if the program accurately achieves our objective Goals Our common goals impertinent goals financial gain "but django makes me $$$!" motivate personal avidities Goals Our goals for today "Why Django Sucks" Goals Our goals for today I could talk all day about why django sucks and I’d be saying lots and lots of true things but not necessarily helpful things Goals Our goals for today I could talk all day about why django sucks and I’d be saying lots and lots of true things but not necessarily helpful things Goals Our goals for today The goal today To equip you with new tools and perspective, with which to explore the question for yourself. Goals Lies Along the way We will visit some of the lies you have been told Goals Lies Tacitly insidious ones "Having to come to grips with Monads just isn’t worth it for most people" Goals Lies Confusingly insidious ones "Imperative vs functional programming vs object-oriented programming" Goals Lies Awkward ones "The real world is mutable" Goals Lies Funny ones "Django: The Web framework for perfectionists with deadlines" Goals Summary What is functional programming? What does monad mean? Functional imperative programming Parametricity —types are documentation What is Functional Programming?
    [Show full text]
  • Usuba, Optimizing & Trustworthy Bitslicing Compiler
    Usuba, Optimizing & Trustworthy Bitslicing Compiler Darius Mercadier, Pierre-Évariste Dagand, Lionel Lacassagne, Gilles Muller To cite this version: Darius Mercadier, Pierre-Évariste Dagand, Lionel Lacassagne, Gilles Muller. Usuba, Optimizing & Trustworthy Bitslicing Compiler. WPMVP’18 - Workshop on Programming Models for SIMD/Vector Processing, Feb 2018, Vienna, Austria. 10.1145/3178433.3178437. hal-01657259v2 HAL Id: hal-01657259 https://hal.archives-ouvertes.fr/hal-01657259v2 Submitted on 16 Oct 2018 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Usuba Optimizing & Trustworthy Bitslicing Compiler Darius Mercadier Pierre-Évariste Dagand Lionel Lacassagne Gilles Muller [email protected] Sorbonne Université, CNRS, Inria, LIP6, F-75005 Paris, France Abstract For example, on input 0110b = 6d, this function returns the Bitslicing is a programming technique commonly used in seventh (note the 0-indexing) element of the table, which is cryptography that consists in implementing a combinational 10d = 1010b. circuit in software. It results in a massively parallel program This implementation suffers from two problems. First, it immune to cache-timing attacks by design. wastes register space: a priori, the input will consume a However, writing a program in bitsliced form requires ex- register of at least 32 bits, leaving 28 bits unused, which treme minutia.
    [Show full text]
  • Functional Programming with Ocaml
    CMSC 330: Organization of Programming Languages Functional Programming with OCaml CMSC 330 - Spring 2019 1 What is a functional language? A functional language: • defines computations as mathematical functions • discourages use of mutable state State: the information maintained by a computation Mutable: can be changed CMSC 330 - Spring 2019 2 Functional vs. Imperative Functional languages: • Higher level of abstraction • Easier to develop robust software • Immutable state: easier to reason about software Imperative languages: • Lower level of abstraction • Harder to develop robust software • Mutable state: harder to reason about software CMSC 330 - Spring 2019 3 Imperative Programming Commands specify how to compute, by destructively changing state: x = x+1; a[i] = 42; p.next = p.next.next; Functions/methods have side effects: int wheels(Vehicle v) { v.size++; return v.numWheels; } CMSC 330 - Spring 2019 4 Mutability The fantasy of mutability: • It's easy to reason about: the machine does this, then this... The reality of mutability: • Machines are good at complicated manipulation of state • Humans are not good at understanding it! • mutability breaks referential transparency: ability to replace an expression with its value without affecting the result • In math, if f(x)=y, then you can substitute y anywhere you see f(x) • In imperative languages, you cannot: f might have side effects, so computing f(x) at one time might result in different value at another CMSC 330 - Spring 2019 5 Mutability The fantasy of mutability: • There is a single state • The computer does one thing at a time The reality of mutability: • There is no single state • Programs have many threads, spread across many cores, spread across many processors, spread across many computers..
    [Show full text]
  • Functional Programming with Scheme
    Functional Programming with Scheme Read: Scott, Chapter 11.1-11.3 1 Lecture Outline n Functional programming languages n Scheme n S-expressions and lists n cons, car, cdr n Defining functions n Examples of recursive functions n Shallow vs. deep recursion n Equality testing Programming Languages CSCI 4430, A. Milanova/B. G. Ryder 2 Racket/PLT Scheme/DrScheme n Download Racket (was PLT Scheme (was DrScheme)) n http://racket-lang.org/ n Run DrRacket n Languages => Choose Language => Other Languages => Legacy Languages: R5RS n One additional textbook/tutorial: n Teach Yourself Scheme in Fixnum Days by Dorai Sitaram: https://ds26gte.github.io/tyscheme/index.html Programming Languages CSCI 4430, A. Milanova 3 First, Imperative Languages n The concept of assignment is central n X:=5; Y:=10; Z:=X+Y; W:=f(Z); n Side effects on memory n Program semantics (i.e., how the program works): state-transition semantics n A program is a sequence of assignment statements with effect on memory (i.e., state) C := 0; for I := 1 step 1 until N do t := a[I]*b[I]; C := C + t; Programming Languages CSCI 4430, A. Milanova 4 Imperative Languages n Functions (also called “procedures”, “subroutines”, or routines) have side effects: Roughly: n A function call affects visible state; i.e., a function call may change state in a way that affects execution of other functions; in general, function call cannot be replaced by result n Also, result of a function call depends on visible state; i.e., function call is not independent of the context of the call Programming Languages CSCI 4430, A.
    [Show full text]
  • Marking Piecewise Observable Purity
    Marking Piecewise Observable Purity Seyed Hossein Haeri∗ and Peter Van Roy INGI, Universit´ecatholique de Louvain, Belgium; fhossein.haeri,[email protected] Abstract We present a calculus for distributed programming with explicit node annotation, built- in ports, and pure blocks. To the best of our knowledge, this is the first calculus for distributed programming, in which the effects of certain code blocks can go unobserved, making those blocks pure. This is to promote a new paradigm for distributed programming, in which the programmer strives to move as much code as possible into pure blocks. The purity of such blocks wins familiar benefits of pure functional programming. 1 Introduction Reasoning about distributed programs is difficult. That is a well-known fact. That difficulty is partly because of the relatively little programming languages (PL) research on the topic. But, it also is partly due to the unnecessary complexity conveyed by the traditional conception of distributed systems. Reasoning about sequential programs written for a single machine, in contrast, is the bread and butter of the PL research. Many models of functional programming, for example, have received considerable and successful attention from the PL community. The mathematically rich nature of those models is the main motivation. In particular, pure functional programming, i.e., programming with no side-effects, has been a major source of attraction to the PL community. In that setting, many interesting properties are exhibited elegantly, e.g., equational reasoning, referential transparency, and idempotence. The traditional conception about distributed programming, however, lacks those nice prop- erties. In that conception, side-effects are inherent in distributed programming.
    [Show full text]
  • Lecture01 -- Introduction
    CS 320: Concepts of Programming Languages Wayne Snyder Computer Science Department Boston University Schedule of Topics 26 lectures: 2 midterms, 1 intro lecture, so 23 lectures Approximately 8 lectures = 1 month for each major area: o 8 lectures (2 – 9) for Introduction to Functional Programming in Haskell, followed by midterm 1 (lecture slot 10) o 8 lectures (11 – 18) for Interpreters: Implementation of Functional Programming Languages, plus further development of Haskell, particularly monads and parsing, followed by midterm 2 (lecture slot 19) o 7 lectures (20 – 26) for Compilers: Implementation of Imperative Programming Languages. Introduction to Functional Programming in Implementation of Functional Implementation of Imperative Haskell Programming Languages -- Interpretation Programming Languages -- Compilation Algebraic (1st order) vs Functional Term Rewriting, Matching, Abstract Attribute Grammars (higher-order) thinking. Unification, Lambda Calculus Evaluation Order, Management of Translation templates Environments for various language Context-Free features. Grammars, Parse Trees, Derivations. Recursive, Higher- Implementation of (Regular Languages Order, List-based various language and DFAs??) Programming features. Intermediate code, machine organization, & assembly language. Parsing in Haskell Bare-bones Basic Haskell: Types, Advanced Haskell: Building an interpreter Building a compiler for Practical Haskell Classes, features for Monads for Mini-Haskell Mini-C HO/ List Programming Lectures: 1 10 19 26 What is a Programming Language? A Programming Language is defined by ² Syntax – The rules for putting together symbols (e.g., Unicode) to form expressions, statements, procedures, functions, programs, etc. It describes legal ways of specifying computations. ² Semantics – What the syntax means in a particular model of computation. This can be defined abstractly, but is implemented by an algorithm + data structures (examples: Python interpreter or C compiler).
    [Show full text]
  • What Is Functional Programming? by by Luis Atencio
    What is Functional Programming? By By Luis Atencio Functional programming is a software development style with emphasis on the use functions. It requires you to think a bit differently about how to approach tasks you are facing. In this article, based on the book Functional Programming in JavaScript, I will introduce you to the topic of functional programming. The rapid pace of web platforms, the evolution of browsers, and, most importantly, the demand of your end users has had a profound effect in the way we design web applications today. Users demand web applications feel more like a native desktop or mobile app so that they can interact with rich and responsive widgets, which forces JavaScript developers to think more broadly about the solution domain push for adequate programming paradigms. With the advent of Single Page Architecture (SPA) applications, JavaScript developers began jumping into object-oriented design. Then, we realized embedding business logic into the client side is synonymous with very complex JavaScript applications, which can get messy and unwieldy very quickly if not done with proper design and techniques. As a result, you easily accumulate lots of entropy in your code, which in turn makes your application harder write, test, and maintain. Enter Functional Programming. Functional programming is a software methodology that emphasizes the evaluation of functions to create code that is clean, modular, testable, and succinct. The goal is to abstract operations on the data with functions in order to avoid side effects and reduce mutation of state in your application. However, thinking functionally is radically different from thinking in object-oriented terms.
    [Show full text]
  • Equality Saturation: a New Approach to Optimization ∗
    Equality Saturation: a New Approach to Optimization ∗ RossTate MichaelStepp ZacharyTatlock SorinLerner Department of Computer Science and Engineering University of California, San Diego {rtate, mstepp, ztatlock, lerner} @cs.ucsd.edu Abstract generated code, a problem commonly known as the phase ordering Optimizations in a traditional compiler are applied sequentially, problem. Another drawback is that profitability heuristics, which with each optimization destructively modifying the program to pro- decide whether or not to apply a given optimization, tend to make duce a transformed program that is then passed to the next op- their decisions one optimization at a time, and so it is difficult for timization. We present a new approach for structuring the opti- these heuristics to account for the effect of future transformations. mization phase of a compiler. In our approach, optimizations take In this paper, we present a new approach for structuring optimiz- the form of equality analyses that add equality information to a ers that addresses the above limitations of the traditional approach, common intermediate representation. The optimizer works by re- and also has a variety of other benefits. Our approach consists of peatedly applying these analyses to infer equivalences between computing a set of optimized versions of the input program and program fragments, thus saturating the intermediate representation then selecting the best candidate from this set. The set of candidate with equalities. Once saturated, the intermediate representation en- optimized programs is computed by repeatedly inferring equiva- codes multiple optimized versions of the input program. At this lences between program fragments, thus allowing us to represent point, a profitability heuristic picks the final optimized program the effect of many possible optimizations at once.
    [Show full text]