DOCSLIB.ORG

Python Functional Programming Python Functional Programming

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Python Functional Programming Python Functional Programming +f(x) = Python Functional Programming Python Functional Programming Functional Programming by Wikipidia: “Functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data". In other words, functional programming promotes code with no side effects, no change of value in variables. It oposes to imperative programming, which enfatizes change of state”. Python Functional Programming What this means? ● No mutable data (no side effect). ● No state (no implicit, hidden state). Once assigned (value binding), a variable (a symbol) does not change its value. All state is bad? No, hidden, implicit state is bad. Functional programming do not eliminate state, it just make it visible and explicit (at least when programmers want it to be). ● Functions are pure functions in the mathematical sense: their output depend only in their inputs, there is not “environment”. ● Same result returned by functions called with the same inputs. Python Functional Programming What are the advantages? ● Cleaner code: "variables" are not modified once defined, so we don't have to follow the change of state to comprehend what a function, a, method, a class, a whole project works. ● Referential transparency: Expressions can be replaced by its values. If we call a function with the same parameters, we know for sure the output will be the same (there is no state anywhere that would change it). There is a reason for which Einstein defined insanity as "doing the same thing over and over again and expecting different results". Python Functional Programming Advantages enabled by referential transparence ● Memoization ○ Cache results for previous function calls. ● Idempotence ○ Same results regardless how many times you call a function. ● Modularization ○ We have no state that pervades the whole code, so we build our project with small, black boxes that we tie together, so it promotes bottom-up programming. ● Ease of debugging ○ Functions are isolated, they only depend on their input and their output, so they are very easy to debug. Python Functional Programming Advantages enabled by referential transparence ● Parallelization ○ Functions calls are independent. ○ We can parallelize in different process/CPUs/computers/… result = func1(a, b) + func2(a, c) We can execute func1 and func2 in paralell because a won’t be modified. Python Functional Programming Advantages enabled by referential transparence ● Concurrence a. With no shared data, concurrence gets a lot simpler: i. No semaphores. ii. No monitors. iii. No locks. iv. No race-conditions. v. No dead-locks. Python Functional Programming Python is a multi paradigm programming language. As a Python programmer why uses functional programming in Python? Python is not a functional language but have a lot of features that enables us to applies functional principles in the development, turning our code more elegant, concise, maintanable, easier to understand and test. Python Functional Programming Don’t Update, Create - String name = 'Geison' name = '{0} Flores'.format(name) FIRSTNAME = 'Geison' LASTNAME = '{0} Flores'.format(FIRSTNAME) NAME = '{0} {1}'.format(FIRSTNAME, LASTNAME) Python Functional Programming Don’t Update, Create - Lists years = [2001, 2002] years.append(2003) years += [2004, 2005] years # [2001, 2002, 2003, 2004, 2005] YEARS = [2001, 2001] ALL_YEARS = YEARS + [2003] + [2004, 2005] Python Functional Programming Don’t Update, Create - Dict ages = {'John': 30} ages['Mary'] = 28 ages # {'John': 30, 'Mary': 28} AGES = {'John': 30} ALL_AGES = dict(AGES.itens() + {'Mary': 28}.itens()) Python Functional Programming Higher Order Functions Functions and methods are first-class objects in Python, so if you want to pass a function to another function, you can just treat it as any other object. def caller(f): f() def say_hello(name): return 'Hello {0}'.format(name) caller(say_hello) Python Functional Programming Higher Order Functions - Map map(lambda word: word.upper(), ["milu", "rantanplan"]) # result ["MILU", "RANTANPLAN"] def add_2(n): n + 2 map(add_2, [1, 2, 3]) # result [3, 4, 5] Python Functional Programming Higher Order Functions - Filter filter(lambda word: len(word) == 4, ["milu", "rantanplan"]) # result ["MILU"] def greater_than_10(num): return num > 10 filter(greater_than_10, range(15)) # result [11, 12, 13, 14, 15] Python Functional Programming Higher Order Functions - Reduce import operator reduce(operator.add, [1, 2, 3, 4, 5]) # result 15 def acumullator(a, b): return len(a) + len(b) reduce(acumullator, ["milu", "rantanplan"]) # result 14 reduce(lambda a,b: len(a) + len(b), ["milu", "rantanplan"]) # result 14 Python Functional Programming Higher Order Functions - Reduce from itertools import izip list1 = [1, 2, 3] list2 = ["a", "b", "c"] [list(x) for x in izip(list1, list2)] # result [[1, "a"], [2, "b"], [3, "c"]] Python Functional Programming Higher Order Functions - Closure def add_x(x): def adder(y): return x + y return adder add_5 = add_x(5) add_7 = add_x(7) add_5(10) # result 15 add_7(10) # result 17 Python Functional Programming Currying and Partial Functions Higher-order functions enable Currying, which the ability to take a function that accepts n parameters and turns it into a composition of n functions each of them take 1 parameter. A direct use of currying is the Partial Functions where if you have a function that accepts n parameters then you can generate from it one of more functions with some parameter values already filled in. from functools import partial plus = lambda a, b: a + b # defining a function that sums 2 numbers plus(3, 5) # result 8 # curring calling partial function by supplying the first parameters with value 1 plus_one = partial(plus, 1) # I can use the new function as normal plus_one(5) # result 6 Python Functional Programming Eager vs Lazy Evaluation ● Eager evaluation: expressions are calculated at the moment that variables is assined, function called... ● Lazy evaluation: delays the evaluation of the expression until it is needed. ○ Memory efficient: no memory used to store complete structures. ○ CPU efficient: no need to calculate the complete result before returning. ○ Laziness is not a requisite for FP, but it is a strategy that fits nicely on the paradigm(Haskell). Python uses eager evaluation (but short-circuits && or ||). Python generators are a mechanism for lazy evaluation. Python arrays are not lazy, use enumarators when necessary. Python Functional Programming Recursion Looping by calling a function from within itself. When you don’t have access to mutable data, recursion is used to build up and chain data construction. This is because looping is not a functional concept, as it requires variables to be passed around to store the state of the loop at a given time. ● Purely functional languages have no imperative for-loops, so they use recursion a lot. ● If every recursion created an stack, it would blow up very soon. ● Tail-call optimization (TCO) avoids creating a new stack when the last call in a recursion is the function itself. ● TCO is not implemented in Python. ● Unfortunarely following recursion style in Python has it’s own tax: Performance. Python Functional Programming Solving Python Lack of TCO(Tail Call Optimization) # The functional solution have problens with big values fib = lambda n: n if n < 2 else fib(n-1) + fib(n-2) # The iterative solution using generators works perfectly with large values def fibs(): a = 0 b = 1 while True: yield a a, b = b, a + b Python Functional Programming FP in OOP? It is possible do FP in OOP? Yes it is! ● OOP is orthogonal to FP. ● Well, at least in theory, because: ○ Typical OOP tends to emphasize change of state in objects. ○ Typical OOP mixes the concepts of identity and state. ○ Mixture of data and code raises both conceptual and practical problems. ● OOP functional languages: Scala, F#, ... Python Functional Programming A Pratical Example Exercise: "What's the sum of the first 10 natural number whose square value is divisible by 5?" Imperative: n, num_elements, s = 1, 0, 0 while num_elements < 10: if n**2 % 5 == 0: s += n num_elements += 1 n += 1 n #275 Functional: sum(filter(lambda x: x**2 % 5 == 0, range(1, 100))[:10]) Python Functional Programming The last advice Learn at least one functional language, it will open your mind to a new paradigm becoming you a better programmer. Some Functional Languages: ● Haskell ● ML (Standard ML, Objective Caml, ...) ● Scheme ● Erlang ● Scala ● Closure ● F# Python Functional Programming Conclusion ● As you can tell, Python helps you write in functional style but it doesn’t force you to it. ● Writing in functional style enhances your code and makes it more self documented. Actually it will make it more thread-safe also. ● The main support for FP in Python comes from the use of list conprehension, lambdas, closures, iterators and generators, also from the modules functools and itertools. ● Python still lack an important aspect of FP: Pattern Matching and Tails Recursion. ● There should be more work on tail recursion optimization, to encourage developers to use recursion. ● Any other thoughts? Python Functional Programming References ● http://en.wikipedia.org/wiki/Functional_programming ● http://www.cse.chalmers.se/~rjmh/Papers/whyfp.pdf ● http://docs.python.org/2/howto/functional.html ● http://www.youtube.com/watch?v=Ta1bAMOMFOI ● http://clojure.org/ ● http://www.defmacro.org/ramblings/fp.html Python Functional Programming Contact me ● Email: ○ [email protected] ● Skype ○ geisonfgf ● Facebook ○ http://www.facebook.com/geisonfgf ● Twitter ○ http://www.twitter.com/geisonfgf.
Load more

Recommended publications
  • Exploring Languages with Interpreters and Functional Programming Chapter 8
    Exploring Languages with Interpreters and Functional Programming Chapter 8 H. Conrad Cunningham 24 September 2018 Contents 8 Evaluation Model 2 8.1 Chapter Introduction . .2 8.2 Referential Transparency Revisited . .2 8.3 Substitution Model . .3 8.4 Time and Space Complexity . .6 8.5 Termination . .7 8.6 What Next? . .7 8.7 Exercises . .8 8.8 Acknowledgements . .8 8.9 References . .9 8.10 Terms and Concepts . .9 Copyright (C) 2016, 2017, 2018, H. Conrad Cunningham Professor of Computer and Information Science University of Mississippi 211 Weir Hall P.O. Box 1848 University, MS 38677 (662) 915-5358 Browser Advisory: The HTML version of this textbook requires a browser that supports the display of MathML. A good choice as of September 2018 is a recent version of Firefox from Mozilla. 1 8 Evaluation Model 8.1 Chapter Introduction This chapter introduces an evaluation model applicable to Haskell programs. As in the previous chapters, this chapter focuses on use of first-order functions and primitive data types. A goal of this chapter and the next one is enable students to analyze Haskell functions to determine under what conditions they terminate normally and how efficient they are. This chapter presents the evaluation model and the next chapter informally analyzes simple functions in terms of time and space efficiency and termination. How can we evaluate (i.e. execute) an expression that “calls” a function like the fact1 function from Chapter 4? We do this by rewriting expressions using a substitution model, as we see in this chapter. This process depends upon a property of functional languages called referential transparency.
    [Show full text]
  • Bringing GNU Emacs to Native Code
    Bringing GNU Emacs to Native Code Andrea Corallo Luca Nassi Nicola Manca [email protected] [email protected] [email protected] CNR-SPIN Genoa, Italy ABSTRACT such a long-standing project. Although this makes it didactic, some Emacs Lisp (Elisp) is the Lisp dialect used by the Emacs text editor limitations prevent the current implementation of Emacs Lisp to family. GNU Emacs can currently execute Elisp code either inter- be appealing for broader use. In this context, performance issues preted or byte-interpreted after it has been compiled to byte-code. represent the main bottleneck, which can be broken down in three In this work we discuss the implementation of an optimizing com- main sub-problems: piler approach for Elisp targeting native code. The native compiler • lack of true multi-threading support, employs the byte-compiler’s internal representation as input and • garbage collection speed, exploits libgccjit to achieve code generation using the GNU Com- • code execution speed. piler Collection (GCC) infrastructure. Generated executables are From now on we will focus on the last of these issues, which con- stored as binary files and can be loaded and unloaded dynamically. stitutes the topic of this work. Most of the functionality of the compiler is written in Elisp itself, The current implementation traditionally approaches the prob- including several optimization passes, paired with a C back-end lem of code execution speed in two ways: to interface with the GNU Emacs core and libgccjit. Though still a work in progress, our implementation is able to bootstrap a func- • Implementing a large number of performance-sensitive prim- tional Emacs and compile all lexically scoped Elisp files, including itive functions (also known as subr) in C.
    [Show full text]
  • Week 3: Scope and Evaluation Order Assignment 1 Has Been Posted!
    Week 3: Scope and Evaluation Order CSC324 Principles of Programming Languages David Liu, Department of Computer Science Assignment 1 has been posted! Closures and static scope static (adjective) determined only by the program source code dynamic (adjective) determined only when the program is run E.g., referential transparency is a static property How are closures implemented? Are they static or dynamic? Or both? Function bodies can be processed statically (e.g., Haskell compiler generates code once per lambda). The closure environment (and therefore the closure itself) can only be generated dynamically. The closure depends only on where the function is evaluated, not where that function is called. (define (make-adder n) (lambda (x) (+ x n))) (define adder (make-adder 1)) (adder 100) So we can determine where each free identifier obtains its values statically, based on where its enclosing function is defined. scope (of an identifier) The parts of program code that may refer to that identifier. static (aka lexical) scope The scope of every identifier is determined by the structure of the source code (e.g., by nesting of lambdas and lets). Every identifier obtains its value from the closest enclosing expression that binds it. (define (make-adder n) (lambda (x) (+ x n))) (define adder (make-adder 1)) ; (0x..., {n: 1}) (let* ([n 100]) (adder 2)) Implementing static scope in an interpreter A simple interpreter (define/match (interpret env expr) [(_ (? number?)) expr] [(_ (? symbol?)) (hash-ref env expr)] [(_ (list '+ l r)) (+ (interpret env l) (interpret env r))]) A simple interpreter (define/match (interpret env expr) [(_ (? number?)) expr] [(_ (? symbol?)) (hash-ref env expr)] [(_ (list '+ l r)) (+ (interpret env l) (interpret env r))]) The environment is passed recursively when interpreting each subexpression.
    [Show full text]
  • Hop Client-Side Compilation
    Chapter 1 Hop Client-Side Compilation Florian Loitsch1, Manuel Serrano1 Abstract: Hop is a new language for programming interactive Web applications. It aims to replace HTML, JavaScript, and server-side scripting languages (such as PHP, JSP) with a unique language that is used for client-side interactions and server-side computations. A Hop execution platform is made of two compilers: one that compiles the code executed by the server, and one that compiles the code executed by the client. This paper presents the latter. In order to ensure compatibility of Hop graphical user interfaces with popular plain Web browsers, the client-side Hop compiler has to generate regular HTML and JavaScript code. The generated code runs roughly at the same speed as hand- written code. Since the Hop language is built on top of the Scheme program- ming language, compiling Hop to JavaScript is nearly equivalent to compiling Scheme to JavaScript. SCM2JS, the compiler we have designed, supports the whole Scheme core language. In particular, it features proper tail recursion. How- ever complete proper tail recursion may slow down the generated code. Despite an optimization which eliminates up to 40% of instrumentation for tail call in- tensive benchmarks, worst case programs were more than two times slower. As a result Hop only uses a weaker form of tail-call optimization which simplifies recursive tail-calls to while-loops. The techniques presented in this paper can be applied to most strict functional languages such as ML and Lisp. SCM2JS can be downloaded at http://www-sop.inria.fr/mimosa/person- nel/Florian.Loitsch/scheme2js/.
    [Show full text]
  • SI 413, Unit 3: Advanced Scheme
    SI 413, Unit 3: Advanced Scheme Daniel S. Roche ([email protected]) Fall 2018 1 Pure Functional Programming Readings for this section: PLP, Sections 10.7 and 10.8 Remember there are two important characteristics of a “pure” functional programming language: • Referential Transparency. This fancy term just means that, for any expression in our program, the result of evaluating it will always be the same. In fact, any referentially transparent expression could be replaced with its value (that is, the result of evaluating it) without changing the program whatsoever. Notice that imperative programming is about as far away from this as possible. For example, consider the C++ for loop: for ( int i = 0; i < 10;++i) { /∗ some s t u f f ∗/ } What can we say about the “stuff” in the body of the loop? Well, it had better not be referentially transparent. If it is, then there’s no point in running over it 10 times! • Functions are First Class. Another way of saying this is that functions are data, just like any number or list. Functions are values, in fact! The specific privileges that a function earns by virtue of being first class include: 1) Functions can be given names. This is not a big deal; we can name functions in pretty much every programming language. In Scheme this just means we can do (define (f x) (∗ x 3 ) ) 2) Functions can be arguments to other functions. This is what you started to get into at the end of Lab 2. For starters, there’s the basic predicate procedure?: (procedure? +) ; #t (procedure? 10) ; #f (procedure? procedure?) ; #t 1 And then there are “higher-order functions” like map and apply: (apply max (list 5 3 10 4)) ; 10 (map sqrt (list 16 9 64)) ; '(4 3 8) What makes the functions “higher-order” is that one of their arguments is itself another function.
    [Show full text]
  • Supporting Return-Value Optimisation in Coroutines
    Document P1663R0 Reply To Lewis Baker <[email protected]> Date 2019-06-18 Audience Evolution Supporting return-value optimisation in coroutines Abstract 2 Background 3 Motivation 4 RVO for coroutines and co_await expressions 6 Return-Value Optimisation for normal functions 6 Return-Value Optimisation for co_await expressions 8 Status Quo 8 Passing a handle to the return-value slot 8 Resuming with an exception 10 Multiple Resumption Paths and Symmetric Transfer 10 Simplifying the Awaiter concept 11 Specifying the return-type of an awaitable 12 Calculating the address of the return-value 12 Benefits and Implications of RVO-style awaitables 14 Generalising set_exception() to set_error<E>() 17 Avoiding copies when returning a temporary 17 A library solution to avoiding construction of temporaries 19 Language support for avoiding construction of temporaries 21 Avoiding copies for nested calls 22 Adding support for this incrementally 23 Conclusion 24 Acknowledgements 24 Appendix - Examples 25 Abstract Note that this paper is intended to motivate the adoption of changes described in P1745R0 for C++20 to allow adding support for async RVO in a future version of C++. The changes described in this paper can be added incrementally post-C++20. The current design of the coroutines language feature defines the lowering of a co_await expression ​ ​ such that the result of the expression is produced by a call to the await_resume() method on the ​ ​ awaiter object. This design means that a producer that asynchronously produces a value needs to store the value somewhere before calling .resume() so that the await_resume() method can then return that ​ ​ ​ ​ value.
    [Show full text]
  • A History of Clojure
    A History of Clojure RICH HICKEY, Cognitect, Inc., USA Shepherd: Mira Mezini, Technische Universität Darmstadt, Germany Clojure was designed to be a general-purpose, practical functional language, suitable for use by professionals wherever its host language, e.g., Java, would be. Initially designed in 2005 and released in 2007, Clojure is a dialect of Lisp, but is not a direct descendant of any prior Lisp. It complements programming with pure functions of immutable data with concurrency-safe state management constructs that support writing correct multithreaded programs without the complexity of mutex locks. Clojure is intentionally hosted, in that it compiles to and runs on the runtime of another language, such as the JVM. This is more than an implementation strategy; numerous features ensure that programs written in Clojure can leverage and interoperate with the libraries of the host language directly and efficiently. In spite of combining two (at the time) rather unpopular ideas, functional programming and Lisp, Clojure has since seen adoption in industries as diverse as finance, climate science, retail, databases, analytics, publishing, healthcare, advertising and genomics, and by consultancies and startups worldwide, much to the career-altering surprise of its author. Most of the ideas in Clojure were not novel, but their combination puts Clojure in a unique spot in language design (functional, hosted, Lisp). This paper recounts the motivation behind the initial development of Clojure and the rationale for various design decisions and language constructs. It then covers its evolution subsequent to release and adoption. CCS Concepts: • Software and its engineering ! General programming languages; • Social and pro- fessional topics ! History of programming languages.
    [Show full text]
  • Tail Recursion • Natural Recursion with Immutable Data Can Be Space- Inefficient Compared to Loop Iteration with Mutable Data
    CS 251 SpringFall 2019 2020 Principles of of Programming Programming Languages Languages Ben Wood Topics λ Ben Wood Recursion is an elegant and natural match for many computations and data structures. Tail Recursion • Natural recursion with immutable data can be space- inefficient compared to loop iteration with mutable data. • Tail recursion eliminates the space inefficiency with a +tail.rkt simple, general pattern. • Recursion over immutable data expresses iteration more clearly than loop iteration with mutable state. • More higher-order patterns: fold https://cs.wellesley.edu/~cs251/s20/ Tail Recursion 1 Tail Recursion 2 Naturally recursive factorial CS 240-style machine model Registers Code Stack (define (fact n) Call frame (if (= n 0) 1 Call frame (* n (fact (- n 1))))) Call frame Heap arguments, variables, fixed size, general purpose general size, fixed return address per function call Space: O( ) Program How efficient is this implementation? Counter cons cells, Time: O( ) Stack data structures, … Pointer Tail Recursion 3 Tail Recursion 4 Evaluation (define (fact n) (if (= n 0) Naturally recursive factorial example 1 (* n (fact (- n 1))))) Call stacks at each step (fact 3) (fact 3): 3*_ (fact 3): 3*_ (fact 3): 3*_ (define (fact n) (fact 2) (fact 2): 2*_ (fact 2): 2*_ (if (= n 0) Compute result so far Base case returns 1 after/from recursive call. (fact 1) (fact 1): 1*_ base result. Remember: n ↦ 2; and (* n (fact (- n 1))))) “rest of function” for this call. (fact 0) Recursive case returns result so far. Compute remaining argument before/for recursive call. (fact 3): 3*_ (fact 3): 3*_ (fact 3): 3*_ (fact 3): 3*2 (fact 2): 2*_ (fact 2): 2*_ (fact 2): 2*1 Space: O( ) (fact 1): 1*_ (fact 1): 1*1 Time: O( ) (fact 0): 1 Tail Recursion 5 Tail Recursion 6 Tail recursive factorial Common patterns of work Accumulator parameter Natural recursion: Tail recursion: provides result so far.
    [Show full text]
  • Functional Programming Functional Vs. Imperative Referential
    Functional vs. Imperative Referential transparency Imperative programming concerned with “how.” The main (good) property of functional programming is Functional Programming referential transparency. Functional programming concerned with “what.” COMS W4115 Every expression denotes a single value. Based on the mathematics of the lambda calculus Prof. Stephen A. Edwards (Church as opposed to Turing). The value cannot be changed by evaluating an expression Spring 2003 or by sharing it between different parts of the program. “Programming without variables” Columbia University No references to global data; there is no global data. Department of Computer Science It is inherently concise, elegant, and difficult to create subtle bugs in. There are no side-effects, unlike in referentially opaque Original version by Prof. Simon Parsons languages. It’s a cult: once you catch the functional bug, you never escape. The Joy of Pascal Strange behavior Variables program example(output) This prints 5 then 4. At the heart of the “problem” is fact that the global data var flag: boolean; flag affects the value of f. Odd since you expect function f(n:int): int In particular begin f (1) + f (2) = f (2) + f (1) if flag then f := n flag := not flag else f := 2*n; gives the offending behavior flag := not flag Mathematical functions only depend on their inputs end They have no memory Eliminating assignments eliminates such problems. begin In functional languages, variables not names for storage. flag := true; What does this print? Instead, they’re names that refer to particular values. writeln(f(1) + f(2)); writeln(f(2) + f(1)); Think of them as not very variables.
    [Show full text]
  • Perhaps There Is a Much Better Way Goals
    DjangoCon 2014 Tony Morris Perhaps There Is A much Better Way Goals Our existing common goals The goals for today Goals Our common goals to implement software efficiently to arrive at an initial result as quickly as possible over time, to reliably arrive at results as quickly as possible (aka maintenance) a result is valid if the program accurately achieves our objective Goals Our common goals impertinent goals financial gain "but django makes me $$$!" motivate personal avidities Goals Our goals for today "Why Django Sucks" Goals Our goals for today I could talk all day about why django sucks and I’d be saying lots and lots of true things but not necessarily helpful things Goals Our goals for today I could talk all day about why django sucks and I’d be saying lots and lots of true things but not necessarily helpful things Goals Our goals for today The goal today To equip you with new tools and perspective, with which to explore the question for yourself. Goals Lies Along the way We will visit some of the lies you have been told Goals Lies Tacitly insidious ones "Having to come to grips with Monads just isn’t worth it for most people" Goals Lies Confusingly insidious ones "Imperative vs functional programming vs object-oriented programming" Goals Lies Awkward ones "The real world is mutable" Goals Lies Funny ones "Django: The Web framework for perfectionists with deadlines" Goals Summary What is functional programming? What does monad mean? Functional imperative programming Parametricity —types are documentation What is Functional Programming?
    [Show full text]
  • Peach Documentation Release
    peach Documentation Release Alyssa Kwan Nov 11, 2017 Contents 1 Table of Contents 3 1.1 Solutions to Common Problems.....................................3 2 Indices and tables 5 i ii peach Documentation, Release Welcome to peach. peach is a functional ETL framework. peach empowers you to easily perform ETL in a way inherent to the functional programming paradigm. Why peach? Please see the Solutions to Common Problems section of the documentation. peach is the culmination of learnings from over a decade of mistakes made by the principal author. As such, it represents best practices to deal with a wide array of ETL and data lake / data warehouse problems. It also represents a sound theoretical framework for approaching this family of problems, namely “how to deal with side effects amidst concurrency and failure in a tractable way”. Contents 1 peach Documentation, Release 2 Contents CHAPTER 1 Table of Contents 1.1 Solutions to Common Problems 1.1.1 Clean Retry on ETL Job Failure Problem Jobs are side-effecting; that is their point: input data is ingested, and changes are made to the data lake or warehouse state in response. If a job fails partway through execution, it leaves all sort of garbage. This garbage gets in the way of retries, or worse yet, any other tries at all. If all of the job artifacts - both intermediate and final - are hosted upon a transactional data store, and jobs are only using features that are included in transactions (for instance, some RDBMS’s don’t include schema changes in transaction scope, so you can’t create tables and have them automatically cleaned up), then congratulations! There is no problem.
    [Show full text]
  • Interprocedural Control Flow Analysis of First-Order Programs with Tail Call Optimization 1 Introduction
    Interpro cedural Control Flow Analysis of FirstOrder Programs with Tail Call Optimization Saumya K Debray and To dd A Pro ebsting Department of Computer Science The University of Arizona Tucson AZ USA Email fdebray toddgcsarizonaedu Decemb er Abstract The analysis of control ow Involves guring out where returns will go How this may b e done With items LR and Is what in this pap er we show Intro duction Most co de optimizations dep end on control ow analysis typically expressed in the form of a control ow graph Traditional algorithms construct intrapro cedural ow graphs which do not account for control ow b etween pro cedures Optimizations that dep end on this limited information cannot consider the b e havior of other pro cedures Interpro cedural versions of these optimizations must capture the ow of control across pro cedure b oundaries Determining interpro cedural control ow for rstorder programs is relatively straightforward in the absence of tail call optimization since pro cedures return control to the p oint imme diately after the call Tail call optimization complicates the analysis b ecause returns may transfer control to a pro cedure other than the active pro cedures caller The problem can b e illustrated by the following simple program that tak es a list of values and prints in their original order all the values that satisfy some prop erty eg exceed To take advantage of tailcall optimization it uses an accumulator to collect these values as it traverses the input list However this causes the order of the values in the
    [Show full text]