Concatenative Programming
From Ivory to Metal Jon Purdy ● Why Concatenative Programming Matters (2012) ● Spaceport (2012–2013) Compiler engineering ● Facebook (2013–2014) Site integrity infrastructure (Haxl) ● There Is No Fork: An Abstraction for Efficient, Concurrent, and Concise Data Access (ICFP 2014) ● Xamarin/Microsoft (2014–2017) Mono runtime (performance, GC) What I Want in a ● Prioritize reading & modifying code over writing it Programming ● Be expressive—syntax closely Language mirroring high-level semantics ● Encourage “good” code (reusable, refactorable, testable, &c.) ● “Make me do what I want anyway” ● Have an “obvious” efficient mapping to real hardware (C) ● Be small—easy to understand & implement tools for ● Be a good citizen—FFI, embedding ● Don’t “assume you’re the world” ● Forth (1970) Notable Chuck Moore Concatenative ● PostScript (1982) Warnock, Geschke, & Paxton Programming ● Joy (2001) Languages Manfred von Thun ● Factor (2003) Slava Pestov &al. ● Cat (2006) Christopher Diggins ● Kitten (2011) Jon Purdy ● Popr (2012) Dustin DeWeese ● … History Three ● Lambda Calculus (1930s) Alonzo Church Formal Systems of ● Turing Machine (1930s) Computation Alan Turing ● Recursive Functions (1930s) Kurt Gödel Church’s Lambdas e ::= x Variables λx.x ≅ λy.y | λx. e Functions λx.(λy.x) ≅ λy.(λz.y) | e1 e2 Applications λx.M[x] ⇒ λy.M[y] α-conversion (λx.λy.λz.xz(yz))(λx.λy.x)(λx.λy.x) ≅ (λy.λz.(λx.λy.x)z(yz))(λx.λy.x) (λx.M)E ⇒ M[E/x] β-reduction ≅ λz.(λx.λy.x)z((λx.λy.x)z) ≅ λz.(λx.λy.x)z((λx.λy.x)z) ≅ λz.z Turing’s Machines
⟨ ⟩ M = Q, Γ, b, Σ, δ, q0, F ● Begin with initial state & tape ● Repeat: Q Set of states ○ If final state, then halt Γ Alphabet of symbols ○ Apply transition function ∈ b Γ Blank symbol ○ Modify tape ⊆ ∖ Σ Γ {b} Input symbols ○ Move left or right ∈ ⊆ q0 Q, F Q Initial & final states δ State transition function
δ : (Q ∖ F) × Γ → Q × Γ × {L, R} Gödel’s Functions
f(x1, x2, …, xk) = n Constant S(x) = x + 1 Successor
k Pi (x1, x2, …, xk) = xi Projection f ∘ g Composition
ρ(f, g) Primitive recursion
μ(f) Minimization Three Four ● Lambda Calculus (1930s) Alonzo Church Formal Systems of ● Turing Machine (1930s) Computation Alan Turing ● Recursive Functions (1930s) Kurt Gödel ● Combinatory Logic (1950s) Moses Schönfinkel, Haskell Curry Combinatory Logic (SKI, BCKW)
Just combinators and applications! Bxyz = x(yz) Compose Cxyz = xzy Flip Sxyz = xz(yz) Application Kxy = x Constant S = λx.λy.λz.xz(yz) “Starling” Wxy = xyy Duplicate
Kxy = x Constant SKKx = Kx(Kx) = x K = λx.λy.x “Kestrel” M = SII = λx.xx Ix = x Identity L = CBM = λf.λx.f(xx) I = λx.x “Idiot” Y = SLL = λf.(λx.f(xx))(λx.f(xx)) What is Turing machines → imperative Lambda calculus → functional concatenative Combinatory logic →* concatenative programming? “A concatenative programming language is a point-free computer programming language in which all expressions denote functions, and the juxtaposition of expressions denotes function composition.”
— Wikipedia, Concatenative Programming Language “…a point-free computer programming language…” Point-Free Programming find . -name '*.txt' define hist (List
Can Programming Be Liberated int inner_product from the Von Neumann Style? (int n, int a[], int b[]) (1977) John Backus { int p = 0; CPU & memory connected by “von for (int i = 0; i < n; ++i) Neumann bottleneck” via primitive p += a[i] * b[i]; “word-at-a-time” style; programming return p; languages reflect that } Value-Level Programming int inner_product n=3; a={1, 2, 3}; b={6, 5, 4}; (int n, int a[], int b[]) p ← 0; { i ← 0; int p = 0; p ← 0 + 1 * 6 = 6; for (int i = 0; i < n; ++i) i ← 0 + 1 = 1; p += a[i] * b[i]; p ← 6 + 2 * 5 = 16; return p; i ← 1 + 1 = 2; } p ← 16 + 3 * 4 = 28; 28 Value-Level Programming
● Semantics & state closely ● No high-level combining forms: coupled: values depend on all everything built from primitives previous states ● No useful algebraic properties: ● Too low-level: ○ Can’t easily factor out ○ Compiler infers structure to subexpressions without optimize (e.g. vectorization) writing “wrapper” code ○ Programmer mentally ○ Can’t reason about subparts executes program or steps of programs without context through it in a debugger (state, history) FP Def InnerProd ≡ (Insert +) ∘ (ApplyToAll ×) ∘ Transpose
Def InnerProd ≡ (/ +) ∘ (α ×) ∘ Trans
innerProd ∷ Num a ⇒ [[a]] → a innerProd = sum . map product . transpose FP
Def InnerProd ≡ InnerProd:⟨⟨1, 2, 3⟩, ⟨6, 5, 4⟩⟩ (Insert +) ∘ (ApplyToAll ×) ∘ ((/ +) ∘ (α ×) ∘ Trans):⟨⟨1,2,3⟩, ⟨6,5,4⟩⟩ Transpose (/ +):((α ×):(Trans:⟨⟨1,2,3⟩, ⟨6,5,4⟩⟩)) (/ +):((α ×):⟨⟨1,6⟩, ⟨2,5⟩, ⟨3,4⟩⟩) ≡ Def InnerProd (/ +):(⟨×:⟨1,6⟩, ×:⟨2,5⟩, ×:⟨3,4⟩⟩) ∘ ∘ (/ +) (α ×) Trans (/ +):⟨6,10,12⟩ +:⟨6, +:⟨10,12⟩⟩ +:⟨6,22⟩ 28 Function-Level Programming
● Stateless: values have no ● Made by only combining forms dependencies over time; all data ● Useful algebraic properties dependencies are explicit ● Easily factor out subexpressions: ● High-level: Def SumProd ≡ (+ /) ∘ (α ×) ○ Expresses intent Def ProdTrans ≡ (α ×) ∘ Trans ○ Compiler knows structure ● Subprograms are all pure ○ Programmer reasons about functions—all context explicit large conceptual units J innerProd =: +/@:(*/"1@:|:) innerProd >1 2 3; 6 5 4
(+/ @: (*/"1 @: |:)) >1 2 3; 6 5 4
+/ (*/"1 (|: >1 2 3; 6 5 4))
+/ (*/"1 >1 6; 2 5; 3 4)
+/ 6 10 12
28 J You can give verbose names to things: sum =: +/ of =: @: products =: */"1 transpose =: |:
innerProduct =: sum of products of transpose
(J programmers don’t.) Function-Level ● Primitive pure functions ● Combining forms: combinators, Programming: HoFs, “forks” & “hooks” Summary ● Semantics defined by rewriting, not state transitions ● Enables purely algebraic reasoning about programs (“plug & chug”) ● Reuse mathematical intuitions from non-programming education ● Simple factoring of subprograms: “extract method” is cut & paste Three Four Five ● Lambda Calculus (1930s) Alonzo Church Formal Systems of ● Turing Machine (1930s) Computation Alan Turing ● Recursive Functions (1930s) Kurt Gödel ● Combinatory Logic (1950s) Moses Schönfinkel, Haskell Curry ● Concatenative Calculus (~2000s) Manfred von Thun, Brent Kirby Concatenative Calculus
The Theory of Concatenative [ A ] dup = [ A ] [ A ] Combinators (2002) Brent Kirby [ A ] [ B ] swap = [ B ] [ A ] E ::= C Combinator | [ E ] Quotation [ A ] drop =
| E1 E2 Composition [ A ] quote = [ [ A ] ] ∘ (E2 E1) [ A ] [ B ] cat = [ A B ]
[ A ] call = A Concatenative Calculus
{ dup, swap, drop, quote, cat, call } is Turing-complete!
Smaller basis:
[ B ] [ A ] k = A [ B ] [ A ] cake = [ [ B ] A ] [ A [ B ] ]
[ B ] [ A ] cons = [ [ B ] A ] [ B ] [ A ] take = [ A [ B ] ] Combinatory Logic (BCKW)
● B — apply functions Bkab = k(ab) Compose/apply ● C — reorder values Ckab = kba Flip ● K — delete values Kka = k Constant ● W — duplicate values Wka = kaa Duplicate
Connection to logic: substructure!
● W — contraction ● C — exchange ● K — weakening Combinatory Logic
● BCKW = SKI ● BI = ordered + linear ○ S = B(BW)(BBC) “Exactly once, in order” ○ K = K (Works in any category!) ○ I = WK ● BCI = linear ● SKI → LC (expand combinators) “Exactly once” ● LC → SKI (abstraction algorithm) ● BCKI = affine ● { B, C, K, W } = LC “At most once” ● BCWI = relevant “At least once” Substructural Type Systems
● Rust, ATS, Clean, Haskell (soon) ● Reason about any resource: ● Rust (affine): if a mutable memory, file handles, locks, reference exists, it must be sockets… unique—eliminate data races & ● Enforce protocols: “consume” synchronization overhead objects that are no longer valid ● Avoid garbage collection: ● Prevent invalid state transitions precisely track lifetimes of ● Reversible computing objects to make memory usage ● Quantum computing deterministic (predictable perf.) Substructural Rules in Concatenative Calculus
[ A ] dup k = [ A ] [ A ] k ● Continuations are no longer Wka = kaa scary or confusing ● “Current continuation” (call/cc) [ A ] [ B ] swap k = [ B ] [ A ] k is simply the remainder of the Ckab = kba program [ A ] drop k = k ● Saving a continuation is as easy Kka = k as saving the stacks and instruction pointer Concatenative Calculus ≈ Combinatory Logic + Continuation-Passing Style “…all expressions denote functions […] juxtaposition…denotes function composition.” Stacks ● Composition is the main way to build programs, but what are we composing functions of? ● We need a convenient data structure to store the program state and allow passing multiple values between functions ● Most concatenative languages use a heterogeneous stack, separate from the call stack, accessible to the programmer ● Other models proposed; stack is convenient & efficient in practice “Everything is an object a list a function”
Literals (“nouns”) take stack & return Term 2 is a function, pushes value 2. it with corresponding value on top. 2 3 + is a function, equal to 5. Can be split into 2 3 and + or 2 and 3 +. 2 : ∀s. s → s × ℤ "hello" : ∀s. s → s × string Higher-order functions (“adverbs”) take functions (“quotations”). Operators & functions (“verbs”) pop inputs from & push outputs to stack. ["ay", "bee", "cee"] { "bo" (+) say } each : ∀s. s × ℤ × ℤ → s × ℤ (+) // aybo beebo ceebo (±) : ∀s. s × ℤ × ℤ → s × ℤ × ℤ Forth : SQ ( n -- n^2 ) DUP * ; 2 SQ
Imperative or pure? Both!
2 SQ ⇒ 2 DUP * ⇒ 2 2 * ⇒ 4 2 ⇒ 2 2 ⇒ 4
: READ ( -- str ) … ; : EVAL ( str -- val ) … ; : PRINT ( val -- ) … ; : LOOP ( -- ) READ EVAL PRINT LOOP ; : REPL LOOP ; Stack Shuffling
: MAX 2DUP < IF SWAP THEN DROP ;
5 3 MAX 3 5 MAX 5 3 2DUP < IF SWAP THEN DROP 3 5 2DUP < IF SWAP THEN DROP 5 3 5 3 < IF SWAP THEN DROP 3 5 3 5 < IF SWAP THEN DROP 5 3 0 IF SWAP THEN DROP 3 5 1 IF SWAP THEN DROP 3 5 SWAP DROP 5 3 DROP 5 3 DROP 5 5 Local Variables
Can be more readable to drop from Locals are simply lambda function to value level with local expressions in disguise—composing variables. instead of applying. “Lambda” is decoupled into “anonymous function” dup2 (<) if { swap } drop and “variable binding”. → x, y; Remember: f g = g ∘ f = λs. g (f s) if (x < y) { y } else { x } f (→ x; g) = λs. (λx. g (snd s)) (fst s) Translation to Lambdas
Simple translation from concatenative terms to lambda terms:
(a b)′ = λs. b′ (a′ s) [ a ]′ = λs. pair (λt. a′ t) s [strict] = λs. pair a′ s [lazy] dup′ = λs. pair (fst s) s swap′ = λs. pair (fst (snd s)) (pair (fst s) (snd (snd s))) … A Spoonful of Sugar
Having the option to write operators (1 + 2) * (3 + 4) infix makes it easier to copy & tweak 1 2 (+) 3 4 (+) (*) math expressions from other languages, even if it breaks b neg concatenativity. + (b ^ 2 - 4 * a * c) sqrt / (2 * a) Same goes for control flow: people are accustomed to if…elif…else b neg b 2 (^) 4 a (*) c (*) and can choose a combinator form if (-) sqrt (+) 2 a (*) (/) they want its specific advantages. Without local variables? Have fun. Close mapping from syntax to semantics
● Data flow order matches ● Not an isomorphism: multiple program order: things happen input programs can map to the the way you write them same semantics ● Syntax monoid: concatenation ● Programs compose! The and empty program; semantic meaning of the concatenation of monoid: function composition two programs is the composition and identity function on stacks of their meanings ● Monoid homomorphism from ● Can be concatenative at the syntax to semantics, preserving lexical level (Forth, Factor) or the identity and joining operation term level (Kitten) Factor(ing) “C”: var price = concatenative.org wiki customer.orders[0].price;
Factor:
orders>> first price>> Factor(ing) var orders = (customer == null ? null : customer.orders); concatenative.org wiki var order = (orders == null ? null : orders[0]); var price = (order == null ? null : order.price);
dup [ orders>> ] when dup [ first ] when dup [ price>> ] when Factor(ing) dup [ orders>> ] when dup [ first ] when dup [ price>> ] when concatenative.org wiki MACRO: maybe ( quots -- ) [ '[ dup _ when ] ] map [ ] join ;
{ [ orders>> ] [ first ] [ price>> ] } maybe Value Propositions ● Pure functions are a good default unit of behavior of Concatenative ● Function composition is a good Programming default means of combining behaviors ● Juxtaposition is a convenient notation for composition ● Having a simple language with a strong mathematical foundation makes it easier to develop tooling and reason about code Implementation How do we make this efficient?
● Forth: typically threaded code to define ite
Implementation of Stack-based ● Conversion to SSA/SSI/CPS Languages on Register Machines ○ Program is post-order (1996) M. Anton Ertl flattened data flow graph ○ No dynamic stack ops ● Spectrum of representations ○ Must know arity of functions ● Represent the stack in memory / generate specializations ● Cache top value in a register ○ Uses standard register (huge win for code size & perf.) allocation techniques ● Cache multiple values ○ Stack shuffling becomes ● FSM of possible registers in calls mov or no-op Linear Lisp ● Variables are consumed when used; copies must be explicit ● Can be compiled efficiently to a Linear Logic and Permutation stack machine architecture Stacks—The Forth Shall Be ● Reduce Von Neumann bottleneck First (1993) Henry Baker “A…stack cache utilizes its space on the chip & memory bandwidth better than a register bank of the same capacity […] A linear stack machine should be even more efficient […] all of the data held in the stack cache is live data and is not just tying up space.” Linear Lisp ● “Most people describe the top several positions of the Forth stack as ‘locations’, but it is more Linear Logic and Permutation productive to think of them as Stacks—The Forth Shall Be ‘busses’, since no addressing is First (1993) Henry Baker required to read from them at all--the ALU is directly connected to these busses.” ● “…one can conceive of multiple arithmetic operations being performed simultaneously on a number of the top items of the ‘stack’…in parallel” Linear Lisp ● Because call rate is so high, and functions are small, you can use the call stack to store not return Linear Logic and Permutation addresses, but functions Stacks—The Forth Shall Be themselves First (1993) Henry Baker ● A “call” copies the contents of a function onto the return stack (queue) and proceeds ● Can be implemented with a cyclic shift register—small loops are just repeated shifts of this register, no branch prediction required Value ● Pros: uniform representation, generic functions are easy—no Representation: need to generate specializations Boxing? ● Cons: performance overhead of indirections; need RC or GC ● With no types or full static types, most things can be unboxed ● Small arrays: put elements directly on the stack; size is known ● Closures: copy captured variables onto stack w/ function pointer; invoking closure is just pop+jump ● Otherwise: COW/RC Static Typing State of Type ● Most concatenative languages are dynamically typed (Joy, Factor, Systems in PostScript) or untyped (Forth) Concatenative ● There have been a handful of Programming Forths with simple type checkers ● Cat was the first concatenative language with static types based on Hindley–Milner; now defunct ● Nobody else was working on a statically typed one, so I started working on Kitten (2011) “Simply Aritied” Approach used in some static Forths: each function has m inputs and n Languages outputs
dup : a -- a a Type Inference for Stack Languages (2017) Rob Kleffner swap : a b -- b a drop : a --
Problem: no stack polymorphism
call1,1 : a ( a -- b ) -- b
call1,2 : a ( a -- b c ) -- b c
call2,1 : a b ( a b -- c ) -- c … Typing with Tuples
Stack represented as a product type Types can get unwieldy—add (tuple); “rest of stack” is polymorphic. syntactic sugar to make it usable.
● dup : ∀sa. s × a → s × a × a define map
● All functions are polymorphic E.g., functional argument to map must wrt. the part of the stack they be applied on different stack states. don’t touch; higher-order ∀ functions are higher-rank; map : sab. → recursion is polymorphic (s × List a × (s × a s × b) → ● Complete and Easy Bidirectional s × List b) Type Checking for Higher-Rank map : ∀sab. Polymorphism Joshua Dunfield, (s × List a × ∀t. (t × a → t × b) Neel Krishnaswami → s × List b) Challenges with Stack Polymorphism
● Higher-order functions can be E.g., functional argument to dip may polymorphic over the have an arbitrary (but known) effect. stack—need to generate ∀ → → specializations based on arity dip : sta. (s × a × (s t) t × a) (and calling convention) { drop } dip swap drop
{ "meow" } dip "meow" swap Representing ● Can’t “do” anything with only pure functions; should we throw up our Effects hands and have an impure language? (Forth, Factor, Cat, &al.) ● Haskell uses monads: represent actions as values, build them with pure functions; under the hood, compile to imperative code ● Problem: monads don’t compose—can’t (always, easily) mix effects ● Solution: algebraic effects Effect Types define newline (-> +IO): "\n" print (Permissions) in Kitten define print_or_fail (Bool -> +IO +Fail): -> x; if (x): "good" print else: "bad" fail
If f needs +A and g needs +B, f g needs +A +B or +B +A (commutative) Effect Types (“Permissions”) in Kitten
Inspired by Koka (2012) Daan Leijen ● Effects: enforce what a function is allowed to do (e.g. I/O, unsafe) Compositional: a function has the ● Coeffects: enforce constraints effects of the functions it calls. on the environment where a Polymorphic: a higher-order function function is called (e.g. platform) has the effect of its argument: ● RAII: “handler” that discharges a permission (e.g. locking) map
Treat preceding terms as stack, evaluating code at compile time to construct new terms:
"%s: %d" #printf Term → Term List
Concatenative programming is both