Outline

• Lua Introduction • Lua History • Lua Features Lua Foundation of PLs

Yu Zhang Acknowledgement: Stanford CS242: Programming Languages, http://cs242.stanford.edu/

Course web site: http://staff.ustc.edu.cn/~yuzhang/fopl

FOPL: Lua 2

What is a scripting language? Scripting langs: so productive

• Common features of a scripting language • Key idea: encode program information as you go – Dynamically typed – e.g. type information, data lifetime – Garbage collected – No one likes commitment! – Rich standard library – Reflection/metaprogramming • Easy to use certain idioms hard to express in static types – Interfaces/duck typing – Polymorphism – Heterogeneous data structures – Extensible classes

FOPL: Lua 3 FOPL: Lua 4

Script. langs: semi-specialized Why Lua?

• Bash-like scripting languages • Simplest, cleanest scripting language still in use – e.g. Python, • “Correct” scoping – For file manipulation,data crunching, command line parsing • No class system (but can build our own!) • Web scripting languages • Easy to learn in a day – e.g. JavaScript, PHP – Specialized constructs for dealing with webpages or HTTP requests • Born in 1993 at PUC-Rio, Brazil • Embedded scripting languages • https://www.lua.org/pil/ – Mostly just Lua • Data structure: tables (associative arrays) – Lightweight, easy to build, simple semantics for games, config • , extensible semantics, embedding files The Evolution of Lua, HOPL III, Jun 9-10, 2007.

FOPL: Lua 5 FOPL: Lua 6

1 Lua: Overview Lua Implementation

• Lua function • https://www.lua.org/download.html -- Recursive impl. -- Iterative impl. – Lua 5.3: Jan 12, 2015, Lua 5.3.5: Jul 10, 2018 function fact(n) function fact(n) if n == 0 then local a = 1 – Smallish, e.g. Lua 5.1 17000 lines of return 1 for i = 1, n do else a = a * i – Portable return n * fact(n-1) end end return a – Embeddable end end • can call Lua from C and C from Lua • C – Clean code -- Recursive impl. -- Iterative impl. int fact(int n) { int fact(int n){ • Good for your “code reading club” if (n == 0) int a = 1, i; – Efficiency return 1; for (i = 1; i

FOPL: Lua 7 FOPL: Lua 8

Lua vs. Modula Syntax Lua: similarities with Scheme

• Designed for productive use “The influence of Scheme has gradually increased during Lua’s evolution.” – “Syntactically, Lua is reminiscent of Modula and uses • Simiarities familiar keywords.” [HOPL] – Dynamic typing, first-class values, anonymous functions, closures, ... -- Lua -- Modula-2 function fact(n) PROCEDURE Fact(n: CARDINAL): • would have wanted first-class CARDINAL; • function foo() ... end is syntactic sugar for foo = function () ... end local a = 1 VAR a: CARDINAL; BEGIN a := 1 – Scheme has lists as its data structuring mechanism, for i = 1, n do FOR i := 1 TO n DO while Lua has tables. a = a * i a := a * i; end END; – No particular object or class model forced onto the return a RETURN a; programmer—choose or implement one yourself. end END Fact;

FOPL: Lua 9 FOPL: Lua 10

Lua History SOL

• Prehistory • SOL as in "Simple Object Language" – Born in 1993 inside Tecgraf – a DSL for a configurable report generator for lithology (Comp. Graphics Tech. Group of PUC-Rio in Brazil) profiles – Lua creators: Roberto Ierusalimschy, Luiz Henrique de • SOL Figueiredo, and Waldemar Celes – read a report description, and syntax and type check – Lua Ancestors: “These languages, called DEL and specified objects and attributes SOL, were the ancestors of Lua.” • syntax influenced by BibTeX • DEL and SOL were domain-specific languages (DSLs) for Tecgraf-developed interactive graphical programs type @track{ x:number, y:number=23, id=0 } – DEL as in "data-entry language“ type @line{ t:@track=@track{x=8}, z:number* } • for describing data-entry tasks: named and typed fields, T = @track{ y=9, x=10, id="1992-34" } data validation rules, how to input and output data L = @line{ t=@track{x=T.y, y=T.x}, z=[2,3,4] }

FOPL: Lua 11 FOPL: Lua 12

2 Motivation for Lua Birth of Lua

– DEL users began to ask for more power, e.g. • "Lua"—"moon" in Portuguese (with conditionals and loops) – cf. "SOL"—"sun" in Portuguese – SOL implementation finished, but not delivered, as • SOL’s syntax for record and list construction support for was soon to be required T = @track{ y=9, x=10, id="1992-34" } – Conclusion: replace both SOL and DEL by a single, valid in both SOL and Lua. more powerful language • Semantics differ: • Existing Alternatives – tables represent both records and lists; – Tcl: "unfamiliar syntax", bad data description support, – track (here) does not name a record type, it names a Unix only function to be applied. No match for the free, – Lisps: "unfriendly syntax" do-it-yourself atmosphere at Tecgraf – Python: still in its infancy FOPL: Lua 13 FOPL: Lua 14

Lua Feature Evolution Lua Types

• Lua designers have shown good judgement. • Lua’s type selection has remained fairly stable. • Learn PL design by asking: – initially: numbers, strings, tables, nil, userdata (pointers to C objects), Lua functions, C functions – What features were added to Lua and why? – unified functions in v3.0; booleans and threads in v5.0 – What features were turned down and why? • Learn PL implementation by asking: • Tables: any value as index – early syntax: @(), @[1,2], @{x=1,y=2} – How were the features implemented? – later syntax: {}, {1,2}, {x=1,y=2}, {1,2,x=1,y=2} – What kind of implementations were not possible due to • sparse arrays OK: {[1000000000]=1} – other implementation choices? – element referencing sugar: a.x for a["x"] – tables with named functions for OO • for inheritance, define a table indexing operation

FOPL: Lua 15 FOPL: Lua 16

Tables Extensible Semantics

• The syntax of tables has evolved, the semantics of • Goals tables in Lua has not changed at all: – allow tables to be used as a basis for objects and classes – tables are still associative arrays and can store arbitrary • fallbacks in Lua 2.1(备选) pairs of values – One function per operation (table indexing, arithmetic • Effort in implementing tables efficiently operations, string concatenation, order comparisons, and – Lua 4.0, tables were implemented as pure hash tables, function calls) 当操作被应用到错误的值时，调用备选函数 with all pairs stored explicitly • tag methods in Lua 3.0 – Lua 5.0, a hybrid representation for tables: every table – tag-specific fallbacks, any value taggable contains a hash part and an array part, and both parts • metatables and metamethods in Lua 5.0 can be empty. Tables automatically adapt their two parts x = {} according to their contents. function f () return -5 end setmetatable(x, { __unm = f }) return -x --> -5 FOPL: Lua 17 FOPL: Lua 18

3 Expressing OOP Concepts Expressing OOP Concepts

FOPL: Lua 19 FOPL: Lua 20

Expressing OOP Concepts

Use metatables to add a layer of Overhead issue! indirection and to provide dynamic lookup on the metatable. According to the definition, each A group of related tables may share a instance of an account contains common metatable (which describes an entry for every method their common behavior). member, which leads to a lot of Line 3 creates object if user does not provide one; line 4 calls setmetatable pointers, and a lot of overhead. to set or change the metatable of any new object t, and make t inherit its Assume you have a class with 30 operations from the A table itself methods, then every time you using the index metamethod, accordingly reducing the overhead make an instance of the class, mentioned before. you have to allocate 30 strings and store them all in a table.

FOPL: Lua 21 FOPL: Lua 22

Expressing OOP Concepts

The derived class LA is just an Scope, Function Calls and instance of A but extended with member l. Storage Management LA inherits new from A. When new at line 20 executes, the self parameter will refer to LA. Therefore, value at “ConceptsinProgrammingLanguages” index index in the metatable of a will  Chapter7:Scope,Functions,andStorage be LA. Thus a inherits from LA, which Management inherits from A. When calling a:w at line 21, Lua cannot find a w field in a, so it looks into LA and there it finds the implementation for LA:w.

FOPL: Lua 23

4 Scope Simplified Machine Model

• Nested blocks, local variables

• Storage management – Enter block: allocate space for variables – Exits block: some or all space may be deallocated

• Static (lexical) scoping (Lua, etc.) – Global refers to declaration in closest enclosing block • Dynamic scoping – Global refers to most recent activation record

FOPL: Lua 25 FOPL: Lua 26

Activation record for in-link block Activation record for function

• Control link • Return address – Pointer to previous record on – Location of code to execute stack on function return • Push record on stack • Return-result address – Set new control link to point to – Address in activation record old env ptr of calling block to store – Set env ptr to new record function return val • Pop record off stack • Parameters – Follow control link of current – Locations to contain data record to reset environment from calling block pointer

FOPL: Lua 27 FOPL: Lua 28

First-order functions Activation record for static scope

• Parameter passing • Control link – pass-by-value: copy value to new activation record – Link to activation record of previous (calling) block – pass-by-reference: copy ptr to new activation record • Access link – Link to activation record of • Access to global variables closest enclosing block in – global variables are contained in an activation record program text higher “up” the stack • Difference – Control link depends on dynamic behavior of prog • Tail – Access link depends on static – an optimization for certain recursive functions form of program text

FOPL: Lua 29 FOPL: Lua 30

5 Higher-order functions Closures

• Language features • Function value is pair – Functions passed as arguments – closure = < env, code > – Functions that return functions from nested blocks – Need to maintain environment of function • When a function represented by a closure is called, Functions as first class values – Allocate activation record for call (as always) • Simpler case – Set the access link in the activation record using the – Function passed as argument environment pointer from the closure – Need pointer to activation record “higher up” in stack • More complicated second case – Function returned as result of function call – Need to keep activation record of returning function FOPL: Lua 31 FOPL: Lua 32

Closures via "Upvalues" Full Lexical Scoping

• Lua authors wanted lexical scoping (词法作用域/静 • Lua 5.0 got the real thing 态作用域) early on • Solution: “Keep local variables in the (array-based) – difficult due to technical restrictions stack and only move them to the heap if they go • wanted to keep a simple array stack for activation records out of scope while being referred by nested • one-pass functions." (JUCS 11 #7) • Lua 3.1 with a compromise called upvalues function f () – In creating a function, make (frozen) copies of the values local b = 1 of any external variables used by a function. local inc_b = (function () b = b + 1 end) inc_b() function f () 高阶函数：void (voidint) return (function () return b end) local b = 1 end return (function () return %b + 1 end) // b是外部的局部变量, upvalue return f()() --> 2 closure: 一个匿名函数加上其可访问的 upvalue end return f()() --> 2 upvalue 有些像C的static局部变量

FOPL: Lua 33 FOPL: Lua 34

Tail Calls Coroutines

• tail calls supported since 5.0 • coroutines—a general control abstraction – called function reuses the stack entry of the calling – term introduced by Melvin Conway in 1963 function – has lacked a precise definition, but implies “the capability • erases information from stack traces of keeping state between successive calls" • only for statements of the form return f(...) • have not been popular in mainstream languages – return n * fact(n-1) does not result in a tail call • classification: – full coroutines are stackful, and first-class objects • stackful coroutines can suspend their execution from within nested functions – an asymmetric is “subordinate” to its caller— can yield, caller can resume

FOPL: Lua 35 FOPL: Lua 36

6 Coroutines in Lua Coroutine Example

• constraints: portability and C integration – cannot manipulate a C in ANSI C > return (string.gsub("abbc", "b", – impossible: first-class continuations (as in Scheme), function (x) return "B" end)) symmetric coroutines (e.g., in Modula-2) aBBc • Lua 5.0 got full asymmetric coroutines, with > return (string.gsub("abbc", "b", create, resume and yield operations coroutine.wrap(function (x) coroutine.yield("B") – ...and PUC-Rio guys gave proof of ample expressive coroutine.yield("C") power end))) – capture only a partial , from yield to aBCc resume — cannot have C parts there 协程有现场保护

FOPL: Lua 37 FOPL: Lua 38

Lua is designed for embedding

• Interpreter is small – Lua: 35 files, 14K LOC, 700 KB, ~1s compile time on Will’s laptop – Python: 4000 files, 900K LOC, 300 MB, ~1.5m compile time Embedding Lua • Language is small – Not much syntax

 ProgramminginLua Part IV • The C API – Few core language concepts (e.g. no classes) – Small standard library   24– AnOverviewoftheCAPI

 25– ExtendingyourApplication • Relatively simple C API

 26– CallingCfromLua – Small language = small API surface

 http://lua-users.org/lists/lua-l/2007-11/msg00248.html – Easily sandboxed FOPL: Lua 40

C embedded in Lua? Issues of Interop between PLs

• Easy access to many libraries • Lua and C – C ABI is the lowest common denominator for many PLs – Type system: dynamic vs. static – Memory management: automatic vs. manual • Improve performance of bottlenecks – In Python, numpy and stdlib • C and Fortran – Fortran: HPC（complex number）, column-major order, and function, call-by-reference • Extend language semantics – C: argc and argv，dynamic memory management, row- – Async I/O, threading, … major order, call-by-value – E.g. generate mpif.h by wrapping mpi library written in C

FOPL: Lua 41 FOPL: Lua 42

7 Issues of Interop between PLs CAPI

• Spark • Issues – Write applications in Scala, Java, Python, R – Static typing in C vs. Dynamic typing in Lua – Combine SQL, streaming and complex analytics (GraphX, – Manual memory management in C vs. GC in Lua MLlib, …) =>Use an abstract stack to exchange data between C & Lua – Run on Hadoop, Mesos, Kubernetes, standalone, or in the function foo(t) return t.x end cloud void foo_l(void) { /* Lua 1.0 */ – Access diverse data sources, e.g. HDFS, Cassandra, HBase, lua_Object t = lua_getparam(1); Hive, Tachyon, Amazon S3 lua_Object r = lua_getfield(t, "x"); lua_pushobject(r); • Javascript and WebGL … } – Zhen Zhang. xWIDL: Modular and Deep JavaScript API int foo_l(lua_State* L) { /* Lua 4.0 */ Misuses Checking Based on eXtended WebIDL, ACM lua_pushstring(L, "x"); SIGPLAN 2016 Student Research Contest, (Poster) lua_gettable(L, 1); return 1; FOPL: Lua 43 FOPL: Lua 44 }

Interop between Lua and C

• Two modes of use: Lua in C vs. C in Lua • Lua uses a stack as an API – Prevent user from having to manage memory – Provide easy way to manage variadic inputs/outputs – Contrasts with Python manual refcounting

• Userdata provides a means of giving C values to Lua • Lua as a configuration language: extending your app.

FOPL: Lua 45

8