Es: A shell with higher-order functions Paul Haahr ± Adobe Systems Incorporated Byron Rakitzis ± Network Appliance Corporation ABSTRACT In the fall of 1990, one of us (Rakitzis) re-implemented the Plan 9 command interpreter, rc, for use as a UNIX shell. Experience with that shell led us to wonder whether a more general approach to the design of shells was possible, and this paper describes the result of that experimentation. We applied concepts from modern functional programming languages, such as Scheme and ML, to shells, which typically are more concerned with UNIX features than language design. Our shell is both simple and highly programmable. By exposing many of the internals and adopting constructs from functional programming languages, we have created a shell which supports new paradigms for programmers. Although most users think of the shell as an At a super®cial level, es looks like most UNIX interactive command interpreter, it is really a pro- shells. The syntax for pipes, redirection, background gramming language in which each statement runs a jobs, etc., is unchanged from the Bourne shell. Es's command. Because it must satisfy both the interac- programming constructs are new, but reminiscent of tive and programming aspects of command execu- rc and Tcl[6]. tion, it is a strange language, shaped as much by Es is freely redistributable, and is available by history as by design. anonymous ftp from ftp.white.toronto.edu. Ð Brian Kernighan & Rob Pike [1] Using es Introduction Commands A shell is both a programming language and For simple commands, es resembles other the core of an interactive environment. The ancestor shells. For example, newline usually acts as a com- of most current shells is the 7th Edition Bourne mand terminator. These are familiar commands shell[2], which is characterized by simple semantics, which all work in es: a minimal set of interactive features, and syntax that is all too reminiscent of Algol. One recent shell, cd /tmp rc[3], substituted a cleaner syntax but kept most of rm Ex* the Bourne shell's attributes. However, most recent ps aux | grep '^byron' | developments in shells (e.g., csh, ksh, zsh) have awk '{print $2}' | xargs kill -9 focused on improving the interactive environment without changing the structure of the underlying For simple uses, es bears a close resemblance language ± shells have proven to be resistant to to rc. For this reason, the reader is referred to the innovation in programming languages. paper on rc for a discussion of quoting rules, redirection, and so on. (The examples shown here, While rc was an experiment in adding modern however, will try to aim for a lowest common syntax to Bourne shell semantics, es is an explora- denominator of shell syntax, so that an understanding tion of new semantics combined with rc-in¯uenced of rc is not a prerequisite for understanding this syntax: es has lexically scoped variables, ®rst-class paper.) functions, and an exception mechanism, which are concepts borrowed from modern programming Functions languages such as Scheme and ML[4, 5]. Es can be programmed through the use of shell functions. Here is a simple function to print the date In es, almost all standard shell constructs (e.g., in yy-mm-dd format: pipes and redirection) are translated into a uniform representation: function calls. The primitive func- fn d { tions which implement those constructs can be mani- date +%y-%m-%d pulated the same way as all other functions: invoked, } replaced, or passed as arguments to other functions. The ability to replace primitive functions in es is key Functions can also be called with arguments. to its extensibility; for example, a user can override Es allows parameters to be speci®ed to functions by the de®nition of pipes to cause remote execution, or placing them between the function name and the the path-searching machinery to implement a path open-brace. This function takes a command cmd and look-up cache. arguments args and applies the command to each argument in turn: 1993 Winter USENIX ± January 25-29, 1993 ± San Diego, CA 53 Es: A shell with higher-order functions Haahr & Rakitzis fn apply cmd args { command-line. This is called a lambda.2 It takes the for (i = $args) form $cmd $i @ parameters { commands } } 1 In effect, a lambda is a procedure ``waiting to For example: happen.'' For example, it is possible to type: es> apply echo testing 1.. 2.. 3.. es> @ i {cd $i; rm -f *} /tmp testing 1.. directly at the shell, and this runs the inlined func- 2.. tion directly on the argument /tmp. 3.. There is one more thing to notice: the inline Note that apply was called with more than two function that was supplied to apply had a parame- arguments; es assigns arguments to parameters one- ter named i, and the apply function itself used a to-one, and any leftovers are assigned to the last reference to a variable called i. Note that the two parameter. For example: uses did not con¯ict: that is because es function parameters are lexically scoped, much as variables es> fn rev3 a b c { are in C and Scheme. echo $c $b $a } Variables es> rev3 1 2 3 4 5 The similarity between shell functions and 3 4 5 2 1 lambdas is not accidental. In fact, function de®nitions are rewritten as assignments of lambdas If there are fewer arguments than parameters, es to shell variables. Thus these two es commands are leaves the leftover parameters null: entirely equivalent: es> rev3 1 fn echon args {echo -n $args} 1 fn-echon = @ args {echo -n $args} So far we have only seen simple strings passed In order not to con¯ict with regular variables, as arguments. However, es functions can also take function variables have the pre®x fn- prepended to program fragments (enclosed in braces) as argu- their names. This mechanism is also used at execu- ments. For example, the apply function de®ned tion time; when a name like apply is seen by es, it above can be used with program fragments typed ®rst looks in its symbol table for a variable by the directly on the command line: name fn-apply. Of course, it is always possible to es> apply @ i {cd $i; rm -f *} \ execute the contents of any variable by dereferencing /tmp /usr/tmp it explicitly with a dollar sign: es> silly-command = {echo hi} This command contains a lot to understand, so let us es> $silly-command break it up slowly. hi In any other shell, this command would usually be split up into two separate commands: The previous examples also show that variables can be set to contain program fragments as well as es> fn cd-rm i { simple strings. In fact, the two can be intermixed: cd $i rm -f * es> mixed = {ls} hello, {wc} world } es> echo $mixed(2) $mixed(4) es> apply cd-rm /tmp /usr/tmp hello, world es> $mixed(1) | $mixed(3) Therefore, the construct 61 61 478 @ i {cd $i; rm -f *} Variables can hold a list of commands, or even is just a way of inlining a function on the a list of lambdas. This makes variables into versatile tools. For example, a variable could be used as a function dispatch table. 1In our examples, we use ``es> '' as es's prompt. The default prompt, which may be overridden, is ``; '' which is interpreted by es as a null command followed by a 2The keyword @ introduces the lambda. Since @ is not a command separator. Thus, whole lines, including special character in es it must be surrounded by white prompts, can be cut and pasted back to the shell for re- space. @ is a poor substitute for the letter λ, but it was execution. In examples, an italic ®xed width font one of the few characters left on a standard keyboard indicates user input. which did not already have a special meaning. 54 1993 Winter USENIX ± January 25-29, 1993 ± San Diego, CA Haahr & Rakitzis Es: A shell with higher-order functions Binding es> trace echo-nl In the section on functions, we mentioned that es> echo-nl a b c function parameters are lexically scoped. It is also calling echo-nl a b c possible to use lexically-scoped variables directly. a For example, in order to avoid interfering with a glo- calling echo-nl b c bal instance of i, the following scoping syntax can b be used: calling echo-nl c c let (var = value) { calling echo-nl commands which use $var } The reader should note that Lexical binding is useful in shell functions, where it !cmd becomes important to have shell functions that do not clobber each others' variables. is es's ``not'' command, which inverts the sense of the return value of cmd, and Es code fragments, whether used as arguments to commands or stored in variables, capture the ~ subject pattern values of enclosing lexically scoped values. For matches subject against pattern and returns true if example, the subject is the same as the pattern. (In fact, the es> let (h=hello; w=world) { matching is a bit more sophisticated, for the pattern hi = { echo $h, $w } may include wildcards.) } Shells like the Bourne shell and rc support a es> $hi form of local assignment known as dynamic binding. hello, world The shell syntax for this is typically: One use of lexical binding is in rede®ning var=value command functions. A new de®nition can store the previous That notation con¯icts with es's syntax for assign- de®nition in a lexically scoped variable, so that it is ment (where zero or more words are assigned to a only available to the new function.