WCL: Delivering Efficient Common Lisp Applications under Unix Wade Hennessey Center for Design Research Stanford University wade@ sunrise. stanf ord. edu Abstract: Common Lisp implementations for Unix from foreign data. However, because Lisp programs can- have traditionally provided a rich development environ- not be provided to other programmers in a form com- ment at the expense of an inefficient delivery environ- patible with the native system linker, Lisp must remain ment. The goal of WCL is to allow hundreds of Lisp “in control” by linking foreign code into the Lisp image. applications to be realistically available at once, while Unfortunately, foreign debugging information is usually allowing several of them to run concurrently. WCL ac- lost in the process, thus making debugging of mixed complishes this by providing Common Lisp as a Unix language programa difficult. shared library that can be linked with Lisp and C code Standalone applications are produced by dynami- to produce efficient applications. For example, the cally loading application code into a running Lisp and executable for a Lisp version of the canonical “Hello then saving a memory image to disk. Autoloading can World!” program requires only 40k bytes under SunOS be used to delay the loading of code until runtime, thus 4.1 for SPARC. WCL also supports a full development reducing image size, but it can substantially increase the environment, including dynamic file loading and debug- startup time of an application. Treeshakers and selec- ging. A modified version of GDB [1], the GNU Debug- tive loading of subparts of the Lisp system are also used ger, is used to debug WCL programs, providing support to reduce the size of saved memory images, but the in- for mixed language debugging. The techniques used in herently intertwined nature of Common Lisp makes this WCL should also be applicable to other high-level lan- difficult to use effectively. Furthermore, the read-only guages that allow runtime mappings from names to ob- portions of a memory image can be shared only with jects. other people running exactly the same image. Thus, every application that is even partially written in Lisp must duplicate both on disk and in memory a signifi- 1 Introduction cant portion of the Lisp runtime library, no matter how small the executable size becomes. 1.1 Lisp Machine Approach Many Common Lisp implementations for Unix are 1.2 Unix and WCL approach modeled after the “Lisp Machine” view of comput- Window oriented Unix programs encountered problems ing. A Lisp development environment is often a multi- similar to Common Lisp when dealing with large li- megabyte executable that contains a compiler, evalua- braries such as X. Every application that wanted to ac- tor, dynamic linker, debugger, runtime library, threads cess the window system was statically linked to include package, editor, window system interface, etc., all run- large portions of the X library, giving each application ning in a single address space. Support for passing data its own copy of the library code. For example, when to foreign languages is typically quite good, although the executable for the X Calculator program xcalc is great effort is usually taken to distinguish Lisp data linked this way, it occupies 773k bytes of disk space un- Permission to copy without fee all or part of this material is der Ultrix 4.1 on a MIPS based DECStation. Dozens of granted provided that the copies are not made or distributed for other X clients have similar disk requirements, leading direct commercial advantage, the ACM copyright notice and the to a poor utilization of disk space and memory. title of the publication and its date appear, and notice is given Shared library support was added to many versions that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee of Unix to overcome this problem. Executable only and/or spacific permission. contain the code that is unique to a specific application, 1992 ACM LISP & F. P.-6 I92ICA @ 1992 ACM 0.89791 -483 -X1921000610260 . ..$I .50 260 while common library code is shared both on disk and heap. A Common Lisp conservative GC is actually a in memory with other applications. For example, under bit simpler than a conservative GC for Scheme because SunOS 4.1 on SPARC, xcalc only requireds 82k bytes Common Lisp does not support heap allocated upward of disk space because it shares the X library code with continuations. A static heap that is not subject to other applications. garbage collection is also provided for permanent ob- WCL takes advantage of shared libraries to produce jects that should never be copied. correspondingly efficient Lisp applications. WCL emits standard “.0” files that can be linked into a sharable 2.2 Runtime Model library using ld, the standard Unix linker. Applications can then be linked with shared libraries using ld again. Every Lisp function is translated into a corresponding Thus, Lisp libraries can be shard with other programs C function, using name mangling to convert Lisp names just as easily as a library written in C or Fortran. into legal C names. Depending upon the compiler’s Most Lisp systems also provide their own unique optimization settings, Lisp functions may be called di- debugger that executes in the same address space as rectly, just as they would normally be called in C, or the Lisp system. This differs from the Unix model of they may be called indirectly through the function cell debugging in which the debugger executes in a supe- of the symbol that names the function. The first argu- rior process that runs and controls the program being ment to every Lisp function is an argument count, fol- debugged in a separate, inferior process. In order to lowed by the required arguments. The varargs/stdarg make WCL programs work well with other languages, support in C is used to support optional, keyword and the GNU debugger GDB has been modified to under- rest arguments. Both heap allocated and stack allo- stand Lisp. GDB was chosen because it is an excel- cated rest argument lists are supported, as well as stack lent debugger with full source code availability. The re- allocated rest argument vectors. sult is that Lisp and C programs can be linked together Multiple values are supported by overloading the and debugged using the native tools used by other Unix purpose of the argument count. If a caller wants to programs. Similarly, native profiling and code analysis receive multiple values, a pointer to a stack allocated tools can also be applied to Lisp programs. structure is passed as the argument count. The ac- tual argument count is stored inside of this structure, along with sufficient space for receiving the values. The 2 WCL Compiler and Runtime structure pointer is distinguishable from an actual argu- Model ment count because argument counts are small integers, while stack pointers appear as large integers greater WCL translates Common Lisp into either K+R or ANSI than call-arguments-limit. C. As Lisp code is translated to C, information about If a callee wants to return multiple values, the first symbol usage, structure definitions, and package oper- value is returned as the function’s normal return value, ations is also recorded. This information is currently while the remaining values are stored into the values stored as a comment at the start of the emitted C file, structure along with a return value count. This sys- although it should eventually be stored in a special sec- tem imposes no overhead on the normal single value tion of the object file. The information is later used by expected, single value returned, function call. The only the linking portions of the WCL development system. complication involves passing a multiple value holder on to other functions when making tail calls. For example, if A calls B expecting multiple values, and B does a tail 2.1 Memory Model call to C, B must pass along the multiple value holder The runtime model for WCL is fairly conventional. All from A to C. However, it must determine at runtime dynamically allocated storage comes from a single heap. if A passed a multiple value holder. This check adds Every object in the heap has a single word header, one a few instructions to tail calls. However, this approach byte of which is used as a type field. The least significant has the advantage of using standard C while still being bit of every Lisp pointer is used as a tag bit. A tag of ef%cient. O indicates that the pointer is really a 31 bit immediate Catch, throw, dynamic return-from, dynamic go, integer, while a tag of 1 signifies that the type field in unwimd-prot ect, and special variable binding are all the object header indicates the object’s type. A single supported by set jmp and long jmp, and a stack allo- table of all possible character objects is maintained to cated “dynamic chain”. For example, a call to throw avoid allocating characters in the heap. first searches the dynamic chain for a corresponding A conservative GC based on the work done in cat ch tag. If found, then the chain is unwound, undoing Scheme->C [2] is used to garbage collect the dynamic special bindings and using longjmp to hop up the stack 261 and execute cleanup code in any unwind-protects. any copying of data, allowing destructive operations to Once the stack is unwound, long j mp transfers control work on returned data as expected. This same tech- to the catch point. nique can also be used to pass and receive arrays of any Set jmp and long j mp present a few problems for rank and element type.