CMU Common Lisp User s Manual
Rob ert A MacLachlan Editor
July
CMU CS
Scho ol of Computer Science
Carnegie Mellon University
Pittsburgh PA
Sup ersedes Technical Rep orts CMU CS and CMU CS
Abstract
CMU Common Lisp is an implementation of that Common Lisp is currently supp orted on MIPS pro cessor
DECstations Sparc based workstations from Sun and the IBM RT PC and other p orts are planned All
architectures are supp orted under Mach aBerkeley Unix binary compatible op erating system The Sparc is
also supp orted under SunOS The largest single part of this do cument describ es the Python compiler and the
programming styles and techniques that the compiler encourages The rest of the do cument describ es extensions
and the implementation dep endentchoices made in developing this implementation of Common Lisp Wehave
added several extensions including a source level debugger an interface to Unix system calls a foreign function
call interface supp ort for interpro cess communication and remote pro cedure call and other features that provide
agoodenvironment for developing Lisp co de
This research was sp onsored by The Defense Advanced Research Pro jects Agency Information Science and Technology
O ce under the title Research on Parallel Computing ARPA Order No issued byDARPA CMO under Contract
MDA C
The views and conclusions contained in this do cument are those of the author and should not b e interpreted as
represen ting the o cial p olicies either expressed or implied of the U S government
Keywords lisp Common Lisp manual compiler programming language implementation programming
environment
Contents
Introduction
Supp ort
Lo cal Distribution of CMU Common Lisp
Net Distribution of CMU Common Lisp
Source Availability
Command Line Options
Credits
Design Choices and Extensions
Data Typ es
Symb ols
Integers
Floats
Characters
Array Initialization
Default Interrupts for Lisp
Packages
The Editor
Garbage Collection
Describ e
The Insp ector
The Graphical Interface
The TTY Insp ector
Load
The Reader
Running Programs from Lisp
Pro cess Accessors
Saving a Core Image
Pathnames
Unix Pathnames
Wildcard Pathnames
Logical Pathnames
Search Lists
Prede ned Search Lists
Search List Op erations
Search List Example
Filesystem Op erations
Wildcard Matching
File Name Completion
Miscellaneous Filesystem Op erations
Time Parsing and Formatting
Lisp Library i
CONTENTS ii
The Debugger
Debugger Intro duction
The Command Lo op
StackFrames
Stack Motion
How Arguments are Printed
Function Names
FunnyFrames
Debug Tail Recursion
Unknown Lo cations and Interrupts
Variable Access
Variable Value Availability
Note On Lexical Variable Access
Source Lo cation Printing
How the Source is Found
Source Lo cation Availability
Compiler Policy Control
Exiting Commands
Information Commands
Breakp oint Commands
Breakp oint Example
Function Tracing
Encapsulation Functions
Sp ecials
The Compiler
Compiler Intro duction
Calling the Compiler
Compilation Units
Unde ned Warnings
Interpreting Error Messages
The Parts of the Error Message
The Original and Actual Source
The Pro cessing Path
Error Severity
Errors During Macro expansion
Read Errors
Error Message Parameterization
Typ es in Python
Compile Time Typ e Errors
Precise Typ e Checking
Weakened Typ e Checking
Getting Existing Programs to Run
Compiler Policy
The Optimize Declaration
The Optimize Interface Declaration
Op en Co ding and Inline Expansion
Advanced Compiler Use and E ciency Hints
Advanced Compiler Intro duction
Typ es
Optimization
Function Call
Represen tation of Ob jects
Writing E cient Code
More Ab out Typ es in Python
CONTENTS iii
More Typ es Meaningful
Canonicalization
Member Typ es
Union Typ es
The EmptyTyp e
Function Typ es
The Values Declaration
Structure Typ es
The Freeze T yp e Declaration
Typ e Restrictions
Typ e Style Recommendations
Typ e Inference
Variable Typ e Inference
Lo cal Function Typ e Inference
Global Function Typ e Inference
Op eration Sp eci c Typ e Inference
Dynamic Typ e Inference
Typ e Check Optimization
Source Optimization
Let Optimization
ConstantFolding
Unused Expression Elimination
Control Optimization
Unreachable Co de Deletion
Multiple Values Optimization
Source to Source Transformation
Style Recommendations
Tail Recursion
Tail Recursion Exceptions
Lo cal Call
Self Recursive Calls
Let Calls
Closures
Lo cal Tail Recursion
Return Values
Blo ck Compilation
Blo ck Compilation Semantics
Blo ck Compilation Declarations
Compiler Arguments
Practical Di culties
Context Declarations
Context Declaration Example
Inline Expansion
Inline Expansion Recording
Semi Inline Expansion
The Mayb e Inline Declaration
Byte Co ded Compilation
Ob ject Represen tation
Think Before You Use a List
Structure Representation
Arrays
Vectors
Bit Vectors
Hashtables
Numb ers
CONTENTS iv
Descriptors
Non Descriptor Representations
Variables
Generic Arithmetic
Fixnums
Word Integers
Floating Point E ciency
Sp ecialized Arrays
Sp ecialized Structure Slots
Interactions With Lo cal Call
Represen tation of Characters
General E ciency Hints
Compile Your Co de
Avoid Unnecessary Consing
Complex ArgumentSyntax
Mapping and Iteration
Trace Files and Disassembly
E ciency Notes
Typ e Uncertainty
E ciency Notes and Typ e Checking
Represen tation E ciency Notes
Verb osityControl
Pro ling
Pro le Interface
Pro ling Techniques
Nested or Recursive Calls
Clock resolution
Pro ling overhead
Additional Timing Utilities
A Note on Timing
Benchmarking Techniques
UNIX Interface
Reading the Command Line
Useful Variables
Lisp Equivalents for C Routines
Typ e Translations
System Area Pointers
Unix System Calls
File Descriptor Streams
Making Sense of Mach Return Co des
Unix Interrupts
Changing Interrupt Handlers
Examples of Signal Handlers
Event Dispatching with SERVE EVENT
Ob ject Sets
The SERVE EVENT Function
Using SERVE EVENT with Unix File Descriptors
Using SERVE EVENT with the CLX Interface to X
Without Ob ject Sets
With Ob ject Sets
A SERVE EVENT Example
Without Ob ject Sets Example
With Ob ject Sets Example
CONTENTS v
Alien Ob jects
Intro duction to Aliens
Alien Typ es
De ning Alien Typ es
Alien Typ es and Lisp Typ es
Alien Typ e Sp eci ers
The C Call Package
Alien Op erations
Alien Access Op erations
Alien Coercion Op erations
Alien Dynamic Allo cation
Alien Variables
Lo cal Alien Variables
External Alien Variables
Alien Data Structure Example
Loading Unix Ob ject Files
Alien Function Calls
The alien funcall Primitive
The def alien routine Macro
def alien routine Example
Calling Lisp from C
Step by Step Alien Example
Interprocess Communication under LISP
The REMOTE Package
Connecting Servers and Clients
Remote Evaluations
Remote Ob jects
Host Addresses
The WIRE Package
Untagged Data
Tagged Data
Making Your Own Wires
Out Of Band Data
Debugger Programmer s Interface
DI Exceptional Conditions
Debug conditions
Debug errors
Debug variables
Frames
Debug functions
Debug blo cks
Breakp oints
Co de lo cations
Debug sources
Source Translation Utilities
Function Index
Variable Index
Typ e Index
Concept Index
Chapter
Intro duction
CMU Common Lisp is a public domain implementation of Common Lisp develop ed in the Computer Science
Department of Carnegie Mellon University CMU Common Lisp is currently supp orted on MIPS pro cessor
DECstations Sparc based workstations from Sun and the IBM RT PC and other p orts are planned Currently
it runs under CMU s Mach op erating system OSF or SunOS This do cument describ es the implementation
based on the Python compiler Previous versions of CMU Common Lisp ran on the IBM RT PC and when known
as Spice Lisp on the Perq workstation See man cmucl man man cmucl for other general information
CMU Common Lisp sources and executables are freely available via anonymousFTP thissoftware is as is
and has no warrantyofany kind CMU and the authors assume no resp onsibility for the consequences of any
use of this software See doc release notes txt for a description of the state of the release you have
Supp ort
The CMU Common Lisp pro ject s goal is to develop a high quality public domain system so wewantyour bug
rep orts bug xes and enhancements However sta limitations prevent us from providing extensive supp ort
to p eople outside of CMU We are lo oking for university and industrial a liates to help us with p orting and
maintenance for hardware and software that is not widely used at CMU
This manual contains only implementation sp eci c information ab out CMU Common Lisp Users will also
need a separate manual describing the Common Lisp standard Common Lisp was initially de ned in Common
Lisp The Language by Guy L Steele Jr Common Lisp is now undergoing standardization by the X J
committee of ANSI The X J sp ec is not y et completed but a numb er of clari cations and modi cation have
b een approved Weintend that CMU Common Lisp will eventually adhere to the X J sp ec and wehave
already implemented many of the changes approved by X J
Until the X J standard is completed the second edition of Common Lisp The Language is probably the
b est available manual for the language and for our implementation of it This b o ok has no o cial role in the
standardization pro cess but it do es include many of the changes adopted since the rst edition was completed
In addition to the language itself this do cument describ es a numb er of useful library modules that run in
CMU Common Lisp Hemlock an Emacs like text editor is included as an integral part of the CMU Common
Lisp environment Two do cuments describ e Hemlock the Hemlock User s Manual and the Hemlock Command
Implementor s Manual
Lo cal Distribution of CMU Common Lisp
At CMU you can get Common Lisp by running modmisc
usr cs etc modmisc cs misc cmucl
This establishes usr misc cmucl as a symb olic link to the release area In your login add CMU CL
to your path
setpath i usr misc cmucl
CHAPTER INTRODUCTION
Then run lisp Note that the rst time you run Lisp it will take AFS several minutes to copy the image into
its lo cal cache Subsequen t starts will b e much faster
Or you can run directly out of the AFS release area whichmay b e necessary on SunOS machines Put this
in your login shell script
setenv CMUCLLIB afs cs misc cmucl sys beta lib
setpath i afs cs misc cmucl sys beta
After setting your path man cmucl will giveanintro duction to CMU CL and man lisp will describ e
command line options For SunOS installation notes see the README le in the SunOS release area
See usr misc cmucl doc for release notes and do cumentation Hardcopy do cumentation is available in
the do cument ro om Do cumentation supplements maybeavailable for recent additions see the README le
Send bug rep orts and questions to cmucl bugs cs cmu edu If you send a bug rep ort to gripe or help
they will just forward it to this mailing list
Net Distribution of CMU Common Lisp
Externally CMU Common Lisp is only available via anonymous FTP We don t have the manpower to make
tap es These are our distribution machines
lisp rt slisp cs cmu edu
lisp rt slisp cs cmu edu
Log in with the user anonymous and username host as password i e your EMAIL address When you
log in the curren t directory should b e set to the CMU Common Lisp release area If y ou haveany trouble with
FTP access please send mail to slisp cs cmu edu
The release area holds compressed tar les with names of the form
version machine os tar Z
FTP compressed tar archives in binary mode To extract cd to the directory that is to b e the ro ot of the tree
then typ e
uncompress file tar Z tar xf
The resulting tree is ab out megabytes For installation directions see the section site initialization in
README le at the ro ot of the tree
If poor network connections make it di cult to transfer a meg le the release is also available split into
ve parts with the su x to To extract from multiple les use
cat file tar Z uncompress tar xf
The release area also contains source distributions and other binary distributions A listing of the current
contents of the release area is in FILES Ma jor release announcements will b e made to comp lang lisp until
there is enough volume to warrantacomp lang lisp cmu
Source Availability
Lisp and do cumentation sources are available via anonymous FTP ftp to any CMU CS machine All CMU
written co de is public domain but CMU CL also makes use of two imported packages PCL and CLX Although
these packages are copyrighted they may b e freely distributed without any licensing agreement or fee See the
README le in the binary distribution for up to date source p ointers
The release area contains a source distribution which is an image of all the lisp source les used to build
a particular system version
v ersion source tar Z meg
All of our les including the release area are actually in the AFS le system On the release machines the
FTP server s home is the release directory afs cs cmu edu project clisp release The actual working
source areas are in other sub directories of clisp and you can directly cd to those directories if you knowthe
name Due to the wayanonymous FTP access control is done it is important to cd to the source directory with a single command and then do a get op eration
CHAPTER INTRODUCTION
Command Line Options
The command line syntax and environment is describ ed in the lisp man page in the man man directory of
the distribution See also cmucl Currently Lisp accepts the following switches
core requires an argument that should b e the name of a core le Rather than using the default core le
usr misc lisp lib lisp core the sp eci ed core le is loaded
edit sp eci es to enter Hemlock A le to edit may b e sp eci ed by placing the name of the le b etween the
program name usually lisp and the rst switch
eval accepts one argumentwhich should b e a Lisp form to evaluate during the start up sequence The value
of the form will not b e printed unless it is wrapp ed in a form that do es output
hinit accepts an argument that should b e the name of the hemlock init le to load the rst time the function
ed is invoked The default is to load hemlock init object typ e or if that do es not exist hemlock
init lisp from the user s home directory If the le is not in the user s home directory the full path
must b e sp eci ed
init accepts an argument that should b e the name of an init le to load during the normal start up sequence
The default is to load init ob ject typ e or if that do es not exist init lisp from the user s home
directory If the le is not in the user s home directory the full path must b e sp eci ed
noinit accepts no arguments and sp eci es that an init le should not b e loaded during the normal start up
sequence Also this switch suppresses the loading of a hemlock init le when Hemlockisstartedupwith
the edit switch
load accepts an argumentwhich should b e the name of a le to load into Lisp b efore entering Lisp s read eval
print loop
slave sp eci es that Lisp should start up as a sla ve Lisp and try to connect to an editor Lisp The
name of the editor to connect to must b e sp eci ed to nd the editor s name use the Hemlock
Accept Slave Connections command The name for the editor Lisp is of the form
machine name so cket
where machine name is the internet host name for the machine and so cket is the decimal number of the
so cket to connect to
For more details on the use of the edit and slave switches see the Hemlock User s Manual
Arguments to the ab ove switches can b e sp eci ed in one of twoways switch value or switch space value
For example to start up the saved core le mylisp core use either of the following two commands
lisp core mylisp core
lisp core mylisp core
Credits
Since many p eople have contributed to the development of CMU Common Lisp The currently active
memb ers are
David Axmark
Miles Bader
Ted Dunning
Scott Fahlman fearless leader
Mike Garland
Paul Gleichauf
Sean Hallgren
Simon Leinen
William Lott
Robert A Maclachlan Tim Moore
CHAPTER INTRODUCTION
Many p eople are voluntarily working on improving CMU Common Lisp means a full time CMU employee
and means a part time student employee A partial listing of signi cant past contributors follows
Rick Busdiecker
Bill Chiles
Chris Hoover
John Kolojejchick
Todd Kaufmann
Dave McDonald
Skef Wholey
This researc h was sp onsored by the Defense Advanced Researc h Pro jects Agency Information Science and
Technology O ce under the title Research on Parallel Computing issued byDARPA CMO under Contract
MDA C ARPA Order No
The views and conclusions contained in this do cument are those of the authors and should not b e interpreted
as represen ting the o cial p olicies either expressed or implied of the Defense Advanced Research Pro jects
Agency or the U S government
Chapter
Design Choices and Extensions
Several design choices in Common Lisp are left to the individual implementation and some essen tial parts of
the programming environment are left unde ned This chapter discusses the most important design choices and
extensions
Data Typ es
Symb ols
As in Common Lisp The Language all symb ols and package names are printed in lower case as a user is likely
to typ e them Internally they are normally stored upp er case only
Integers
The fixnum typ e is equivalentto signed byte Integers outside this range are represen ted as a bignum or
aword integer see section page Almost all integers that app ear in programs can b e represen ted as
a fixnum sointeger numb er consing is rare
Floats
CMU Common Lisp supp orts two oating p oint formats single float and double float These are im
plemented with IEEE single and double oat arithmetic resp ectively short float is a synonym for single
float andlong float is a synonym for double float The initial value of read default float format
is single float
Both single float and double float are represen ted with a p ointer descriptor so oat op erations can cause
numb er consing Numb er consing is greatly reduced if programs are written to allow the use of non descriptor
represen tations see section page
IEEE Special Values
CMU Common Lisp supp orts the IEEE in nity and NaN sp ecial values These non numeric values will only b e
generated when trapping is disabled for some oating p oint exception see section page so users of
the default con guration need not concern themselves with sp ecial values
extensions short float positive infinity Constant
extensions short float negative infinity Constant
extensions single float positive infinity Constant
extensions single float negative infinity Constant
extensions double float positive infinity Constant
extensions double float negative infinity Constant
CHAPTER DESIGN CHOICES AND EXTENSIONS
extensions long float positive infinity Constant
extensions long float negative infinity Constant
The values of these constants are the IEEE p ositive and negativein nity ob jects for each oat format
extensions float infinity p x Function
This function returns true if x is an IEEE oat in nity of either sign x must b e a oat
extensions float nan p x Function
extensions float trapping nan p x Function
float nan p returns true if x is an IEEE NaN Not A Numb er ob ject float trapping nan p returns
true only if x is a trapping NaN With either function x must b e a oat
NegativeZero
The IEEE oat format provides for distinct p ositive and negative zeros To test the sign on zero or anyother
oat use the Common Lisp float sign function Negative zero prints as f or d
Denormalized Floats
CMU Common Lisp supp orts IEEE denormalized oats Denormalized oats provide a mechanism for grad
ual under ow The Common Lisp float precision function returns the actual precision of a denormalized
oat which will b e less than float digits Note that in order to generate or even prin t denormalized
oats trapping must b e disabled for the under ow exception see section page The Common Lisp
least positive format float constants are denormalized
extensions float normalized p x Function
This function returns true if x is a denormalized oat x must b e a oat
Floating Point Exceptions
The IEEE oating p oint standard de nes several exceptions that o ccur when the result of a oating p oint
op eration is unclear or undesirable Exceptions can b e ignored in which case some default action is taken such
as returning a sp ecial value When trapping is enabled for an exception a error is signalled whenever that
exception o ccurs These are the p ossible oating p oint exceptions
underflow This exception o ccurs when the result of an op eration is to o small to b e represen ted as a normal
ized oat in its format If trapping is enabled the floating point underflow condition is signalled
Otherwise the op eration results in a denormalized oat or zero
overflow This exception o ccurs when the result of an op eration is to o large to b e represen ted as a oat in
its format If trapping is enabled the floating point overflow exception is signalled Otherwise the
op eration results in the appropriate in nity
inexact This exception o ccurs when the result of a oating p oint op eration is not exact i e the result
was rounded If trapping is enabled the extensions floating point inexact condition is signalled
Otherwise the rounded result is returned
invalid This exception o ccurs when the result of an op eration is ill de ned such as If trapping
is enabled the extensions floating point invalid condition is signalled Otherwise a quiet NaN is
returned
divide by zero This exception o ccurs when a oat is divided by zero If trapping is enabled the divide
by zero condition is signalled Otherwise the appropriate in nity is returned
CHAPTER DESIGN CHOICES AND EXTENSIONS
Floating Point Rounding Mo de
IEEE oating p oint sp eci es four p ossible rounding modes
nearest In this mode the inexact results are rounded to the nearer of the two p ossible result values If the
neither p ossibility is nearer then the even alternativeischosen This form of rounding is also called round
to even and is the form of rounding sp eci ed for the Common Lisp round function
positive infinity This mode rounds inexact results to the p ossible value closer to p ositive in nity Thisis
analogous to the Common Lisp ceiling function
negative infinity This mode rounds inexact results to the p ossible value closer to negative in nity Thisis
analogous to the Common Lisp floor function
zero This mode rounds inexact results to the p ossible value closer to zero This is analogous to the Common
Lisp truncate function
Warning Although the rounding mode can b e changed with set floating point modes use of anyvalue
other than the default nearest can cause unusual b ehavior since it will a ect rounding done by Common
Lisp system co de as well as rounding in user co de In particular the unary round function will stop doing
round to nearest on oats and instead do the selected form of rounding
Accessing the Floating PointModes
These functions can b e used to modify or read the oating p oint modes
extensions set floating point modes key traps rounding mode Function
fast mode accrued exceptions
current exceptions
extensions get floating point mode Function s
The keyword arguments to set floating point modes set various modes controlling how oating p oint
arithmetic is done
traps A list of the exception conditions that should cause traps Possible exceptions are underflow
overflow inexact invalid and divide by zero Initially all traps except inexact are enabled
See section page
rounding mode The rounding mode to use when the result is not exact Possible values are nearest
positive infinity negative infinity and zero Initially the rounding mode is nearest See
the warning in section ab out use of other rounding modes
current exceptions accrued exceptions Lists of exception keywords used to set the exception ags The
curren t exceptions are the exceptions for the previous op eration so setting it is not very useful The
accrued exceptions are a cumulative record of the exceptions that o ccurred since the last time these ags
were cleared Sp ecifying will clear any accrued exceptions
fast mode Set the hardware s fast mode ag if any When set IEEE conformance or debuggabilitymaybe
impaired Some machines maynothave this feature in which case the value is always nil No currently
supp orted machines have a fast mode
Ifakeyword argument is not supplied then the asso ciated state is not changed
get floating point mo des returns a list represen ting the state of the oating p ointmodes The list is in
the same format as the keyword arguments to set floating point modes so apply could b e used with set
floating point modes to restore the modes in e ect at the time of the call to get floating point modes
CHAPTER DESIGN CHOICES AND EXTENSIONS
Characters
CMU Common Lisp implements characters according to Common Lisp the Language I I The main di erence
from the rst version is that character bits and fonthave b een eliminated and the names of the typ es have b een
changed base character is the new equivalentoftheoldstring char In this implementation all characters
are base characters there are no extended characters Character co des range b etween and using the
ASCI I enco ding
Array Initialization
If no initial value is sp eci ed arrays are initialized to zero
Default Interrupts for Lisp
CMU Common Lisp has several interrupt handlers de ned when it starts up as follows
SIGINT c causes Lisp to enter a break lo op This puts you into the debugger which allows you to lo ok at the
curren t state of the computation If you pro ceed from the break loop the computation will pro ceed from
where it was interrupted
SIGQUIT causes Lisp to do a throw to the top level This causes the curren t computation to b e ab orted and
control returned to the top level read eval printloop
SIGTSTP z causes Lisp to susp end execution and return to the Unix shell If control is returned to Lisp the
computation will pro ceed from where it was interrupted
SIGILL SIGBUS SIGSEGV and SIGFPE cause Lisp to signal an error
For keyb oard interrupt signals the standard interrupt c haracter is in parentheses Your login may set up
di erent interrupt characters When a signal is generated there may b e some delay b efore it is pro cessed since
Lisp cannot b e interrupted safely in an arbitrary place The computation will continue until a safe p ointis
reached and then the interrupt will b e pro cessed See section page to de ne your own signal handlers
Packages
When CMU Common Lisp is rst started up the default package is the user package The user package uses
the common lisp extensions andpcl packages The symb ols exp orted from these three packages can b e
referenced without package quali ers This section describ es packages whichhave exp orted interfaces that may
concern users The numerous internal packages which implement parts of the system are not describ ed here
Package nicknames are in parenthesis after the full name
alien c call Exp ort the features of the Alien foreign data structure facility see section page
pcl This package contains PCL Portable CommonLoops which is a p ortable implementation of CLOS the
Common Lisp Ob ject System This implements most but not all of the features in the CLOS chapter
of Common Lisp The Language
debug The debug package contains the command line oriented debugger It exp orts utilityv arious functions and
switches
debug internals The debug internals package exp orts the primitives used to write debuggers See section
page
extensions ext The extensions packages exp orts local extensions to Common Lisp that are do cumented in
this manual Examples include the save lisp function and time parsing
hemlock ed The hemlock package contains all the co de to implement Hemlock commands The hemlock
package currently exp orts no symb ols
CHAPTER DESIGN CHOICES AND EXTENSIONS
hemlock internals hi The hemlock internals package contains co de that implements low level primitives
and exp orts those symb ols used to write Hemlock commands
keyword The keyword package contains keywords e g start All symb ols in the keyword package are
exp orted and evaluate to themselves i e the value of the symb ol is the symbol itself
profile The profile package exp orts a simple run time pro ling facility see section page
common lisp cl lisp The common lisp package exp orts all the symb ols de ned by Common Lisp the Lan
guage and only those symb ols Strictly p ortable Lisp co de will dep end only on the symb ols exp orted from
the lisp package
unix mach These packages exp ort system call interfaces to generic BSD Unix and Mach see section page
system sys The system package contains functions and information necessary for system interfacing This
package is used by the lisp package and exp orts several symb ols that are necessary to interface to system
co de
common lisp user user cl user The common lisp user package is the default package and is where a user s
co de and data is placed unless otherwise sp eci ed This package exp orts no sym b ols
xlib The xlib package contains the Common Lisp X interface CLX to the X proto col This is mostly Lisp
co de with a couple of functions that are de ned in C to connect to the server
wire The wire package exp orts a remote pro cedure call facility see section page
The Editor
The ed function invokes the Hemlock editor whichisdescribedinHemlock User s Manual and Hemlock Command
Implementor s Manual Most users at CMU prefer to use Hemlock s slaveCommon Lisp mechanism which
provides an interactive bu er for the read eval print lo op and editor commands for evaluating and compiling
text from a bu er into the slave Common Lisp Since the editor runs in the Common Lisp using slaves keeps
users from trashing their editor by developing in the same Common Lisp with Hemlock
Garbage Collection
CMU Common Lisp uses a stop and copy garbage collector that compacts the items in dynamic space every time
it runs Most users cause the system to garbage collect GC frequently long b efore space is exhausted With
or megabytes of memory causing GC s more frequently on less garbage allows the system to GC without
much if any paging
The following functions invoke the garbage collector or control whether automatic garbage collection is in
e ect
extensions gc Function
This function runs the garbage collector If ext gc verbose is non nil then it invokes ext gc notify
before b efore GC ing and ext gc notify after afterwards
extensions gc off Function
This function inhibits automatic garbage collection After calling it the system will not GC unless you call
ext gc or ext gc on
extensions gc on Function
This function reinstates automatic garbage collection If the system would have GC ed while automatic GC
was inhibited then this will call ext gc
The following variables control the b ehavior of the garbage collector
CHAPTER DESIGN CHOICES AND EXTENSIONS
extensions bytes consed between gcs Variable
CMU Common Lisp automatically GC s whenever the amount of memory allocated to dynamic ob jects
exceeds the value of an internal variable After each GC the system sets this internal variable to the amount
of dynamic space in use at that p ointplusthevalue of the variable ext bytes consed between gcs The
default value is
extensions gc verbose Variable
This variable controls whether ext gc invokes the functions in ext gc notify before and ext gc
notify after If gc verbose is nil ext gc forego es printing any messages The default value is T
extensions gc notify before Variable
This variable s value is a function that should notify the user that the system is ab out to GC It takes one
argument the amount of dynamic space in use b efore the GC measured in bytes The default value of this
variable is a function that prints a message similar to the following
GC threshold exceeded with bytes in use Commencing GC
extensions gc notify after Variable
This variable s value is a function that should notify the user when a GC nishes The function must take
three arguments the amount of dynamic spaced retained by the GC the amoun t of dynamic space freed and
the new threshold which is the minimum amount of space in use b efore the next GC will o ccur All values are
byte quantities The default value of this variable is a function that prints a message similar to the following
GC completed with bytes retained and bytes freed
GC will next o ccur when at least bytes are in use
Note that a garbage collection will not happ en at exactly the new threshold printed by the default ext gc
notify after function The system p erio dically checks whether this threshold has b een exceeded and only
then do es a garbage collection
extensions gc inhibit hook Variable
This variable s value is either a function of one argumentornil When the system has triggered an automatic
GC if this variable is a function then the system calls the function with the amount of dynamic space currently
in use measured in bytes If the function returns nil then the GC o ccurs otherwise the system inhibits
automatic GC as if you had called ext gc off The writer of this ho ok is resp onsible for knowing when
automatic GC has b een turned o and for calling or providing a way to call ext gc on The default value of
this variable is nil
extensions before gc hooks Variable
extensions after gc hooks Variable
These variables values are lists of functions to call b efore or after any GC o ccurs The system provides these
purely for side e ect and the functions take no arguments
Describe
In addition to the basic function describ ed b elow there are a numb er of switches and other things that can b e
used to control describe s b ehavior
Function describe ob ject optional stream
The describe function prints useful information ab out ob ject on stream which defaults to standard
output For any ob ject describe will printoutthetyp e Then it prints other information based on the typ e
of ob ject The typ es which are presently handled are
CHAPTER DESIGN CHOICES AND EXTENSIONS
hash table describe prints the number of entries currently in the hash table and the numb er of buckets
curren tly allocated
function describe prints a list of the function s name if any and its formal parameters If the name has
function do cumentation then it will b e printed If the function is compiled then the le where it is de ned
will b e printed as well
fixnum describe prints whether the integer is prime or not
symbol The symbol s value prop erties and do cumentation are printed If the symb ol has a function de nition
then the function is describ ed
If there is anything interesting to b e said ab out some component of the ob ject describ e will invoke itself recur
sively to describ e that ob ject The level of recursion is indicated by indenting output
extensions describe level Variable
The maximum level of recursive description allowed Initially two
extensions describe indentation Variable
The numb er of spaces to indent for each level of recursive description initially three
extensions describe print level Variable
extensions describe print length Variable
The values of print level and print length during description Initially twoand ve
The Insp ector
CMU Comm on Lisp has b oth a graphical insp ector that uses X windows and a simple terminal based insp ector
inspect optional ob ject Function
Inspect calls the insp ector on the optional argument ob ject Ifob ject is unsupplied inspect immediately
returns nil Otherwise the b ehavior of insp ect dep ends on whether Lisp is running under X When inspect is
eventually exited it returns some selected Lisp ob ject
The Graphical Interface
CMU Common Lisp has an interface to Motif which is functionally similar to CLM but works b etter in CMU
CL See
doc motif toolkit doc
doc motif internals doc
This motif interface has b een used to write the insp ector and graphical debugger There is also a Lisp control
panel with a simple le management facility aprop os and insp ector dialogs and controls for setting global
options See the interface and toolkit packages
interface lisp control panel Function
This function creates a control panel for the Lisp pro cess
interface interface style Variable
When the graphical interface is loaded this variable controls whether it is used by inspect and the error
system If the value is graphics the default and the DISPLAY environmentvariable is de ned the graphical
insp ector and debugger will b e invoked by inspect or when an error is signalled Possible values are graphics
and tty If the value is graphics but there is no X display thenwe quietly use the TTY interface
CHAPTER DESIGN CHOICES AND EXTENSIONS
The TTY Insp ector
If X is unavailable a terminal insp ector is invoked The TTY insp ector is a crude interface to describe which
allows ob jects to b e traversed and maintains a history This insp ector prints information ab out and ob ject and a
numb ered list of the components of the ob ject The command line based interface is a normal read eval print
lo op but an integer n descends into the n th component of the current ob ject and symb ols with these sp ecial
names are interpreted as commands
U Move back to the enclosing ob ject As you descend into the components of an ob ject a stack of all the ob jects
previously seen is kept This command p ops you up one level of this stack
Q E Return the current ob ject from inspect
R Recompute ob ject display and print again Useful if the ob ject mayhave changed
D Display again without recomputing
H Showhelpmessage
Load
load lename key verbose print if does not exist Function
if source newer contents
As in standard Common Lisp this function loads a le containing source or ob ject co de into the running Lisp
Several CMU extensions have b een made to load to conveniently supp ort a variety of program le organizations
lename may b e a wildcard pathname such as lisp in which case all matching les are loaded
If lename has a pathname type or extension then that exact le is loaded If the le has no extension
then this tells load to use a heuristic to load the right le The load source types and load object
types variables b elow are used to determine the default source and ob ject le typ es If only the source or the
ob ject le exists but not b oth then that le is quietly loaded Similarly if b oth the source and ob ject le exist
and the ob ject le is newer than the source le then the ob ject le is loaded The value of the if source newer
argument is used to determine what action to take when b oth the source and ob ject les exist but the ob ject
le is out of date
load ob ject The ob ject le is loaded even though the source le is newer
load source The source le is loaded instead of the older ob ject le
compile The source le is compiled and then the new ob ject le is loaded
query Theuserisasked a yes or no question to determine whether the source or ob ject le is loaded
This argument defaults to the value of ext load if source newer initially load object
The contents argument can b e used to override the heuristic based on the le extension that normally
determines whether to load the le as a source le or an ob ject le If non null this argumentmust b e either
source or binary which forces loading in source and binary mode resp ectively You really shouldn t ever
need to use this argument
extensions load source types Variable
extensions load object types Variable
These variables are lists of p ossible pathname type values for source and ob ject les to b e passed to load
These variables are only used when the le passed to load has no typ e in this case the p ossible source and
ob ject typ es are used to default the typ e in order to determine the names of the source and ob ject les
extensions load if source newer Variable
This variable determines the default value of the if source newer argumenttoload Its initial value is
load object
CHAPTER DESIGN CHOICES AND EXTENSIONS
The Reader
extensions ignore extra close parentheses Variable
If this variable is t the default then the reader merely prints a warning when an extra close parenthesis is
detected instead of signalling an error
Running Programs from Lisp
It is p ossible to run programs from Lisp by using the following function
extensions run program program args key env wait pty input Function
if input does not exist
output
Run program runs program in a child pro cess Program should b e a pathname or string naming the program
Args should b e a list of strings which this passes to program as normal Unix parameters For no arguments
sp ecify args as nil The value returned is either a pro cess structure or nil The pro cess interface follows the
description of run program Ifrun program fails to fork the child pro cess it returns nil
Except for sharing le descriptors as explained in keyword argument descriptions run program closes all le
descriptors in the child pro cess b efore running the program When you are done using a pro cess call process
close to reclaim system resources You only need to do this when you supply stream for one of input
output or error oryou supply pty non nil You can call process close regardless of whether you must
to reclaim resources without p enaltyifyou feel safer
run program accepts the following keyword arguments
env This is an a list mapping keywords and simple strings The default is ext environment list If env
is sp eci ed run program uses the value given and do es not combine the environment passed to Lisp with
the one sp eci ed
wait If non nil the default wait until the child pro cess terminates If nil continue running Lisp while the
child pro cess runs
pty This should b e one of t nil or a stream If sp eci ed non nil the subpro cess executes under a Unix
PTY If sp eci ed as a stream the system collects all output to this pty and writes it to this stream If
sp eci ed as t the process pty slot contains a stream from whichyou can read the program s output and
to whichyou can write input for the program The default is nil
input This sp eci es how the program gets its input If sp eci ed as a string it is the name of a le that contains
input for the child pro cess run program op ens the le as standard input If sp eci ed as nil the default
then standard input is the le dev null If sp eci ed as t the program uses the current standard input
This may cause some confusion if wait is nil since two pro cesses may use the terminal at the same time
If sp eci ed as stream then the process input slot contains an output stream Anything written to this
stream go es to the program as input Input may also b e an input stream that already contains all the
input for the pro cess In this case run program reads all the input from this stream b efore returning so
this cannot b e used to in teract with the pro cess
if input does not exist This sp eci es what to do if the input le do es not exist The following values are
valid nil the default causes run program to return nil without doing anything create creates the
named le and error signals an error
output This sp eci es what happ ens with the program s output If sp eci ed as a pathname it is the name of
a le that contains output the program writes to its standard output If sp eci ed as nil the default all
output go es to dev null If sp eci ed as t the program writes to the Lisp pro cess s standard output
This may cause confusion if wait is nil since two pro cesses may write to the terminal at the same time
If sp eci ed as stream then the process output slot contains an input stream from whichyou can read the program s output
CHAPTER DESIGN CHOICES AND EXTENSIONS
if output exists This sp eci es what to do if the output le already exists The following values are valid
nil causes run program to return nil without doing anything error the default signals an error
supersede overwrites the curren t le and append app ends all output to the le
error This is similar to output except the le b ecomes the program s standard error Additionally error
can b e output in which case the program s error output is routed to the same place sp eci ed for output
If sp eci ed as stream the process error contains a stream similar to the process output slot when
sp ecifying the output argument
if error exists This sp eci es what to do if the error output le already exists It accepts the same values
as if output exists
status hook This sp eci es a function to call whenever the pro cess changes status This is esp ecially useful
when sp ecifying wait as nil The function takes the pro cess as a required argument
before execve This sp eci es a function to run in the child pro cess b efore it b ecomes the program to run This
is useful for actions such as authenticating the child pro cess without modifying the parent Lisp pro cess
Pro cess Accessors
The following functions interface the pro cess returned by run program
extensions process p thing Function
This function returns t if thing is a pro cess Otherwise it returns nil
extensions process pid pro cess Function
This function returns the pro cess ID an integer for the pro cess
extensions process status pro cess Function
This function returns the curren t status of pro cess which is one of running stopped exited or
signaled
extensions process exit code pro cess Function
This function returns either the exit co de for pro cess ifitis exited or the termination signal pro cess if it
is signaled The result is unde ned for pro cesses that are still alive
extensions process core dumped pro cess Function
This function returns t if someone used a Unix signal to terminate the pro cess and caused it to dump a Unix
core image
extensions process pty pro cess Function
This function returns either the two way stream connected to pro cess s Unix PTY connection or nil if there
is none
extensions process input pro cess Function
extensions process output pro cess Function
extensions process error pro cess Function
If the corresp onding stream was created these functions return the input output or error le descriptor nil is returned if there is no stream
CHAPTER DESIGN CHOICES AND EXTENSIONS
extensions process status hook pro cess Function
This function returns the curren t function to call whenever pro cess s status changes This function takes the
pro cess as a required argument process status hook is setf able
extensions process plist pro cess Function
This function returns annotations supplied by users and it is setf able This is available solely for users to
asso ciate information with pro cess without having to build a lists or hash tables of pro cess structures
extensions process wait pro cess optional check for stopp ed Function
This function waits for pro cess to nish If check for stopped is non nil this also returns when pro cess stops
extensions process kill pro cess signal optional whom Function
This function sends the Unix signal to pro cess Signal should b e the numb er of the signal or a keyword with
the Unix name for example sigsegv Whom should b e one of the following
pid This is the default and it indicates sending the signal to pro cess only
process group This indicates sending the signal to pro cess s group
pty process group This indicates sending the signal to the pro cess group curren tly in the foreground on the
Unix PTY connected to pro cess This last option is useful if the running program is a shell and you wish
to signal the program running under the shell not the shell itself If process pty of pro cess is nil using
this option is an error
extensions process alive p pro cess Function
This function returns t if pro cess s status is either running or stopped
extensions process close pro cess Function
This function closes all the streams asso ciated with pro cess Whenyou are done using a pro cess call this to
reclaim system resources
Saving a Core Image
A mechanism has b een provided to save a running Lisp core image and to later restore it This is convenientif
you don t want to load several les into a Lisp when you rst start it up The main problem is the large size of
each saved Lisp image typically at least megabytes
extensions save lisp le key purify root structures init function Function
load init file print herald site init
process command line
The save lisp function saves the state of the currently running Lisp core image in le The keyword
arguments have the following meaning
purify If non NIL the default the core image is puri ed b efore it is saved This means moving accessible Lisp
ob jects from dynamic space into read only and static space This reduces the amountofwork the garbage
collector must do when the resulting core image is b eing run Also if more than one Lisp is running on
the same machine this maximizes the amount of memory that can b e shared b etw een the two pro cesses
Ob jects in read only and static space can never b e reclaimed even if all p ointers to them are dropp ed
root structures This should b e a list of the main entry p oints for the resulting core image The puri cation
pro cess tries to localize symb ols functions etc in the core image so that paging p erformance is improved
The default value is NIL which means that Lisp ob jects will still b e localized but probably not as optimally
as they could b e This argument has no meaning if purify is NIL
CHAPTER DESIGN CHOICES AND EXTENSIONS
init function This is the function that starts running when the created core le is resumed The default
function simply invokes the top level read eval print lo op If the function returns the lisp will exit
load init file If non NIL then load an init le either the one sp eci ed on the command line or
init fasl typ e or if init fasl typ e do es not exist init lisp from the user s home directory
If the init le is found it is loaded into the resumed core le b efore the read eval print lo op is entered
site init If non NIL the name of the site init le to quietly load The default is library site init No
error is signalled if the le do es not exist
print herald If non NIL the default then print out the standard Lisp herald when starting
process command line If non NIL the default pro cesses the command line switches and p erforms the ap
propriate actions
To resume a saved le typ e
lisp core file
Pathnames
In Common Lisp quite a few asp ects of pathname semantics are left to the implementation
Unix Pathnames
Unix pathnames are always parsed with a unix host ob ject as the host and nil as the device The last two
dots in the namestring mark the typ e and version however if the rst character is a dot it is considered
part of the name If the last character is a dot then the pathname has the empty string as its typ e The typ e
defaults to nil and the version defaults to newest
defun parse x
values pathname name x pathname type x pathname version x
parse foo foo NIL NEWEST
parse foo bar foo bar NEWEST
parse foo foo NIL NEWEST
parse foo bar foo bar NEWEST
parse NEWEST
parse foo foo NEWEST
parse foo bar foo bar
parse foo bar baz foo bar baz NEWEST
The directory of pathnames b eginning with a slash or a search list see section page is starts
absolute others start with relative The directory is parsed as up there is no namestring for back
pathname directory usr foo bar baz ABSOLUTE usr foo
pathname directory foo bar baz RELATIVE UP foo
Wildcard Pathnames
Wildcards are supp orted in Unix pathnames If is sp eci ed for a part of a pathname that is parsed as
wild can b e used as a directory name to indicate wild inferiors Filesystem op erations treat wild
inferiors the same as wild but pathname pattern matching e g for logical pathname translation see section
page matches anynumb er of directory parts with see see section page
emb edded in a pathname part matches anynumber of characters Similarly matches exactly one
character and a b matches the characters a or b These pathname parts are parsed as pattern ob jects
Backslash can b e used as an escap e character in namestring parsing to prevent the next character from b eing
treated as a wildcard Note that if typ ed in a string constant the backslash must b e doubled since the string
reader also uses backslash as a quote pathname name foo bar foo bar
CHAPTER DESIGN CHOICES AND EXTENSIONS
Logical Pathnames
If a namestring b egins with the name of a de ned logical pathname host followed by a colon then it will b e parsed
as a logical pathname Both and wildcards are implemented load logical pathname defaults on
name lo oks for a logical host de nition le in library name translations Note that library designates
the search list see section page initialized to the CMU CommonLisp lib directory notalogical
pathname The format of the le is a single list of two lists of the from and to patterns
foo text usr ram foo txt
foo lisp usr ram foo l
Search Lists
Search lists are an extension to Common Lisp pathnames They serve a function somewhat similar to Common
Lisp logical pathnames but work more likeUnixPATH variables Search lists are used for two purp oses
They provide a convenient shorthand for commonly used directory names and
They allow the abstract directory structure indep endent sp eci cation of le lo cations in program path
name constants similar to logical pathnames
Each search list has an asso ciated list of directories represen ted as pathnames with no name or typ e component
The namestring for any relative pathname may b e pre xed with slist indicating that the pathname is relative
to the search list slist instead of to the current working directory Once quali ed with a search list the pathname
is no longer considered to b e relative
When a search list quali ed pathname is passed to a le system op eration suchasopen load or truename
each directory in the search list is successiv ely used as the ro ot of the pathname until the le is located When
a le is written to a searc h list directory the leisalways written to the rst directory in the list
Prede ned Search Lists
These search lists are initialized from the Unix environment or when Lisp was built
default The curren t directory at startup
home The user s home directory
library The CMU Common Lisp lib directory CMUCLLIB environmentvariable
path The Unix command path PATH environmentvariable
target The ro ot of the tree where CMU Common Lisp was compiled
It can b e useful to rede ne these search lists for example library can b e augmented to allow logical pathname
translations to b e lo cated and target can b e rede ned to p oint to where CMU Common Lisp system sources
are locally installed
Search List Op erations
These op erations de ne and access search list de nitions A searc h list name maybeparsedinto a pathname
b efore the search list is actually de ned but the search list must b e de ned b efore it can actually b e used in a
lesystem op eration
extensions search list name Function
This function returns the list of directories asso ciated with the search list name Ifname is not a de ned
searc h list then an error is signalled When set with setf the list of directories is changed to the new value If
the new value is just a namestring or pathname then it is interpreted as a one element list Note that unlike
Unix pathnames search list names are case insensitive
CHAPTER DESIGN CHOICES AND EXTENSIONS
extensions search list defined p name Function
extensions clear search list name Function
search list defined p returns t if name is a de ned search list name nil otherwise clear search list
make the searc h list name unde ned
extensions enumerate search list var pathname result fform g Macro
This macro provides an interface to search list resolution The b o dy forms are executed with var b ound to
each successiv e p ossible expansion for name Ifname do es not contain a searc h list then the b o dy is executed
exactly once Everything is wrapp ed in a blo cknamednil so return can b e used to terminate early Theresult
form default nil isevaluated to determine the result of the iteration
Search List Example
The search list code can b e de ned as follows
setf ext search list code usr lisp code
It is now p ossible to use code as an abbreviation for the directory usr lisp code in all le op erations For
example you can now sp ecify code eval lisp to refer to the le usr lisp code eval lisp
To obtain the value of a search list name use the function search list as follows
ext search list name
Where name is the name of a search list as describ ed ab ove For example calling ext search list on code
as follows
ext search list code
returns the list usr lisp code
Filesystem Op erations
CMU Common Lisp provides a numb er of extensions and optional features b eyond those require by Common
Lisp
Wildcard Matching
Unix lesystem op erations such as open will accept wildcard pathnames that match a single le of course
directory allows anynumb er of matches Filesystem op erations treat wild inferiors the same as wild
directory wildname key all check for subdirs Function
follow links
The keyword arguments to this Common Lisp function are a CMU extension The arguments all default to
t have the following functions
all Include les b eginning with dot suchas login similar to ls a
check for subdirs Test whether les are directories similar to ls F
follow links Call truename on each le which expands out all symb olic links Note that this option can
easily result in pathnames b eing returned whichhave a di erent directory from the one in the wildname
argument
Function extensions print directory wildname optionalstream key all verbose
return list
Print a directory of wildname listing to stream default standard output all and verbose both
default to nil and corresp ond to the a and l options of ls Normally this function returns nil but if
return list is true a list of the matched pathnames are returned
CHAPTER DESIGN CHOICES AND EXTENSIONS
File Name Completion
extensions complete file pathname key defaults ignore types Function
Attempt to complete a le name to the longest unambiguous pre x If supplied directory from defaults
is used as the working directory when doing completion ignore types is a list of strings of the pathname
typ es a k a extensions that should b e disregarded as p ossible matches binary le names etc
extensions ambiguous files pathname optionaldefaults Function
Return a list of pathnames for all the p ossible completions of pathname with resp ect to defaults
Miscellaneous Filesystem Op erations
extensions default directory Function
Return the current working directory as a pathname If set with setf set the working directory
extensions file writable name Function
This function accepts a pathname and returns t if the curren t pro cess can write it and nil otherwise
extensions unix namestring pathname optionalfor input Function
This function converts pathname into a string that can b e used with UNIX system calls Search lists and
wildcards are expanded for input controls the treatment of search lists when true the default and the le
exists anywhere on the search list then that absolute pathname is returned otherwise the rst elementofthe
searc h list is used as the directory
Time Parsing and Formatting
Functions are provided to allow parsing strings containing time information and printing time in various formats
are available
extensions parse time time string key error on mismatch default seconds Function
default minutes default hours
default day
parse time accepts a string containing a time e g Jan and returns the universal time if it is
successful If it is unsuccessful and the keyword argument error on mismatch is non nil it signals an error
Otherwise it returns nil The other keyword arguments have the following meaning
default seconds sp eci es the default value for the seconds value if one is not provided by time string The
default value is
default minutes sp eci es the default value for the minutes value if one is not provided by time string The
default value is
default hours sp eci es the default value for the hours value if one is not provided by time string The default
value is
default day sp eci es the default value for the dayvalue if one is not provided by time string The default
value is the current day
default month sp eci es the default value for the month value if one is not provided by time string The default
value is the current month
default year sp eci es the default value for the year value if one is not provided b y time string The default
value is the current year
CHAPTER DESIGN CHOICES AND EXTENSIONS
default zone sp eci es the default value for the time zone value if one is not provided by time string The
default value is the curren t time zone
default weekday sp eci es the default value for the dayoftheweek if one is not provided by time string The
default value is the curren t day of the week
Any of the ab ovekeywords can b e given the value current which means to use the curren t value as determined
by a call to the op erating system
extensions format universal time dest universal time key timezone Function
style date first
print seconds
extensions format decoded time dest seconds minutes hours daymonth year key Function
format universal time formats the time sp eci ed by universal time format decoded time formats the
time sp eci ed by seconds minutes hours day month and year Dest is any destination accepted by the format
function The keyword argumen ts have the following meaning
timezone is an integer sp ecifying the hours west of Greenwich Timezone defaults to the curren t time zone
style sp eci es the style to use in formatting the time The legal values are
short sp eci es to use a numeric date
long sp eci es to format months and weekdays as words instead of numbers
abbreviated is similar to long except the words are abbreviated
government is similar to abbreviated except the date is of the form daymonth year instead of month
day year
date first if non nil default will place the date rst Otherwise the time is placed rst
print seconds if non nil default will format the seconds as part of the time Otherwise the seconds will
b e omitted
print meridian if non nil default will format AM or PM as part of the time Otherwise the AM
or PM will b e omitted
print timezone if non nil default will format the time zone as part of the time Otherwise the time zone
will b e omitted
print seconds if non nil default will format the seconds as part of the time Otherwise the seconds will
b e omitted
print weekday if non nil default will format the weekday as part of date Otherwise the weekday will b e
omitted
Lisp Library
The CMU Common Lisp pro ject maintains a collection of useful or interesting programs written by users of our
system The library is in lib contrib Two les there that users should read are
CATALOG TXT This le contains a page for eachentry in the library Itcontains information such as the
author p ortability or dep endency issues how to load the entry etc
READ ME TXT This le describ es the library s organization and all the p ossible pieces of information an
entry s catalog description could contain
Hemlock has a command Library Entry that displays a list of the curren t library entries in an editor bu er
There are mode sp eci c commands that display catalog descriptions and load entries This is a simple and
convenient waytobrowse the library
Chapter
The Debugger
By Rob ert MacLachlan
Debugger Intro duction
The CMU Common Lisp debugger is unique in its level of supp ort for source level debugging of compiled co de
Although some other debuggers allow access of variables by name this seems to b e the rst Common Lisp
debugger that
Tells you when a variable do esn t haveavalue b ecause it hasn t b een initialized yet or has already b een
deallo cated or
Can display the precise source location corresp onding to a co de location in the debugged program
These features allow the debugging of compiled co de to b e made almost indistinguishable from interpreted co de
debugging
The debugger is an interactive command lo op that allows a user to examine the function call stack The
debugger is invoked when
A serious condition is signalled and it is not handled or
error is called and the condition it signals is not handled or
The debugger is explicitly invoked with the Common Lisp break or debug functions
Note thereare two debugger interfaces in CMU CL the TTY debugger describedbelow and the Motif
debugger Since the di erence is only in the user interface much of this chapter also applies to the Motif
version SeeSeesection page for a very brief discussion of the graphical interface
When you enter the TTY debugger it lo oks something like this
Error in function CAR
Wrong type argument should have been of type LIST
Restarts
Return to Top Level
Debug type H for help
CAR
The rst group of lines describ e what the error was that put us in the debugger In this case car was called
on After Restarts isalistofallthewa ys that we can restart execution after this error In this case the
only option is to return to top level After printing its banner the debugger prints the current frame and the
debugger prompt
CHAPTER THE DEBUGGER
The Command Lo op
The debugger is an interactive read eval printloopmuch like the normal top level but some symb ols are inter
preted as debugger commands instead of b eing evaluated A debugger command starts with the symb ol name
of the command p ossibly followed by some arguments on the same line Some commands prompt for additional
input Debugger commands can b e abbreviated byanyunambiguous pre x help can b e typ ed as h he etc
For convenience some commands haveambiguous one letter abbreviations f for frame
The package is not signi cant in debugger commands any symb ol with the name of a debugger command
will work If you wanttoshow the value of a variable that happ ens also to b e the name of a debugger command
you can use the list locals command or the debug var function or you can wrap the variable in a progn to
hide it from the command loop
The debugger prompt is frame where frame is the numb er of the current frame Frames are numb ered
starting from zero at the top most recen t call increasing down to the b ottom The curren t frame is the frame
that commands refer to The current frame also provides the lexical environment for evaluation of non command
forms
The debugger evaluates forms in the lexical environment of the functions b eing debugged The debugger
can only access variables You can t go or return from into a function and you can t call lo cal functions
Sp ecial variable references are evaluated with their current value the innermost binding around the debugger
invo cation you don t get the value that the sp ecial had in the curren t frame See section page for
more information on debugger variable access
StackFrames
A stack frame is the run time represen tation of a call to a function the frame stores the state that a function
needs to rememb er what it is doing Frames have
Variables see section page which are the values b eing op erated on and
Arguments to the call which are really just particularly interesting variables and
A current location see section page which is the place in the program where the function was
running when it stopp ed to call another function or b ecause of an interrupt or error
Stack Motion
These commands move to a new stack frame and print the name of the function and the values of its arguments
in the style of a Lisp function call
up Move up to the next higher frame More recen t function calls are considered to b e higher on the stack
down Movedown to the next lower frame
top Move to the highest frame
bottom Move to the lowest frame
frame n Move to the frame with the sp eci ed numb er Prompts for the numb er if not supplied
How Arguments are Printed
A frame is printed to lo ok like a function call but with the actual argumentvalues in the argument p ositions
So the frame for this call in the source
myfun a
would look like this MYFUN A
CHAPTER THE DEBUGGER
All keyword and optional arguments are displayed with their actual values if the corresp onding argumentwas
not supplied the value will b e the default So this call
subseq foo
would look like this
SUBSEQ foo
And this call
string upcase test case
would look like this
STRING UPCASE test case START END NIL
The arguments to a function call are displayed by accessing the argumentvariables Although those variables
are initialized to the actual argumentvalues they can b e set inside the function in this case the new value will
b e displayed
rest arguments are handled somewhat di erently Thevalue of the rest argumentvariable is displayed as
the spread out arguments to the call so
format t A is a A This test
would look like this
FORMAT T A is a A This TEST
Rest arguments cause an exception to the normal displayofkeyword arguments in functions that haveboth
rest and key arguments In this case the keyword argumentvariables are not displayed at all the rest arg
is displayed instead So for these functions only the keywords actually supplied will b e shown and the values
displayed will b e the argumentvalues not values of the p ossibly modi ed variables
If the variable for an argument is never referenced by the function it will b e deleted The variable value is
then unavailable so the debugger prints unused arg instead of the value Similarly ifforanyofanumber
of reasons describ ed in more detail in section the value of the variable is unavailable or not known to b e
available then unavailable arg will b e printed instead of the argumentvalue
Printing of argumentvalues is controlled by debug print level and debug print length page
Function Names
If a function is de ned by defun labels or flet then the debugger will print the actual function name after
the op en parenthesis like
STRING UPCASE test case START END NIL
SETF AREF a for
Otherwise the function name is a string and will b e printed in quotes
DEFUN MYFUN BAR
DEFMACRO DO DO I I I NIL
SETQ GC NOTIFY BEFORE
This string name is derived from the defmumble form that encloses or expanded into the lamb da or the
outermost enclosing form if there is no defmumble
CHAPTER THE DEBUGGER
FunnyFrames
Sometimes the evaluator intro duces new functions that are used to implement a user function but are not directly
sp eci ed in the source The main place this is done is for checking argumenttyp e and syntax Usually these
functions do their thing and then go away andthus are not seen on the stack in the debugger But when you
get some sort of error during lamb da list pro cessing you end up in the debugger on one of these funny frames
These funny frames are agged byprinting keyword after the parentheses For example this call
car a b
will look like this
CAR A EXTERNAL
And this call
string upcase test case end
would look like this
DEFUN STRING UPCASE test case OPTIONAL
As you can see these frames haveonlyavague resemblance to the original call Fortunately the error
message displayed when you enter the debugger will usually tell you what problem is in these cases to o many
arguments and odd keyword arguments Also if you go down the stack to the frame for the calling function
you can display the original source see section page
With recursive or block compiled functions see section page an EXTERNAL frame may app ear
b efore the frame represen ting the rst call to the recursive function or entry to the compiled blo ck This is a
consequence of the way the compiler do es block compilation there is nothing o dd with your program You will
also see CLEANUP frames during the execution of unwind protect cleanup co de Note that inline expansion and
op en co ding a ect what frames are presen t in the debugger see sections and
Debug Tail Recursion
Both the compiler and the interpreter are prop erly tail recursive If a function call is in a tail recursive
p osition the stack frame will b e deallocated at the time of the call rather than after the call returns Consider
this backtrace
BAR
FOO
Because of tail recursion it is not necessarily the case that FOO directly called BAR Itmay b e that FOO called
some other function FOO which then called BAR tail recursively as in this example
defun foo
foo
defun foo
bar
defun bar
Usually the elimination of tail recursive frames makes debugging more pleasant since theses frames are mostly
uninformative If there is any doubt ab out how one function called another it can usually b e eliminated by
nding the source location in the calling frame section
For a more thorough discussion of tail recursion see section page
CHAPTER THE DEBUGGER
Unknown Lo cations and Interrupts
The debugger op erates using sp ecial debugging information attached to the compiled co de This debug infor
mation tells the debugger what it needs to know ab out the locations in the co de where the debugger can b e
invoked If the debugger somehow encounters a location not describ ed in the debug information then it is said
to b e unknown If the co de location for a frame is unknown then some variables may b e inaccessible and the
source location cannot b e precisely displayed
There are three reasons why a co de lo cation could b e unknown
There is inadequate debug information due to the value of the debug optimization quality See section
page
The debugger was entered b ecause of an interrupt such as C
A hardware error suchas bus error o ccurred in co de that was compiled unsafely due to the value of
the safety optimization quality See section page
In the last two cases the values of argumentvariables are accessible but may b e incorrect See section
page for more details on when variable values are accessible
It is p ossible for an interrupt to happ en when a function call or return is in progress The debugger may
then ame out with some obscure error or insist that the b ottom of the stack has b een reached when the real
problem is that the current stack frame can t b e located If this happ ens return from the interrupt and try
again
When running interpreted co de all lo cations should b e known However an interrupt might catch some
subfunction of the interpreter at an unknown location In this case you should b e able to go up the stacka
frame or two and reach an interpreted frame which can b e debugged
Variable Access
There are three ways to access the curren t frame s lo cal variables in the debugger The simplest is to typ e the
v ariable s name into the debugger s read eval print lo op The debugger will evaluate the variable reference as
though it had app eared inside that frame
The debugger doesn t really understand lexical scoping it has just one namespace for all the variables in
a function If a symb ol is the name of multiple variables in the same function then the reference app ears
ambiguous even though lexical scoping sp eci es whichvalue is visible at anygiven source location If the scop es
of the twovariables are not nested then the debugger can resolve the ambiguityby observing that only one
variable is accessible
When there are ambiguous variables the evaluator assigns eachoneasmallinteger identi er The debug var
function and the list locals command use this identi er to distinguish b etween ambiguous variables
list locals pre x
This command prints the name and value of all variables in the curren t frame whose name has the sp eci ed
pre x pre x may b e a string or a symb ol If no pre x is given then all available variables are printed If
avariable has a p otentially ambiguous name then the name is printed with a identi er su x where
identi er is the small integer used to make the name unique
debug var name optional identi er Function
This function returns the value of the variable in the current frame with the sp eci ed name If supplied
identi er determines whichvalue to return when there are ambiguous v ariables
When name is a symb ol it is interpreted as the symb ol name of the variable i e the package is signi cant
If name is an uninterned symb ol gensym then return the value of the uninterned variable with the same name
If name is a string debug var interprets it as the pre x of a variable name and must unambiguously complete
to the name of a valid variable
This function is useful mainly for accessing the value of uninterned or ambiguous variables since most
variables can b e evaluated directly
CHAPTER THE DEBUGGER
Variable Value Availability
The value of a variable may b e unavailable to the debugger in p ortions of the program where Common Lisp says
that the variable is de ned If a variable value is not available the debugger will not let you read or write that
variable With one exception the debugger will never displayanincorrectvalue for a variable Rather than
displaying incorrect values the debugger tells you the value is unavailable
The one exception is this if you interrupt e g with C or if there is an unexp ected hardware error such
as bus error which should only happ en in unsafe co de then the values displayed for arguments to the
interrupted frame might b e incorrect This exception applies only to the interrupted frame any frame farther
down the stack will b e ne
The value of a variable maybeunavailable for these reasons
The value of the debug optimization qualitymayhave omitted debug information needed to determine
whether the variable is available Unless a variable is an argument its value will only b e available when
debug is at least
The compiler did lifetime analysis and determined that the value was no longer needed even though its
scop e had not b een exited Lifetime analysis is inhibited when the debug optimization qualityis
The v ariable s name is an uninterned symb ol gensym Tosave space the compiler only dumps debug
information ab out uninterned variables when the debug optimization qualityis
The frame s lo cation is unknown see section page b ecause the debugger was entered due to an
interrupt or unexp ected hardware error Under these conditions the values of arguments will b e available
but might b e incorrect This is the exception ab ove
The variable was optimized out of existence Variables with no reads are always optimized away even
in the interpreter The degree to which the compiler deletes variables will dep end on the value of the
compile speed optimization quality but most source level optimizations are done under all compilation
p olicies
Since it is esp ecially useful to b e able to get the arguments to a function argumentvariables are treated
sp ecially when the speed optimization quality is less than and the debug quality is at least With this
compilation p olicy thevalues of argumentvariables are almost always available everywhere in the function even
at unknown lo cations For non argumentvariables debug must b e at least for values to b e available and even
then values are only available at known lo cations
Note On Lexical Variable Access
When the debugger command lo op establishes v ariable bindings for available variables these variable bindings
have lexical scop e and dynamic extent You can close over them but such closures can t b e used as upward
funargs
You can also set local variables using setq but if the variable was closed over in the original source and
never set then setting the variable in the debugger maynotchange the value in all the functions the variable is
de ned in Another risk of setting variables is that you may assign a value of a typ e that the compiler proved
the variable could never take on This may result in bad things happ ening
Source Lo cation Printing
One of CMU Common Lisp s unique capabilities is source level debugging of compiled co de These commands
display the source lo cation for the current frame
source context
This command displays the le that the current frame s function was de ned from if it was de ned from a
le and then the source form resp onsible for generating the co de that the curren t frame was executing If
context is sp eci ed then it is an integer sp ecifying the numb er of enclosing levels of list structure to print
Since the lo cation of an interrupt or hardware error will always b e an unknown lo cation see section page non argument
variable values will never b e available in the interrupted frame
The variable bindings are actually created using the Common Lisp symbol macro let sp ecial form
CHAPTER THE DEBUGGER
vsource context
This command is identical to source except that it uses the global values of print level and print
length instead of the debugger printing control variables debug print level and debug print
length
The source form for a lo cation in the co de is the innermost list presen t in the original source that encloses
the form resp onsible for generating that co de If the actual source form is not a list then some enclosing list
will b e printed For example if the source form was a reference to the variable some random special then
the innermost enclosing evaluated form will b e printed Here are some p ossible enclosing forms
let a some random special
some random special
If the co de at a location was generated from the expansion of a macro or a source level compiler opti
mization then the form in the original source that expanded into that co de will b e printed Supp ose the le
usr me mystuff lisp looked like this
defmacro mymac
myfun
defun foo
mymac
If foo has called myfun and is waiting for it to return then the source command would print
File usr me mystuff lisp
MYMAC
Note that the macro use was printed not the actual function call form myfun
If enclosing source is printed by giving an argumenttosource or vsource then the actual source form is
marked by wrapping it in a list whose rst elementis HERE In the previous example source would
File usr me mystuff lisp
DEFUN FOO
HERE
MYMAC
How the Source is Found
If the co de was de ned from Common Lisp by compile or eval then the source can always b e reliably located
If the co de was de ned from a fasl le created by compile file then the debugger gets the source forms it
prints by reading them from the original source le This is a p otential problem since the source le mighthave
moved or changed since the time it was compiled
The source le is op ened using the truename of the source le pathname originally given to the compiler
This is an absolute pathname with all logical names and symb olic links expanded If the le can t b e lo cated
using this name then the debugger gives up and signals an error
If the source le can b e found but has b een modi ed since the time it was compiled the debugger prints
this warning
File has been modified since compilation
lename Using form offset instead of character position
CHAPTER THE DEBUGGER
where lename is the name of the source le It then pro ceeds using a robust but not fo olproof heuristic for
lo cating the source This heuristic works if
No top level forms b efore the top level form containing the source have b een added or deleted and
The top level form containing the source has not b een modi ed much More precisely noneofthelist
forms b eginning b efore the source form have b een added or deleted
If the heuristic do esn t work the displayed source will b e wrong but will probably b e near the actual source
If the shap e of the top level form in the source le is to o di erent from the original form then an error will
b e signalled When the heuristic is used the the source location commands are noticeably slowed
Source lo cation printing can also b e confused if after the source was compiled a read macro you used in the
co de was rede ned to expand into something di erent or if a read macro ever returns the same eq list twice If
you don t de ne read macros and don t use in p erverted ways you don t need to worry ab out this
Source Lo cation Availability
Source lo cation information is only available when the debug optimization quality is at least If source lo cation
information is unavailable the source commands will give an error message
If source lo cation information is available but the source location is unknown b ecause of an interrupt or
unexp ected hardware error see section page then the command will print
Unknown location using block start
and then pro ceed to print the source lo cation for the start of the basic blo ck enclosing the co de location It s a
bit complicated to explain exactly what a basic blo ck is but here are some prop erties of the blo ck start location
The block start location may b e the same as the true location
The block start location will never b e later in the the program s owofcontrol than the true lo cation
No conditional control structures such as if cond or will intervene b etween the blo ck start and the
true lo cation but note that some conditionals presen t in the original source could b e optimized away
Function calls do not end basic blo cks
The head of a loop will b e the start of a block
The programming language concept of block structure and the Common Lisp block sp ecial form are
totally unrelated to the compiler s basic blo ck
In other words the true lo cation lies b etween the printed location and the next conditional but watch out
b ecause the compiler mayhavechanged the program on you
Compiler Policy Control
The compilation p olicy sp eci ed by optimize declarations a ects the b ehavior seen in the debugger The
debug quality directly a ects the debugger bycontrolling the amount of debugger information dumped Other
optimization qualities have indirect but observable e ects due to changes in the way compilation is done
Unlike the other optimization qualities which are compared in relativevalue to evaluate tradeo s the debug
optimization quality is directly translated to a level of debug information This absolute interpretation allows
the user to count on a particular amount of debug information b eing available even when the values of the other
qualities are changed during compilation These are the levels of debug information that corresp ond to the values
of the debug quality
Only the function name and enough information to allow the stac ktobeparsed
Any level greater than gives level plus all argumentvariables Values will only b e accessible if the
argumentvariable is never set and speed is not CMU Common Lisp allows anyrealvalue for opti
mization qualities It may b e useful to sp ecify to get backtrace argumentdisplay without argument
do cumentation
CHAPTER THE DEBUGGER
Level provides argument do cumentation printed arglists and derived argument result typ e information
This makes describe more informative and allows the compiler to do compile time argumentcount and
typ e checking for any calls compiled at run time
Level plus all interned local variables source lo cation information and lifetime information that tells the
debugger when arguments are available even when speed is or the argument is set This is the default
Level plus all uninterned variables In addition lifetime analysis is disabled even when speed is ensuring
that all variable values are available at anyknown location within the scop e of the binding This has a
sp eed p enalty in addition to the obvious space p enalty
As you can see if the speed qualityis debugger p erformance is degraded This e ect comes from the
elimination of argumentvariable sp ecial casing see section page Some degree of sp eed debuggability
tradeo is unavoidable but the e ect is not to o drastic when debug is at least
In addition to inline and notinline declarations the relativevaluesofthe speed and space qualities also
change whether functions are inline expanded see section page If a function is inline expanded then
there will b e no frame to represen t the call and the arguments will b e treated likeany other lo cal v ariable
Functions mayalsobe semi inline inwhich case there is a frame to represen t the call but the call is to an
optimized local version of the function not to the original function
Exiting Commands
These commands get you out of the debugger
quit Throw to top level
restart n
Invokes the nth restart case as displayed bytheerror command If n is not sp eci ed the available restart
cases are rep orted
go Calls continue on the condition given to debug If there is no restart case named continue then an error is
signaled
abort Calls abort on the condition given to debug This is useful for p opping debug command lo op levels or
ab orting to top level as the case maybe
Information Commands
Most of these commands print information ab out the curren t frame or function but a few show general infor
mation
help Displays a synopsis of debugger commands
describe Calls describe on the current function displays number of local variables and indicates whether the
function is compiled or interpreted
print Displays the current function call as it would b e displayed bymoving to this frame
vprint or pp verb osity
Displa ys the current function call using print level and print length instead of debug print
level and debug print length verb osity is a small integer default that controls other dimensions
of verb osity
error Prints the condition given to invoke debugger and the active pro ceed cases
backtrace n
Displays all the frames from the current to the b ottom Only shows n frames if sp eci ed The printing is
controlled by debug print level and debug print length
CHAPTER THE DEBUGGER
Breakp oint Commands
CMU Common Lisp supp orts setting of breakp oints inside compiled functions and stepping of compiled co de
Breakp oints can only b e set at at known lo cations see section page so these commands are largely
useless unless the debug optimize quality is at least see section page These commands manipulate
breakp oints
breakpoint lo cation foption value g
Set a breakp oint in some function location maybeaninteger co de location numb er as displayed by list
locations ora keyword The keyword can b e used to indicate setting a breakp oint at the function start
start s or function end end e The breakpoint command has condition break print and
function options whichwork similarly to the trace options
list locations or ll function
List all the co de locations in the current frame s function or in function if it is supplied The display
format is the co de lo cation numb er a colon and then the source form for that lo cation
N
If consecutive lo cations have the same source then a numeric range like will b e printed For example
a default function call has a known lo cation b oth immediately b efore and after the call whichwould result
in two co de locations with the same source The listed function b ecomes the new default function for
breakp oin t setting via the breakpoint command
list breakpoints or lb
List all currently active breakp oints with their breakp ointnumber
delete breakpoint or db number
Delete a breakp oint sp eci ed by its breakp ointnumb er If no numb er is sp eci ed delete all breakp oints
step
Step to the next p ossible breakp oint lo cation in the current function This always steps over function calls
instead of stepping into them
Breakp oint Example
Consider this de nition of the factorial function
defun n
if zerop n
n n
This debugger session demonstrates the use of breakp oints
common lisp user break Invoke debugger
Break
Restarts
CONTINUE Return from BREAK
ABORT Return to Top Level
Debug type H for help
INTERACTIVE EVAL BREAK
ll
LAMBDA N BLOCK IF ZEROP N
CHAPTER THE DEBUGGER
N N
N
N
N N
LAMBDA N BLOCK IF
br
N N
in
Added
q
common lisp user Call the function
Breakpoint hit
Restarts
CONTINUE Return from BREAK
ABORT Return to Top Level
Debug type H for help
We are now in first call arg before the multiply
Source N N
st
Step
We have finished evaluation of n
Source N
st
Breakpoint hit
Restarts
CONTINUE Return from BREAK
ABORT Return to Top Level
Debug type H for help
We hit the breakpoint in the recursive call
Source N N
Function Tracing
The tracer causes selected functions to print their arguments and their results whenever they are called Options
allow conditional printing of the trace information and conditional breakp oints on function entry or exit
trace foption global value g fname foption value g g Macro
trace is a debugging to ol that prints information when sp eci ed functions are called In its simplest form
trace name name
trace causes a printout on trace output each time that one of the named functions is entered or returns
the names are not evaluated Trace output is indented according to the numb er of p ending traced calls and
CHAPTER THE DEBUGGER
this trace depth is printed at the b eginning of each line of output Printing verb osity of arguments and return
values is controlled by debug print level and debug print length
If no names or options are are given trace returns the list of all currently traced functions traced
function list
Trace options can cause the normal printout to b e suppressed or cause extra information to b e printed
Each option is a pair of an option keyword and a value form Options maybeintersp ersed with function names
Options only a ect tracing of the function whose name they app ear immediately after Global options are
sp eci ed b efore the rst name and a ect all functions traced byagiven use of trace If an already traced
function is traced again any new options replace the old options The following options are de ned
condition form condition after form condition all form
If condition is sp eci ed then trace do es nothing unless form evaluates to true at the time of the call
condition after is similar but suppresses the initial printout and is tested when the function returns
condition all tries b oth b efore and after
wherein names
If sp eci ed names is a function name or list of names trace do es nothing unless a call to one of those
functions encloses the call to this function i e it would app ear in a backtrace Anonymous functions
have string names like DEFUN FOO
break form break after form break all form
If sp eci ed and form evaluates to true then the debugger is inv oked at the start of the function at the
end of the function or b oth according to the resp ective option
print form print after form print all form
In addition to the usual printout the result of evaluating form is printed at the start of the function at
the end of the function or b oth according to the resp ective option Multiple print options cause multiple
values to b e printed
function function form
This is a not really an option but rather another way of sp ecifying what function to trace The function
form is evaluated immediately and the resulting function is traced
encapsulate f default t nil g
In CMU Common Lisp tracing can b e done either by temporarily rede ning the function name encap
sulation or using breakp oints When breakp oints are used the function ob ject itself is destructiv ely
modi ed to cause the tracing action The advantage of using breakp oints is that tracing works even when
the function is anonymously called via funcall
When encapsulate is true tracing is done via encapsulation default is the default and means
to use encapsulation for interpreted functions and funcallable instances breakp oints otherwise When
encapsulation is used forms are not evaluated in the function s lexical environment but debug arg can
still b e used
condition break and print forms are evaluated in the lexical environment of the called function
debug var and debug arg can b e used The after and all forms are evaluated in the null environment
Macro untrace rest function names
This macro turns o tracing for the sp eci ed functions and removes their names from traced function
list Ifnofunction names are given then all curren tly traced functions are untraced
extensions traced function list Variable
A list of function names maintained and used by trace untrace and untrace all This list should contain
the names of all functions currently b eing traced
extensions max trace indentation Variable
The maximum numb er of spaces which should b e used to indent trace printout This variable is initially set to
CHAPTER THE DEBUGGER
Encapsulation Functions
The encapsulation functions provide a mechanism for intercepting the arguments and results of a function
encapsulate changes the function de nition of a symbol and saves it so that it can b e restored later The new
de nition normally calls the original de nition The Common Lisp fdefinition function always returns the
original de nition stripping o any encapsulation
The original de nition of the symb ol can b e restored at any time bytheunencapsulate function
encapsulate and unencapsulate allow a symbol to be multiply encapsulated in such a way that di erent
encapsulations can b e completely transparent to each other
Each encapsulation has a typ e whichmay b e an arbitrary lisp ob ject If a symb ol has several encapsulations
of di erent typ es then any one of them can b e removed without a ecting more recen t ones A symbol mayhave
more than one encapsulation of the same typ e but only the most recen t one can b e undone
extensions encapsulate symbol typ e b o dy Function
Saves the current de nition of symbol and replaces it with a function which returns the result of evaluating
the form body Typ e is an arbitrary lisp ob ject whichisthetyp e of encapsulation
When the new function is called the following variables are b ound for the evaluation of body
extensions argument list A list of the arguments to the function
extensions basic definition The unencapsulated de nition of the function
The unencapsulated de nition may b e called with the original arguments by including the form
apply extensions basic definition extensions argument list
encapsulate always returns symbol
extensions unencapsulate symbol typ e Function
Undo es symbol s most recen t encapsulation of typ e typ e Typ e is compared with eq Encapsulations of other
typ es are left in place
extensions encapsulated p symbol typ e Function
Returns t if symbol has an encapsulation of typ e typ e Returns nil otherwise Typ e is compared with eq
Sp ecials
These are the sp ecial variables that control the debugger action
extensions debug print level Variable
extensions debug print length Variable
print level and print length are b ound to these values during the execution of some debug com
mands When evaluating arbitrary expressions in the debugger the normal values of print level and print
length are in e ect These variables are initially set to and resp ectively
Chapter
The Compiler
Compiler Intro duction
This chapter contains information ab out the compiler that every CMU Common Lisp user should b e familiar
with Chapter go es into greater depth describing ways to use more advanced features
The CMU Common Lisp compiler also known as Python has many features that are seldom or never
supp orted byconventional Common Lisp compilers
Source level debugging of compiled co de see chapter
Typ e error compiler warnings for typ e errors detectable at compile time
Compiler error messages that provide a go o d indication of where the error app eared in the source
Full run time checking of all p otential typ e errors with optimization of typ e checks to minimize the cost
Scheme like features such as prop er tail recursion and extensive source level optimization
Advanced tuning and optimization features such as comprehensivee ciencynotes ow analysis and
untagged numb er represen tations see chapter
Calling the Compiler
Functions may b e compiled using compile compile file or compile from stream
compile name optional de nition Function
This function compiles the function whose name is name Ifname is nil the compiled function ob ject is
returned If de nition is supplied it should b e a lamb da expression that is to b e compiled and then placed in
the function cell of name As p er the prop osed X J cleanup compile argument problems de nition may
also b e an interpreted function
The return values are as p er the prop osed X J cleanup compiler diagnostics The rst value is the
function name or function ob ject The second value is nil if no compiler diagnostics were issued and t otherwise
The third value is nil if no compiler diagnostics other than style warnings were issued A non nil value indicates
that there were serious compiler diagnostics issued or that other conditions of typ e error or warning but
not style warning were signalled during compilation
compile file input pathname key output file error file trace file Function
error output verbose print progress
load block compile entry points
byte compile
The CMU Common Lisp compile file is extended through the addition of several new keywords and an
additional interpretation of input pathname
CHAPTER THE COMPILER
input pathname If this argument is a list of input les rather than a single input pathname then all the source
les are compiled into a single ob ject le In this case the name of the rst le is used to determine the
default output le names This is esp ecially useful in combination with block compile
output le This argument sp eci es the name of the output le t gives the default name nil suppresses the
output le
error le A listing of all the error output is directed to this le If there are no errors then no error le is
pro duced and any existing error le is deleted t gives name err the default and nil suppresses
the output le
error output If t the default then error output is sent to error output If a stream then output is sent to
that stream instead If nil then error output is suppressed Note that this error output is in addition to
but the same as the output placed in the error le
verb ose If t the default then the compiler prints to error output at the start and end of compilation of each
le See compile verbose page
print If t the default then the compiler prints to error output when each function is compiled See compile
print page
progress If t default nil then the compiler prints to error output progress information ab out the phases of
compilation of each function This is a CMU extension that is useful mainly in large block compilations
See compile progress page
trace le If true several of the intermediate represen tations including annotated assembly co de are dumped
out to this le t gives name trace Trace output is o by default See section page
load If true load the resulting output le
blo ck compile Controls the compile time resolution of function calls By default only self recursive calls are
resolved unless an ext block start declaration app ears in the source le See section page
entry p oints If non null then this is a list of the names of all functions in the le that should have global
de nitions installed b ecause they are referenced in other les See section page
byte compile If true compiling to a compact interpreted byte co de is enabled Possible values are t nil and
maybe the default See byte compile default page and See section page
The return values are as p er the prop osed X J cleanup compiler diagnostics The rst value from
compile file is the truename of the output le or nil if the le could not b e created The interpretation of
the second and third values is describ ed ab ove for compile
compile verbose Variable
compile print Variable
compile progress Variable
These variables determine the default values for the verbose print and progress arguments to compile
file
extensions compile from stream input stream key error stream Function
trace stream
block compile entry points
byte compile
This function is similar to compile file but it takes all its arguments as streams It reads Common Lisp
co de from input stream until end of le is reached compiling into the curren t environment This function returns
the same twovalues as the last tw ovalues of compile No output les are pro duced
CHAPTER THE COMPILER
Compilation Units
CMU Common Lisp supp orts the with compilation unit macro added to the language by the prop osed X J
with compilation unit compiler cleanup This provides a mechanism for eliminating spurious unde ned warn
ings when there are forward references across les and also provides a standard way to access compiler extensions
with compilation unit fkey value g fformg Macro
This macro evaluates the forms in an environment that causes warnings for unde ned variables functions
and typ es to b e delayed until all the forms have b een evaluated Eachkeyword value is an evaluated form These
keyword options are recognized
override If uses of with compilation unit are dynamically nested the outermost use will take precedence
suppressing printing of unde ned warnings by inner uses However when the override option is true this
shadowing is inhibited an inner use will print summary warnings for the compilations within the inner
scop e
optimize This is a CMU extension that sp eci es of the global compilation p olicy for the dynamic extentof
the b o dy The argument should evaluate to an optimize declare form lik e
optimize speed safety
See section page
optimize interface Similar to optimize but sp eci es the compilation p olicy for function interfaces argu
ment countandtyp e checking for the dynamic extent of the b o dy See section page
context declarations This is a CMU extension that pattern matches on function names automatically splic
ing in any appropriate declarations at the head of the function de nition See section page
Unde ned Warnings
Warnings ab out unde ned variables functions and typ es are delayed until the end of the current compilation unit
The compiler entry functions compile etc implicitly use with compilation unit so unde ned warnings will
b e printed at the end of the compilation unless there is an enclosing with compilation unit In order the gain
the b ene t of this mechanism you should wrap a single with compilation unit around the calls to compile
file i e
with compilation unit
compile file file
compile file file
Unlike for functions and typ es unde ned warnings for variables are not suppressed when a de nition e g
defvar app ears after the reference but in the same compilation unit This is b ecause doing sp ecial declarations
out of order just do esn t work although early references will b e compiled as sp ecial bindings will b e done
lexically
Unde ned warnings are printed with full source context see section page which tremendously
simpli es the problem of nding unde ned references that resulted from macroexpansion After printing detailed
information ab out the unde ned uses of each name with compilation unit also prints summary listings of
the names of all the unde ned functions typ es and variables
undefined warning limit Variable
This variable controls the numb er of unde ned warnings for each distinct name that are printed with full
source context when the compilation unit ends If there are more unde ned references than this then they are
condensed into a single warning
Warning count more uses of undefined function name
When the value is then the unde ned warnings are not broken down by name at all only the summary listing
of unde ned names is printed
CHAPTER THE COMPILER
Interpreting Error Messages
One of Python s unique features is the level of source lo cation information it provides in error messages The
error messages contain a lot of detail in a terse format to they may b e confusing at rst Error messages will b e
illustrated using this example program
defmacro zoq x
roq ploq x
defun foo y
declare symbol y
zoq y
The main problem with this program is that it is trying to add to a symb ol Note also that the functions roq
and ploq aren t de ned anywhere
The Parts of the Error Message
The compiler will pro duce this warning
File usr me stuff lisp
In DEFUN FOO
ZOQ Y
ROQ PLOQ
Y
Warning Result is a SYMBOL not a NUMBER
In this example we see each of the six p ossible parts of a compiler error message
File usr me stuff lisp This is the le that the compiler read the relevant co de from The le name is
displayed b ecause it may not b e immediately obvious when there is an error during compilation of a large
system esp ecially when with compilation unit is used to delay unde ned warnings
In DEFUN FOO This is the de nition or top level form resp onsible for the error It is obtained by taking the rst
two elements of the enclosing form whose rst elementisasymb ol b eginning with DEF If there is no
enclosing def mumble then the outermost form is used If there are multiple def mumbles then they are
all printed from the out in separated by s In this example the problem was in the defun for foo
ZOQ Y This is the original source form resp onsible for the error Original source means that the form directly
app eared in the original input to the compiler i e in the lamb da passed to compile or the top level form
read from the source le In this example the expansion of the zoq macro was resp onsible for the error
ROQ PLOQ This is the pro cessing path that the compiler used to pro duce the errorful co de The pro cessing
path is a represen tation of the evaluated forms enclosing the actual source that the compiler encountered
when pro cessing the original source The path is the rst elementofeach form or the form itself if the
form is not a list These forms result from the expansion of macros or source to source transformation done
by the compiler In this example the enclosing evaluated forms are the calls to roq ploq and These
calls resulted from the expansion of the zoq macro
Y This is the actual source resp onsible for the error If the actual source app ears in the explanation then
we print the next enclosing evaluated form instead of printing the actual source twice This is the form
that would otherwise have b een the last form of the pro cessing path In this example the problem is with
the evaluation of the reference to the variable y
Warning Result is a SYMBOL not a NUMBER This is the explanation the problem In this example the prob
lem is that y evaluates to a symbol but is in a context where a numb er is required the argumentto
CHAPTER THE COMPILER
Note that each part of the error message is distinctively marked
File and In mark the le and de nition resp ectively
The original source is an indented form with no pre x
Each line of the pro cessing path is pre xed with
The actual source form is indented like the original source but is marked by a preceding line This is
like the macroexpands to notation used in Common Lisp The Language
The explanation is pre xed with the error severity see section page either Error Warning
or Note
Each part of the error message is more sp eci c than the preceding one If consecutive error messages are for
nearby locations then the front part of the error messages would b e the same In this case the compiler omits
as much of the second message as in common with the rst For example
File usr me stuff lisp
In DEFUN FOO
ZOQ Y
ROQ
PLOQ Y
Warning Undefined function PLOQ
ROQ PLOQ Y
Warning Undefined function ROQ
In this example the le de nition and original source are identical for the two messages so the compiler omits
them in the second message If consecutive messages are entirely identical then the compiler prints only the
rst message followed by
Last message occurs rep eats times
where rep eats is the numb er of times the message was given
If the source was not from a le then no le line is printed If the actual source is the same as the original
source then the pro cessing path and actual source will b e omitted If no forms intervene b etween the original
source and the actual source then the pro cessing path will also b e omitted
The Original and Actual Source
The original source displayed will almost alwa ys b e a list If the actual source for an error message is a symbol
the original source will b e the immediately enclosing evaluated list form So even if the o ending symbol does
app ear in the original source the compiler will print the enclosing list and then print the symb ol as the actual
source as though the symbol were intro duced by a macro
When the actual source is displayed and is not a symb ol it will always b e co de that resulted from the
expansion of a macro or a source to source compiler optimization This is co de that did not app ear in the
original source program it was intro duced by the compiler
Keep in mind that when the compiler displays a source form in an error message it always displays the most
sp eci c innermost resp onsible form For example compiling this function
defun bar x
let a
declare fixnum a
setq a foo x a
CHAPTER THE COMPILER
Gives this error message
In DEFUN BAR
LET A DECLARE FIXNUM A SETQ A FOO X A
Warning The binding of A is not a FIXNUM
NIL
This error message is not saying there s a problem somewhere in this let itissaying that there is a problem
with the let itself In this example the problem is that a s nil initial value is not a fixnum
The Pro cessing Path
The pro cessing path is mainly useful for debugging macros so if you don t write macros you can ignore the
pro cessing path Consider this example
defun foo n
dotimes i n undefined
Compiling results in this error message
In DEFUN FOO
DOTIMES I N UNDEFINED
DO BLOCK LET TAGBODY RETURN FROM
PROGN UNDEFINED
Warning Undefined variable UNDEFINED
Note that do app ears in the pro cessing path This is b ecause dotimes expands into
do i i g n
i g undefined
declare type unsigned byte i
The rest of the pro cessing path results from the expansion of do
block nil
let i g n
declare type unsigned byte i
tagbody go g
g psetq i i
g unless i g go g
return from nil progn undefined
In this example the compiler descended into the block let tagbody and return from to reach the progn
printed as the actual source This is a place where the actual source app ears in explanation rule was applied
The innermost actual source form was the symbol undefined itself but that also app eared in the explanation
so the compiler backed out one level
Error Severity
There are three levels of compiler error severity
Error This severity is used when the compiler encounters a problem serious enough to prevent normal pro cessing
of a form Instead of compiling the form the compiler compiles a call to error Errors are used mainly
for signalling syntax errors If an error happ ens during macroexpansion the compiler will handle it The
compiler also handles and attempts to pro ceed from read errors
Warning Warnings are used when the compiler can prove that something bad will happ en if a p ortion of the
program is executed but the compiler can pro ceed by compiling co de that signals an error at runtime if the problem has not b een xed
CHAPTER THE COMPILER
Violation of typ e declarations or
Function calls that have the wrong numb er of arguments or malformed keyword argument lists or
Referencing a variable declared ignore or unrecognized declaration sp eci ers
In the language of the Common Lisp standard these are situations where the compiler can determine
that a situation with unde ned consequences or that would cause an error to b e signalled would result at
runtime
Note Notes are used when there is something that seems a bit o dd but that might reasonably app ear in correct
programs
Note that the compiler does not fully conform to the prop osed X J compiler diagnostics cleanup Errors
warnings and notes mostly corresp ond to errors warnings and style warnings but many things that the cleanup
considers to b e style warnings are printed as warnings rather than notes Also warnings style warnings and
most errors aren t really signalled using the condition system
Errors During Macro expansion
The compiler handles errors that happ en during macroexpansion turning them into compiler errors If you want
to debug the error to debug a macro you can set break on signals to error For example this de nition
defun foo e l
do current l cdr current
atom current nil
when eq car current e return current
gives this error
In DEFUN FOO
DO CURRENT L NIL WHEN EQ E RETURN CURRENT
Error during macroexpansion
Error in function LISP DO DO BODY
DO step variable is not a symbol ATOM CURRENT
Read Errors
The compiler also handles errors while reading the source For example
Error Read error at
foo
Error in function LISP COMMA MACRO
Comma not inside a backquote
The at refers to the character p osition in the source le at which the error was signalled which is generally
immediately after the erroneous text The next line foo is the line in the source that contains the
error le p osition The indicates the error p osition within that line in this example immediately after the
o ending comma
When in Hemlock orany other EMACS like editor you can go to a character p osition with
M C u p osition C f
Note that if the source is from a Hemlock bu er then the p osition is relative to the start of the compiled region
or defun not the le or bu er start
After printing a read error message the compiler attempts to recover from the error bybacking up to the
start of the enclosing top level form and reading again with read suppress true If the compiler can recover
from the error then it substitutes a call to cerror for the unreadable form and pro ceeds to compile the rest of
the le normally
If there is a read error when the le p osition is at the end of the le i e an unexp ected EOF error then
the error message lo oks like this
CHAPTER THE COMPILER
Error Read error in form starting at
defun test
Error in function LISP FLUSH WHITESPACE
EOF while reading Stream for file usr me test lisp
In this case starting at indicates the character p osition at which the compiler started reading i e the
p osition b efore the start of the form that was missing the closing delimiter The line defun test is rst
line after the starting p osition that the compiler thinks mightcontain the unmatched op en delimiter
Error Message Parameterization
There is some control over the verb osity of error messages See also undefined warning limit page
efficiency note limit and efficiency note cost threshold page
enclosing source cutoff Variable
This variable sp eci es the numb er of enclosing actual source forms that are printed in full rather than in the
abbreviated pro cessing path format Increasing the value from its default of allows you to see more of the guts
of the macroexpanded source which is useful when debugging macros
error print length Variable
error print level Variable
These variables are the printlevel and print length used in printing error messages The default values are
and Ifnull the global values of print level and print length are used
extensions def source context name lambda list fform g Macro
This macro de nes how to extract an abbreviated source context from the name d form when it app ears in
the compiler input lamb da list is a defmacro style lamb da list used to parse the arguments The body should
return a list of subforms that can b e printed on ab out one line There are prede ned methods for defstruct
defmethod etc If no method is de ned then the rst two subforms are returned Note that this facility
implicitly determines the string name asso ciated with anonymous functions
Typ es in Python
A big di erence b etween Python and all other Common Lisp compilers is the approachtotyp e checking and
amount of knowledge ab out typ es
Python treats typ e declarations much di erently that other Lisp compilers do Python doesn t blindly
b elievetyp e declarations it considers them assertions ab out the program that should b e checked
Python also has a tremendously greater knowledge of the Common Lisp typ e system than other compilers
Supp ort is incomplete only for the not and and satisfies typ es
See also sections and
Compile Time Typ e Errors
If the compiler can prove at compile time that some p ortion of the program cannot b e executed without a typ e
error then it will giveawarning at compile time It is p ossible that the o ending co de would never actually b e
executed at run time due to some higher level consistency constraintunknown to the compiler so a typ e warning
do esn t always indicate an incorrect program For example consider this co de fragment
CHAPTER THE COMPILER
defun raz foo
let x case foo
this
that
the other
declare fixnum x
foo x
Compilation pro duces this warning
In DEFUN RAZ
CASE FOO THIS THAT THE OTHER
LET COND IF COND IF COND IF
COND
Warning This is not a FIXNUM
NIL
In this case the warning is telling you that if foo isn t anyof this that or the other then x will b e
initialized to nil which the fixnum declaration makes illegal The warning will go awayifecase is used instead
of case orif the other is changed to t
This sort of spurious typ e warning happ ens moderately often in the expansion of complex macros and in
inline functions In such cases there may b e dead co de that is impossible to correctly execute The compiler
can t always prove this co de is dead could never b e executed so it compiles the erroneous co de which will
always signal an error if it is executed and gives a warning
extensions required argument Function
This function can b e used as the default value for keyword arguments that must always b e supplied Since it
is known by the compiler to never return it will avoid any compile time typ e warnings that would result from
a default value inconsistent with the declared typ e When this function is called it signals an error indicating
that a required keyword argumentwas not supplied This function is also useful for defstruct slot defaults
corresp onding to required arguments See section page
Although this function is a CMU extension it is relatively harmless to use it in otherwise p ortable co de since
you can easily de ne it yourself
defun required argument
error A required keyword argument was not supplied
Typ e warnings are inhibited when the extensions inhibit warnings optimization qualityis see section
page This can b e used in a local declaration to inhibit typ e warnings in a co de fragmentthathas
spurious warnings
Precise Typ e Checking
With the default compilation p olicy all typ e assertions are precisely checked Precise checking means that the
check is done as though typep had b een called with the exact typ e sp eci er that app eared in the declaration
Python uses p olicy to determine whether to trust typ e assertions see section page Typ e assertions
from declarations are indistinguishable from the typ e assertions on arguments to built in functions In Python
adding typ e declarations makes co de safer
If a variable is declared to b e integer then its value must always always b e an integer b etween
and Ifmultiple typ e declarations apply to a single variable then all the declarations must b e correct it is
as though all the typ es were intersected pro ducing a single and typ e sp eci er
Argumenttyp e declarations are automatically enforced If you declare the typ e of a function argument a
typ e check will b e done when that function is called In a function call the called function do es the argument
There are a few circumstances where a typ e declaration is discarded rather than b eing used as typ e assertion This do esn t
a ect safetymuch since such discarded declarations are also not b elieved to b e true by the compiler
CHAPTER THE COMPILER
typ e checking which means that a more restrictive typ e assertion in the calling function e g from the may
b e lost
The typ es of structure slots are also checked The value of a structure slot must always b e of the typ e indicated
in any type slot option Because of precise typ e checking the arguments to slot accessors are checked to b e
the correct typ e of structure
In traditional Common Lisp compilers not all typ e assertions are checked and typ e checks are not precise
Traditional compilers blindly trust explicit typ e declarations but maycheck the argumenttyp e assertions for
built in functions Typ e checking is not precise since the argumenttyp e checks will b e for the most general typ e
legal for that argument In many systems typ e declarations suppress what little typ e checking is b eing done
so adding typ e declarations makes co de unsafe This is a problem since it discourages writing typ e declarations
during initial co ding In addition to b eing more error prone adding typ e declarations during tuning also loses
all the b ene ts of debugging with checked typ e assertions
To gain maximum b ene t from Python s typ e checking you should always declare the typ es of function
arguments and structure slots as precisely as p ossible This often involves the use of or member and other
list style typ e sp eci ers Paradoxically even though adding typ e declarations intro duces typ e checks it usually
reduces the overall amountoftyp e chec king This is esp ecially true for structure slot typ e declarations
Python uses the safety optimization quality rather than presence or absence of declarations to cho ose one
of three levels of run time typ e error checking see section page See section page for more
information ab out typ es in Python
Weakened Typ e Checking
When the value for the speed optimization quality is greater than safety andsafety is not then typ e
checking is weakened to reduce the sp eed and space p enalty In structure intensive co de this can double the
sp eed yet still catch most typ e errors Weakened typ e checks provide a level of safety similar to that of safe
co de in other Common Lisp compilers
Atyp e check is weakened bychanging the check to b e for some convenient sup ertyp e of the asserted typ e For
example integer is changed to fixnum simple vector to simple vector and structure typ es
are changed to structure A complex check like
or node hunk member foo bar baz
will b e omitted entirely i e the check is weakened to If a precise check can b e done for no extra cost then
no weakening is done
Although weakened typ e checking is similar to typ e checking done by other compilers it is sometimes safer
ened checks are done in the same places is precise checks so all the preceding and sometimes less safe Weak
discussion ab out where checking is done still applies Weakened checking is sometimes somewhat unsafe b ecause
although the check is weakened the precise typ e is still input into typ e inference In some contexts this will
result in typ e inferences not justi ed bytheweakened check and hence deletion of some typ e checks that would
b e done by conventional compilers
For example if this co de was compiled with weakened checks
defstruct foo
a nil type simple string
defstruct bar
a nil type single float
defun myfun x
declare type bar x
bar a x
and myfun was passed a foo thennotyp e error would b e signalled and wewould try to multiply a simple
vector as though it were a oat with unpredictable results This is b ecause the check for bar was weakened
to structure yet when compiling the call to bar a the compiler thinks it knows it has a bar
The initial value need not b e of this typ e as long as the corresp onding argument to the constructorisalways supplied but this
will cause a compile timetyp e warning unless required argument is used
CHAPTER THE COMPILER
Note that normally even weakened typ e checks rep ort the precise typ e in error messages For example if
myfun s bar check is weakened to structure and the argumentisnil then the error will b e
Type error in MYFUN
NIL is not of type BAR
However there is some sp eed and space cost for signalling a precise error so the weakened typ e is rep orted if
the speed optimization qualityis or debug qualityislessthan
Type error in MYFUN
NIL is not of type STRUCTURE
See section page for further discussion of the optimize declaration
Getting Existing Programs to Run
Since Python do es much more comprehensivetyp e checking than other Lisp compilers Python will detect typ e
errors in many programs that have b een debugged using other compilers These errors are mostly incorrect
declarations although compile timetyp e errors can nd actual bugs if parts of the program have never b een
tested
Some incorrect declarations can only b e detected by run time typ e checking It is very important to initially
compile programs with full typ e checks and then test this version After the checking version has b een tested
then you can consider weakening or eliminating typ e checks This applies even to previously debugged
programs Python do es much more typ e inference than other Common Lisp compilers so b elieving an incorrect
declaration do es much more damage
The most common problem is with variables whose initial value do esn t match the typ e declaration Incorrect
initial values will always b e agged by a compile time typ e error and they are simple to x once located Consider
this co de fragment
prog foo
declare fixnum foo
setq foo
Here the variable foo is given an initial value of nil but is declared to b e a fixnum Even if it is never read
the initial value of a variable must match the declared typ e There are twoways to x this problem Change
the declaration
prog foo
declare type or fixnum null foo
setq foo
or change the initial value
prog foo
declare fixnum foo
setq foo
It is generally preferable to change to a legal initial value rather than to weaken the declaration but sometimes
it is simpler to weaken the declaration than to try to make an initial value of the appropriate typ e
Another declaration problem o ccasionally encountered is incorrect declarations on defmacro arguments This
probably usually happ ens when a function is converted into a macro Consider this macro
defmacro my x
declare fixnum x the fixnum x
CHAPTER THE COMPILER
Although legal and well de ned Common Lisp this meaning of this de nition is almost certainly not what the
writer intended For example this call is illegal
my
The call is illegal b ecause the argument to the macro is whichisalist not a fixnum Because of
macro semantics it is hardly ever useful to declare the typ es of macro arguments If you really want to assert
something ab out the typ e of the result of evaluating a macro argument then put a the in the expansion
defmacro my x
the fixnum the fixnum x
In this case it would b e stylistically preferable to change this macro back to a function and declare it inline
Macros have no e ciency advantage over inline functions when using Python See section page
Some more subtle problems are caused by incorrect declarations that can t b e detected at compile time
Consider this co de
do pos position a string start pos
null pos
declare fixnum pos
Although pos is almost always a fixnum itis nil at the end of the loop If this example is compiled with full
typ e checks the default then running it will signal a typ e error at the end of the lo op If compiled without
typ e checks the program will go into an in nite lo op or p erhaps position will complain b ecause nil isn t
a sensible start Why Because if y ou compile without typ e checks the compiler just quietly b elieves the typ e
declaration Since pos is always a fixnum it is never nil so null pos is never true and the lo op exit test is
optimized away Such errors are sometimes agged by unreachable co de notes see section page but
it is still importan t to initially compile any system with full typ e checks even if the system works ne when
compiled using other compilers
In this case the x is to weaken the typ e declaration to or fixnum null Note that there is usually
little p erformance p enaltyforweakening a declaration in this way Anynumeric op erations in the b ody can still
assume the variable is a fixnum since nil is not a legal numeric argument Another p ossible x would b e to
say
do pos position a string start pos
null pos
let pos pos
declare fixnum pos
This would b e preferable in some circumstances since it would allow a non standard represen tation to b e used
for the local pos variable in the lo op b o dy see section
In summary rememb er that all values that a variable ever has must b e of the declared typ e and that you
should test using safe co de initially
Compiler Policy
The p olicy is what tells the compiler how to compile a program This is logically and often textually distinct
from the program itself Broad control of p olicy is provided bytheoptimize declaration other declarations and
variables control more sp eci c asp ects of compilation
Actually this declaration is totally unnecessary in Python since it already knows position returns a non negative fixnum or
nil
CHAPTER THE COMPILER
The Optimize Declaration
The optimize declaration recognizes six di erent qualities The qualities are conceptually indep endent asp ects
of program p erformance In reality increasing one quality tends to haveadverse e ects on other qualities The
compiler compares the relativevalues of qualities when it needs to make a trade o i e if speed is greater than
safety then improve sp eed at the cost of safety
The default for all qualities except debug is Whenever qualities are equal ties are broken according to
a broad idea of what a go o d default environment is supp osed to b e Generally this downplays speed compile
speed and space in favor of safety and debug Novice and casual users should stick to the default p olicy
Advanced users often wanttoimprove sp eed and memory usage at the cost of safety and debuggability
If the value for a qualityis or thenitmayhave a sp ecial interpretation A value of means totally
unimportant and a means ultimately important These extreme optimization values enable heroic com
pilation strategies that are not always desirable and sometimes self defeating Sp ecifying more than one quality
as is not desirable since it do esn t tell the compiler which quality is most important
These are the optimization qualities
speed How fast the program should is run speed enables some optimizations that hurt debuggability
compilation speed How fast the compiler should run Note that increasing this ab ove safety weakens typ e
checking
space Howmuch space the compiled co de should take up Inline expansion is mostly inhibited when space
is greater than speed Avalue of enables promiscuous inline expansion Wide use of a value is not
recommended as it maywaste so much space that run time is slowed See section page for a
discussion of inline expansion
debug How debuggable the program should b e The quality is treated di erently from the other qualities each
value indicates a particular level of debugger information it is not compared with the other qualities See
section page for more details
safety Howmuch error checking should b e done If speed space or compilation speed is more important
than safety thentyp e checking is weakened see section page If safety if then no run time
error checking is done In addition to suppressing typ e checks also suppresses argumentcountchecking
unb ound symb ol checking and arrayboundschecks
extensions inhibit warnin gs This is a CMU extension that determines how little or howmuch diagnostic
output should b e printed during compilation This quality is compared to other qualities to determine
whether to printstyle notes and warnings concerning those qualities If speed is greater than inhibit
warnings then notes ab out howtoimprove sp eed will b e printed etc The default value is so raising
the value for any standard qualityabove its default enables notes for that quality Ifinhibit warnings
is then all notes and most non serious warnings are inhibited This is useful with declare to suppress
warnings ab out unavoidable problems
The Optimize Interface Declaration
The extensions optimize interface declaration is identical in syntax to the optimize declaration but it
sp eci es the p olicy used during compilation of co de the compiler automatically generates to checkthenumber
and typ e of arguments supplied to a function It is useful to sp ecify this p olicy separately since even thoroughly
debugged functions are vulnerable to b eing passed the wrong arguments The optimize interface declaration
can sp ecify that arguments should b e checked even when the general optimize p olicy is unsafe
Note that this argumentchecking is the checking of user supplied arguments to any functions de ned within
the scop e of the declaration not the checking of arguments to Common Lisp primitives that app ear in those
de nitions
The idea b ehind this declaration is that it allows the de nition of functions that app ear fully safe to other
callers but that do no internal error checking Of course it is p ossible that arguments maybeinvalid in wa ys
other than having incorrect typ e Functions compiled unsafely must still protect themselves against things like
user supplied array indices that are out of b ounds and improper lists See also the context declarations
option to with compilation unit page
CHAPTER THE COMPILER
Op en Co ding and Inline Expansion
Since Common Lisp forbids the rede nition of standard functions the compiler can have sp ecial knowledge of
these standard functions emb edded in it This sp ecial knowledge is used in various ways op en co ding inline
expansion source transformation but the implications to the user are basically the same
Attempts to rede ne standard functions may b e frustrated since the function maynever b e called Al
though it is technically illegal to rede ne standard functions users sometimes want to implicitly rede ne
these functions when they are debugging using the trace macro Sp ecial casing of standard functions can
b e inhibited using the notinline declaration
The compiler can havemultiple alternate implementations of standard functions that implement di erent
trade o s of sp eed space and safety This selection is based on the compiler p olicy see section page
When a function call is op en co ded inline co de whose e ect is equivalent to the function call is substituted
for that function call When a function call is closed co ded it is usually left as is although it might b e turned
into a call to a di erent function with di erent arguments As an example if nthcdr were to b e op en co ded
then
nthcdr foobar
might turn into
cdr cdr cdr cdr foobar
or even
do i i
list foobar cdr foobar
i list
If nth is closed co ded then
nth x l
might stay the same or turn into something like
car nthcdr x l
In general op en co ding sacri ces space for sp eed but some functions suchascar are so simple that they
are always op en co ded Even when not op en co ded a call to a standard function may b e transformed into a
di erent function call as in the last example or compiled as static call Static function call uses a more e cient
calling convention that forbids rede nition
See the prop osed X J lisp symbol rede nition cleanup
Chapter
Advanced Compiler Use and E ciency
Hints
By Rob ert MacLachlan
Advanced Compiler Intro duction
In CMU Common Lisp as is any language on any computer the path to e cient co de starts with goo d algo
rithms and sensible programming techniques but to avoid ine ciency pitfalls you need to know some of this
implementation s quirks and features This chapter is mostly a fairly long and detailed overview of what opti
mizations Python do es Although there are the usual negative suggestions of ine cient features to avoid the
main emphasis is on describing the things that programmers can count on b eing e cient
The optimizations describ ed here can have the e ect of sp eeding up existing programs written in conventional
styles but the p otential for new programming styles that are clearer and less error prone is at least as signi cant
For this reason several sections end with a discussion of the implications of these optimizations for programming
style
Typ es
Python s supp ort for typ es is unusual in three ma jor ways
Precise typ e checking encourages the sp eci c use of typ e declarations as a form of run time consistency
checking This sp eeds developmentby localizing typ e errors and giving more meaningful error messages
See section page Python pro duces completely safe co de optimized typ e checking maintains
reasonable e ciency on conventional hardware see section page
Comprehensive supp ort for the Common Lisp typ e system makes complex typ e sp eci ers useful Using typ e
sp eci ers such as or and member has b oth e ciency and robustness advantages See section page
Typ e inference eliminates the need for some declarations and also aids compile time detection of typ e
errors Given detailed typ e declarations typ e inference can often eliminate typ e checks and enable more
e cient ob ject represen tations and co de sequences Checking all typ es results in fewer typ e checks See
sections and
Optimization
The main barrier to e cient Lisp programs is not that there is no e cientway to co de the program in Lisp but
that it is di cult to arrive at that e cient co ding Common Lisp is a highly complex language and usually has
many semantically equivalent reasonable ways to co de a given problem It is desirable to make all of these
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
equivalent solutions have comparable e ciency so that programmers don t havetowaste time discovering the
most e cient solution
Source level optimization increases the numb er of e cientways to solve a problem This e ect is much
larger than the increase in the e ciency of the b est solution Source level optimization transforms the orig
inal program into a more e cient but equivalent program Although the optimizer isn t doing anything the
programmer couldn t have done this high level optimization is important b ecause
The programmer can co de simply and directly rather than obfuscating co de to please the compiler
When presented with a choice of similar co ding alternatives the programmer can chose whichever happ ens
to b e most convenient instead of worrying ab out which is most e cient
Source level optimization eliminates the need for macros to optimize their expansion and also increases the
e ectiveness of inline expansion See sections and
E cient supp ort for a safer programming style is the biggest advantage of source level optimization Existing
tuned programs typically won t b ene t much from source optimization since their source has already b een
optimized by hand However even tuned programs tend to run faster under Python b ecause
Lowlevel optimization and register allocation provides modest sp eedups in any program
Blo ck compilation and inline expansion can reduce function call overhead but may require some program
restructuring See sections and
E ciency notes will p oint out importanttyp e declarations that are often missed even in highly tuned
programs See section page
Existing programs can b e compiled safely without prohibitive speed penalty although they w ould b e faster
and safer with added declarations See section page
The context declaration mechanism allows b oth space and runtime of large systems to b e reduced without
sacri cing robustness by semi automaticallyvarying compilation p olicy without addition any optimize
declarations to the source See section page
Byte compilation can b e used to dramatically reduce the size of co de that is not sp eed critical See section
page
Function Call
The sort of symb olic programs generally written in Common Lisp often favor recursion over iteration or have
inner loops so complex that they involvemultiple function calls Such programs sp end a larger fraction of their
time doing function calls than is the norm in other languages for this reason Common Lisp implementations
strive to make the general or full function call as inexp ensive as p ossible Python go es b eyond this byproviding
two go od alternatives to full call
Lo cal call resolves function references at compile time allowing b etter calling sequences and optimization
across function calls See section page
Inline expansion totally eliminates call overhead and allows manycontext dep endent optimizations This
provides a safe and e cient implementation of op erations with function semantics eliminating the need
for error prone macro de nitions or manual case analysis Although most Common Lisp implementations
supp ort inline expansion it b ecomes a more p owerful to ol with Python s source level optimization See
sections and
Generally Python provides simple implementations for simple uses of function call rather than having only
a single calling convention These features allow a more natural programming style
Prop er tail recursion See section page
Relatively e cient closures
A funcall that is as e cient as normal named call
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Calls to local functions such as from labels are optimized
Control transfer is a direct jump
The closure environment is passed in registers rather than heap allocated
Keyword arguments and multiple values are implemented more e ciently
See section page
Representation of Ob jects
Sometimes traditional Common Lisp implementation techniques compare so p o orly to the techniques used in
other languages that Common Lisp can b ecome an impractical language choice Terrible ine ciencies app ear in
numb er crunc hing programs since Common Lisp numeric op erations often involvenumb er consing and generic
arithmetic Python supp orts e cient natural represen tations for numb ers and some other typ es and allows
these e cient represen tations to b e used in more contexts Python also provides goo d e ciency notes that warn
when a crucial declaration is missing
See section for more ab out ob ject represen tations and numeric typ es Also see section page
ab out e ciency notes
Writing E cientCode
Writing e cient co de that works is a complex and prolonged pro cess It is important not to get so involved in
the pursuit of e ciency that you lose sight of what the original problem demands Remember that
The program should b e correct it doesn t matter how quickly you get the wrong answer
Both the programmer and the user will make errors so the program mustberobust itmust detect
errors in a waythatallows easy correction
A small p ortion of the program will consume most of the resources with the bulk of the co de b eing
virtually irrelevant to e ciency considerations Even exp erienced programmers familiar with the problem
area cannot reliably predict where these hot sp ots will b e
The b est way to get e cient co de that is still worth using is to separate co ding from tuning During co ding
you should
Use a co ding style that aids correctness and robustness without b eing incompatible with e ciency
Cho ose appropriate data structures that allow e cient algorithms and ob ject represen tations see section
page Try to makeinterfaces abstract enough so that you can change to a di erent represen tation
if pro ling reveals a need
Whenever you make an assumption ab out a function argument or global data structure add consistency
assertions either with typ e declarations or explicit uses of assert ecase etc
During tuning you should
Identify the hot sp ots in the program through pro ling section
Identify ine cient constructs in the hot sp ot with e ciency notes more pro ling or manual insp ection of
the source See sections and
Add declarations and consider the application of optimizations See sections and
If all else fails consider algorithm or data structure changes If you did a go o d job co ding changes will b e
easy to intro duce
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
More Ab out Typ es in Python
This section go es into more detail describing what typ es and declarations are recognized by Python The area
where Python di ers most radically from previous Common Lisp compilers is in its supp ort for typ es
Precise typ e checking helps to nd bugs at run time
Compile timetyp e checking helps to nd bugs at compile time
Typ e inference minimizes the need for generic op erations and also increases the e ciency of run time typ e
checking and the e ectiveness of compile time typ e checking
Supp ort for detailed typ es provides a wealth of opp ortunity for op eration sp eci c typ e inference and opti
mization
More Typ es Meaningful
Common Lisp has a very p owerful typ e system but conventional Common Lisp implementations typically only
recognize the small set of typ es sp ecial in that implementation In these systems there is an unfortunate paradox
a declaration for a relatively general typ e like fixnum will b e recognized by the compiler but a highly sp eci c
declaration such as integer is totally ignored
This is obviously a problem since the user has to knowhow to sp ecify the typ e of an ob ject in the waythe
compiler wants it A very minimal but rarely satis ed criterion for typ e system supp ort is that it b e no worse
to make a sp eci c declaration than to make a general one Python go es b eyond this by exploiting a number of
advantages obtained from detailed typ e information
Using more restrictive typ es in declarations allows the compiler to do b etter typ e inference and more compile
time typ e checking Also when typ e declarations are considered to b e consistency assertions that should b e
veri ed conditional on p olicy then complex t yp es are useful for making more detailed assertions
Python understands the list style or member function array and number typ e sp eci ers Understanding
means that
If the typ e contains more information than is used in a particular context then the extra information is
simply ignored rather than derailing typ e inference
In many contexts the extra information from these typ e sp eci er is used to go od e ect In particular
typ e checking in Python is precise so these complex typ es can b e used in declarations to makeinteresting
assertions ab out functions and data structures see section page More sp eci c declarations also
aid typ e inference and reduce the cost for typ e checking
For related information see section page for numeric typ es and section for arraytyp es
Canonicalization
When given a typ e sp eci er Python will often rewrite it into a di erent butequivalent typ e This is the
mechanism that Python uses for detecting typ e equivalence For example in Python s canonical represen tation
these typ es are equivalent
or list member end or cons member nil end
This has two implications for the user
The standard symbol typ e sp eci ers for atom null fixnum etc are in no way magical The null typ e
is actually de ned to b e member nil list is or cons null and fixnum is signed byte
When the compiler prints out a typ e it may not look likethetyp e sp eci er that originally app eared in the
program This is generally not a problem but it must b e taken into consideration when reading compiler error messages
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Member Typ es
The member typ e sp eci er can b e used to represen t symbolic values analogous to the enumerated typ es of
Pascal For example the second value of find symbol has this typ e
member internal external inherited nil
Member typ es are very useful for expressing consistency constraints on data structures for example
defstruct ice cream
flavor vanilla type member vanilla chocolate strawberry
Member typ es are also useful in typ e inference as the number of memb ers can sometimes b e pared down to one
in which case the value is a known constant
Union Typ es
The or union typ e sp eci er is understo o d and is meaningfully applied in manycontexts The use of or allows
assertions to b e made ab out typ es in dynamically typ ed programs For example
defstruct box
next nil type or box null
top removed type or box top member removed
The typ e assertion on the top slot ensures that an error will b e signalled when there is an attempt to store an
illegal value suchas rmoved Although somewhat weak these union typ e assertions provide a useful input
into typ e inference allowing the cost of typ e checking to b e reduced For example this lo op is safely compiled
with no typ e checks
defun find box with top box
declare type or box null box
do current box box next current
null current
unless eq box top current removed
return current
Union typ es are also useful in typ e inference for represen ting typ es that are partially constrained For example
the result of this expression
if foo
logior x y
list x y
can b e expressed as or integer cons
The EmptyTyp e
The typ e nil is also called the emptytyp e since no ob ject is of typ e nil The union of no typ es or is also
empty Python s interpretation of an expression whose typ e is nil is that the expression never yields anyvalue
but rather fails to terminate or is thrown out of For example the typ e of a call to error orauseofreturn
is nil When the typ e of an expression is empty compile timetyp e warnings ab out its value are suppressed
presumably someb o dy else is signalling an error If a function is declared to havereturntyp e nil but do es in
fact return then in safe compilation p olicies a NIL Function returned error will b e signalled See also the
function required argument page
Function Typ es
function typ es are understo o d in the restrictive sense sp ecifying
The argumentsyntax that the function must b e called with This is information ab out what argument
counts are acceptable and whichkeyword arguments are recognized In Python warnings ab out argument
syntax are a consequence of function typ e checking
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
The typ es of the argumentvalues that the caller must pass If the compiler can prove that some argument
to a call is of a typ e disallowed by the called function s typ e then it will give a compile time typ e warning
In addition to b eing used for compile time typ e checking these typ e assertions are also used as output typ e
assertions in co de generation For example if foo is declared to haveafixnum argument then the in
foo x is compiled with knowledge that the result must b e a xnum
The typ es the values that will b e b ound to argumentvariables in the function s de nition Declaring a
function s typ e with ftype implicitly declares the typ es of the arguments in the de nition Python checks
for consistency b etween the de nition and the ftype declaration Because of precise typ e checking an
error will b e signalled when a function is called with an argumentofthewrongtyp e
The typ e of return value s that the caller can exp ect This information is a useful input to typ e inference
For example if a function is declared to return a fixnum then when a call to that function app ears in an
expression the expression will b e compiled with knowledge that the call will return a fixnum
The typ e of return value s that the de nition must return The result typ e in an ftype declaration is
treated like an implicit the wrapp ed around the b o dy of the de nition If the de nition returns a value of
the wrong typ e an error will b e signalled If the compiler can prove that the function returns the wrong
typ e then it will give a compile timewarning
This is consisten t with the new interpretation of function typ es and the ftype declaration in the prop osed
X J function typ e argument typ e semantics cleanup Note also that if you don t explicitly declare the typ e
of a function using a global ftype declaration then Python will compute a function typ e from the de nition
providing a degree of inter routine typ e inference see section page
The Values Declaration
CMU Common Lisp supp orts the values declaration as an extension to Common Lisp The syntax is
values typ e typ e typ en This declaration is semantically equivalenttoathe form wrapp ed around the
b o dy of the sp ecial form in whichthevalues declaration app ears The advantage of values over the is purely
syntactic it do esn t intro duce more indentation For example
defun foo x
declare values single float
ecase x
this
that
the other
is equivalentto
defun foo x
the single float
ecase x
this
that
the other
and
defun floor number optional divisor
declare values integer real
is equivalentto
defun floor number optional divisor
the values integer real
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
In addition to b eing recognized by lambda and hence by defun the values declaration is recognized byall
the other sp ecial forms with b o dies and declarations let let labels and flet Macros with declarations
usually splice the declarations into one of the ab ove forms so they will accept this declaration to o but the exact
e ect of a values declaration will dep end on the macro
If you declare the typ es of all arguments to a function and also declare the return value typ es with values
you have describ ed the typ e of the function Python will use this argument and result typ e information to derive
a function typ e that will then b e applied to calls of the function see section page This provides a
way to declare the typ es of functions that is much less syntactically awkward than using the ftype declaration
with a function typ e sp eci er
Although the values declaration is non standard it is relatively harmless to use it in otherwise p ortable co de
since anywarning in non CMU implementations can b e suppressed with the standard declaration pro clamation
Structure Typ es
Because of precise typ e checking structure typ es are much b etter supp orted by Python than by conventional
compilers
The structure argument to structure accessors is precisely checked if you call foo a on a bar an error
will b e signalled
The typ es of slot v alues are precisely checked if you pass the wrong typ e argument to a constructor or
a slot setter then an error will b e signalled
This error checking is tremendously useful for detecting bugs in programs that manipulate complex data struc
tures
An additional advantage of checking structure typ es and enforcing slot typ es is that the compiler can safely
b elieve slot typ e declarations Python e ectively moves the typ e checking from the slot access to the slot setter
or constructor call This is more e cient since caller of the setter or constructor often knows the typ e of the
value entirely eliminating the need to check the value s typ e Consider this example
defstruct coordinate
x nil type single float
y nil type single float
defun make it
make coordinate x y
defun use it it
declare type coordinate it
sqrt expt coordinate x it expt coordinate y it
make it and use it are compiled with no checking on the typ es of the oat slots yet use it can use single
float arithmetic with p erfect safety Note that make coordinate must still check the values of x and y unless
the call is block compiled or inline expanded see section page But even without this advantage it is
almost always more e cienttocheck slot values on structure initialization since slots are usually written once
and read many times
The Freeze Typ e Declaration
The extensions freeze type declaration is a CMU extension that enables more e cient compilation of user
de ned typ es by asserting that the de nition is not going to change This declaration may only b e used glob
ally with declaim or proclaim Currently freeze type only a ects structure typ e testing done by typep
typecase etc Here is an example
declaim freeze type foo bar
This asserts that the typ es foo and bar and their subtyp es are not going to change This allows more e cient
typ e testing since the compiler can op en co de a test for all p ossible subtyp es rather than having to examine
the typ e hierarchy at run time
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Typ e Restrictions
Avoid use of the and not and satisfies typ es in declarations since typ e inference has problems with them
When these typ es do app ear in a declaration they are still checked precisely butthetyp e information is of
limited use to the compiler and typ es are e ective as long as the intersection can b e canonicalized to a typ e
that do esn t use and For example
and fixnum unsigned byte
is ne since it is the same as
integer most positive xnum
but this typ e
and symbol not member end
will not b e fully understo o d bytyp e interference since the and can t b e removed by canonicalization
Using any of these typ e sp eci ers in a typ e test with typep or typecase is ne since as tests these typ es
can b e translated into the and macro the not function or a call to the satis es predicate
Typ e Style Recommendations
Python provides go od supp ort for some curren tly unconventional ways of using the Common Lisp typ e system
With Python it is desirable to make declarations as precise as p ossible but typ e inference also makes some
declarations unnecessary Here are some general guidelines for maximum robustness and e ciency
Declare the typ es of all function arguments and structure slots as precisely as p ossible while avoiding not
and and satisfies Put these declarations in during initial co ding so that typ e assertions can nd bugs
for you during debugging
Use the member typ e sp eci er where there are a small numb er of p ossible symbol values for example
member red blue green
Use the or typ e sp eci er in situations where the typ e is not certain but there are only a few p ossibilities
for example or list vector
Declare integer typ es with the tightest b ounds that you can such as integer
De ne deftype or defstruct typ es b efore they are used De nition after use is legal pro ducing no
unde ned typ e warnings but typ e tests and structure op erations will b e compiled much less e ciently
Use the extensions freeze type declaration to sp eed up typ e testing for structure typ es whichwon t
have new subtyp es added later See section page
In addition to declaring the arrayelementtyp e and simpleness also declare the dimensions if they are
xed for example
simple array single float
This b ounds information allows arrayindexingformulti dimensional arrays to b e compiled much more
e ciently and may also allowarray b ounds checking to b e done at compile time See section
page
Avoid use of the the declaration within expressions Not only do es it clutter the co de but it is also almost
worthless under safe p olicies If the need for an output typ e assertion is revealed by e ciency notes during
tuning then you can consider the but it is preferable to constrain the argumenttyp es more allowing the
compiler to prove the desired result typ e
Don t b other declaring the typ e of let or other non argumentvariables unless the typ e is non obvious If
you declare function return typ es and structure slot typ es then the typ e of a variable is often obvious b oth
to the programmer and to the compiler An important case where the typ e isn t obvious and a declaration
is appropriate is when the value for a variable is pulled out of untyp ed structure e g the result of car
or comes from some weakly typ ed function suchasread
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Declarations are sometimes necessary for integer loop variables since the compiler can t always prove that
the value is of a go o d integer typ e These declarations are b est added during tuning when an e ciency
note indicates the need
Typ e Inference
Typ e inference is the pro cess by which the compiler tries to gure out the typ es of expressions and variables
given an inevitable lack of complete typ e information Although Python do es muchmoretyp e inference than
most Common Lisp compilers rememb er that the more precise and comprehensivetyp e declarations are the
more typ e inference will b e able to do
Variable Typ e Inference
The typeofavariable is the union of the typ es of all the de nitions In the degenerate case of a let the typ e of
the variable is the typ e of the initial value This inferred typ e is intersected with any declared typ e and is then
propagated to all the variable s references The typ es of multiple value bind variables are similarly inferred
from the typ es of the individual values of the values form
If multiple typ e declarations apply to a single variable then all the declarations must b e correct it is as
though all the typ es were intersected pro ducing a single and typ e sp eci er In this example
defmacro my dotimes var count body body
do var var
var count
declare type integer var
body
my dotimes i
declare fixnum i
the two declarations for i are intersected so i is known to b e a non negative xnum
In practice this typ e inference is limited to lets and local functions since the compiler can t analyze all the
calls to a global function But typ e inference works w ell enough on lo cal variables so that it is often unnecessary
to declare the typ e of lo cal variables This is esp ecially likely when function result typ es and structure slot typ es
are declared The main areas where typ e inference breaks down are
When the initial value of a variable is a untyp ed expression such as car x and
When the typ e of one of the variable s de nitions is a function of the variable s current value as in setq
x x
Lo cal Function Typ e Inference
The typ es of arguments to lo cal functions are inferred in the same was as any other local variable the typ e is
the union of the argumenttyp es across all the calls to the function intersected with the declared typ e If there
are any assignments to the argumentvariables the typ e of the assigned value is unioned in as well
The result typ e of a local function is computed in a sp ecial way that takes tail recursion see section
page into consideration The result typ e is the union of all p ossible return values that aren t tail recursive
calls For example Python will infer that the result typ e of this function is integer
defun n res
declare integer n res
if zerop n
res
n n res
Although this is a rather obvious result it b ecomes somewhat less trivial in the presence of mutual tail recursion
of multiple functions Lo cal function result typ e inference interacts with the mechanisms for ensuring prop er tail
recursion mentioned in section
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Global Function Typ e Inference
As describ ed in section a global function typ e ftype declaration places implicit typ e assertions on the
call arguments and also guarantees the typ e of the return value So wherever a call to a declared function
app ears there is no doubt as to the typ es of the arguments and return value Furthermore Python will infer
a function typ e from the function s de nition if there is no ftype declaration Anytyp e declarations on the
argumentvariables are used as the argumenttyp es in the derived function typ e and the compiler s b est guess
for the result typ e of the function is used as the result typ e in the derived function typ e
This method of deriving function typ es from the de nition implicitly assumes that functions won t b e rede ned
at run time Consider this example
defun foo p x
let res and consp x eq car x foo
format t It is not foo res
defun frob it
if foo p it
setf cadr it yow
it
Presumably the programmer really meant to return res from foo p but he seems to have forgotten When
he tries to call do frob list foo nil frob will ame out when it tries to add to a cons Realizing his
error he xes foo p and recompiles it But when he retries his test case he is ba ed b ecause the error is
still there What happ ened in this example is that Python proved that the result of foo p is null and then
pro ceeded to optimize awaythe setf in frob
Fortunately in this example the error is detected at compile time due to notes ab out unreachable co de see
section page Still some users maynotwanttoworry ab out this sort of problem during incremental
development so there is a variable to control deriving function typ es
V extensions derive function types ariable
If true the default argument and result typ e information derived from compilation of defuns is used when
compiling calls to that function If false only information from ftype pro clamations will b e used
Op eration Sp eci c Typ e Inference
Many of the standard Common Lisp functions havespecialtyp e inference pro cedures that determine the result
typ e as a function of the argumenttyp es For example the result typ e of aref is the arrayelementtyp e Here
are some other examples of typ e inferences
logand x xFF unsigned byte
the integer x the integer y integer
ash the unsigned byte x unsigned byte
Dynamic Typ e Inference
Python uses ow analysis to infer typ es in dynamically typ ed programs For example
ecase x
list length x
Here the compiler knows the argumenttolength is a list b ecause the call to length is only done when x is a
list The most signi cant e ciency e ect of inference from assertions is usually in typ e check optimization
Dynamic typ e inference has two inputs explicit conditionals and implicit or explicit typ e assertions Flow
analysis propagates these constraints on variable typ e to any co de that can b e executed only after passing though
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
the constraint Explicit typ e constraints come from ifs where the test is either a lexical variable or a function
of lexical variables and constants where the function is either a typ e predicate a numeric comparison or eq
If there is an eq or eql test then the compiler will actually substitute one argument for the other in the
true branch For example
when eq x yow return x
b ecomes
when eq x yow return yow
This substitution is done when one argument is a constant or one argument has b etter typ e information than
the other This transformation reveals opp ortunities for constant folding or typ e sp eci c optimizations If the
test is against a constant then the compiler can prove that the variable is not that constantvalue in the false
branch or not member yow in the example ab ove This can eliminate redundant tests for example
if eq x nil
if x a b
is transformed to this
if eq x nil
a
Variables app earing as if tests are interpreted as not eq var nil tests The compiler also converts into
eql where p ossible It is di cult to do inference directly on since it do es implicit co ercions
When there is an explicit or test on integer variables the compiler makes inferences ab out the ranges
the variables can assume in the true and false branches This is mainly useful when it proves that the values
are small enough in magnitude to allow op en co ding of arithmetic op erations For example in many uses of
dotimes with a fixnum rep eat count the compiler proves that xnum arithmetic can b e used
Implicit t yp e assertions are quite common esp ecially if you declare function argumenttyp es Dynamic
inference from implicit typ e assertions sometimes helps to disambiguate programs to a useful degree but is most
noticeable when it detects a dynamic typ e error For example
defun foo x
car x x
results in this warning
In DEFUN FOO
CAR X X
X
Warning Result is a LIST not a NUMBER
Note that Common Lisp s dynamic typ e checking semantics make dynamic typ e inference useful even in
programs that aren t really dynamically typ ed for example
car x length x
Here x presumably always holds a list but in the absence of a declaration the compiler cannot assume x is a
list simply b ecause list sp eci c op erations are sometimes done on it The compiler must consider the program
to b e dynamically typ ed until it proves otherwise Dynamic typ e inference proves that the argumenttolength
is always a list b ecause the call to length is only done after the list sp eci c car op eration
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Typ e Check Optimization
Python backs up its supp ort for precise typ e checking by minimizing the cost of run time typ e checking This is
done b oth through typ e inference and though optimizations of typ e checking itself
Typ e inference often allows the compiler to prove thatavalue is of the correct typ e and thus no typ e check
is necessary For example
defstruct foo a b c
defstruct link
foo required argument type foo
next nil type or link null
foo a link foo x
Here there is no need to check that the result of link foo is a foo since it always is Even when some typ e
checks are necessary typ e inference can often reduce the number
defun test x
let a foo a x
b foo b x
c foo c x
In this example only one foo p x check is needed This applies to a lesser degree in list op erations suchas
if eql car x cdr x y
Here we only havetocheck that x is a list once
Since Python recognizes explicit typ e tests co de that explicitly protects itself against typ e errors has little
intro duced overhead due to implicit typ e checking For example this loop compiles with no implicit checks
checks for car and cdr
defun memq e l
do current l cdr current
atom current nil
when eq car current e return current
Python reduces the cost of checks that must b e done through an optimization called complementing A
complemented checkfortyp e is simply a chec kthatthevalue is not of the typ e not typ e This is only
interesting when something is known ab out the actual typ e in which case we can test for the complementof
and known typ e not typ e or the di erence b etween the known typ e and the assertion An example
link foo link next x
Here we change the typ e checkforlink foo from a test for foo to a test for
not and or foo null not foo
or more simply not null This is probably the most important use of complementing since the situation is
fairly common and a null test is much cheap er than a structure typ e test
Here is a more complicated example that illustrates the combination of complementing with dynamic typ e
inference
defun find a a x
declare type or link null x
do current x link next current
null current nil
let foo link foo current
when eq foo a foo a return foo
This lo op can b e compiled with no typ e checks The link test for link foo and link next is complemented to
not null and then deleted b ecause of the explicit null test As b efore no check is necessary for foo a since
the link foo is always a foo This sort of situation shows how precise typ e checking combined with precise
declarations can actually result in reduced typ e checking
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Source Optimization
This section describ es source level transformations that Python do es on programs in an attempt to make them
more e cient Although source level optimizations can make existing programs more e cient the biggest advan
tage of this sort of optimization is that it makes it easier to write e cient programs If a clean straightforward
implementation is can b e transformed into an e cient one then there is no need for tricky and dangerous hand
optimization
Let Optimization
The primary optimization of let variables is to delete them when they are unnecessary Whenever the value of
a let variable is a constant a constant variable or a constant lo cal or non notinline function the variable is
deleted and references to the variable are replaced with references to the constant expression This is useful
primarily in the expansion of macros or inline functions where argumentvalues are often constant in anygiven
call but are in general non constant expressions that must b e b ound to preserv e order of evaluation Let variable
optimization eliminates the need for macros to carefully avoid spurious bindings and also makes inline functions
just as e cient as macros
A particularly interesting class of constant is a local function Substituting for lexical variables that are
b ound to a function can substantially improve the e ciency of functional programming styles for example
let a lambda x zow x
funcall a
e ectively transforms to
zow
This transformation is done even when the function is a closure as in
let a let y zug
lambda x zow x y
funcall a
b ecoming
zow zug
A constantvariable is a lexical variable that is never assigned to always keeping its initial value Whenever
p ossible avoid setting lexical variables instead bind a new variable to the new value Except for loop variables
it is almost always p ossible to avoid setting lexical variables This form
let x f x
is more e cient than this form
setq x f x
Setting variables makes the program more di cult to understand b oth to the compiler and to the program
mer Python compiles assignments at least as e ciently as any other Common Lisp compiler but most let
optimizations are only done on constant variables
Constantvariables with only a single use are also optimized away even when the initial value is not constant
For example this expansion of incf
let g x
setq x G
b ecomes
The source transformation in this example do esn t represent the preservation of evaluation order implicit in the compiler s
internal representation Where necessary the back end will reintro duce temp oraries to preserve the semantics
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
setq x x
The typ e semantics of this transformation are more important than the eliminationofthevariable itself Consider
what happ ens when x is declared to b e a fixnum after the transformation the compiler can compile the addition
knowing that the result is a fixnum whereas b efore the transformation the addition would have to allow for
xnum over ow
Another variable optimization deletes anyvariable that is never read This causes the initial value and any
assigned values to b e unused allowing those expressions to b e deleted if they have no side e ects
Note that a let is actually a degenerate case of local call see section page and that let optimization
can b e done on calls that weren t created by a let Also lo cal call allows an applicative style of iteration that is
totally assignment free
ConstantFolding
Constant folding is an optimization that replaces a call of constant arguments with the constant result of that
call Constant folding is done on all standard functions for which it is legal Inline expansion allows folding of
any constant parts of the de nition and can b e done even on functions that have side e ects
It is convenient to rely on constant folding when programming as in this example
defconstant limit
defun foo
limit
Constant folding is also helpful when writing macros or inline functions since it usually eliminates the need to
write a macro that sp ecial cases constant arguments
Constant folding of a user de ned function is enabled bytheextensions constant function pro clamation
In this example
declaim ext constant function myfun
defun myexp x y
declare single float x y
exp log x y
myexp
The call to myexp is constant folded to
Unused Expression Elimination
If the value of any expression is not used and the expression has no side e ects then it is deleted As with
constant folding this optimization applies most often when cleaning up after inline expansion and other op
timizations Any function declared an extensions constant function is also sub ject to unused expression
elimination
Note that Python will eliminate parts of unused expressions known to b e side e ect free even if there are
other unknown parts For example
let a list foo bar
if t
zow
raz a
b ecomes
progn foo bar zow
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Control Optimization
The most important optimization of control is recognizing when an if test is known at compile time then
deleting the if the test expression and the unreachable branchoftheif This can b e considered a sp ecial case
of constant folding although the test do esn t have to b e truly constant as long as it is de nitely not nil Note
also that typ e inference propagates the result of an if test to the true and false branches see section
page
A related if optimization is this transformation
if if a b c x y
into
if a
if b x y
if c x y
The opp ortunity for this sort of optimization usually results from a conditional macro For example
if not a x y
is actually implemented as this
if if a nil t x y
which is transformed to this
if a
if nil x y
if t x y
which is then optimized to this
if a y x
Note that due to Python s internal represen tations the if if situation will b e recognized even if other forms
are wrapp ed around the inner if like
if let g
loop
return not g
xy
In Python all the Common Lisp macros really are macros written in terms of if block and tagbody so
user de ned control macros can b e just as e cient as the standard ones Python emits basic blocks using a
heuristic that minimizes the numb er of unconditional branches The co de in a tagbody will not b e emitted in
the order it app eared in the source so there is no p oint in arranging the co de to makecontrol drop through to
the target
Unreachable Co de Deletion
Python will delete co de whenever it can prov e that the co de can never b e executed Co de b ecomes unreachable
when
An if is optimized away or
There is an explicit unconditional control transfer such as go or return from or
The last reference to a local function is deleted or there never was any reference
Note that the co de for x and y isn t actually replicated
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
When co de that app eared in the original source is deleted the compiler prints a note to indicate a p ossible
problem or at least unnecessary co de For example
defun foo
if t
write line True
write line False
will result in this note
In DEFUN FOO
WRITE LINE False
Note Deleting unreachable code
It is importanttopay attention to unreachable co de notes since they often indicate a subtle typ e error For
example
defstruct foo a b
defun lose x
let a foo a x
b if x foo b x none
results in this note
In DEFUN LOSE
IF X FOO B X NONE
NONE
Note Deleting unreachable code
The none is unreachable b ecause typ e inference knows that the argumentto foo a mustbeafoo andthus
can t b e nil Presumably the programmer forgot that x could b e nil when he wrote the binding for a
Here is an example with an incorrect declaration
defun count a string
do pos position a string start pos
count count
null pos count
declare fixnum pos
This time our note is
In DEFUN COUNT A
DO POS COUNT
NULL POS COUNT
DECLARE FIXNUM POS
BLOCK LET TAGBODY RETURN FROM PROGN
COUNT
Note Deleting unreachable code
The problem here is that pos can never b e null since it is declared a fixnum
It takes some exp erience with unreachable co de notes to b e able to tell what they are trying to say In
non obvious cases the b est thing to do is to call the function in a way that should cause the unreachable co de
to b e executed Either you will get a typ e error or you will nd that there truly is no wayforthecodetobe
executed Not all unreachable co de results in a note
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
A note is only given when the unreachable co de textually app ears in the original source This prevents
spurious notes due to the optimization of macros and inline functions but sometimes also forego es a note
that would have b een useful
Since accurate source information is not available for non list forms there is an element of heuristic in
determining whether or not to give a note about an atom Spurious notes maybegiven when a macro
or inline function de nes a variable that is also present in the calling function Notes ab out nil and t
are never given since it is to o easy to confuse these constants in expanded co de with ones in the original
source
Notes are only given ab out co de unreachable due to control ow There is no note when an expression is
deleted b ecause its value is unused since this is a common consequence of other optimizations
Somewhat spurious unreachable co de notes can also result when a macro inserts multiple copies of its argu
ments in di erent contexts for example
defmacro t and f var form
if var form form
defun foo x
t and f x if x True False
results in these notes
In DEFUN FOO
IF X True False
False
Note Deleting unreachable code
True
Note Deleting unreachable code
It seems like it has deleted b oth branches of the if but it has really deleted one branch in one copy and the
other branchintheothercopy Note that these messages are only spurious in not satisfying the intent of the
rule that notes are only given when the deleted co de app ears in the original source there is always some co de
b eing deleted when a unreachable co de note is printed
Multiple Values Optimization
Within a function Python implements uses of multiple values particularly e ciently Multiple values can b e kept
in arbitrary registers so using multiple values do esn t imply stack manipulation and represen tation conversion
For example this co de
let a if x foo x u
b if x bar x v
is actually more e cien t written this way
multiple value bind
a b
if x
values foo x bar x
values u v
Also see section page for information on how lo cal call provides e cient supp ort for multiple function
return values
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Source to Source Transformation
The compiler implements a numb er of op eration sp eci c optimizations as source to source transformations You
will often see unfamiliar co de in error messages for example
defun my zerop zerop x
gives this warning
In DEFUN MY ZEROP
ZEROP X
X
Warning Undefined variable X
The original zerop has b een transformed into a call to This transformation is indicated with the same
used to mark macro and function inline expansion Although it can b e confusing display of the transformed
source is important since warnings are given with resp ect to the transformed source This a more obscure
example
defun foo x logand x
gives this e ciency note
In DEFUN FOO
LOGAND X
LOGAND C Y C X
Note Forced to do static function Two arg and cost
Unable to do inline fixnum arithmetic cost because
The first argument is a INTEGER not a FIXNUM
etc
Here the compiler commuted the call to logand intro ducing temp oraries The note complains that the rst
argumentisnotafixnum when in the original call it was the second argument Tomake things more confusing
the compiler intro duced temp oraries called c x and c y that are b ound to y and resp ectively
You will also notice source to source optimizations when e ciency notes are enabled see section
page When the compiler is unable to do a transformation that might b e p ossible if there was more in
formation then an e ciency note is printed For example my zerop ab ove will also give this e ciency note
In DEFUN FOO
ZEROP X
X
Note Unable to optimize because
Operands might not be the same type so can t open code
Style Recommendations
Source level optimization makes p ossible a clearer and more relaxed programming style
Don t use macros purely to avoid function call If you want an inline function write it as a function and
declare it inline It s clearer less error prone and works just as well
Don t write macros that try to optimize their expansion in trivial ways such as avoiding binding variables
for simple expressions The compiler do es these optimizations to o and is less likely to make a mistake
Make use of local functions i e labels or flet and tail recursion in places where it is clearer Lo cal
function call is faster than full call
Avoid setting local variables when p ossible Binding a new let variable is at least as e cient as setting an
existing variable and is easier to understand b oth for the compiler and the programmer
Instead of writing similar co de over and over again so that it can b e hand customized for each use de ne
a macro or inline function and let the compiler do the work
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Tail Recursion
A call is tail recursive if nothing has to b e done after the the call returns i e when the call returns the
returned value is immediately returned from the calling function In this example the recursive call to myfun is
tail recursive
defun myfun x
if oddp random x
isqrt x
myfun x
Tail recursion is interesting b ecause it is form of recursion that can b e implemented much more e ciently
than general recursion In general a recursive call requires the compiler to allocate storage on the stackat
run time for every call that has not yet returned This memory consumption makes recursion unacceptably
ine cient for represen ting rep etitive algorithms having large or unb ounded size Tail recursion is the sp ecial case
of recursion that is semantically equivalent to the iteration constructs normally used to represen t rep etition in
programs Because tail recursion is equivalent to iteration tail recursive programs can b e compiled as e ciently
as iterative programs
So whywould you want to write a program recursively when you can write it using a lo op Well the main
answer is that recursion is a more general mechanism so it can express some solutions simply that are awkward
to write as a loop Some programmers also feel that recursion is a stylistically preferable way to write loops
b ecause it avoids assigning variables For example instead of writing
defun fun x
something that uses x
defun fun y
something that uses y
do x something fun fun x
nil
You can write
defun fun x
fun something that uses x
defun fun y
fun something that uses y
fun something
The tail recursive de nition is actually more e cient in addition to b eing arguably clearer As the number of
functions and the complexity of their call graph increases the simplicity of using recursion b ecomes compelling
Consider the advantages of writing a large nite state mac hine with separate tail recursive functions instead of
using a single huge prog
It helps to understand how to use tail recursion if you think of a tail recursivecallasapsetq that assigns
the argumentvalues to the called function s variables followed bya go to the start of the called function This
makes clear an inherent e ciency advantage of tail recursive call in addition to not having to allocate a stack
frame there is no need to prepare for the call to return e g by computing a return PC
Is there any disadvantage to tail recursion Other than an increase in e ciency the only wayyou can tell
that a call has b een compiled tail recursively is if you use the debugger Since a tail recursive call has no stack
frame there is no way the debugger can print out the stack frame represen ting the call The e ect is that
backtrace will not show some calls that would have b een displayed in a non tail recursive implementation In
practice this is not as bad as it sounds in fact it isn t really clearly worse just di erent See section
page for information ab out the debugger implications of tail recursion
In order to ensure that tail recursion is preserv ed in arbitrarily complex calling patterns across separately
compiled functions the compiler must compile any call in a tail recursive p osition as a tail recursive call This is
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
done regardless of whether the program actually exhibits any sort of recursive calling pattern In this example
the call to fun will always b e compiled as a tail recursivecall
defun fun x
fun x
So tail recursion do esn t necessarily haveanything to do with recursion as it is normally thought of See section
page for more discussion of using tail recursion to implement loops
Tail Recursion Exceptions
Although Python is claimed to b e prop erly tail recursive some might dispute this since there are situations
where tail recursion is inhibited
When the call is enclosed by a sp ecial binding or
When the call is enclosed bya catch or unwind protect or
When the call is enclosed bya block or tagbody and the blo cknameorgo tag has b een closed over
These dynamic extent binding forms inhibit tail recursion b ecause they allocate stack space to represen t the
binding Shallow binding implementations of dynamic scoping also require cleanup co de to b e evaluated when
the scop e is exited
Lo cal Call
Python supp orts two kinds of function call full call and lo cal call Full call is the standard calling convention
its late binding and generalitymake Common Lisp what it is but create unavoidable overheads When the
compiler can compile the calling function and the called function simultaneously it can use local call to avoid
some of the overhead of full call Lo cal call is really a collection of compilation strategies If some asp ect of call
overhead is not needed in a particular lo cal call then it can b e omitted In some cases local call can b e totally
free Lo cal call provides two main advantages to the user
Lo cal call makes the use of the lexical function binding forms flet and labels much more e cient A
localcallisalways faster than a full call and in many cases is much faster
Lo cal call is a natural approachtoblock compilation a compilation technique that resolves function
references at compile time Block compilation sp eeds function call but increases compilation times and
prevents function rede nition
Self RecursiveCalls
Lo cal call is used when a function de ned by defun calls itself For example
defun fact n
if zerop n
n fact n
This use of local call sp eeds recursion but can also complicate debugging since trace will only show the rst call
to fact and not the recursiv e calls This is b ecause the recursive calls directly jump to the start of the function
and don t indirect through the symbol function Self recursive lo cal call is inhibited when the block compile
argumenttocompile file is nil see section page
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Let Calls
Because local call avoids unnecessary call overheads the compiler internally uses lo cal call to implement some
macros and sp ecial forms that are not normally thoughtofasinvolving a function call For example this let
let a foo
b bar
is internally represen ted as though it was macroexpanded into
funcall lambda a b
foo
bar
This implementation is acceptable b ecause the simple cases of lo cal call equivalenttoalet result in go o d co de
This do esn t make let any more e cient but do es make local calls that are semantically the same as let much
more e cient than full calls For example these de nitions are all the same as far as the compiler is concerned
defun foo
some other stuff
let a something
some stuff
defun foo
flet localfun a
some stuff
some other stuff
localfun something
defun foo
let funvar lambda a
some stuff
some other stuff
funcall funvar something
Although lo cal call is most e cient when the function is called only once a call do esn t havetobeequivalent
to a let to b e more e cient than full call All lo cal calls avoid the overhead of argumentcountchecking and
keyword argument parsing and there are a numb er of other advantages that apply in many common situations
See section page for a discussion of the optimizations done on let calls
Closures
Lo cal call allows for much more e cient use of closures since the closure environment do esn t need to b e allocated
on the heap or even stored in memory at all In this example there is no p enalty for localfun referencing a
and b
defun foo a b
flet localfun x
a b x
if a b
localfun x
localfun x
In local call the compiler e ectively passes closed over values as extra arguments so there is no need for you to
optimize lo cal function use by explicitly passing in lexically visible values Closures may also b e sub ject to let
optimization see section page
Note indirect value cells are curren tly always allocated on the heap when a variable is b oth assigned to with
setq or setf and closed over regardless of whether the closure is a lo cal function or not This is another reason
to avoid setting variables when you don t have to
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Lo cal Tail Recursion
Tail recursive local calls are particularly e cient since they are in e ect an assignment plus a control transfer
Scheme programmers write lo ops with tail recursive lo cal calls instead of using the imperative go and setq This
has not caught on in the Common Lisp community since conventional Common Lisp compilers don t implement
lo cal call In Python users can cho ose to write lo ops suchas
defun n
labels loop n total
if zerop n
total
loop n n total
loop n
extensions iterate name f var initial value g fdeclarationg fformg Macro
This macro provides syntactic sugar for using labels to do iteration It creates a local function name with
the sp eci ed var s as its arguments and the declarationsandformsasitsbody This function is then called with
the initial values and the result of the call is return from the macro
Here is our factorial example rewritten using iterate
defun n
iterate loop
n n
total
if zerop n
total
loop n n total
The main advantage of using iterate over do is that iterate naturally allows stepping to b e done di erently
dep ending on conditionals in the b o dy of the loop iterate can also b e used to implement algorithms that
aren t really iterativeby simply doing a non tail call For example the standard recursive de nition of factorial
can b e written like this
iterate fact
n n
if zerop n
n fact n
Return Values
One of the more subtle costs of full call comes from allowing arbitrary numb ers of return values This overhead
can b e avoided in lo cal calls to functions that always return the same number of values For e ciency reasons
as well as stylistic ones you should write functions so that they always return the same number of values This
may require passing extra nil arguments to values in some cases but the result is more e cient not less so
When e ciency notes are enabled see section page and the compiler wants to use known values
return but can t prove that the function always returns the same number of values then it will print a note like
this
In DEFUN GRUE
DEFUN GRUE X DECLARE FIXNUM X COND NIL T
Note Return type not fixed values so can t use known return convention
VALUES OR INTEGER NULL REST T
In order to implement prop er tail recursion in the presence of known values return see section page
the compiler sometimes must provethatmultiple functions all return the same number of values When this
can t b e proven the compiler will print a note like this
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
In DEFUN BLUE
DEFUN BLUE X DECLARE FIXNUM X COND T
Note Return value count mismatch prevents known return from
these functions
BLUE
SNOO
See section page for the interaction b etween lo cal call and the represen tation of numeric typ es
Blo ck Compilation
Blo ck compilation allows calls to global functions de ned by defun to b e compiled as lo cal calls The function
call can b e in a di erent top level form than the defun oreven in a di erent le
In addition blo ck compilation allows the declaration of the entry p oints to the blo ck compiled p ortion An
entry p ointisany function that may b e called from outside of the block compilation If a function is not an
entry p oint then it can b e compiled more e ciently since all calls are known at compile time In particular if
a function is only called in one place then it will b e let converted This e ectively inline expands the function
but without the co de duplication that results from de ning the function normally and then declaring it inline
The main advantage of blo ck compilation is that it it preserv es e ciency in programs even when for read
ability and syntactic convenience they are broken up into many small functions There is absolutely no overhead
for calling a non entry p oint function that is de ned purely for modularity i e called only in one place
Blo ck compilation also allows the use of non descriptor arguments and return values in non trivial programs
see section page
Blo ck Compilation Semantics
The e ect of block compilation can b e envisioned as the compiler turning all the defuns in the block compilation
into a single labels form
declaim start block fun fun
defun fun
defun fun
fun
defun fun x
if x
fun
fun
declaim end block
b ecomes
labels fun
fun
fun
fun x
if x fun
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
fun
setf fdefinition fun fun
setf fdefinition fun fun
Calls b etween the block compiled functions are lo cal calls so changing the global de nition of fun will haveno
e ect on what fun do es fun will keep calling the old fun
The entry p oints fun and fun are still installed in the symbol function as the global de nitions of the
functions so a full call to an entry p ointworks just as b efore However fun is not an entry p oint so it is not
globally de ned In addition fun is only called in one place so it will b e let converted
Blo ck Compilation Declarations
The extensions start block and extensions end block declarations allow ne grained control of blockcom
pilation These declarations are only legal as a global declarations declaim or proclaim
The start block declaration has this syntax
start block fentry p oint name g
When pro cessed by the compiler this declaration marks the start of block compilation and sp eci es the entry
p oints to that blo ck If no entry p oints are sp eci ed then all functions are made into entry p oints If already
blo ck compiling then the compiler ends the curren t block and starts a new one
The end block declaration has no arguments
end block
The end block declaration ends a blo ck compilation unit without starting a new one This is useful mainly
when only a p ortion of a le is worth blo ck compiling
Compiler Arguments
The block compile and entry points arguments to extensions compile from stream and compile file
page provide overall control of blo ck compilation and allow blo ck compilation without requiring modi cation
of the program source
There are three p ossible values of the block compile argument
nil Do no compile time resolution of global function names not even for self recursive calls This inhibits any
start block declarations app earing in the le allowing all functions to b e incrementally rede ned
t Start compiling in block compilation mode This is mainly useful for block compiling small les that contain
no start block declarations See also the entry points argument
specified Start compiling in form at a time mode but exploit start block declarations and compile self
recursive calls as local calls Normally specified is the default for this argument see block compile
default page
The entry points argument can b e used in conjunction with block compile t to sp ecify the entry p oints
to a blo ck compiled le If not sp eci ed or nil all global functions will b e compiled as entry p oints When
block compile is not t this argument is ignored
block compile default Variable
This variable determines the default value for the block compile argumenttocompile file and compile
from stream The initial value of this v ariable is specified but nil is sometimes useful for totally inhibiting
blo ck compilation
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Practical Di culties
The main problem with blo ck compilation is that the compiler uses large amounts of memory when it is block
compiling This places an upp er limit on the amount of co de that can b e block compiled as a unit To makebest
use of blo ck compilation it is necessary to lo cate the parts of the program containing manyinternal calls and
then add the appropriate start block declarations When writing new co de it is a goo d idea to put in block
compilation declarations from the very b eginning since writing block declarations correctly requires accurate
knowledge of the program s function call structure If you want to initially develop co de with full incremental
rede nition you can compile with block compile default page set to nil
Note if a defun app ears in a non null lexical environment then calls to it cannot b e blo ck compiled
Unless les are very small it is probably impractical to block compile multiple les as a unit by sp ecifying a
list of les to compile file Semi inline expansion see section page provides another way to extend
blo ck compilation across le b oundaries
Context Declarations
CMU Common Lisp has a context sensitive declaration mechanism which is useful b ecause it allows exible
control of the compilation p olicy in large systems without requiring changes to the source les The primary
use of this feature is to allow the exp orted interfaces of a system to b e compiled more safely than the system
internals The context used is the name b eing de ned and the kind of de nition function macro etc
The context declarations option to with compilation unit page has dynamic scop e a ecting all
compilation done during the evaluation of the b o dy The argument to this option should evaluate to a list of
lists of the form
context sp ec fdeclare formg
In the indicated context the sp eci ed declare forms are inserted at the head of each de nition The declare
forms for all contexts that match are app ended together with earlier declarations getting precedence o ver later
ones A simple example
context declarations
external declare optimize safety
This will cause all functions that are named by external symb ols to b e compiled with safety
The full syntax of context sp ecs is
internal external True if the symbol is internal external in its home package
uninterned True if the symb ol has no home package
package fpackage name g True if the symb ol s home packageisinany of the named packages false if
uninterned
anonymous True if the function do esn t haveanyinteresting name not defmacro defun labels or flet
macro function macro is a global defmacro macro function is anything else
local global local is a labels or flet global is anything else
or fcontext sp ec g True when any supplied context sp ec is true
and fcontext sp ec g True only when all supplied context sp ec s are true
not fcontext sp ec g True when context sp ec is false
member fname g True when the de ned name is one of these names equal test
match fpatterng True when any of the patterns is a substring of the name The name is wrapp ed with s
so FOO matches names b eginning with FOO etc
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Context Declaration Example
Here is a more complex example of with compilation unit options
optimize optimize speed space inhibit warnings
debug safety
optimize interface optimize interface safety debug
context declarations
or external and match match SET
declare optimize interface safety
or and external macro
match PARSE
declare optimize safety
The optimize and extensions optimize interface declarations see section page set up the global
compilation p olicy The b o dies of functions are to b e compiled completely unsafe safety but argument
count and weakened argumenttyp e checking is to b e done when a function is called speed safety
The rst declaration sp eci es that all functions that are external or whose names contain b oth and SET
are to b e compiled compiled with completely safe interfaces safety The reason for this particular match
rule is that setf inverse functions in this system tend to have b oth strings in their name somewhere Wewant
setf inverses to b e safe b ecause they are implicitly called by users even though their name is not exp orted
The second declaration makes external macros or functions whose names start with PARSE have safe b o dies
as well as interfaces This is desirable b ecause a syntax error in a macro may cause a typ e error inside the b ody
The match rule is used b ecause macros often have auxiliary functions whose names b egin with this string
This particular example is used to build part of the standard CMU Common Lisp system Note however
that context declarations must b e set up according to the needs and co ding conventions of a particular system
di erent parts of CMU Common Lisp are compiled with di erent context declarations and y our system will
probably need its own declarations In particular any use of the match option dep ends on naming conventions
used in co ding
Inline Expansion
Python can expand almost any function inline including functions with keyword arguments The only restrictions
are that keyword argumentkeywords in the call must b e constant and that global function de nitions defun
must b e done in a null lexical environment not nested in a let or other binding form Lo cal functions flet
can b e inline expanded in anyenvironment Combined with Python s source level optimization inline expansion
can b e used for things that formerly required macros for e cient implementation In Python macros don t have
any e ciency advantage so they need only b e used where a macro s syntactic exibilityisrequired
Inline expansion is a compiler optimization technique that reduces the overhead of a function call by simply
not doing the call instead the compiler e ectively rewrites the program to app ear as though the de nition of the
called function was inserted at each call site In Common Lisp this is straightforwardly expressed by inserting
the lambda corresp onding to the original de nition
proclaim inline my
defun my x x
my someval lambda x x someval
When the function expanded inline is large the program after inline expansion may b e substantially larger
than the original program If the program b ecomes to o large inline expansion hurts sp eed rather than helping
it since hardware resources suchasphysical memory and cache will b e exhausted Inline expansion is called for
When pro ling has shown that a relatively simple function is called so often that a large amount of time
is b eing wasted in the calling of that function as opp osed to running in that function If a function is
complex it will take a long time to run relative the time sp ent in call so the sp eed advantage of inline
expansion is diminished at the same time the space cost of inline expansion is increased Of course if a
function is rarely called then the overhead of calling it is also insigni cant
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
With functions so simple that they take less space to inline expand than would b e taken to call the
function such as my ab ove It would require intimate knowledge of the compiler to b e certain when
inline expansion would reduce space but it is generally safe to inline expand functions whose de nition is
a single function call or a few calls to simple Common Lisp functions
In addition to this sp eed space tradeo from inline expansion s avoidance of the call inline expansion can
also reveal opp ortunities for optimization Python s extensive source level optimization can make use of context
information from the caller to tremendously simplify the co de resulting from the inline expansion of a function
The main form of caller context is local information ab out the actual argumentvalues what the argument
typ es are and whether the arguments are constant Knowledge ab out argumenttyp es can eliminate run time
typ e tests e g for generic arithmetic Constant arguments in a call provide opp ortunities for constant folding
optimization after inline expansion
A hidden way that constant arguments are often supplied to functions is through the defaulting of unsupplied
optional or keyword arguments There can b e a huge e ciency advantage to inline expanding functions that
have complex keyword based interfaces suchasthisde nitionofthemember function
proclaim inline member
defun member item list key
key identity
test eql testp
test not nil notp
do list list cdr list
null list nil
let car car list
if cond testp
funcall test item funcall key car
notp
not funcall test not item funcall key car
t
funcall test item funcall key car
return list
After inline expansion this call is simpli ed to the obvious co de
member a l key foo a test char
do list list cdr list
null list nil
let car car list
if char item foo a car
return list
In this example there could easily b e more than an order of magnitude improvement in sp eed In addition
to eliminating the original call to member inline expansion also allows the calls to char and foo a to b e
op en co ded We go from a lo op with three tests and two calls to a lo op with one test and no calls
See section page for more discussion of source level optimization
Inline Expansion Recording
Inline expansion requires that the source for the inline expanded function to b e available when calls to the
function are compiled The compiler do esn t rememb er the inline expansion for every function since that would
take an excessive ab out of space Instead the programmer must tell the compiler to record the inline expansion
b efore the de nition of the inline expanded function is compiled This is done by globally declaring the function
inline b efore the function is de ned by using the inline and extensions maybe inline see section
page declarations
In addition to recording the inline expansion of inline functions at the time the function is compiled compile
file also puts the inline expansion in the output le When the output le is loaded the inline expansion is
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
made available for subsequen t compilations there is no need to compile the de nition again to record the inline
expansion
If a function is declared inline but no expansion is recorded then the compiler will give an e ciency note
like
Note MYFUN is declared inline but has no expansion
When you get this note check that the inline declaration and the de nition app ear b efore the calls that are
to b e inline expanded This note will also b e given if the inline expansion for a defun could not b e recorded
b ecause the defun was in a non null lexical environment
Semi Inline Expansion
Python supp orts semi inline functions Semi inline expansion shares a single copy of a function across all the calls
in a component byconverting the inline expansion into a lo cal function see section page This takes up
less space when there are multiple calls but also provides less opp ortunity for context dep endent optimization
When there is only one call the result is identical to normal inline expansion Semi inline expansion is done
when the space optimization qualityis and the function has b een declared extensions maybe inline
This mechanism of inline expansion combined with lo cal call also allows recursive functions to b e inline
expanded If a recursiv e function is declared inline calls will actually b e compiled semi inline Although
recursiv e functions are often so complex that there is little advantage to semi inline expansion it can still b e
useful in the same sort of cases where normal inline expansion is esp ecially advantageous i e functions where
the calling context can help a lot
The Mayb e Inline Declaration
The extensions maybe inline declaration is a CMU Common Lisp extension It is similar to inline but
indicates that inline expansion may sometimes b e desirable rather than saying that inline expansion should
almost always b e done When used in a global declaration extensions maybe inline causes the expansion for
the named functions to b e recorded but the functions aren t actually inline expanded unless space is or the
function is eventually p erhaps lo cally declared inline
Use of the extensions maybe inline declaration followed bythe defun is preferable to the standard idiom
of
proclaim inline myfun
defun myfun
proclaim notinline myfun
Anycallstomyfun here are not inline expanded
defun somefun
declare inline myfun
Calls to myfun here are inline expanded
The problem with using notinline in this way is that in Common Lisp it do es more than just suppress inline
expansion it also forbids the compiler to use anyknowledge of myfun untilalater inline declaration overrides
the notinline This prevents compiler warnings ab out incorrect calls to the function and also prevents block
compilation
The extensions maybe inline declaration is used like this
proclaim extensions maybe inline myfun
defun myfun
Anycallstomyfun here are not inline expanded defun somefun
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
declare inline myfun
Calls to myfun here are inline expanded
defun someotherfun
declare optimize space
Calls to myfun here are expanded semi inline
In this example the use of extensions maybe inline causes the expansion to b e recorded when the defun for
somefun is compiled and do esn t waste space through doing inline expansion by default Unlike notinline
this declaration still allows the compiler to assume that the known de nition really is the one that will b e called
when giving compiler warnings and also allows the compiler to do semi inline expansion when the p olicy is
appropriate
When the goal is merely to control whether inline expansion is done by default it is preferable to use
extensions maybe inline rather than notinline Thenotinline declaration should b e reserved for those
sp ecial o ccasions when a function may b e rede ned at run time so the compiler mustbetoldthattheobvious
de nition of a function is not necessarily the one that will b e in e ect at the time of the call
Byte Co ded Compilation
Python supp orts byte compilation to reduce the size of Lisp programs by allowing functions to b e compiled more
compactly Byte compilation provides an extreme sp eed space tradeo byte co de is typically six times more
compact than native co de but runs fty times or more slower This is ab out ten times faster than the standard
interpreter which is itself considered fast in comparison to other Common Lisp interpreters
Large Lisp systems such as CMU Common Lisp itself often have large amounts of user interface co de
compile time macro co de debugging co de or rarely executed sp ecial case co de This co de is a go o d target for
byte compilation very little time is sp ent running in it but it can take up quite a bit of space Straight line
co de with many function calls is much more suitable than inner loops
When byte compiling the compiler compiles ab out twice as fast and can pro duce a hardware indep endent
ob ject le bytef typ e This le can b e loaded like a normal fasl le on any implementation of CMU CL
with the same byte ordering DEC PMAX has lbytef t yp e
The decision to byte compile or native compile can b e done on a p er le or p er co de ob ject basis The
byte compile argumenttocompile file page has these p ossible values
nil Don t byte compile anything in this le
t Byte compile everything in this le and pro duce a pro cessor indep endent bytef le
maybe Pro duce a normal fasl le but byte compile any functions for which the speed optimization qualityis
and the debug quality is not greater than
extensions byte compile top level Variable
If this variable is true the default and the byte compile argumenttocompile file is maybe thenbyte
compile top level co de co de outside of any defun defmethod etc
extensions byte compile default Variable
This variable determines the default value for the byte compile argumenttocompile file initially
maybe
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Ob ject Representation
A somewhat subtle asp ect of writing e cient Common Lisp programs is cho osing the correct data structures
so that the underlying ob jects can b e implemented e ciently This is partly b ecause of the need for multiple
represen tations for a given value see section page but is also due to the sheer numb er of ob ject
typ es that Common Lisp has built in The numb er of p ossible represen tations complicates the choice of a go od
represen tation b ecause semantically similar ob jects mayvary in their e ciency dep ending on how the program
op erates on them
Think Before You Use a List
Although Lisp s creator seemed to think that it was for LISt Pro cessing the astute observer mayhave noticed
that the chapter on list manipulation makes up less that three p ercen t of Common Lisp the Language I I The
language has grown since Lisp new data typ es sup ersede lists for many purp oses
Structure Representation
One of the b est ways of building complex data structures is to de ne appropriate structure typ es using defstruct
In Python access of structure slots is always at least as fast as list or vector access and is usually faster In
comparison to a list represen tation of a tuple structures also have a space advantage
Even if structures weren t more e cient than other represen tations structure use would still b e attractive
b ecause programs that use structures in appropriate ways are much more maintainable and robust than programs
written using only lists For example
rplaca caddr cadddr x caddr y
could have b een written using structures in this way
setf beverage flavor astronaut beverage x beverage flavor y
The second version is more maintainable b ecause it is easier to understand what it is doing It is more robust
b ecause structures accesses are typ e checked An astronaut will nev er b e confused with a beverage and the
result of beverage flavor is always a avor See sections and for more information ab out structure
typ es See section page for a numb er of examples that make clear the advantages of structure typing
Note that the structure de nition should b e compiled b efore any uses of its accessors or typ e predicate so
that these function calls can b e e ciently op en co ded
Arrays
Arrays are often the most e cient represen tation for collections of ob jects b ecause
Array represen tations are often the most compact An arrayisalways more compact than a list containing
the same numb er of elements
Arrays allow fast constant time access
Arrays are easily destructiv ely modi ed which can reduce consing
Array elementtyp es can b e sp ecialized which reduces b oth overall size and consing see section
page
Access of arrays that are not of typ e simple array is less e cient so declarations are appropriate when an
array is of a simple typ e like simple string or simple bit vector Arrays are almost always simple but the
compiler may not b e able to prove simpleness at every use The only way to get a non simple arrayistousethe
displaced to fill pointer or adjustable arguments to make array Ifyou don t use these hairy options
then arrays can always b e declared to b e simple
Because of the many sp ecialized arraytyp es and the p ossibility of non simple arrays array access is much
like generic arithmetic see section page In order for arra y accesses to b e e ciently compiled the
elementtyp e and simpleness of the arraymust b e known at compile time If there is inadequate information
the compiler is forced to call a generic array access routine You can detect ine cientarray accesses by enabling e ciency notes see section page
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Vectors
Vectors one dimensional arrays are particularly useful since in addition to their obvious array like applications
they are also well suited to represen ting sequences In comparison to a list represen tation vectors are faster
to access and takeupbetween two and sixty four times less space dep ending on the elementtyp e As with
arbitrary arrays the compiler needs to knowthatvectors are not complex so you should use simple string in
preference to string etc
The only advantage that lists haveover vectors for represen ting sequences is that it is easy to change the
length of a list add to it and remove items from it Likely signs of archaic slow lisp co de are nth and nthcdr
If you are using these functions you should probably b e using a vector
Bit Vectors
Another thing that lists have b een used for is set manipulation In applications where there is a known reasonably
small universe of items bit vectors can b e used to improve p erformance This is much less convenient than using
lists b ecause instead of symb ols each element in the universe must b e assigned a numeric index into the bit
vector Using a bit vector will nearly always b e faster and can b e tremendously faster if the numb er of elements
in the set is not small The logical op erations on simple bit vectors are e cient since they op erate on a word
at a time
Hashtables
Hashtables are an e cient and general mechanism for main taining asso ciations such as the asso ciation b etween
an ob ject and its name Although hashtables are usually the b est way to maintain asso ciations e ciency and
style considerations sometimes favor the use of an asso ciation list a list
assoc is fairly fast when the test argumentiseq or eql and there are only a few elements but the time
go es up in prop ortion with the numb er of elements In contrast the hash table lookup has a somewhat higher
overhead but the sp eed is largely una ected bythenumb er of entries in the table For an equal hash table or
alist hash tables haveaneven greater advantage since the test is more exp ensive Whatever you do b e sure to
use the most restrictive test function p ossible
The style argument observes that although hash tables and alists overlap in function they do not do all
things equally well
Alists are go o d for maintaining scop ed environments They were originally invented to implement scoping
in the Lisp interpreter and are still used for this in Python With an alist one can non destructively change
an asso ciation simply by consing a new element on the front This is something that cannot b e done with
hash tables
Hashtables are go o d for maintaining a global asso ciation The value asso ciated with an entry can easily
be changed with setf With an alist one has to go through contortions either rplacd ing the cons if the
entry exists or pushing a new one if it do esn t The side e ecting nature of hash table op erations is an
advantage here
Historically symb ol prop erty lists were often used for global name asso ciations Prop erty lists provide an
awkward and error prone combination of name asso ciation and record structure If you must use the prop erty
list please store all the related values in a single structure under a single prop erty rather than using many
dicum of typing and abstraction See section prop erties This makes access more e cient and also adds a mo
page for information on typ es in CMU Common Lisp
Numb ers
Numb ers are interesting b ecause numb ers are one of the few Common Lisp data typ es that have direct supp ort
in conventional hardware If a numb er can b e represen ted in the way that the hardware exp ects it then there is
a big e ciency advantage
Using hardware represen tations is problematical in Common Lisp due to dynamic typing where the typ e
of a value may b e unknown at compile time It is p ossible to compile co de for statically typ ed p ortions of
a Common Lisp program with e ciency comparable to that obtained in statically typ ed languages suchasC
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
but not all Common Lisp implementations succeed There are two main barriers to e cientnumerical co de in
Common Lisp
The compiler must prove that the numerical expression is in fact statically typ ed and
The compiler must b e able to somehow reconcile the con icting demands of the hardware mandated number
represen tation with the Common Lisp requirements of dynamic typing and garbage collecting dynamic
storage allocation
Because of its typ e inference see section page and e ciency notes see section page Python
is b etter than conventional Common Lisp compilers at ensuring that numerical expressions are statically typ ed
Python also go es somewhat farther than existing compilers in the area of allowing nativemachine number
represen tations in the presence of garbage collection
Descriptors
Common Lisp s dynamic typing requires that it b e p ossible to represen t anyvalue with a xed length ob ject
known as a descriptor This xed length requirement is implicit in features such as
Data typ es like simple vector that can contain anytyp e of ob ject and that can b e destructiv ely
modi ed to contain di erent ob jects of p ossibly di erent typ es
Functions that can b e called with anytyp e of argument and that can b e rede ned at run time
In order to save space a descriptor is invariably represen ted as a single word Ob jects that can b e directly
represen ted in the descriptor itself are said to b e immediate Descriptors for ob jects larger than one word are in
reality p ointers to the memory actually containing the ob ject
Representing ob jects using p ointers has two ma jor disadvantages
The memory p ointed to m ust b e allocated on the heap so it must eventually b e freed by the garbage
collector Excessiv e heap allocation of ob jects or consing is ine cient in several ways See section
page
Represen ting an ob ject in memory requires the compiler to emit additional instructions to read the actual
value in from memory and then to write the value back after op erating on it
The intro duction of garbage collection makes things even worse since the garbage collector must b e able
to determine whether a descriptor is an immediate ob ject or a p ointer This requires that a few bits in each
descriptor b e dedicated to the garbage collector The loss of a few bits do esn t seem likemuch but it has a ma jor
e ciency implication ob jects whose natural machine represen tation is a full word integers and single oats
cannot have an immediate represen tation So the compiler is forced to use an unnatural immediate represen tation
such as fixnum or a natural p ointer represen tation with the attendantconsingoverhead
Non Descriptor Representations
From the discussion ab ove we can see that the standard descriptor represen tation has many problems the
worst b eing numb er consing Common Lisp compilers try to avoid these descriptor e ciency problems byusing
non descriptor represen tations A compiler that uses non descriptor represen tations can compile this function
so that it do es no numb er consing
defun multby vec n
declare type simple array single float vec
single float n
dotimes i length vec
setf aref vec i
n aref vec i
If a descriptor represen tation were used each iteration of the loop might cons two oats and do three times as
many memory references
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
As its negative de nition suggests the range of p ossible non descriptor represen tations is large The p er
formance improvement from non descriptor represen tation dep ends up on b oth the number of typ es that have
non descriptor represen tations and the number of contexts in which the compiler is forced to use a descriptor
represen tation
Many Common Lisp compilers supp ort non descriptor represen tations for oat typ es such as single float
and double float section Python adds supp ort for full word integers see section page
characters see section page and system area p ointers unconstrained p ointers see section
page Many Common Lisp compilers supp ort non descriptor represen tations for variables section
and array elements section Python adds supp ort for non descriptor arguments and return values in
lo cal call see section page and structure slots see section page
Variables
In order to use a non descriptor represen tation for a variable or expression intermediate value the compiler
must b e able to prove that the value is always of a particular typ e having a non descriptor represen tation Typ e
inference see section page often needs some help from user supplied declarations The b est kind of typ e
declaration is a variable typ e declaration placed at the binding p oint
let x car l
declare single float x
Use of the orofvariable declarations not at the binding form is insu cienttoallow non descriptor represen tation
of the variable with these declarations it is not certain that all values of the variable are of the righttyp e
It is sometimes useful to intro duce a gratuitous binding that allows the compiler to change to a non descriptor
represen tation like
etypecase x
signed byte
let x x
declare type signed byte x
The declaration on the inner x is necessary here due to a phase ordering problem Although the compiler will
eventually prove that the outer x is a signed byte within that etypecase branch the inner x would have
b een optimized awayby that time Declaring the typ e makes let optimization more cautious
Note that storing a value into a global or special variable always forces a descriptor represen tation
Wherever p ossible you should op erate only on local variables binding any referenced globals to local variables
at the b eginning of the function and doing any global assignments at the end
E ciency notes signal use of ine cient represen tations so programmer s needn t continuously worry ab out
the details of represen tation selection see section page
Generic Arithmetic
In Common Lisp arithmetic op erations are generic The function can b e passed fixnums bignums ratios
and various kinds of floatsandcomplexes in anycombination In addition to the inherent complexityofbignum
and ratio op erations there is also a lot of overhead in just guring out which op eration to do and what contagion
and canonicalization rules apply The complexity of generic arithmetic is so great that it is inconceivable to op en
co de it Instead the compiler do es a function call to a generic arithmetic routine consuming many instructions
b efore the actual computation even starts
This is ridiculous since even Common Lisp programs do a lot of arithmetic and the hardware is capable of
doing op erations on small integers and oats with a single instruction To get acceptable e ciency the compiler
sp ecial cases uses of generic arithmetic that are directly implemented in the hardware In order to op en co de
arithmetic several constraints must b e met
As Steele notes in CLTL I I this is a generic conception of generic and is not to b e confused with the CLOS concept of a generic function
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
All the arguments must b e known to b e a goo d typ e of numb er
The result must b e known to be a good typ e of number
Anyintermediate values such as the result of a b in the call a b c must b e knowntobeagood
typ e of numb er
All the ab ovenumb ers with go o d typ es must b e of the same goo d typ e Don t try to mix integers and
oats or di erent oat formats
The good typ es are signed byte unsigned byte single float and double float See
sections and for more discussion of goo d numeric typ es
float is not a go o d typ e since it might mean either single float or double float integer isnotagood
typ e since it mightmeanbignum rational is not a good typ e since it might mean ratio Notehowever that
these typ es are still useful in declarations since typ e inference maybeabletostrengthenaweak declaration into
a go o d one when it would b e at a loss if there was no declaration at all see section page The integer
and unsigned byte or non negativeinteger typ es are esp ecially useful in this regard since they can often b e
strengthened to a go od integer typ e
Arithmetic with complex numb ers is ine cient in comparison to oat and integer arithmetic Complex
numb ers are always represen ted with a p ointer descriptor causing consing overhead and complex arithmetic is
always closed co ded using the general generic arithmetic functions But arithmetic with complex typ es such as
complex float
complex fixnum
is still faster than bignum or ratio arithmetic since the implementation is much simpler
Note don t use to divide integers unless you wanttheoverhead of rational arithmetic Use truncate even
when you know that the arguments divide evenly
You don t need to rememb er all the rules for how to get op en co ded arithmetic since e ciency notes will tell
you when and where there is a problem see section page
Fixnums
A xnum is a FIXed precision NUMb er In modern Common Lisp implementations xnums can b e represen ted
with an immediate descriptor so op erating on xnums requires no consing or memory references Clever choice
of represen tations also allows some arithmetic op erations to b e done on xnums using hardware supp orted
word integer instructions somewhat reducing the sp eed p enalty for using an unnatural integer represen tation
It is useful to distinguish the fixnum typ e from the xnum represen tation of integers In Python there
is absolutely nothing magical ab out the fixnum typ e in comparison to other nite integer typ es fixnum is
equiv alent to is de ned with deftype to b e signed byte fixnum is simply the largest subset of integers
that can b e represen ted using an immediate xnum descriptor
Unlike in other Common Lisp compilers it is in no way desirable to use the fixnum typ e in declarations
in preference to more restrictive integer typ es suchasbit integer and unsigned byte Since
Python do es understand these integer typ es it is preferable to use the more restrictive typ e as it allows b etter
typ e inference see section page
The small e cient xnum is contrasted with bignum or BIG NUMb er This is another descriptor repre
sentation for integers but this time a p ointer represen tation that allows for arbitrarily large integers Bignum
op erations are less e cient than xnum op erations b oth b ecause of the consing and memory reference over
heads of a p ointer descriptor and also b ecause of the inherent complexity of extended precision arithmetic
While xnum op erations can often b e done with a single instruction bignum op erations are so complex that
they are always done using generic arithmetic
A crucial p oint is that the compiler will use generic arithmetic if it can t prove that all the arguments
intermediate values and results are xnums With b ounded integer typ es such as fixnum the result typ e proves
to b e esp ecially problematical since these typ es are not closed under common arithmetic op erations such as
and For example the fixnum x do es not necessarily evaluate to a fixnum Bign ums were added
to Common Lisp to get around this problem but they really just transform the correctness problem if this
add over ows you will get the wrong answer to the e ciency problem if this add might over ow then your
program will run slowly b ecause of generic arithmetic
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
There is just no getting around the fact that the hardware only directly supp orts short integers To get the
most e cient op en co ding the compiler must b e able to prove that the result is a go o d integer typ e This is an
argumentinfavor of using more restrictive integer typ es the fixnum x maynotalways b e a fixnum
but the unsigned byte x always is Of course you can also assert the result typ e by putting in
lots of the declarations and then compiling with safety
Word Integers
Python is unique in its e cient implementation of arithmetic on full word integers through non descriptor rep
resen tations and op en co ding Arithmetic on anysubtyp e of these typ es
signed byte
unsigned byte
is reasonably e cient although subtyp es of fixnum remain somewhat more e cient
If a word integer must b e represen ted as a descriptor then the bignum represen tation is used with its
asso ciated consing overhead The supp ort for word integers in no way changes the language semantics it just
makes arithmetic on small bignums vastly more e cient It is ne to do arithmetic op erations with mixed fixnum
and word integer op erands just declare the most sp eci c integer typ e you can and let the compiler decide what
represen tation to use
In fact to most users the greatest advantage of word integer arithmetic is that it e ectively provides a few
guard bits on the xnum represen tation If there are missing assertions on intermediate values in a xnum
expression the intermediate results can usually b e pro ved to t in a word After the whole expression is
evaluated there will often b e a xnum assertion on the nal result allowing creation of a xnum result without
even checking for over ow
The remarks in section ab out xnum result typ e also apply to word integers you must b e careful to
give the compiler enough information to prove that the result is still a word integer This time though when
we blow out of word integers welandininto generic bignum arithmetic whichismuch worse than sleazing from
fixnumstoword integers Note that mixing unsigned byte arguments with arguments of any signed typ e
such as fixnum is a no no since the result might not b e unsigned
Floating Point E ciency
Arithmetic on ob jects of typ e single float and double float is e ciently implemented using non descriptor
represen tations and op en co ding As for integer arithmetic the arguments mustbeknown to b e of the same
oat typ e Unlikeforinteger arithmetic the results and intermediate values usually take care of themselves due
to the rules of oat contagion i e the single float x is always a single float
Although they are not sp ecially implemented short float and long float are also acceptable in decla
rations since they are synonyms for the single float and double float typ es resp ectively Itisharmless
to use list style oat typ e sp eci ers such as single float but Python currently makes little use of
b ounds on oat typ es
When a oat must b e represen ted as a descriptor a p ointer represen tation is used creating consing overhead
For this reason you should try to avoid situations such as full call and non sp ecialized data structures that
force a descriptor represen tation See sections and
See section page for information on the extensions to supp ort IEEE oating p oint
Sp ecialized Arrays
Common Lisp supp orts sp ecialized arrayelementtyp es through the element type argumenttomake array
When an array has a sp ecialized elementtyp e only elements of that typ e can b e stored in the array From this
restriction comes two ma jor e ciency advantages
A sp ecialized array can savespaceby packing multiple elements into a single word For example a base
char array can have elements p er word and a bit arraycanhave This space e cient represen tation
is p ossible b ecause it is not necessary to separately indicate the typ e of eachelement
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
The elements in a sp ecialized arraycanbegiven the same non descriptor represen tation as the one used in
registers and on the stack eliminating the need for represen tation conversions when reading and writing
array elements For ob jects with p ointer descriptor represen tations such as oats and word integers there
is also a substantial consing reduction b ecause it is not necessary to allocate a new ob ject every time an
array elementismodi ed
These are the sp ecialized elementtyp es curren tly supp orted
bit
unsigned byte
unsigned byte
unsigned byte
unsigned byte
unsigned byte
base character
single float
double float
Although a simple vector can hold anytyp e of ob ject t should still b e considered a sp ecialized arraytyp e
since arrays with elementtyp e t are sp ecialized to hold descriptors
When using non descriptor represen tations it is particularly importanttomake sure that array accesses
are op en co ded since in addition to the generic op eration overhead e ciency is lost when the array element
is converted to a descriptor so that it can b e passed to or from the generic access routine You can detect
ine cient array accesses by enabling e ciency notes see section page See section page
Sp ecialized Structure Slots
Structure slots declared by the type defstruct slot option to have certain known numeric typ es are also given
non descriptor represen tations These typ es and subtyp es of these typ es are supp orted
unsigned byte
single float
double float
The primary advantage of sp ecialized slot represen tations is a large reduction spurious memory allocation
and access overhead of programs that intensively use these typ es
Interactions With Lo cal Call
Lo cal call has manyadvantages see section page one relevant to our discussion here is that local call
extends the usefulness of non descriptor represen tations If the compiler knows from the argumenttyp e that
an argument has a non descriptor represen tation then the argument will b e passed in that represen tation The
easiest way to ensure that the argumenttyp e is known at compile time is to always declare the argumenttyp e
in the called function like
defun f x
declare single float x
x
The advantages of passing arguments and return values in a non descriptor represen tation are the same as for
non descriptor represen tations in general reduced consing and memory access see section page This
extends the applicative programming styles discussed in section to numeric co de Also if source les are kept
reasonably small blo ck compilation can b e used to reduce numb er consing to a minimum
Note that non descriptor return values can only b e used with the known return convention section If
the compiler can t prove that a function always returns the same number of values then it must use the unknown
values return convention which requires a descriptor represen tation Pay attention to the known return e ciency
notes to avoid numb er consing
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Representation of Characters
Python also uses a non descriptor represen tation for characters when convenient This improves the e ciency of
string manipulation but is otherwise pretty invisible characters have an immediate descriptor represen tation so
there is not a great p enalty for converting a character to a descriptor Nonetheless it may sometimes b e helpful
to declare character valued variables as base character
General E ciency Hints
This section is a summary of various implementation costs and ways to get around them These hints are
relatively unrelated to the use of the Python compiler and probably also apply to most other Common Lisp
implementations In each section there are references to related in depth discussion
Compile Your Co de
At this p oint the advantages of compiling co de relativetorunningitinterpreted probably need not b e emphasized
to o much but rememb er that in CMU Common Lisp compiled co de typically runs hundreds of times faster
than interpreted co de Also compiled fasl les load signi cantly faster than source les so it is worthwhile
compiling les which are loaded many times even if the sp eed of the functions in the le is unimportant
Even disregarding the e ciency advantages compiled co de is as go od or b etter than interpreted co de Com
piled co de can b e debugged at the source level see chapter and compiled co de do es more error checking For
these reasons the interpreter should b e regarded mainly as an interactive command interpreter rather than as
a programming language implementation
Do not b e concerned ab out the p erformance of your program until you see its sp eed compiled Some
techniques that make compiled co de run faster makeinterpreted co de run slow er
Avoid Unnecessary Consing
Consing is another name for allocation of storage as done bythe cons function hence its name cons is by
no means the only function which conses so do es make array and many other functions Arithmetic and
function call can also have hidden consing overheads Consing hurts p erformance in the following ways
Consing reduces memory access locality increasing paging activity
Consing takes time just likeanything else
Any space allocated eventually needs to b e reclaimed either by garbage collection or by starting a new
lisp pro cess
Consing is not undiluted evil since programs do things other than consing and appropriate consing can
sp eed up the real work It would certainly save time to allocate a vector of intermediate results that are reused
hundreds of times Also if it is necessary to copy a large data structure many times it may b e more e cient
to up date the data structure non destructively this somewhat increases up date overhead but makes copying
trivial
Note that the remarks in section ab out the importance of separating tuning from co ding also apply to
consing overhead The majority of consing will b e done by a small p ortion of the program The consing hot sp ots
are even less predictable than the CPU hot sp ots so don t waste time and create bugs by doing unnecessary
consing optimization During initial co ding avoid unnecessary side e ects and cons where it is convenient If
pro ling reveals a consing problem then go back and x the hot sp ots
See section page for a discussion of howtoavoid numb er consing in Python
Complex Argument Syntax
Common Lisp has very p owerful argument passing mec hanisms Unfortunately twoofthemostpowerful mech
anisms rest arguments and keyword arguments have a signi cant p erformance p enalty
With keyword arguments the called function has to parse the supplied keywords by iterating over them
and checking them against the desired keywords
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
With rest arguments the function must cons a list to hold the arguments If a function is called many
times or with many arguments large amounts of memory will b e allocated
Although rest argument consing is worse than keyword parsing neither problem is serious unless thousands
of calls are made to such a function The use of keyword arguments is strongly encouraged in functions with
many arguments or with interfaces that are likely to b e extended and rest arguments are often natural in user
interface functions
Optional arguments have some e ciency advantage over keyword arguments but their syntactic clumsiness
and lack of extensibility has caused many Common Lisp programmers to abandon use of optionals except in
functions that haveobviously simple and immutable interfaces such as subseq or in functions that are only
called in a few places When de ning an interface function to b e used by other programmers or users use of
only required and keyword arguments is recommended
Parsing of defmacro keyword and rest arguments is done at compile time so a macro can b e used to provide
a convenientsyntax with an e cient implementation If the macro expanded form contains no keyword or rest
arguments then it is p erfectly acceptable in inner lo ops
Keyword argument parsing overhead can also b e avoided by use of inline expansion see section page
and blo ck compilation section
Note the compiler op en co des most heavily used system functions whichhavekeyword or rest arguments
so that no run time overhead is inv olved
Mapping and Iteration
One of the traditional Common Lisp programming styles is a highly applicative one involving the use of mapping
functions and many lists to store intermediate results To compute the sum of the square ro ots of a list of numbers
one mightsay
apply mapcar sqrt list of numbers
This programming style is clear and elegant but unfortunately results in slow co de There are two reasons
why
The creation of lists of intermediate results causes much consing see
Each level of application requires another scan down the list Thus disregarding other e ects the ab ove
co de would probably taketwice as long as a straightforward iterativeversion
An example of an iterativeversion of the same co de
do num list of numbers cdr num
sum sqrt car num sum
null num sum
See sections and for a discussion of the interactions of iteration constructs with typ e inference and
variable optimization Also section discusses an applicativestyle of iteration
Trace Files and Disassembly
In order to write e cient co de you need to know the relative costs of di erent op erations The main reason
why writing e cient Common Lisp co de is di cult is that there are so many op erations and the costs of these
op erations vary in obscure context dep endent ways Although e ciency notes p oint out some problem areas
the only way to ensure generation of the b est co de is to look at the assembly co de output
The disassemble function is a convenientway to get the assembly co de for a function but it can b e very
di cult to interpret since the corresp ondence with the original source co de is weak A b etter but more awkward
option is to use the trace file argumenttocompile file to generate a trace le
A trace le is a dump of the compiler s internal represen tations including annotated assembly co de Each
comp onent in the program gets four pages in the trace le separated by L
The implicit continuation or IR represen tation of the optimized source This is a dump of the ow graph
represen tation used for source level optimizations As you will quickly notice it is not really very close
to the source This represen tation is not very useful to even sophisticated users
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
The Virtual Machine VM or IR represen tation of the program This dump represen ts the generated
co de as sequences of Virtual OPerations VOPs This represen tation is intermediate b etween the source
and the assembly co de eachVOP corresp onds fairly directly to some primitive function or construct
but a given VOP also has a fairly predictable instruction sequence An op eration such as mayhave
multiple implementations with di erent cost and applicability The choice of a particular VOP such as
fixnum or single float represen ts this choice of implementation Once you are familiar with it the
VM represen tation is probably the most useful for determining what implementation has b een used
An assembly listing annotated with the VOP resp onsible for generating the instructions This listing is
useful for guring out what a VOP do es and howitisimplemented in a particular context but its large
size makes it more di cult to read
A disassembly of the generated co de which has all pseudo op erations expanded out but is not annotated
with VOPs
Note that trace le generation takes much space and time since the trace le is tens of times larger than the
source le Toavoid huge confusing trace les and muchwasted time it is b est to separate the critical program
p ortion into its own le and then generate the trace le from this small le
E ciency Notes
E ciency notes are messages that warn the user that the compiler has chosen a relatively ine cient implemen
tation for some op eration Usually an e ciency note re ects the compiler s desire for more typ e information If
the typ e of the values concerned is known to the programmer then additional declarations can b e used to get a
more e cient implementation
E ciency notes are controlled by the extensions inhibit warnings optimization quality see section
page When speed is greater than extensions inhibit warnings e ciency notes are enabled Note that
this implicitly enables e ciency notes whenever speed is increased from its default of
Consider this program with an obscure missing declaration
defun eff note x y z
declare fixnum x y z
the fixnum x y z
If compiled with speed safety this note is given
In DEFUN EFF NOTE
X Y Z
X Y Z
Note Forced to do inline signed byte arithmetic cost
Unable to do inline fixnum arithmetic cost because
The first argument is a INTEGER
not a FIXNUM
This e ciency note tells us that the result of the intermediate computation x y is not known to b e a fixnum
so the addition of the intermediate sum to z must b e done less e ciently This can b e xed by changing the
de nition of eff note
defun eff note x y z
declare fixnum x y z
the fixnum the fixnum x y z
Typ e Uncertainty
The main cause of ine ciency is the compiler s lack of adequate information ab out the typ es of function argument
and result values Many important op erations such as arithmetic have an ine cient general generic case but
have e cient implementations that can usually b e used if there is su cient argumenttyp e information
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Typ e e ciency notes are given when a value s typ e is uncertain There is an important distinction b etween
values that are not known to b e of a goo d typ e uncertain and values that are known not to b e of a goo d typ e
E ciency notes are given mainly for the rst case uncertain typ es If it is clear to the compiler that that there
is not an e cient implementation for a particular function call then an e ciency note will only b e given if the
extensions inhibit warnings optimization qualityis see section page
In other words the default e ciency notes only suggest that you add declarations not that you change the
semantics of your program so that an e cient implementation will apply For example compilation of this form
will not give an e ciency note
elt the list l i
even though a vector access is more e cient than indexing a list
E ciency Notes and Typ e Checking
It is importantthatthe eff note example ab ove used safety When typ e checking is enabled you may
get apparently spurious e ciency notes With safety the note has this extra line on the end
The result is a INTEGER not a FIXNUM
This seems strange since there is a the declaration on the result of that second addition
In fact the ine ciency is real and is a consequence of Python s treating declarations as assertions to b e
veri ed The compiler can t assume that the result typ e declaration is true it must generate the result and
then test whether it is of the appropriate typ e
In practice this means that when you are tuning a program to run without typ e checks you should work
from the e ciency notes generated by unsafe compilation If you want co de to run e ciently with typ e checking
then y ou should pay attention to all the e ciency notes that you get during safe compilation Since user supplied
output typ e assertions e g from the are disregarded when selecting op eration implementations for safe co de
you must somehowgive the compiler information that allows it to prove that the result truly mustbeofagood
typ e In our example it could b e done by constraining the argumenttyp es more
defun eff note x y z
declare type unsigned byte x y z
x y z
Of course this declaration is acceptable only if the arguments to eff note always are unsigned byte
integers
Representation E ciency Notes
When operating on values that have non descriptor represen tations see section page there can b e
a substantial time and consing p enalty for converting to and from descriptor represen tations For this reason
the compiler gives an e ciency note whenever it is forced to do a represen tation co ercion more exp ensivethan
efficiency note cost threshold page
Ine cient represen tation co ercions maybeduetotyp e uncertainty asinthisexample
defun set flo x
declare single float x
prog var
setq var gorp
setq var x
return var
which pro duces this e ciency note
In DEFUN SET FLO
SETQ VAR X Note Doing float to pointer coercion cost from X to VAR
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
The variable var is not known to always hold values of typ e single float so a descriptor represen tation must
b e used for its value In sort of situation and adding a declaration will eliminate the ine ciency
Often ine cient represen tation conversions are not due to typ e uncertainty instead they result from
evaluating a non descriptor expression in a context that requires a descriptor result
Assignment to or initialization of any data structure other than a sp ecialized array see section
page or
Assignmenttoaspecial variable or
Passing as an argument or returning as a value in any function call that is not a local call see section
page
If such ine cient co ercions app ear in a hot sp ot in the program data structures redesign or program
reorganization may b e necessary to improve e ciency See sections and
Because represen tation selection is done rather late in compilation the source context in these e ciency notes
is somewhat vague making interpretation more di cult This is a fairly straightforward example
defun cf x y
declare single float x y
cons x y t
which gives this e ciency note
In DEFUN CF
CONS X Y T
Note Doing float to pointer coercion cost for
The first argument of CONS
The source context form is almost always the form that receives the value b eing co erced as it is in the preceding
example but can also b e the source form which generates the co erced value Compiling this example
defun if cf x y
declare single float x y
cons if grue x y snoc t
pro duces this note
In DEFUN IF CF
X Y
Note Doing float to pointer coercion cost
In either case the note s text explanation attempts to include additional information ab out what locations
are the source and destination of the co ercion Here are some example notes
IF GRUE X SNOC
Note Doing float to pointer coercion cost from X
SETQ VAR X
Note Doing float to pointer coercion cost from X to VAR
Note that the return value of a function is also a place to which co ercions mayha vetobedone
DEFUN F X Y DECLARE SINGLE FLOAT X Y X Y
Note Doing float to pointer coercion cost to return value
Sometimes the compiler is unable to determine a name for the source or destination in which case the source context is the only clue
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Verb osityControl
These variables control the verb osity of e ciency notes
efficiency note cost threshold Variable
Before printing some e ciency notes the compiler compares the value of this variable to the di erence in
cost b etween the chosen implementation and the b est p otential implementation If the di erence is not greater
than this limit then no note is printed The units are implementation dep endent the initial value suppresses
notes ab out trivial ine ciencies A value of will note any ine ciency
efficiency note limit Variable
When printing some e ciency notes the compiler rep orts p ossible e cient implementations The initial
value of prevents excessively long e ciency notes in the common case where there is no typ e information so
all implementations are p ossible
Pro ling
The rst step in improving a program s p erformance is to pro le the activity of the program to nd where it
sp ends its time The b est way to do this is to use the pro ling utility found in the profile package This
package provides a macro profile that encapsulates functions with statistics gathering co de
Pro le Interface
timed functions Variable
This variable holds a list of all functions that are currently b eing pro led
profile fname j Macro callers tg
This macro wraps pro ling co de around the named functions As in trace the name s are not evaluated If a
function is already pro led then the function is unpro led and repro led useful to notice function rede nition
Awarning is printed for each name that is not a de ned function
If CALLERS T is sp eci ed then each function that calls this function is recorded along with the number of
calls made
unprofile fname g Macro
This macro removes pro ling co de from the named functions If no name s are supplied all currently pro led
functions are unpro led
report time fnameg Macro
This macro prints a rep ort for each name d function of the following information
The total CPU time used in that function for all calls
the total number of bytes consed in that function for all calls
the total numb er of calls
the average amount of CPU time p er call
Summary totals of the CPU time consing and calls columns are printed An estimate of the pro ling overhead is
also printed see b elow If no name s are supplied then the times for all curren tly pro led functions are printed
reset time fname g Macro
This macro resets the pro ling counters asso ciated with the name d functions If no name s are supplied then all curren tly pro led functions are reset
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Pro ling Techniques
Start by pro ling big pieces of a program then carefully cho ose which functions close to but not in the inner
lo op are to b e pro led next Avoid pro ling functions that are called by other pro led functions since this op ens
the p ossibility of pro ling overhead b eing included in the rep orted times
If the p er call time rep orted is less than second then consider the clo ck resolution and pro ling overhead
b efore you b elieve the time It may b e that you will need to run your program many times in order to average
out to a higher resolution
Nested or Recursive Calls
The pro ler attempts to compensate for nested or recursive calls Time and consing overhead will b e charged
to the dynamically innermost most recen t call to a pro led function So pro ling a subfunction of a pro led
function will cause the rep orted time for the outer function to decrease However if an inner function has a large
numb er of calls some of the pro ling overhead may leak into the rep orted time for the outer function In
general b e wary of pro ling short functions that are called many times
Clo ck resolution
Unless you are very lucky the length of your machine s clo ck tick is probably much longer than the time it
takes simple function to run For example on the IBM RT the clo ck resolution is second This means that
if a function is only called a few times then only the rst couple decimal places are really meaningful
Note however that if a function is called many times then the statistical averaging across all calls should
result in increased resolution For example on the IBM RT if a function is called a thousand times then a
resolution of tens of microseconds can b e exp ected
Pro ling overhead
The added pro ling co de takes time to run every time that the pro led function is called whic h can disrupt the
attempt to collect timing information In order to avoid serious in ation of the times for functions that take
little time to run an estimate of the overhead due to pro ling is subtracted from the times rep orted for each
function
Although this correction works fairly well it is not totally accurate resulting in times that b ecome increasingly
meaningless for functions with short runtimes This is only a concern when the estimated pro ling overhead is
many times larger than rep orted total CPU time
The estimated pro ling overhead is not represen ted in the rep orted total CPU time The sum of total CPU
time and the estimated pro ling overhead should b e close to the total CPU time for the entire pro ling run as
determined by the time macro Time unaccounted for is probably b eing used by functions that you forgot to
pro le
Additional Timing Utilities
time form Macro
This macro evaluates form prints some timing and memory allocation information to trace output and
returns anyvalues that form returns The timing information includes real time user run time and system
run time This macro executes a form and rep orts the time and consing overhead If the time form is not
compiled e g it was typ ed at top level then compile will b e called on the form to give more accurate timing
information If you really wanttotimeinterpreted sp eed you can say
time eval form
Things that execute fairly quickly should b e timed more than once since there may b e more paging overhead in
the rst timing To increase the accuracy of very short times you can time multiple evaluations
time dotimes i form
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
extensions get bytes consed Function
This function returns the number of bytes allocated since the rst time you called it The rst time it is
called it returns zero The ab ove pro ling routines use this to rep ort consing information
extensions gc run time Variable
This variable accumulates the run time consumed by garbage collection in the units returned by get
internal run time
internal time units per second Constant
The value of internal time units per second is
A Note on Timing
There are two general kinds of timing information provided bythe time macro and other pro ling utilities real
time and run time Real time is elapsed wall clo ck time It will b e a ected in a fairly obvious waybyany
other activity on the machine The more other pro cesses contending for CPU and memory the more real time
will increase This means that real time measurements are di cult to replicate though this is less true on a
dedicated workstation The advantage of real time is that it is real It tells you really how long the program to ok
to run under the b enchmarking conditions The problem is that you don t know exactly what those conditions
were
Run time is the amount of time that the pro cessor supp osedly sp ent running the program as opp osed to
waiting for I O or running other pro cesses User run time and system run time are numb ers rep orted by
the Unix kernel They are supp osed to b e a measure of howmuch time the pro cessor sp ent running y our user
program which will include GC overhead etc and the amount of time that the kernel sp ent running on your
b ehalf
Ideally user time should b e totally una ected by b enchmarking conditions in reality user time do es dep end
on other system activity though in rather non obvious ways
System time will clearly dep end on b enchmarking conditions In Lisp b enchmarking paging activity increases
system run time but not byasmuch as it increases real time since the kernel sp ends some time waiting for the
disk and this is not run time kernel or otherwise
In my exp erience the biggest trap in interpreting kernel user run time is to look only at user time In reality
it seems that the sum of kernel and user time is more repro ducible The problem is that as system activity
increases there is a spurious decrease in user run time In e ect as paging etc increases user time leaks into
system time
So in practice the only way to get truly repro ducible results is to run with the same competing activityon
the system Try to run on a machine with nob o dy else logged in and check with ps aux to see if there are
any system pro cesses munching large amounts of CPU or memory If the ratio b etween real time and the sum
of user and system time varies much b etween runs then you have a problem
Benchmarking Techniques
Given these imperfect timing to ols how do should you do b enchmarking The answer dep ends on whether you
are trying to measure improvements in the p erformance of a single program on the same hardware or if you are
trying to compare the p erformance of di erent programs and or di erent hardware
For the rst use measuring the e ect of program modi cations with constant hardware you should lo ok at
b oth system user and real time to understand what e ect the change had on CPU use and on I O including
paging If you are working on a CPU intensive program the change in system user time will giveyou a moder
ately repro ducible measure of p erformance across a fairly wide range of system conditions For a CPU intensive
program you can think of system user as howlongitwould havetaken to run if I had myown machine So
in the case of comparing CPU intensive programs system user time is relatively real and reasonable to use
For programs that sp end a substantial amount of their time paging you really can t predict elapsed time
under a given op erating condition without b enchmarking in that condition User or system user time maybe
fairly repro ducible but it is also relatively meaningless since in a paging or I O intensive program the program
is sp ending its time waiting not running and system time and user time are b oth measures of run time A
change that reduces run time might increase real time by increasing paging
CHAPTER ADVANCED COMPILER USE AND EFFICIENCY HINTS
Another common use for b enchmarking is comparing the p erformance of the same program on di erent hard
ware You wanttoknowwhich machine to run your program on For comparing di erent machines op erating
systems etc the only way to compare that makes sense is to set up the machines in exactly the way that they
will normally b e run and then measure real time If the program will normally b e run along with X then run X
If the program will normally b e run on a dedicated workstation then b e sure nob o dy else is on the b enchmarking
machine If the program will normally b e run on a machine with three other Lisp jobs then run three other
Lisp jobs If the program will normally b e run on a machine with meg of memory then run with meg Here
normal means normal for that machine Ifyou the choice of an unloaded RT or a heavily loaded PMAX
do your b enchmarking on an unloaded RTandaheavily loaded PMAX
If you have a program you b elievetobeCPUintensive then you might b e tempted to compare run times
across systems hoping to get a meaningful result even if the b enchmarking isn t done under the exp ected running
condition Don t to this for two reasons
The op erating systems might not compute run time in the same way
Under the real running condition the program mightnotbeCPUintensive after all
In the end only real time means anything it is the amount of time you havetowait for the result The
only valid uses for run time are
To develop insightinto the program For example if run time is muc h less than elapsed time then you are
probably sp ending lots of time paging
Toevaluate the relative p erformance of CPU intensive programs in the same environment
Chapter
UNIX Interface
By Robert MacLachlan Skef Wholey
Bill Chiles and William Lott
CMU Common Lisp attempts to make the full p ower of the underlying environmentavailable to the Lisp
programmer This is done using combination of hand co ded interfaces and foreign function calls to C libraries
Although the techniques di er the style of interface is similar This chapter provides an overview of the facilities
available and general rules for using them as well as describing sp eci c features in detail It is assumed that the
reader has a working familiarity with Mach Unix and X as well as access to the standard system do cumentation
Reading the Command Line
The shell parses the command line with which Lisp is invoked and passes a data structure containing the parsed
information to Lisp This information is then extracted from that data structure and put into a set of Lisp data
structures
extensions command line strings Variable
extensions command line utility name Variable
extensions command line words Variable
extensions command line switches Variable
The value of command line words is a list of strings that make up the command line one word p er string
The rst word on the command line i e the name of the program invoked usually lisp is stored in command
line utility name The value of command line switches is a list of command line switch structures
with a structure for eachword on the command line starting with a hyphen All the command line words b etween
the program name and the rst switch are stored in command line words
The following functions may b e used to examine command line switch structures
extensions cmd switch name switch Function
Returns the name of the switch less the preceding hyphen and trailing equal sign if any
extensions cmd switch value switch Function
Returns the value designated using an emb edded equal sign if any If the switch has no equal sign then this
is null
extensions cmd switch words switch Function
Returns a list of the words b etween this switch and the next switch or the end of the command line
CHAPTER UNIX INTERFACE
Useful Variables
system stdin Variable
system stdout Variable
system stderr Variable
Streams connected to the standard input output and error le descriptors
system tty Variable
A stream connected to dev tty
Lisp Equivalents for C Routines
The UNIX do cumentation describ es the system interface in terms of C pro cedure headers The corresp onding
Lisp function will have a somewhat di erent interface since Lisp argument passing conventions and datatyp es
are di erent
The main di erence in the argument passing conventions is that Lisp do es not supp ort passing values by
reference In Lisp all argument and results are passed byvalue Interface functions take some xed number
of arguments and return some xed number of values A given parameter in the C sp eci cation will app ear
as an argument return value or b oth dep ending on whether it is an In parameter Out parameter or In Out
parameter The basic transformation one makes to come up with the Lisp equivalent of a C routine is to remove
the Out parameters from the call and treat them as extra return values In Out parameters app ear b oth as
arguments and return values Since Out and In Out parameters are only conventions in C you must determine
the usage from the do cumentation
Thus the C routine declared as
kern return t lookup servport portsname portsid
port servport
char portsname
int portsid out
portsid expression to compute portsid field
return KERN SUCCESS
has as its Lisp equivalent something like
defun lookup ServPort PortsName
values
success
expression to compute portsid field
If there are multiple out or in out arguments then there are multiple additional returns values
Fortunately CMU Common Lisp programmers rarely havetoworry ab out the nuances of this translation
pro cess since the names of the arguments and return values are do cumented in a waysothatthe describe
function and the Hemlock Describe Function Call command invoked with C M Shift A will list this infor
mation Since the names of arguments and return values are usually descriptive the information that describe
prints is usually all one needs to write a call Most programmers use this on line do cumentation nearly all of the
time and thereby avoid the need to handle bulky manuals and p erform the translation from barbarous tongues
CHAPTER UNIX INTERFACE
Typ e Translations
Lisp data typ es havevery di erent represen tations from those used byconventional languages such as C Since
the system interfaces are designed for conventional languages Lisp must translate ob jects to and from the
Lisp represen tations Many simple ob jects have a direct translation integers characters strings and oating
p ointnumb ers are translated to the corresp onding Lisp ob ject A number of typ es however are implemented
di erently in Lisp for reasons of clarity and e ciency
Instances of enumerated typ es are expressed as keywords in Lisp Records arrays and p ointer typ es are
implemented with the Alien facility see page Access functions are de ned for these typ es whichconvert
elds of records elements of arrays or data referenced bypointers into Lisp ob jects p ossibly another ob ject to
b e referenced with another access function
One should disp ose of Alien ob jects created by constructor functions or returned from remote pro cedure calls
when they are no longer of any use freeing the virtual memory asso ciated with that ob ject Since Aliens contain
p ointers to non Lisp data the garbage collector cannot do this itself If the memory was obtained from make
alien page or from a foreign function call to a routine that used malloc then free alien page
should b e used If the Alien was created using MACH memory allocation e g vm allocate then the storage
should b e freed using vm deallocate
System Area Pointers
Note that in some cases an address is represen ted by a Lisp integer and in other cases it is represen ted by a real
p ointer Pointers are usually used when an ob ject in the curren t address space is b eing referred to The MACH
virtual memory manipulation calls m ust use integers since in principle the address could b e in any pro cess and
Lisp cannot abide random p ointers Because these typ es are represen ted di erently in Lisp one must explicitly
co erce b etween these represen tations
System Area Pointers SAPs provide a mechanism that bypasses the Alien typ e system and accesses virtual
memory directly A SAP is a rawbyte p ointer into the lisp pro cess address space SAPs are represen ted
with a p ointer descriptor so SAP creation can cause consing However the compiler uses a non descriptor
represen tation for SAPs when p ossible so the consing overhead is generally minimal See section page
system sap int sap Function
system int sap int Function
The function sap int is used to generate an integer corresp onding to the system area p ointer suitable for
passing to the kernel interfaces whichwant all addresses sp eci ed as integers The function int sap is used to
do the opp osite conversion The integer represen tation of a SAP is the byte o set of the SAP from the start of
the address space
system sap sap o set Function
This function adds a byte o set to sap returning a new SAP
system sap ref sap o set Function
system sap ref sap o set Function
system sap ref sap o set Function
These functions return the or bit unsigned integer at o set from sap Theo set is alwaysabyte
o set regardless of the numb er of bits accessed setf may b e used with the these functions to dep osit values
into virtual memory
system signed sap ref sap o set Function
system signed sap ref sap o set Function
system signed sap ref sap o set Function
These functions are the same as the ab ove unsigned op erations except that they sign extend returning a
negativenumb er if the high bit is set
CHAPTER UNIX INTERFACE
Unix System Calls
You probably won t havemuch cause to use them but all the Unix system calls are available The Unix system
call functions are in the Unix package The name of the interface for a particular system call is the name of
the system call prep ended with unix The system usually de nes the asso ciated constants without anypre x
name To nd out how to use a particular system call try using describe on it If that is unhelpful look at the
source in syscall lisp or consult your system maintainer
The Unix system calls indicate an error by returning nil as the rst value and the Unix error number as the
second value If the call succeeds then the rst value will always b e non nil often t
Unix get unix error msg error Function
This function returns a string describing the Unix error number error
File Descriptor Streams
Many of the UNIX system calls return le descriptors Instead of using other UNIX system calls to p erform I O
on them you can create a stream around them For this purp ose fd streams exist
system make fd stream descriptor key input output element type Function
buffering name file original
delete original auto close
timeout pathname
This function creates a le descriptor stream using descriptor If input is non nil input op erations are
allowed If output is non nil output op erations are allowed The default is input only These keywords are
de ned
element typ e is the typ e of the unit of transaction for the stream which defaults to string char Seethe
Common Lisp description of open for valid values
bu ering is the kind of output bu ering desired for the stream Legal values are none for no bu ering line
for bu ering up to each newline and full for full bu ering
name is a simple string name to use for descriptive purp oses when the system prints an fd stream When printing
fd streams the system prep ends the streams name with Stream for Ifname is unsp eci ed it defaults to
a string containing le or descriptor in order of preference
le original le sp eci es the defaulted namestring of the asso ciated le when creating a le stream must b e
a simple string original is the simple string name of a backup le containing the original contents
of le while writing le
When you ab ort the stream by passing t to close as the second argument if you supplied b oth le and
original close will rename the original name to the le name When you close the stream normally
if you supplied original and delete original is non nil close deletes original Ifauto close is true the
default then descriptor will b e closed when the stream is garbage collected
pathname The original pathname passed to op en and returned by pathname not defaulted or translated
timeout if non null then timeout is an integer numb er of seconds after which an input wait should time out If
a read do es time out then the system io timeout condition is signalled
system fd stream p ob ject Function
This function returns t if ob ject is an fd stream and nil if not Obsolete use the p ortable typep x
file stream
system fd stream fd stream Function
This returns the le descriptor asso ciated with stream
CHAPTER UNIX INTERFACE
Making Sense of Mach Return Co des
Whenever a remote pro cedure call returns a Unix error co de such as kern return t it is usually prudent to
check that co de to see if the call was successful To relieve the programmer of the hassle of testing this value
himself and to centralize the information ab out the meaning of non success return co des CMU Common Lisp
provides a numb er of macros and functions See also get unix error msg page
system gr error function gr optional context Function
Signals a Lisp error printing a message indicating that the call to the sp eci ed function failed with the
return co de gr If supplied the context string is printed after the function name and b efore the string asso ciated
with the gr For example
gr error nukegarbage lost big
Error in function GR ERROR
NUKEGARBAGE lost big no space
Proceed cases
Return to Top Level
Debug type H for help
Signal Conditions Simple Error FDE
system gr call function rest args Macro
system gr call function rest args Macro
These macros can b e used to call a function and automatically check the GeneralReturn co de and signal
an appropriate error in case of non successful return gr call returns nil if no error o ccurs while gr call
returns the second value of the function called
gr call mach port allocate task self
NIL
system gr bind fvar g Macro function farg g fformg
This macro can b e used much like multiple value bind to bind the var storeturnvalues resulting from
calling the function with the given arg s The rst return value is not b ound to a variable but is checked as a
GeneralReturn co de as in gr call
gr bind port list port list cnt
mach port select task self
format t The port count is S port list cnt
port list
The port count is
Alien value
Unix Interrupts
CMU Common Lisp allows access to all the Unix signals that can b e generated under Unix It should b e noted
that if this capability is abused it is p ossible to completely destroy the running Lisp The following macros
and functions allow access to the Unix interrupt system The signal names as sp eci ed in section of the Unix
Programmer s Manual are exp orted from the Unix package
CHAPTER UNIX INTERFACE
Changing Interrupt Handlers
system with enabled interrupts sp ecs rest body Macro
This macro should b e called with a list of signal sp eci cations sp ecs Each elementofsp ecs should b e a list
of two elements the rst should b e the Unix signal for which a handler should b e established the second should
b e a function to b e called when the signal is received One or more signal handlers can b e established in this way
with enabled interrupts establishes the correct signal handlers and then executes the forms in body The
forms are executed in an unwind protect so that the state of the signal handlers will b e restored to what it was
b efore the with enabled interrupts was entered A signal handler function sp eci ed as NIL will set the Unix
signal handler to the default which is normally either to ignore the signal or to cause a core dump dep ending on
the particular signal
system without interrupts rest body Macro
It is sometimes necessary to execute a piece a co de that can not b e interrupted This macro the forms in
body with interrupts disabled Note that the Unix interrupts are not actually disabled rather they are queued
until after body has nished executing
system with interrupts rest body Macro
When executing an interrupt handler the system disables interrupts as if the handler was wrapp ed in in a
without interrupts The macro with interrupts can b e used to enable interrupts while the forms in body
are evaluated This is useful if body is going to enter a break loop or do some long computation that might need
to b e interrupted
system without hemlock rest body Macro
For some interrupts such as SIGTSTP susp end the Lisp pro cess and return to the Unix shell it is necessary
to leave Hemlock and then return to it This macro executes the forms in body after exiting Hemlock When
body has b een executed control is returned to Hemlock
system enable interrupt signal function Function
This function establishes function as the handler for signal Unless you want to establish a global sig
nal handler you should use the macro with enabled interrupts to temporarily establish a signal handler
enable interrupt returns the old function asso ciated with the signal
system ignore interrupt signal Function
Ignore interrupt sets the Unix signal mechanism to ignore signal which means that the Lisp pro cess will never
see the signal Ignore interrupt returns the old function asso ciated with the signal or nil if none is currently
de ned
system default interrupt signal Function
Default interrupt can b e used to tell the Unix signal mechanism to p erform the default action for signal For
details on what the default action for a signal is see section of the Unix Programmer s Manual In general it
is likely to ignore the signal or to cause a core dump
CHAPTER UNIX INTERFACE
Examples of Signal Handlers
The following co de is the signal handler used by the Lisp system for the SIGINT signal
defun ih sigint signal code scp
declare ignore signal code scp
without hemlock
with interrupts
break Software Interrupt t
The without hemlock form is used to make sure that Hemlock is exited b efore a break loop is entered The
with interrupts form is used to enable interrupts b ecause the user maywant to generate an interrupt while
in the break lo op Finally break is called to enter a break loop so the user can lo ok at the current state of the
computation If the user pro ceeds from the break lo op the computation will b e restarted from where it was
interrupted
The following function is the Lisp signal handler for the SIGTSTP signal which susp ends a pro cess and
returns to the Unix shell
defun ih sigtstp signal code scp
declare ignore signal code scp
without hemlock
Unix unix kill Unix unix getpid Unix sigstop
Lisp uses this interrupt handler to catch the SIGTSTP signal b ecause it is necessary to get out of Hemlockina
clean way b efore returning to the shell
To set up these interrupt handlers the following is recommended
with enabled interrupts Unix SIGINT ih sigint
Unix SIGTSTP ih sigtstp
user code to execute with the above signal handlers enabled
Chapter
Event Dispatching with
SERVE EVENT
By Bill Chiles and Rob ert MacLachlan
It is common to havemultiple activities simultaneously op erating in the same Lisp pro cess Furthermore
Lisp programmers tend to exp ect a exible developmentenvironment It must b e p ossible to load and modify
application programs without requiring modi cations to other running programs CMU Common Lisp achieves
this byhaving a central scheduling mechanism based on an event driven ob ject oriented paradigm
An event is some interesting happ ening that should cause the Lisp pro cess to wake up and do something
These events include X events and activity on Unix le descriptors The ob ject oriented mechanism is only
available with the rst two and it is optional with X events as describ ed later in this chapter In an X event the
window ID is the ob ject capability and the X eventtyp e is the op eration co de The Unix le descriptor input
mechanism simply consists of an asso ciation list of a handler to call when input shows up on a particular le
descriptor
Ob ject Sets
An ob ject set is a collection of ob jects that have the same implementation for each op eration Externally the
ob ject is represen ted by the ob ject capability and the op eration is represen ted by the op eration co de Within Lisp
the ob ject is represen ted by an arbitrary Lisp ob ject and the implementation for the op eration is represen ted
by an arbitrary Lisp function The ob ject set mechanism maintains this translation from the external to the
internal represen tation
system make object set name optional default handler Function
This function makes a new ob ject set Name is a string used only for purp oses of identifying the ob ject set
when it is printed Default handler is the function used as a handler when an unde ned op eration occurs on
an ob ject in the set You can de ne op erations with the serve operation functions exp orted the extensions
package for X events see section page Ob jects are added with system add xwindow object Initially
the ob ject set has no ob jects and no de ned op erations
system object set operation ob ject set op eration co de Function
This function returns the handler function that is the implementation of the op eration corresp onding to
op eration co de in ob ject set When set with setf the setter function establishes the new handler The serve
op eration functions exp orted from the extensions packageforXevents see section page call this on
b ehalf of the user when announcing a new op eration for an ob ject set
system add xwindow object window ob ject ob ject set Function
These functions add port or window to ob ject set Ob ject is an arbitrary Lisp ob ject that is asso ciated with
the port or window capability Window is a CLX window When an event o ccurs system serve event passes
ob ject as an argument to the handler function
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
The SERVE EVENT Function
The system serve event function is the standard way for an application to wait for something to happ en For
example the Lisp system calls system serve event when it wants input from X or a terminal stream The idea
b ehind system serve event is that it knows the appropriate action to takewhenanyinteresting event happ ens
If an application calls system serve event when it is idle then any other applications with p ending events can
run This allows several applications to run at the same time without interference even though there is only
one thread of control Note that if an application is waiting for input of any kind then other applications will
get events
system serve event optional timeout Function
This function waits for an event to happ en and then dispatches to the correct handler function If sp eci ed
timeout is the numb er of seconds to wait b efore timing out A time out of zero seconds is legal and causes
system serve event to p oll for anyevents immediately available for pro cessing system serve event returns
t if it serviced at least one event and nil otherwise Dep ending on the application when system serve event
returns t you mightwant to call it rep eatedly with a timeout of zero until it returns nil
If input is available on any designated le descriptor then this calls the appropriate handler function supplied
by system add fd handler
Since events for many di erent applications may arrive simultaneously an application waiting for a sp eci c
eventmust loop on system serve event until the desired event happ ens Since programs such as Hemlockcall
system serve event for input applications usually do not need to call system serve event at all Hemlock
allows other application s handlers to run when it go es into an input wait
system serve all events optional timeout Function
This function is similar to system serve event except it serves all the p ending events rather than just one
It returns t if it serviced at least one event and nil otherwise
Using SERVE EVENT with Unix File Descriptors
Ob ject sets are not available for use with le descriptors as there are only two op erations p ossible on le
descriptors input and output Instead a handler for either input or output can b e registered with system serve
event for a sp eci c le descriptor Whenever any input shows up or output is p ossible on this le descriptor the
function asso ciated with the handler for that descriptor is funcalled with the descriptor as it s single argument
system add fd handler fd direction function Function
This function installs and returns a new handler for the le descriptor fd Direction can b e either input if
the system should invoke the handler when input is available or output if the system should invoke the handler
when output is p ossible This returns a unique ob ject represen ting the handler and this is a suitable argument
for system remove fd handler Function must take one argument the le descriptor
system remove fd handler handler Function
This function removes handler thatadd fd handler must have previously returned
system with fd handler direction fd function fformg Macro
This macro executes the supplied forms with a handler installed using fd direction andfunction See
system add fd handler
system wait until fd usable direction fd optional timeout Function
This function waits for up to timeout seconds for fd to b ecome usable for direction either input or output
If timeout is nil or unsp eci ed this waits forever
system invalidate descriptor fd Function
This function removes all handlers asso ciated with fd This should only b e used in drastic cases such as I O
errors but not necessarily EOF Normally you should use remove fd handler to remove the sp eci c handler
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
Using SERVE EVENT with the CLX Interface to X
Rememb er from section an ob ject set is a collection of ob jects CLX windows in this case with some
set of op erations event keywords with corresp onding implementations the same handler functions Since X
allows multiple display connections from a given pro cess you can avoid using ob ject sets if every windowinan
application or display connection b ehaves the same If a particular X application on a single display connection
has windows that want to handle certain events di erently then using ob ject sets is a convenientway to organize
this since you need some way to map the window event combination to the appropriate functionality
The following is a discussion of functions exp orted from the extensions package that facilitate handling
CLX events through system serve event The rst two routines are useful regardless of whether you use
system serve event
ext open clx display optional string Function
This function parses string for an X display sp eci cation including display and screen numb ers String
defaults to the following
cdr assoc display ext environment list test eq
If any eld in the display sp eci cation is missing this signals an error ext open clx display returns the CLX
display and screen
ext flush display ev Function ents display
This function ushes all the events in display s event queue including the current event in case the user calls
this from within an event handler
Without Ob ject Sets
Since most applications that use CLX can avoid the complexity of ob ject sets these routines are describ ed in
a separate section The routines describ ed in the next section that use the ob ject set mechanism are based on
these interfaces
ext enable clx event handling display handler Function
This function causes system serve event to notice when there is input on display s connection to the X
server When this happ ens system serve event invokes handler on display in a dynamic context with an
error handler b ound that ushes all events from display and returns By returning the error handler declines
to handle the error but it will have cleared all events thus entering the debugger will not result in in nite
errors due to streams that wait via system serve event for input Calling this rep eatedly on the same display
establishes handler as a new handler replacing any previous one for display
ext disable clx event handling display Function
This function undo es the e ect of ext enable clx event handling
ext with clx event handling display handler Macro fform g
This macro evaluates each form in a context where system serve event invokes handler on display whenever
there is input on display s connection to the X server This destroys any previously established handler for display
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
With Ob ject Sets
This section discusses the use of ob ject sets and system serve event to handle CLX events This is necessary
when a single X application has distinct windows that want to handle the same events in di erent ways Basically
you need some way of asking for a given windowwhichwayyou want to handle some event b ecause this event
is handled di erently dep ending on the window Ob ject sets provide this feature
For each CLX event key symb ol name XXX for example key press there is a function serve XXX of two
arguments an ob ject set and a function The serve XXX function establishes the function as the handler for
the XXX event in the ob ject set Recall from section system add xwindow object asso ciates some Lisp
ob ject with a CLX window in an ob ject set When system serve event notices activity on a window it calls
the function given to ext enable clx event handling If this function is ext object set event handler
it calls the function given to serve XXX passing the ob ject given to system add xwindow object and the
event s slots as well as a couple other arguments describ ed b elow
To use ob ject sets in this way
Create an ob ject set
De ne some op erations on it using the serve XXX functions
Add an ob ject for every windowonwhichyou receive requests This can b e the CLX window itself or some
structure more meaningful to your application
Call system serve event to service an X event
ext object set event handler display Function
This function is a suitable argumentto ext enable clx event handling The actual event handlers de ned
for particular events within a given ob ject set must take an argument for every slot in the appropriate event In
addition to the event slots ext object set event handler passes the following arguments
The ob ject as established by system add xwindow object onwhichtheevent o ccurred
event key see xlib event case
send event p see xlib event case
Describing any ext serve event key name function where event key name is an event key symb ol name for
example ext serve key press indicates exactly what all the argumen ts are in their correct order
When creating an ob ject set for use with ext object set event handler sp ecify ext default clx
event handler as the default handler for events in that ob ject set If no default handler is sp eci ed and
the system invokes the default default handler it will cause an error since this function takes arguments suitable
for handling p ort messages
A SERVE EVENT Example
This section contains two examples using system serve event The rst one does not use ob ject sets and the
second slightly more complicated one do es
Without Ob ject Sets Example
This example de nes an input handler for a CLX display connection It only recognizes key press events The
b o dy of the example loops over system serve event to get input
in package SERVER EXAMPLE
defun my input handler display
xlib event case display timeout
key press event window code state format t KEY PRESSED Window D S
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
xlib window id event window
See Hemlock Command Implementor s Manual for convenient
input mapping function
ext translate character display code state
Make XLIB EVENT CASE discard the event
t
defun server example
An example of using the SYSTEM SERVE EVENT function and object sets to
handle CLX events
let display ext open clx display
screen display default screen display
black screen black pixel screen
white screen white pixel screen
window create window parent screen root screen
x y width height
background white border black
border width
event mask
xlib make event mask key press
Wrap code in UNWIND PROTECT so we clean up after ourselves
unwind protect
progn
Enable event handling on the display
ext enable clx event handling display my input handler
Map the windows to the screen
map window window
Make sure we send all our requests
display force output display
Call serve event for events or immediate timeouts
dotimes i system serve event
Disable event handling on this display
ext disable clx event handling display
Get rid of the window
destroy window window
Pick off any events the X server has already queued for our
windows so we don t choke since SYSTEM SERVE EVENT is no longer
prepared to handle events for us
loop
unless deleting window drop event display window
return
Close the display
xlib close display display
defun deleting window drop event display win
Check for any events on win If there is one remove it from the
event queue and return t otherwise return nil
xlib display finish output display
let result nil
xlib process event
display timeout
handler lambda key event window allow other keys
if eq event window win
setf result t
nil result
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
With Ob ject Sets Example
This example involves more work but you get a little more for your e ort It de nes two ob jects input box
and slider and establishes a key press handler for each ob ject key pressed and slider pressed Wehave
two ob ject sets b ecause we handle events on the windows manifesting these ob jects di erently but the events
come over the same display connection
in package SERVER EXAMPLE
defstruct input box print function print input box
constructor make input box display window
Our program knows about input boxes and it doesn t care how they
are implemented
display The CLX display on which my input box is displayed
window The CLX window in which the user types
defun print input box object stream n
declare ignore n
format stream Input Box S input box display object
defvar input box windows
system make object set Input Box Windows
ext default clx event handler
defun key pressed input box event key event window root child
same screen p x y root x root y modifiers time
key code send event p
This is our key press event handler
declare ignore event key root child same screen p x y
root x root y time send event p
format t KEY PRESSED Window D S
xlib window id event window
See Hemlock Command Implementor s Manual for convenient
input mapping function
ext translate character input box display input box
key code modifiers
ext serve key press input box windows key pressed
defstruct slider print function print slider
include input box
constructor make slider
display window window width max
Our program knows about sliders too and these provide input values
zero to max
bits per value bits per discrete value up to max
max End value for slider
defun print slider object stream n
declare ignore n
format stream Slider S D
input box display object
slider max object
defun make slider display window max
make slider display window
truncate xlib drawable width window max
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
max
defvar slider windows
system make object set Slider Windows
ext default clx event handler
defun slider pressed slider event key event window root child
same screen p x y root x root y modifiers time
key code send event p
This is our key press event handler for sliders Probably this is
a mouse thing but for simplicity here we take a character typed
declare ignore event key root child same screen p x y
root x root y time send event p
format t KEY PRESSED Window D S D
xlib window id event window
See Hemlock Command Implementor s Manual for convenient
input mapping function
ext translate character input box display slider
key code modifiers
truncate x slider bits per value slider
ext serve key press slider windows slider pressed
defun server example
An example of using the SYSTEM SERVE EVENT function and object sets to
handle CLX events
let display ext open clx display
screen display default screen display
black screen black pixel screen
white screen white pixel screen
iwindow create window parent screen root screen
x y width height
background white border black
border width
event mask
xlib make event mask key press
swindow create window parent screen root screen
x y width height
background white border black
border width
event mask
xlib make event mask key press
input box make input box display iwindow
slider make slider display swindow
Wrap code in UNWIND PROTECT so we clean up after ourselves
unwind protect
progn
Enable event handling on the display
ext enable clx event handling display
ext object set event handler
Add the windows to the appropriate object sets
system add xwindow object iwindow input box
input box windows
system add xwindow object swindow slider
slider windows Map the windows to the screen
CHAPTER EVENT DISPATCHING WITH SERVE EVENT
map window iwindow
map window swindow
Make sure we send all our requests
display force output display
Call server for events or immediate timeouts
dotimes i system serve event
Disable event handling on this display
ext disable clx event handling display
delete window iwindow display
delete window swindow display
Close the display
xlib close display display
defun delete window window display
Remove the windows from the object sets before destroying them
system remove xwindow object window
Destroy the window
destroy window window
Pick off any events the X server has already queued for our
windows so we don t choke since SYSTEM SERVE EVENT is no longer
prepared to handle events for us
loop
unless deleting window drop event display window
return
defun deleting window drop event display win
Check for any events on win If there is one remove it from the
event queue and return t otherwise return nil
xlib display finish output display
let result nil
xlib process event
display timeout
handler lambda key event window allow other keys
if eq event window win
setf result t
nil result
Chapter
Alien Ob jects
By Robert MacLachlan and William Lott
Intro duction to Aliens
Because of Lisp s emphasis on dynamic memory allocation and garbage collection Lisp implementations use
unconventional memory represen tations for ob jects This represen tation mismatch creates problems when a Lisp
program must share ob jects with programs written in another language There are three di erent approaches to
establishing communication
The burden can b e placed on the foreign program and programmer by requiring the use of Lisp ob ject
represen tations The main di culty with this approach is that either the foreign program must b e written
with Lisp interaction in mind or a substantial amount of foreign glue co de must b e written to p erform
the translation
The Lisp system can automatically convert ob jects back and forth b etween the Lisp and foreign represen ta
tions This is convenient but translation b ecomes prohibitively slow when large or complex data structures
must b e shared
The Lisp program can directly manipulate foreign ob jects through the use of extensions to the Lisp lan
guage Most Lisp systems make use of this approach but the language for describing typ es and expressing
accesses is often not p owerful enough for complex ob jects to b e easily manipulated
CMU Common Lisp relies primarily on the automatic conversion and direct manipulation approaches Aliens of
simple scalar typ es are automatically converted while complex typ es are directly manipulated in their foreign
represen tation Any foreign ob jects that can t automatically b e converted into Lisp values are represen ted by
ob jects of typ e alien value Since Lisp is a dynamically typ ed language even foreign ob jects must havea
run time typ e this typ e information is provided by encapsulating the rawpointer to the foreign data within an
alien value ob ject
The Alien typ e language and op erations are most similar to those of the C language but Aliens can also b e
used when communicating with most other languages that can b e linked with C
Alien Typ es
Alien typ es have a description language based on nested list structure For example
struct foo
int a
struct foo b
CHAPTER ALIEN OBJECTS
has the corresp onding Alien typ e
struct foo
a int
b array struct foo
De ning Alien Typ es
Typ es may b e either named or anonymous With structure and union typ es the name is part of the typ e
sp eci er allowing recursively de ned typ es such as
struct foo a struct foo
An anonymous structure or union typ e is sp eci ed by using the name nil Thewith alien page macro
de nes a lo cal scop e which captures any named typ e de nitions Other typ es are not inherently named but
can b e given named abbreviations using def alien type
alien def alien type name typ e Macro
This macro globally de nes name as a shorthand for the Alien typ e typ e Whenintro ducing global structure
and union typ e de nitions name maybe nil inwhich case the name to de ne is taken from the typ e s name
Alien Typ es and Lisp Typ es
The Alien typ es form a subsystem of the CMU Common Lisp typ e system An alien typ e sp eci er provides a
way to use any Alien typ e as a Lisp typ e sp eci er For example
typep foo alien int
can b e used to determine whether foo is a p ointer to an int alien typ e sp eci ers can b e used in the same ways
as ordinary typ e sp eci ers like string Alien typ e declarations are sub ject to the same precise typ e c hecking
as any other declaration section See section page
Note that the Alien typ e system overlaps with normal Lisp typ e sp eci ers in some cases For example the
typ e sp eci er alien single float is identical to single float since Alien oats are automatically converted
to Lisp oats When type of is called on an Alien value that is not automatically converted to a Lisp value
then it will return an alien typ e sp eci er
Alien Typ e Sp eci ers
Some Alien typ e names are Common Lispsymb ols but the names are still exp orted from the alien package so
it is legal to say alien single float These are the basic Alien typ e sp eci ers
typ e Alien typ e
A p ointer to an ob ject of the sp eci ed typ e Iftyp e is t then it means a p ointer to anything
similar to void in ANSI C Currently the only way to detect a null p ointer is
zerop sap int alien sap ptr
See section page
array typ e fdimensiong Alien typ e
An array of the sp eci ed dimensions holding elements of typ e typ e Notethat int and array
int are considered to b e di erent typ es when typ e checking is done p ointer and arrayt yp es must
b e explicitly co erced using cast
Arrays are accessed using deref passing the indices as additional arguments Elements are stored
in column major order as in C so the rst dimension determines only the size of the memory
blo ck and not the layout of the higher dimensions An array whose rst dimension is variable
may b e sp eci ed by using nil as the rst dimension Fixed size arrays can b e allocated as array
elements structure slots or with alien variables Dynamic arrays can only b e allocated using
make alien page
CHAPTER ALIEN OBJECTS
struct name f eld typ e bits g Alien typ e
A structure typ e with the sp eci ed name and elds Fields are allocated at the same p ositions used
by the implementation s C compiler bits is intended for C like bit eld supp ort but is currently
unused If name is nil then the typ e is anonymous
If a named Alien struct sp eci er is passed to def alien type page or with alien
page then this de nes resp ectively a new global or local Alien structure typ e If no elds are
sp eci ed then the elds are taken from the curren t local or global Alien structure typ e de nition
of name
union name f eld typ e bits g Alien typ e
Similar to struct but de nes a union typ e All elds are allocated at the same o set and the size
of the union is the size of the largest eld The programmer must determine which eld is active
from context
enum name fsp ec g Alien typ e
An enumeration typ e that maps b etween integer values and keywords If name is nil then the
typ e is anonymous Eac h sp ec is either a keyword or a list keyword value Ifinteger is not
supplied then it defaults to one greater than the value for the preceding sp ec or to zero if it is the
rst sp ec
signed bits Alien typ e
A signed integer with the sp eci ed numb er of bits precision The upp er limit on integer precision
is determined by the machine s word size If no size is sp eci ed the maximum size will b e used
integer bits Alien typ e
Identical to signed the distinction b etween signed and integer is purely stylistic
unsigned bits Alien typ e
Like signed but sp eci es an unsigned integer
boolean bits Alien typ e
Similar to an enumeration typ e that maps to nil and all other values to t bits determines the
amount of storage allocated to hold the truth value
single float Alien typ e
A oating p ointnumb er in IEEE single format
double float Alien typ e
A oating p ointnumb er in IEEE double format
farg typ e g function result typ e Alien typ e
A Alien function that takes arguments of the sp eci ed arg typ es and returns a result of typ e result
typ e Note that the only context where a function typ e is directly sp eci ed is in the argument
to alien funcall see section In all other contexts functions are represen ted by function
p ointer typ es function
system area pointer Alien typ e
A p ointer which is represen ted in Lisp as a system area pointer ob ject see section page
The C Call Package
The c call package exp orts these typ e equivalents to the C typ e of the same name char short int long
unsigned char unsigned short unsigned int unsigned long float double c call also exp orts these
typ es
CHAPTER ALIEN OBJECTS
void Alien typ e
This typ e is used in function typ es to declare that no useful value is returned Evaluation of an
alien funcall form will return zero values
c string Alien typ e
This typ e is similar to char but is interpreted as a null terminated string and is automatically
converted into a Lisp string when accessed If the p ointer is C NULL or then accessing gives
Lisp nil
Assigning a Lisp string to a c string structure eld or variable stores the contents of the string
to the memory already p ointed to bythatvariable When an Alien of typ e char is assigned
to a c string then the c string pointer is assigned to This allows c string p ointers to b e
initialized For example
def alien type nil struct foo str c string
defun make foo str
let my foo make alien struct foo
setf slot my foo str make alien char length str
setf slot my foo str str
my foo
Storing Lisp nil writes C NULL to the c string pointer
Alien Op erations
This section describ es the basic op erations on Alien values
Alien Access Op erations
alien deref p ointer or array rest indices Function
This function returns the value p ointed to by an Alien p ointer or the value of an Alien array element If a
p ointer an optional single index can b e sp eci ed to give the equivalentofCpointer arithmetic this index is
scaled b y the size of the typ e p ointed to If an array thenumb er of indices must b e the same as the number of
dimensions in the arraytyp e deref can b e set with setf to assign a new value
alien slot struct or union slot name Function
This function extracts the value of slot slot name from the an Alien struct or union Ifstruct or union is
a p ointer to a structure or union then it is automatically dereferenced This can b e set with setf to assign a
new value Note that slot name is evaluated and need not b e a compile time constant but only constant slot
accesses are e ciently compiled
Alien Co ercion Op erations
alien addr alien expr Macro
This macro returns a p ointer to the location sp eci ed by alien expr whichmust b e either an Alien variable
a use of deref a use of slot or a use of extern alien page
alien cast alien new typ e Macro
This macro converts alien to a new Alien with the sp eci ed new typ e Bothtyp es must b e an Alien p ointer
array or function typ e Note that the result is not eq to the argument but do es refer to the same data bits
CHAPTER ALIEN OBJECTS
alien sap alien sap typ e Macro
alien alien sap alien value Function
sap alien converts sap a system area p ointer see section page to an Alien value with the sp eci ed
typ e typ e is not evaluated
alien sap returns the SAP whichpoints to alien value s data
The typ e to sap alien and the typ e of the alien value to alien sap must some Alien p ointer arrayor
record typ e
Alien Dynamic Allo cation
Dynamic Aliens are allocated using the malloc library so foreign co de can call free on the result of make alien
and Lisp co de can call free alien on ob jects allocated by foreign co de
alien make alien typ e size Macro
This macro returns a dynamically allocated Alien of the sp eci ed typ e which is not evaluated The allocated
memory is not initialized and may contain arbitrary junk If supplied size is an expression to evaluate to compute
the size of the allocated ob ject There are two ma jor cases
When typ e is an arraytyp e an arrayofthatt yp e is allocated and a pointer to it is returned Note that
you must use deref to change the result to an array b efore you can use deref to read or write elements
defvar foo make alien array char
type of foo
alien array signed
setf deref deref foo
If supplied size is used as the rst dimension for the array
When typ e is any other typ e then then an ob ject for that typ e is allocated and a pointer to it is returned
So make alien int returns a int Ifsize is sp eci ed then a blo ck of that many ob jects is allocated
with the result p ointing to the rst one
alien free alien alien Function
This function frees the storage for alien whichmust have b een allocated with make alien or malloc
See also with alien page which stack allocates Aliens
Alien Variables
Both local stack allocated and external C global Alien variables are supp orted
Lo cal Alien Variables
alien with alien f name typ e initial value g fform g Macro
This macro establishes local alien v ariables with the sp eci ed Alien typ es and names for dynamic extent
of the b o dy Thevariable names are established as symb ol macros the bindings have lexical scop e and may
b e assigned with setq or setf This form is analogous to de ning a lo cal variable in C additional storage is
allocated and the initial value is copied
with alien also establishes a new scop e for named structures and unions Any typ e sp eci ed for a variable
may contain name structure or union typ es with the slots sp eci ed Within the lexical scop e of the binding
sp eci ers and b o dy a locally de ned structure typ e foo can b e referenced by its name using struct foo
CHAPTER ALIEN OBJECTS
External Alien Variables
External Alien names are strings and Lisp names are symb ols When an external Alien is represen ted using a
Lisp variable there must b e a way to convert from one name syntax into the other The macros extern alien
def alien variable and def alien routine page use this conversion heuristic
Alien names are converted to Lisp names by upp ercasing and replacing underscores with hyphens
Conversely Lisp names are converted to Alien names bylowercasing and replacing hyphens with under
scores
Both the Lisp symb ol and Alien string names may b e separately sp eci ed by using a list of the form
lisp symb ol alien string
alien def alien variable name typ e Macro
This macro de nes name as an external Alien variable of the sp eci ed Alien typ e name and typ e are not
evaluated The Lisp name of the variable see ab ove b ecomes a global Alien variable in the Lisp namespace
Global Alien variables are e ectively global symb ol macros a reference to the variable fetches the contents
of the external variable Similarly setting the variable stores new contents the new contents mustbeofthe
declared typ e
For example it is often necessary to read the global C variable errno to determine why a particular function
call failed It is p ossible to de ne errno and mak e it accessible from Lisp by the following
def alien variable errno int
Now it is possible to get the value of the C variable errno simply by
referencing that Lisp variable
print errno
alien extern alien name typ e Macro
This macro returns an Alien with the sp eci ed typ e whichpoints to an externally de ned value name is not
evaluated and may b e sp eci ed either as a string or a symb ol typ e is an unevaluated Alien typ e sp eci er
Alien Data Structure Example
Now that wehave Alien typ es op erations and variables we can manipulate foreign data structures This C
declaration can b e translated into the following Alien typ e
struct foo
int a
struct foo b
def alien type nil
struct foo
a int
b array struct foo
With this de nition the following C expression can b e translated in this way
CHAPTER ALIEN OBJECTS
struct foo f
f b a
with alien f struct foo
slot deref slot f b a
Do something with f
Or consider this example of an external C variable and some accesses
struct c struct
short x y
char a b
int z
c struct n
extern struct c struct my struct
my struct x
my struct a
my struct my struct n
which can b e made b e manipulated in Lisp like this
def alien type nil
struct c struct
x short
y short
a char
b char
z int
n c struct
def alien variable my struct c struct
incf slot my struct x
setf slot my struct a
setq my struct slot my struct n
Loading Unix Ob ject Files
Foreign ob ject les are loaded into the running Lisp pro cess by load foreign First it runs the linker on the les
and libraries creating an absolute Unix ob ject le This ob ject le is then loaded into into the currently running
Lisp The external symb ols de ning routines and variables are made available for future external references e g
by extern alien load foreign must b e run b efore any of the de ned symb ols are referenced
Note that if a Lisp core image is saved using save lisp page all loaded foreign co de is lost when the
image is restarted
alien load foreign les key libraries base le env Function
les is a simple string or list of simple strings sp ecifying the names of the ob ject les libraries is a list of
simple strings sp ecifying libraries in a format that ld the Unix linker exp ects The default value for libraries
is lc i e the standard C library base le is the le to use for the initial symb ol table information
CHAPTER ALIEN OBJECTS
The default is the Lisp start up co de path lisp env should b e a list of simple strings in the format of Unix
environmentvariables i e A B where A is an environmentvariable and B is its value The default value for
env is the environment information available at the time Lisp was invoked Unless you are certain that you want
to change this you should just use the default
Alien Function Calls
The foreign function call interface allows a Lisp program to call functions written in other languages The current
implementation of the foreign function call interface assumes a C calling convention and thus routines written
in any language that adheres to this convention may b e called from Lisp
Lisp sets up various interrupt handling routines and other environment information when it rst starts up
and exp ects these to b e in place at all times The C functions called by Lisp should either not change the
environment esp ecially the interrupt entry p oints or should make sure that these entry p oints are restored
when the C function returns to Lisp If a C function makes changes without restoring things to the waythey
were when the C function was entered there is no telling what will happ en
The alien funcall Primitive
alien alien funcall alien function rest arguments Function
This function is the foreign function call primitive alien function is called with the supplied arguments and
its value is returned The alien function is an arbitrary run time expression to call a constant function use
extern alien page or def alien routine
The typ e of alien function must b e alien function or alien function See section
page The function typ e is used to determine how to call the function as through it was declared with
a prototyp e The typ e need not b e known at compile time but only known typ e calls are e ciently compiled
Limitations
Structure typ e return values are not implemented
Passing of structures byvalue is not implemented
Here is an example which allocates a struct foo calls a foreign function to initialize it then returns a
Lisp vector of all the struct foo ob jects lled in by the foreign call
Allocate a foo on the stack
with alien f struct foo
Call some C function to fill in foo fields
alien funcall extern alien mangle foo function void foo
addr f
Find how many foos to use by getting the A field
let num slot f a
result make array num
Get a pointer to the array so that we don t have to keep extracting it
with alien a array struct foo addr slot f b
Loop over the first N elements and stash them in the result vector
dotimes i num
setf svref result i deref deref a i result
CHAPTER ALIEN OBJECTS
The def alien routine Macro
alien def alien routine name result typ e f aname atyp e style g Macro
This macro is a convenience for automatically generating Lisp interfaces to simple foreign functions The
primary feature is the parameter style sp eci cation which translates the C pass by reference idiom into additional
return values
name is usually a string external symb ol but may also b e a symb ol Lisp name or a list of the Lisp name and
the foreign name If only one name is sp eci ed the other is automatically derived see section page
result typ e is the Alien typ e of the return value Each remaining subform sp eci es an argumenttothe
foreign function aname is the symb ol name of the argument to the constructed function for documentation
and atyp e is the Alien typ e of corresp onding foreign argument The semantics of the actual call are the same as
for alien funcall page style should b e one of the following
in sp eci es that the argumentispassedbyvalue This is the default in arguments have no corresp onding
return value from the Lisp function
out sp eci es a pass by reference output value The typ e of the argumentmustbeapointer to a xed sized
ob ject such as an integer or p ointer out and in out cannot b e used with p ointers to arrays records or
functions An ob ject of the correct size is allocated and its address is passed to the foreign function When
the function returns the contents of this location are returned as one of the values of the Lisp function
copy is similar to in but the argument is copied to a pre allo cated ob ject and a p ointer to this ob ject is
passed to the foreign routine
in out is a combination of copy and out The argument is copied to a pre allocated ob ject and a p ointer
to this ob ject is passed to the foreign routine On return the contents of this lo cation is returned as an
additional value
Any e ciency critical foreign interface function should b e inline expanded by preceding def alien routine
with
declaim inline lisp name
In addition to avoiding the Lisp call overhead this allows p ointers word integers and oats to b e passed using
non descriptor represen tations avoiding consing see section page
def alien routine Example
Consider the C function cfoo with the following calling convention
cfoo a i
char str
char a update
int i out
Body of cfoo
which can b e describ ed by the following call to def alien routine
def alien routine cfoo void
str c string
a char in out
i int out
The Lisp function cfoo will havetwo arguments str and a andtwo return values a and i
CHAPTER ALIEN OBJECTS
Calling Lisp from C
Calling Lisp functions from C is sometimes p ossible but is rather hackish See call into lisp and funcall
funcall in the C startup co de The arguments must b e valid CMU CL ob ject descriptors e g xnums
must b e left shifted by See compiler generic objdef lisp or the derived le lisp internals h for
details of the ob ject represen tation Note that the garbage collector moves ob jects and won tbeableto xup
any references in C variables so either turn GC o or don t keep Lisp p ointers in C data unless they are to
statically allocated ob jects
Step by Step Alien Example
This section presents a complete example of an interface to a somewhat complicated C function This example
should giveafairlygoodideaofhow to get the e ect you want for almost any kind of C function Supp ose you
have the following C function whichyou want to b e able to call from Lisp in the le test c
struct c struct
int x
char s
struct c struct c function i s r a
int i
char s
struct c struct r
int a
int j
struct c struct r
printf i d n i
printf s s n s
printf r x d n r x
printf r s s n r s
for j j j printf a d d n j a j
r struct c struct malloc sizeof struct c struct
r x i
r s A C string
return r
It is p ossible to call this function from Lisp using the le test lisp whose contents is
Package test c call
in package TEST C CALL
use package ALIEN
use package C CALL
Define the record c struct in Lisp
def alien type nil
struct c struct
x int
s c string
Define the Lisp function interface to the C routine It returns a
pointer to a record of type c struct It accepts four parameters i an int s a pointer to a string r a pointer to a c struct
CHAPTER ALIEN OBJECTS
record and a a pointer to the array of ints
The INLINE declaration eliminates some efficiency notes about heap
allocation of Alien values
declaim inline c function
def alien routine c function
struct c struct
i int
s c string
r struct c struct
a array int
A function which sets up the parameters to the C function and
actually calls it
defun call cfun
with alien ar array int
c struct struct c struct
dotimes i Fill array
setf deref ar i i
setf slot c struct x
setf slot c struct s A Lisp String
with alien res struct c struct
c function Another Lisp String addr c struct ar
format t Returned from C function
multiple value prog
values slot res x
slot res s
Deallocate result after we are done using it
free alien res
To execute the ab ove example it is necessary to compile the C routine as follows
cc c test c
In order to enable incremental loading with some linkers you mayneedtosay
cc G c test c
Once the C co de has b een compiled you can start up Lisp and load it in
lisp
Lisp should start up with its normal prompt
Compile the Lisp file This step can be done separately You don t have
to recompile every time
compile file test lisp
Load the foreign object file to define the necessary symbols This must
be done before loading any code that refers to these symbols next block
of comments are actually the output of LOAD FOREIGN Different linkers
will give different warnings but some warning about redefining the code
size is typical
load foreign test o
Running library load foreign csh Loading object file
CHAPTER ALIEN OBJECTS
Parsing symbol table
Warning gp moved from x C C to x C
Warning end moved from x C to x C E
o k now load the compiled Lisp object file
load test
Now we can call the routine that sets up the parameters and calls the C
function
test c call call cfun
The C routine prints the following information to standard output
i
s Another Lisp string
r x
r s A Lisp string
a
a
a
a
a
a
a
a
a
a
Lisp prints out the following information
Returned from C function
Return values from the call to test c call call cfun
A C string
If any of the foreign functions do output they should not b e called from within Hemlock Dep ending on the
situation various strange b ehavior o ccurs Under X the output go es to the windowinwhich Lisp was started
on a terminal the output will overwrite the Hemlock screen image in a Hemlockslave standard output is
dev null by default so any output is discarded
Chapter
Interpro cess Communication under
LISP
Written by William Lott and Bill Chiles
CMU Common Lisp o ers a facilityforinterpro cess communication IPC on top of using Unix system calls
and the complications of that level of IPC There is a simple remote pro cedure call RPC package build on top
of TCP IP so ckets
The REMOTE Package
The remote package provides simple RPC facility including interfaces for creating servers connecting to already
existing servers and calling functions in other Lisp pro cesses The routines for establishing a connection b e
tween two pro cesses create request server and connect to remote server return wire structures A wire
maintains the curren t state of a connection and all the RPC forms require a wire to indicate where to send
requests
Connecting Servers and Clients
Before a client can connect to a server it must knowthenetwork address on whichtheserver accepts connections
Network addresses consist of a host address or name and a p ort numb er Host addresses are either a string of
the form VANCOUVER SLISP CS CMU EDU or a bit unsigned integer Port numb ers are bit unsigned integers
Note p ort in this context has nothing to do with Mach p orts and message passing
When a pro cess wants to receive connection requests that is b ecome a server it rst picks an integer to
use as the p ort Only one server Lisp or otherwise can use a given p ort numb er on a given machine at any
particular time This can b e an iterative pro cess to nd a free p ort picking an integer and calling create
request server This function signals an error if the c hosen p ort is unusable You will probably want to write
a lo op using handler case catching conditions of typ e error since this function do es not signal more sp eci c
conditions
wire create request server port optional on connect Function
create request server sets up the curren t Lisp to accept connections on the given p ort If p ort is un
available for any reason this signals an error When a client connects to this p ort the acceptance mechanism
makes a wire structure and invokes the on connect function Invoking this function has a couple purp oses and
on connect maybe nil in which case the system forego es invoking any function at connect time
The on connect function is b oth a ho ok that allows you access to the wire created by the acceptance mecha
nism and it con rms the connection This function takes two arguments the wire and the host address of the
connecting pro cess See the section on host addresses b elow When on connect is nil the request server allows
all connections When it is non nil the function returns twovalues whether to accept the connection and a
function the system should call when the connection terminates Either value maybe nil but when the rst
value is nil the acceptance mechanism destroys the wire
CHAPTER INTERPROCESS COMMUNICATION UNDER LISP
create request server returns an ob ject that destroy request server uses to terminate a connection
wire destroy request server server Function
destroy request server takes the result of create request server and terminates that server Any
existing connections remain intact but all additional connection attempts will fail
wire connect to remote server host p ort optional on death Function
connect to remote server attempts to connect to a remote server at the given p ort on host and returns a
wire structure if it is successful If on death is non nil it is a function the system invokes when this connection
terminates
Remote Evaluations
After the server and clienthave connected they each have a wire allowing function evaluation in the other
pro cess This RPC mechanism has three avors for side e ect only for a single value and for multiple values
Only a limited numb er of data typ es can b e sent across wires as arguments for remote function calls and
as return values integers inclusively less than bits in length symb ols lists and remote ob jects see section
page The system sends symb ols as two strings the package name and the symb ol name and if the
package do esn t exist remotely the remote pro cess signals an error The system ignores other slots of symb ols
Lists maybeany tree of the ab ovevalid data typ es To send other data typ es you must represen t them in
terms of these supp orted typ es For example you could use prin to string lo cally send the string and use
read from string remotely
wire remote wire fcal l specs g Macro
The remote macro arranges for the pro cess at the other end of wire to invokeeach of the functions in
the call sp ecs Tomake sure the system sends the remote evaluation requests over the wire you must call
wire force output
Eachofcall specs lo oks like a function call textually but it has some o dd constraints and semantics The
function p osition of the form must b e the symb olic name of a function remote evaluates each of the argument
subforms for each of the call sp ecs locally in the current context sending these values as the arguments for the
functions
Consider the following example
defun write remote string str
declare simple string str
wire remote wire
write string str
The value of str in the lo cal pro cess is passed over the wire with a request to invoke write string on the value
The system do es not exp ect to remotely evaluate str for a v alue in the remote pro cess
wire wire force output wire Function
wire force output ushes all internal bu ers asso ciated with wire sending the remote requests This is
necessary after a call to remote
wire remote value wire call sp ec Macro
The remote value macro is similar to the remote macro remote value only takes one call sp ec and it
returns the value returned by the function call in the remote pro cess The value must b e a valid typ e the system
can send over a wire and there is no need to call wire force output in conjunction with this interface
If clientunwinds past the call to remote value the server continues running but the system ignores the
value the server sends back
If the server unwinds past the remotely requested call instead of returning normally remote value returns
twovalues nil and t Otherwise this returns the result of the remote evaluation and nil
CHAPTER INTERPROCESS COMMUNICATION UNDER LISP
wire remote value bind wire fvariable g remote form flocal forms g Macro
remote value bind is similar to multiple value bind except the values b ound come from remote form s
evaluation in the remote pro cess The local forms execute in an implicit progn
If the clientunwinds past the call to remote value bind the server continues running but the system
ignores the values the server sends back
If the server unwinds past the remotely requested call instead of returning normally the local forms never
execute and remote value bind returns nil
Remote Ob jects
The wire mechanism only directly supp orts a limited numb er of data typ es for transmission as arguments
for remote function calls and as return values integers inclusively less than bits in length symb ols lists
Sometimes it is useful to allow remote pro cesses to refer to local data structures without allowing the remote
pro cess to op erate on the data Wehave remote ob jects to supp ort this without the need to represen t the data
structure in terms of the ab ove data typ es to send the represen tation to the remote pro cess to deco de the
represen tation to later enco de it again and to send it back along the wire
You can convert any Lisp ob ject into a remote ob ject When you send a remote ob ject along a wire the
system simply sends a unique token for it In the remote pro cess the system lo oks up the token and returns a
remote ob ject for the token When the remote pro cess needs to refer to the original Lisp ob ject as an argument
to a remote call back or as a return value it uses the remote ob ject it has which the system converts to the
unique token sending that along the wire to the originating pro cess Up on receipt in the rst pro cess the system
converts the token back to the same eq remote ob ject
wire make remote object ob ject Function
make remote object returns a remote ob ject that has ob ject as its value The remote ob ject can b e passed
across wires just like the directly supp orted wire data typ es
wire remote object p ob ject Function
The function remote object p returns t if ob ject is a remote ob ject and nil otherwise
wire remote object local p remote Function
The function remote object local p returns t if remote refers to an ob ject in the local pro cess This is
can only o ccur if the lo cal pro cess created remote with make remote object
wire remote object eq ob j ob j Function
The function remote object eq returns t if ob j and ob j refer to the same eq lisp ob ject regardless of
which pro cess created the remote ob jects
wire remote object value remote Function
This function returns the original ob ject used to create the given remote ob ject It is an error if some other
pro cess originally created the remote ob ject
Function wire forget remote translation ob ject
This function removes the information and storage necessary to translate remote ob jects backinto ob ject
so the next gc can reclaim the memory You should use this when you no longer exp ect to receive references to
ob ject If some remote pro cess do es send a reference to ob ject remote object value signals an error
CHAPTER INTERPROCESS COMMUNICATION UNDER LISP
Host Addresses
The op erating system maintains a database of all the valid host addresses You can use this database to convert
between host names and addresses and vice versa
ext lookup host entry host Function
lookup host entry searches the database for the given host and returns a host entry structure for it If it
fails to nd host in the database it returns nil Host is either the address as an integer or the name as a
string of the desired host
ext host entry name host entry Function
ext host entry aliases host entry Function
ext host entry addr list host entry Function
ext host entry addr host entry Function
host entry name host entry aliases and host entry addr list each return the indicated slot from
the host entry structure host entry addr returns the primary rst address from the list returned by host
entry addr list
The WIRE Package
The wire package provides for sending data along wires The remote package sits on top of this package All
data sent with a giv en output routine must b e read in the remote pro cess with the complementary fetching
routine For example if you send so a string with wire output string the remote pro cess must knowtouse
wire get string Toavoid rigid data transfers and complicated co de the interface supp orts sending tagged
data With tagged data the system sends a tag announcing the typ e of the next data and the remote system
takes care of fetching the appropriate typ e
When using interfaces at the wire level instead of the RPC level the remote pro cess must read everything
sent by these routines If the remote pro cess leaves any input on the wire it will later mistake the data for an
RPC request causing unknown lossage
Untagged Data
When using these routines b oth ends of the wire knowexactlywhattyp es are coming and going and in what
order This data is restricted to the following typ es
bit unsigned bytes
bit unsigned bytes
bit integers
simple strings less than in length
wire wire output byte wire byte Function
wire wire get byte wire Function
wire wire output number wire number Function
wire wire get number wire optional signed Function
wire wire output string wire string Function
wire wire get string Function wire
These functions either output or input an ob ject of the sp eci ed data typ e When you use any of these output
routines to send data across the wire you must use the corresp onding input routine interpret the data
CHAPTER INTERPROCESS COMMUNICATION UNDER LISP
Tagged Data
When using these routines the system automatically transmits and interprets the tags for you so b oth ends can
gure out what kind of data transfers o ccur Sending tagged data allows a greater varietyofdatatyp es integers
inclusively less than bits in length symb ols lists and remote ob jects see section page The system
sends symb ols as two strings the package name and the symb ol name and if the package do esn t exist remotely
the remote pro cess signals an error The system ignores other slots of symb ols Lists maybeany tree of the
ab ovevalid data typ es To send other data typ es you must represen t them in terms of these supp orted typ es
For example you could use prin to string locally send the string and use read from string remotely
wire wire output object wire ob ject optional cache it Function
wire wire get object wire Function
The function wire output object sends ob ject over wire preceded by a tag indicating its typ e
If cache it is non nil this function only sends ob ject the rst time it gets ob ject Each end of the wire
asso ciates a token with ob ject similar to remote ob jects allowing you to send the ob ject more e ciently on
successiv e transmissions Cache it defaults to t for symb ols and nil for other typ es Since the RPC level requires
function names a high level proto col based on a set of function calls saves time in sending the functions names
rep eatedly
The function wire get object reads the results of wire output object and returns that ob ject
Making Your Own Wires
You can create wires manually in addition to the remote package s interface creating them for you To create a
wire you need a Unix le descriptor Ifyou are unfamiliar with Unix le descriptors see section of the Unix
manual pages
wire make wire descriptor Function
The function make wire creates a new wire when supplied with the le descriptor to use for the underlying
I O op erations
wire wire p ob ject Function
This function returns t if ob ject is indeed a wire nil otherwise
wire wire fd wire Function
This function returns the le descriptor used bythewire
Out Of Band Data
The TCP IP proto col allows users to send data asynchronously otherwise known as out of band data When
using this feature the op erating system interrupts the receiving pro cess if this pro cess has chosen to b e noti ed
ab out out of band data The receiver can grab this input without a ecting any information curren tly queued
on the so cket Therefore you can use this without interfering with any current activity due to other wire and
remote interfaces
Unfortunately most implementations of TCP IP are broken so use of out of band data is limited for safety
reasons Y ou can only reliably send one character at a time
This routines in this section provide a mechanism for establishing handlers for out of band characters and
for sending them out of band These all take a Unix le descriptor instead of a wire but you can fetch a wire s
le descriptor with wire fd
wire add oob handler fd char handler Function
The function add oob handler arranges for handler to b e called whenever char shows up as out of band
data on the le descriptor fd
CHAPTER INTERPROCESS COMMUNICATION UNDER LISP
wire remove oob handler fd char Function
This function removes the handler for the character char on the le descriptor fd
wire remove all oob handlers fd Function
This function removes all handlers for the le descriptor fd
wire send character out of band fd char Function
This function Sends the character char down the le descriptor fd out of band
Chapter
Debugger Programmer s Interface
The debugger programmers interface is exp orted from from the DEBUG INTERNALS or DI package This is a
CMU extension that allows debugging to ols to b e written without detailed knowledge of the compiler or run time
system
Some of the interface routines take a co de lo cation as an argument As describ ed in the section on co de
lo cations some co de locations are unknown When a function calls for a basic co de location it takes either
typ e but when it sp eci cally names the argument co de location the routine will signal an error if you giveitan
unknown co de lo cation
DI Exceptional Conditions
Some of these op erations fail dep ending on the availability debugging information In the most severe case when
someone saved a Lisp image stripping all debugging data structures no op erations are valid In this case even
backtracing and nding frames is impossible Some interfaces can simply return values indicating the lackof
information or their return values are naturally meaningful in light missing data Other routines as do cumented
b elow will signal serious conditions when they discover awkward situations This interface do es not provide
for programs to detect these situations other than by calling a routine that detects them and signals a condition
These are serious conditions b ecause the program using the interface must handle them b efore it can correctly
continue execution These debugging conditions are not errors since it is no fault of the programmers that the
conditions o ccur
Debug conditions
The debug internals interface signals conditions when it can t adhere to its contract These are serious conditions
b ecause the program using the interface must handle them b efore it can correctly continue execution These
debugging conditions are not errors since it is no fault of the programmers that the conditions o ccur The
interface do es not provide for programs to detect these situations other than calling a routine that detects them
and signals a condition
debug condition Condition
This condition inherits from serious condition and all debug conditions inherit from this These
must b e handled but they are not programmer errors
no debug info Condition
This condition indicates there is absolutely no debugging information available
no debug function returns Condition
This condition indicates the system cannot return values from a frame since its debug function
lacks debug information details ab out returning values
no debug blocks Condition
This condition indicates that a function was not compiled with debug blo ck information but this
information is necessary necessary for some requested op eration
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
no debug variables Condition
Similar to no debug blocks except that variable information was requested
lambda list unavailable Condition
Similar to no debug blocks except that lamb da list information was requested
invalid value Condition
This condition indicates a debug variable has invalid or unknown value in a particular frame
ambiguous variable name Condition
This condition indicates a user supplied debug variable name identi es more than one valid variable
in a particular frame
Debug errors
These are programmer errors resulting from misuse of the debugging to ols programmers interface You could
haveavoided an o ccurrence of one of these by using some routine to check the use of the routine generating the
error
debug error Condition
This condition inherits from error and all user programming errors inherit from this condition
unhandled condition Condition
This error results from a signalled debug condition o ccurring without anyone handling it
unknown code location Condition
This error indicates the invalid use of an unknown co de lo cation
unknown debug variable Condition
This error indicates an attempt to use a debug variable in conjunction with an inappropriate debug
function for example checking the variable s validity using a co de lo cation in the wrong debug
function will signal this error
frame function mismatch Condition
This error indicates you called a function returned by preprocess for eval on a frame other than
the one for which the function had b een prepared
Debug variables
Debug variables represen t the constant information ab out where the system stores argument and local variable
values The system uniquely iden ti es with an integer every instance of a variable with a particular name and
package To access a value you must supply the frame along with the debug variable since these are particular
to a function not every instance of a variable on the stack
debug variable name debug variable Function
This function returns the name of the debug variable The name is the name of the symb ol used as an
identi er when writing the co de
debug variable package debug variable Function
This function returns the package name of the debug variable This is the package name of the symbol used
as an identi er when writing the co de
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
debug variable symbol debug variable Function
This function returns the symbol from interning debug variable name in the package named by debug
variable package
debug variable id debug variable Function
This function returns the integer that makes debug variable s name and package name unique with resp ect
to other debug variable s in the same function
debug variable validity debug variable basic co de lo cation Function
This function returns three values re ecting the validityofdebug variable s value at basic co de lo cation
valid The value is known to b e available
invalid The value is known to b e unavailable
unknown The value s availabilityisunknown
debug variable value debug variable frame Function
This function returns the value stored for debug variable in frame The value maybeinvalid This is
SETF able
debug variable valid value debug variable frame Function
This function returns the value stored for debug variable in frame Ifthevalue is not valid then this
signals an invalid value error
Frames
Frames describ e a particular call on the stack for a particular thread This is the environment for name resolution
getting arguments and lo cals and returning values The stack conceptually growsup sothetopofthestackis
the most recently called function
top frame frame down frame up and frame debug function can only fail when there is absolutely no
debug information available This can only happ en when someone saved a Lisp image sp ecifying that the system
dump all debugging data
top frame Function
This function never returns the frame for itself always the frame b efore calling top frame
frame down frame Function
This returns the frame immediately b elow frame on the stack When frame is the b ottom of the stack this
returns nil
frame up frame Function
This returns the frame immediately ab ove frame on the stack When frame is the top of the stack this
returns nil
frame debug function frame Function
This function returns the debug function for the function whose call frame represen ts
frame code location frame Function
This function returns the co de lo cation where frame s debug function will continue running when program
execution returns to frame If someone interrupted this frame the result could b e an unknown co de lo cation
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
frame catches frame Function
This function returns an a list for all active catches in frame mapping catch tags to the co de lo cations at
which the catch re enters
eval in frame frame form Function
This evaluates form in frame s environment This can signal several di erent debug conditions since its
success relies on a variety of inexact debug information invalid value ambiguous variable name frame
function mismatch See also preprocess for eval page
Debug functions
Debug functions represen t the static information ab out a function determined at compile time argument and
variable storage their lifetime information etc The debug function also contains all the debug blo cks repre
senting basic blocks of co de and these contains information ab out sp eci c co de lo cations in a debug function
do debug function blocks blo ck var debug function result form fform g Macro
This executes the forms in a context with blo ck var b ound to each debug blo ckindebug function successiv ely
Result form is an optional form to execute for a return value and do debug function blocks returns nilif
there is no result form This signals a no debug blocks condition when the debug function lacks debug block
information
debug function lambda list debug function Function
This function returns a list represen ting the lamb da list for debug function The list has the following
structure
required var required var
optional var suppliedp var
optional var
rest var rest var
keyword keyword symbol var suppliedp var
keyword keyword symbol var
Each varn isadebug variable however the symbol deleted app ears instead whenever the argumentremains
unreferenced throughout debug function
If there is no lamb da list information this signals a lambda list unavailable condition
do debug function variables var debug function result fform g Macro
This macro executes each form in a context with var b ound to each debug variable in debug function This
returns the value of executing result defaults to nil This may iterate over only some of debug function s
variables or none dep ending on debug p olicy for example p ossibly the compilation only preserv ed argument
information
debug variable info available debug function Function
This function returns whether there is anyvariable information for debug function This is useful for distin
guishing whether there were no lo cals in a function or whether there was no variable information For example
if do debug functi on variables executes its forms zero times then you can use this function to determine the reason
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
debug function symbol variables debug function symbol Function
This function returns a list of debug variables in debug function having the same name and package as
symbol Ifsymbol is uninterned then this returns a list of debug variables without package names and with
the same name as symbol The result of this function is limited to the availabilityofvariable information in
debug function for example p ossibly debug function only knows ab out its arguments
ambiguous debug variables debug function name pre x string Function
This function returns a list of debug variables in debug function whose names contain name pre x string as
an initial substring The result of this function is limited to the availabilityofvariable information in debug
function for example p ossibly debug function only knows about its arguments
preprocess for eval form basic co de lo cation Function
This function returns a function of one argumentthatevaluates form in the lexical context of basic co de
lo cation This allows e cient rep eated evaluation of form at a certain place in a function which could b e useful
for conditional breaking This signals a no debug variables condition when the co de location s debug function
has no debug variable information available The returned function tak es a frame as an argument See also
eval in frame page
function debug function function Function
This function returns a debug function that represen ts debug information for function
debug function kind debug function Function
This function returns the kind of function debug function represen ts The value is one of the following
optional This kind of function is an entry p oint to an ordinary function It handles optional defaulting
parsing keywords etc
external This kind of function is an entry p oint to an ordinary function It checks argumentvalues and count
and calls the de ned function
top level This kind of function executes one or more random top level forms from a le
cleanup This kind of function represen ts the cleanup forms in an unwind protect
nil This kind of function is not one of the ab ove that is it is not sp ecially marked in anyway
debug function function debug function Function
This function returns the Common Lisp function asso ciated with the debug function This returns nilif the
function is unavailable or is non existent as a user callable function ob ject
debug function name debug function Function
This function returns the name of the function represen ted by debug function This may b e a string or a
cons do not assume it is a symbol
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
Debug blo cks
Debug blo cks contain information p ertinent to a sp eci c range of co de in a debug function
do debug block locations co de var debug blo ck result fform g Macro
This macro executes each form inacontext with co de var b ound to each co de location in debug blo ck This
returns the value of executing result defaults to nil
debug block successors debug blo ck Function
This function returns the list of p ossible co de lo cations where execution maycontinue when the basic block
represen ted by debug blo ck completes its execution
debug block elsewhere p debug blo ck Function
This function returns whether debug blo ck represen ts elsewhere co de This is co de the compiler has moved
out of a function s co de sequence for optimization reasons Co de locations in these blocks are unsuitable for
stepping to ols and the rst co de lo cation has nothing to do with a normal starting location for the block
Breakp oints
A breakp oint represen ts a function the system calls with the curren t frame when execution passes a certain
co de lo cation A break p oint is active or inactive indep endent of its existence They also have an extra slot for
users to tag the breakp oint with information
make breakpoint ho ok function what key kind info function end cookie Function
This function creates and returns a breakp oint When program execution encounters the breakp oint the
system calls ho ok function Ho ok function takes the curren t frame for the function in which the program is
running and the breakp oint ob ject
what and kind determine where in a function the system invokes ho ok function what is either a co de lo cation
or a debug function kind is one of code location function start or function end Since the starts and
ends of functions maynothave co de lo cations represen ting them designate these places by supplying what as a
debug function and kind indicating the function start or function end When what is a debug function
and kind is function end then ho ok function must taketwo additional arguments a list of values returned
by the function and a function end co okie
info is information supplied by and used by the user
function end co okie is a function To implement function end breakp oints the system uses starter breakp oints
to establish the function end breakp oint for each invo cation of the function Up on each entry the system creates
a unique co okie to identify the invo cation and when the user supplies a function for this argument the system
invokes it on the co okie The system later invokes the function end breakp oint ho ok on the same co okie The
user maysave the co okie when passed to the function end co okie function for later comparison in the ho ok
function
This signals an error if what is an unknown co de lo cation
Note Breakp oints in interpreted co de or byte compiled co de are not implemented F unction end breakp oints
are not implemented for compiled functions that use the known local return convention e g for block compiled
or self recursive functions
activate breakpoint breakp oint Function
This function causes the system to invokethe breakp oint s ho ok function until the next call to deactivate
breakpoint or delete breakpoint The system invokes breakp oint ho ok functions in the opp osite order that
you activate them
deactivate breakpoint breakp oint Function
This function stops the system from invoking the breakp oint s ho ok function
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
breakpoint active p breakp oint Function
This returns whether breakp oint is currently active
breakpoint hook function breakp oint Function
This function returns the breakp oint s function the system calls when execution encounters breakp oint and
it is active This is SETF able
breakpoint info breakp oint Function
This function returns breakp oint s information supplied by the user This is SETF able
breakpoint kind breakp oint Function
This function returns the breakp oint s kind sp eci cation
breakpoint what breakp oint Function
This function returns the breakp oint s what sp eci cation
delete breakpoint breakp oint Function
This function frees system storage and removes computational overhead asso ciated with breakp oint After
calling this breakp oint is useless and can never b ecome active again
Co de lo cations
Co de lo cations represen t places in functions where the system has correct information ab out the function s
environment and where interesting op erations can o ccur asking for a local variable s value setting breakp oin ts
evaluating forms within the function s environment etc
Sometimes the interface returns unknown co de lo cations These represen t places in functions but there is
no debug information asso ciated with them Some op erations accept these since they may succeed even with
missing debug data These op erations argumentisnamedbasic co de lo cation indicating they take known and
unknown co de lo cations If an op eration names its argument co de location and you supply an unknown one
it will signal an error For example frame code location may return an unknown co de lo cation if someone
interrupted Lisp in the given frame The system knows where execution will continue but this place in the co de
may not b e a place for which the compiler dumped debug information
code location debug function basic co de lo cation Function
This function returns the debug function represen ting information ab out the function corresp onding to the
co de lo cation
code location debug block basic co de lo cation Function
This function returns the debug block containing co de lo cation if it is available Some debug p olicies inhibit
debug blo ck information and if none is available then this signals a no debug blocks condition
code location top level form offset co de lo cation Function
This function returns the numb er of top level forms b efore the one containing co de lo cation as seen bythe
compiler in some compilation unit A compilation unit is not necessarily a single le see the section on debug
sources
code location form number co de lo cation Function
This function returns the numb er of the form corresp onding to co de lo cation The form numb er is derived
alking the subforms of a top level form in depth rst order While walking the top level form count one in byw
depth rst order for each subform that is a cons See form number translations page
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
code location debug source co de lo cation Function
This function returns co de lo cation s debug source
code location unknown p basic co de lo cation Function
This function returns whether basic co de lo cation is unknown It returns nilwhen the co de lo cation is known
code location co de lo cation co de lo cation Function
This function returns whether the two co de lo cations are the same
Debug sources
Debug sources represen t how to get back the source for some co de The source is either a le compile file or
load a lamb da expression compile defun defmacro or a stream something particular to CMU Common
Lisp compile from stream
When compiling a source the compiler counts each top level form it pro cesses but when the compiler handles
multiple les as one blo ck compilation the top level form count continues past le b oundaries Therefore code
location top level form offset returns an o set that do es not always start at zero for the co de lo cation s
debug source The o set into a particular source is code location top level form offset minus debug
source root number
Inside a top level form a co de location s form numb er indicates the subform corresp onding to the co de
lo cation
debug source from debug source Function
This function returns an indication of the typ e of source The following are the p ossible values
file from a le obtained by compile file if compiled
lisp from Lisp obtained by compile if compiled
stream from a non le stream CMU Common Lisp supp orts compile from stream
debug source name debug source Function
This function returns the actual source in some sense represen ted by debug source which is related to debug
source from
file the pathname of the le
lisp a lamb da expression
stream some descriptive string that s otherwise useless
debug source created debug source Function
This function returns the universal time someone created the source This maybe nilif it is unavailable
debug source compiled debug source Function
This function returns the time someone compiled the source This is nilif the source is uncompiled
debug source root number debug source Function
This returns the numb er of top level forms pro cessed by the compiler b efore compiling this source If this
source is uncompiled this is zero This maybezeroeven if the source is compiled since the rst form in the rst
le compiled in one compilation for example must have a ro ot numb er of zero the compiler sawnoother
top level forms b efore it
CHAPTER DEBUGGER PROGRAMMER S INTERFACE
Source Translation Utilities
These two functions provide a mechanism for converting the rather obscure but highly compact represen tation
of source locations into an actual source form
debug source start positions debug source Function
This function returns the le p osition of each top level form a vector if debug source is from a file If
debug source from is lisp or stream orthe leisbyte compiled then the result is nil
form number translations form tlf number Function
This function returns a table mapping form numb ers see code location form number to source paths A
source path indicates a descent into the top level form form going directly to the subform corresp onding to a
form numb er Tlf number is the top level form number of form
source path context form path context Function
This function returns the subform of form indicated by the source path Form is a top level form and path
is a source path into it Context is the numb er of enclosing forms to return instead of directly returning the
source path form When context is non zero the form returned contains a marker HERE immediately
b efore the form indicated by path
Function Index
A debug source compil ed
debug source created
activate breakp oint
debug source from
add fd handler
debug source name
add o ob handler
debug source ro ot numb er
add xwindo w ob ject
debug source start p ositions
addr
debug variable id
alien funcal l
debug variable info a vailable
alien sap
debug variable name
ambiguous debug v ariables
debug variable pac kage
ambiguous les
debug variable symbol
debug variable v alid value
B
debug variable v alidit y
break
debug variable value
breakp oint active p
def alien routine
breakp oint ho ok function
def alien t yp e
breakp oint info
def alien variable
breakp oint kind
def source context
breakp oint what
default directory
default in terrupt
C
defstruct
cast
deftyp e
ceiling
defun
clear searc h list
delete breakp oint
cmd switch name
deref
cmd switch value
describe
cmd switch words
destroy request server
co de lo cation debug blo ck
directory
co de lo cation debug function
disable clx ev ent handling
co de lo cation debug source
do debug blo ck lo cations
co de lo cation form number
do debug function bl o cks
co de lo cation top lev el form o set
do debug function v ariables
co de lo cation unkno wn p
co de lo cation
E
compile
ed
compile le
enable clx ev ent handling
compile from stream
enable in terrupt
complete le
encapsulate
connect to remote serv er
encapsulated p
create request serv er
enumerate search list
error
D
eval in frame
deactivate breakp oint
extern alien
debug
F
debug blo c k elsewhere p
debug blo ck successors
fd stream fd
debug function functi on
fd stream p
debug function ki nd
fde nition
debug function l am b da list
le writable
debug function name
et
debug function sym b ol variables
oat digits
FUNCTION INDEX
oat in nit y p make fd stream
oat nan p make ob ject set
oat normalized p make remote ob ject
oat precision make wire
oat sign multiple v alue bind
oat trapping nan p
oor
O
ush displ a y events
ob ject set event handler
forget remote translation
ob ject set op eration
form numb er translations
op en clx displa y
format deco ded time
format univ ersal time
P
frame catches
parse time
frame co de lo cation
prepro cess for eval
frame debug function
prin t directory
frame down
pro cess aliv e p
frame up
pro cess close
free alien
pro cess core dump ed
function
pro cess error
function debug functi on
pro cess exit co de
process input
G
process kill
gc
pro cess output
gc o
pro cess p
gc on
process pid
get bytes consed
pro cess plist
get oating p oint mo des
pro cess pty
get internal run time
pro cess status
get unix error msg
pro cess status ho ok
gr bind
pro cess wait
gr call
pro le
gr call
gr error
R
remote
H
remote ob ject eq
host entry addr
remote ob ject lo cal p
host entry addr list
remote object p
host entry aliases
remote ob ject value
host entry name
remote value
remote value bind
I
remove all o ob handlers
if
remove fd handler
ignore in terrupt
remove o ob handler
insp ect
report time
int sap
required argumen t
invalidate descriptor
reset time
iterate
round
run program
L
lab els S
let
sap
lisp con trol panel
sap alien
load
sap int
load foreign
sap ref
load logica l pa thname defaul ts
sap ref
lo okup host en try
sap ref
save lisp
M
search list
search list de ned p make alien
send character out of band make breakp oint
FUNCTION INDEX
serve all events
serve event
set oating p oint mo des
signed sap ref
signed sap ref
signed sap ref
slot
source path con text
T
the
time
top frame
trace
truncate
U
unencapsula te
unix namestrin g
unpro le
untrace
V
var
W
wait until fd usable
wire fd
wire force output
wire get byte
wire get number
wire get ob ject
wire get string
wire output b yte
wire output n umber
wire output ob ject
wire output string
wire p
with alien
with clx ev ent handling
with compilati on un it
with enabled in terrupts
with fd handler
with in terrupts
without hemlo ck
without in terrupts
Variable Index
A L
after gc ho oks load if source new er
load ob ject typ es
B load source t yp es
long oat negativ e in nit y
b efore gc ho oks
long oat p ositiv e in nit y
blo ck compile default
byte compile default
M
byte compile top lev el
bytes consed b etween gcs max trace inden tation
C R
command line stri ngs read default oat format
command line switc hes
command line uti l it y name S
command line w ords
short oat negativ e in nit y
compile prin t
short oat p ositive in nit y
compile progress
single oat negati v e in nit y
compile v erb ose
single oat p ositiv e in nit y
stderr
D
stdin
debug prin t length stdout
debug prin t level
deriv e function typ es T
describ e inden tation
timed functions
describ e level
trace output
describ e print length
traced function lis t
describ e print level
tty
double oat negati v e in nit y
double oat p ositiv e in nit y
U
unde ned w arning limit
E
e ciency note cost threshol d
e ciency note li mit
enclosing sou rce cuto
error prin t length
error prin t level
G
gc inhibi t ho ok
gc notify after
gc notify b efore
gc run time
gc verb ose
I
ignore extra close pa ren theses
interface style
internal time units p er second
Typ e Index
N
no debug blo cks
no debug function return s
A
no debug info
no debug variables
ambiguous variable name
null
and
array
O
or
B
base character
P
bign um
b o olean pathname
C
S
c string
serious conditi on
signed
D
single oat
string char
debug conditi on
struct
debug error
style warning
divide b y zero
symb ol
double oat
system area p ointer
E
U
enum
unhandled con di tion
error
union
F unkno wn co de lo cation
unkno wn debug variable
xnum
unsigned
oating p oint over ow
oating p oint under ow
V
frame function mismatc h
ftyp e
void
function
W
H
warning
hash table
I
integer
invalid value
L
lamb da list una vailable
list
M
member
Concept Index
A
actual source
advising
aliens
argumen t syntax
e ciency
arithmetic
generic
arithmetic typ e inference
array typ es
sp ecialized
arrays
e ciency of
assembly listing
availabili t y of debug variables
B
b enchmarking techniques
bign ums
bit vectors
e ciency of
blo ck
basic
blo ck compilation
debugger implication s
start lo cation
byte co ded compilation
C
call
inline
local
numeric op erands
canonicali zati on of types
characters
clean up
stack frame kind
closures
compatibil it y with other Lisps
compilation
blo ck
units
whyto
compilation sp eed optimization qualit y
compile time typ e errors
compile le
blo ck compilation arguments
compiler error messages
compiler error severity
compiler p olicy
CONCEPT INDEX
compiling
complemen ted typ e checks
Concept Index
conditional typ e inference
consing
overhead of
constan t folding
constan t function declaration
context sensitiv e declarations
continuations
implicit representation
control optimization
CPU time
interpretation of
D
dead co de elimination
debug optimization qualit y
debug variables
debugger
declarations
optimize interface
optimize
blo ck compilation
context sensitiv e
defstruct typ es
derivation of typ es
descriptor representations
forcing of
descriptors
ob ject
dynamic typ e inference
E
e ciency
general hints
e ciency notes
for representation
verb osity
of argument syntax
of memory use
of numeric variables
ofobjects
of typ e checking
empty typ e
the
encapsulatio n
end blo ck declaration
entry p oints
external
equivalence of types
error messages
compiler
verb osity
errors
result typ e of
run time
evaluation
debugger
existing programs
to run
expansion
inline
external entry p oints
CONCEPT INDEX
stack frame kind
F
xnums
oating p oint e ciency
folding
constant
frames
stack
free
C function
freeze typ e declaration
function call
inline
local
Function Index
function
names
tracing
typ e inference
types
G
garbage collection
generic arithmetic
H
hash tables
e ciency of
I
implici t continuation representation IR
inference of types
inhibit w arnings optimization qualit y
inline expansion
interpretation of run time
interrupts
K
keyword argument e ciency
L
let optimization
listing les
trace
lists
e ciency of
lo cal call
numeric op erands
return values
typ e inference
lo cations
unknown
logical pathnames
M
macro expansion
errors during
mallo c
C function
mapping
e ciency of
CONCEPT INDEX
mayb e inline declaration
member typ es
memory allo cation
multiple value optimization
N
names
function
NIL type
non descriptor representations
notes
e ciency
numb ers in lo cal call
numeric
op eration e ciency
typ e inference
types
O
ob ject representation
ob ject representation e ciency notes
ob ject sets
op en co ding
op eration sp eci c typ e inference
optimization
control
function call
let
multiple value
typ e check
optimize declaration
optimize in terface declaration
optional
stack frame kind
or union typ es
original source
P
p ointers
p olicy
compiler
debugger
precise typ e checking
pro cessing path
pro ling
R
read errors
compiler
recording of inline expansions
recursion
self
tail
represen tation e ciency notes
ob ject
rest argumen t e ciency
return values
lo cal call
run time
interpretation of
S
safety optimization qualit y
CONCEPT INDEX
search lists
semi inli ne expansion
severity of compiler errors
source lo cation prin ting
debugger
source to source transformation
space optimization
space optimization qualit y
sp ecialized arraytypes
sp ecialized structure slots
sp eed optimization qualit y
stack frames
stack numb ers
start blo ck declaration
static functions
strings
structure types
e ciency of
numeric slots
style recommendations
T
tail recursion
time formatting
time parsing
timing
trace les
tracing
transformation
source to source
tuning
typ e checking
at compile time
e ciency of
optimization
precise
weakened
typ e declarations
variable
Typ e Index
typ e inference
dynamic
typ es
alien
equivalence
foreign language
function
in python
numeric
p ortabilit y
restrictions on
sp ecialized array
structure
uncertainty
U
uncertain tyoftypes
unde ned warnings
union or typ es
unix interrupts
pathnames
unkno wn co de lo cations
CONCEPT INDEX
unreachable co de deletion
unused expression elimination
V
validit y of debug variables
values declaration
Variable Index
variables
debugger access
non descriptor
vectors
e ciency of
verb osity
of e ciency notes
of error messages
Virtual Machine VM or IR representation
W
weakened typ e checking
word integers