Writing a Best-Effort Portable Code Walker in Common Lisp
Total Page:16
File Type:pdf, Size:1020Kb
Writing a best-effort portable code walker in Common Lisp Michael Raskin∗ LaBRI, University of Bordeaux [email protected] ABSTRACT ACM Reference format: One of the powerful features of the Lisp language family is Michael Raskin. 2017. Writing a best-effort portable code walker possibility to extend the language using macros. Some of in Common Lisp. In Proceedings of 10th European Lisp Simpo- sium, Vrije Universiteit Brussel, Belgium, April 2017 (ELS2017), possible extensions would benefit from a code walker, i.e. a 8 pages. library for processing code that keeps track of the status DOI: 10.5281/zenodo.3254669 of different part of code, for their implementation. But in practice code walking is generally avoided. In this paper, we study facilities useful to code walkers 1 INTRODUCTION provided by \Common Lisp: the Language" (2nd edition) and the Common Lisp standard. We will show that the Much of the power of Common Lisp comes from its extensibil- features described in the standard are not sufficient to write ity. Abstractions that cannot be expressed by functions can a fully portable code walker. still be expressed by macros; actually, many of the features One of the problems is related to a powerful but rarely described in the standard must be implemented as macros. discussed feature. The macrolet special form allows a macro Whilst macros are powerful on their own, some ways of function to pass information easily to other macro invocations extending Lisp would benefit from using higher-level code inside the lexical scope of the expansion. transformation abstractions. For many such abstractions the Another problem for code analysis is related to the usage of most natural way to implement them includes code walk- non-standard special forms in expansions of standard macros. ing, i.e. enumeration of all the subforms of a piece of code, We review the handling of defun by popular free software identification of function calls, variable bindings etc. among Common Lisp implementations. these subforms, and application of some code transforma- We also survey the abilities and limitations of the avail- tions. Unfortunately, code walking is complicated and the able code walking and recursive macro expansion libraries. libraries to perform it are non-portable. Even portable im- Some examples of apparently-conforming code that exhibit plementations of recursive macro expansion, also known as avoidable limitations of the portable code walking tools are macroexpand-all, are missing. provided. Guy Steele writes (in the second edition of the book \Com- We present a new attempt to implement a portable best- mon Lisp: the Language" [3]) that Common Lisp implemen- effort code walker for Common Lisp called Agnostic Lizard. tations are expected to provide all the functionality needed for code walking code analysis tools. The version of the CCS CONCEPTS language described in the book includes support for changing and inspecting the lexical environment objects; the book •Software and its engineering →Macro languages; Soft- explicitly recommends to use macroexpand on the macros in ware testing and debugging; the language core. Lexical environment objects are defined by the Common KEYWORDS Lisp standard [2] are more opaque. Many Common Lisp code walker, macro expansion, code transformation implementations do include some functions for environment inspection and manipulation, and even some code walking function. Unfortunately, both the naming and the feature ∗The author acknowledges support from the Danish National Research set vary between implementations. Foundation and The National Science Foundation of China (under the grant 61361136003) for the Sino-Danish Center for the Theory We will show that (unlike CL:tL2) the Common Lisp stan- of Interactive Computation. This work was supported by the French dard does not allow to implement a portable code walker cor- National Research Agency (ANR project GraphEn / ANR-15-CE40- rectly. We suggest an approach that approximates the desired 0009). functionality fairly well and remains implementation-agnostic even when doing the implementation-specific workarounds. Permission to make digital or hard copies of part or all of this work We present an implementation of this approach, a library for personal or classroom use is granted without fee provided that called Agnostic Lizard. copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. 2 RELATED WORK For all other uses, contact the owner/author(s). ELS2017, Vrije Universiteit Brussel, Belgium 2.1 Portable tools © 2017 Copyright held by the owner/author(s). 978-x-xxxx-xxxx- x/YY/MM. $15.00 The iterate library [4] provides an alternative iteration DOI: 10.5281/zenodo.3254669 construct also called iterate. It is more flexible than the ELS2017, April 2017, Vrije Universiteit Brussel, Belgium Michael Raskin standard loop macro, and it uses code walking for implemen- the implementation name). Even if two Common Lisp imple- tation. It doesn't work exactly as expected for some code, mentations are closely related and have the same names for though. The code snippet the environment processing functions, support for these two (iterate:iterate implementations has to be added separately. This limitation (for x :from 1 :to 3) makes some of the implementation-dependent tools not work (flet on some newer implementations (such as Clasp [15]) even ((f (y) (collect y))) when the tool contains all the code needed for supporting (f x))) the implementation. On the other hand, if different versions of the same Com- works the same as mon Lisp implementation behave in a different way, such (iterate:iterate checks can lead to breakage on some of the relatively recent (for x :from 1 :to 3) versions of a supported Common Lisp implementation. (collect x)) Richards Waters has described [5, 6] a clean, almost portable But the following version with a local macro definition will implementation of macroexpand-all that needs only a single not work environment-related function to be implemented separately for each of the Common Lisp implementations. (iterate:iterate CLWEB [9] by Alex Plotnick follows a similar approach for (for x :from 1 :to 3) its custom code walker, and the same approach is described (macrolet as a \proper code walker" in an essay [10] by Christophe ((f (y) `(collect ,y))) Rhodes. (f x))) hu.dwim.walker [11] is a comprehensive code walking li- The macroexpand-dammit library [7] is an attempt to pro- brary that uses a lot of implementation-specific functions for vide implementation of the full recursive macro expansion inspecting lexical environments etc. Unfortunately, the cur- functionality (macroexpand-all) including an optional lexi- rent versions of some previously supported Common Lisp im- cal environment parameter, but it has some bugs. It is not plementations are not supported because of relatively recent clear if these bugs can be fixed without major changes. For changes in environment handling. Also macroexpand-dammit example, both the original version and the freshest known is implemented in such a way that it removes macrolet and fork [8] return 1 instead of 2 in the following case: symbol-macrolet from the code completely after expanding the local macros and the local symbol macros. (defmacro f () 1) The trivial-macroexpand-all library [13] provides the functionality by wrapping the best func- (defmacro macroexpand-dammit-here macroexpand-all tion provided by each Common Lisp implementation. Un- (form &environment env) fortunately, some implementations don't support the lexical `(quote environment argument (for example, CLISP [16]). The same ,(macroexpand-dammit:macroexpand-dammit approach is used by SLIME (Common Lisp editing and debug- form env))) ging support for Emacs) [12]. Apparently, no more generic code walking functionality is provided in a consistent way by (macrolet ((f () 2)) multiple implementations. (macroexpand-dammit-here (macrolet () (f)))) Additionally, the macroexpand-dammit library includes the macroexpand-dammit-as-macro macro that the environment 3 PROBLEMS handling system of the Common Lisp implementation. Both the function and the macro versions of the recursive macro In this section we present a brief overview of the problems expander remove macrolet and symbol-macrolet forms from that the portable code walkers face. the code (replacing them with progn if necessary). As SICL [14] aims to be a modular and portable con- forming implementation of Common Lisp in Common Lisp, and the standard requires compile to do an equivalent of 3.1 Interpretations and violations of the macroexpand-all, there probably will be a usable code walker standard in SICL at some point; to the best of our knowledge, there is The Common Lisp standard allows Common Lisp imple- currently none. mentations to implement some standard-defined macros as additional special operators. But all special operators added 2.2 Implementation-specific tools instead of macros must also have a macro definition avail- Unfortunately, implementation-specific tools often check the able. This requirement seems to imply that the standard name of the Common Lisp implementation to choose the code expects the code walkers to succeed if they implement special path (for example, using #+ reader conditionals to check for handling only for the special operators listed in the standard. Writing a best-effort portable code walker in Common Lisp ELS2017, April 2017, Vrije