TUGboat, Volume 40 (2019), No. 2 119 Quickref: Common Lisp reference extensible than its industrial competitors such as documentation as a stress test for Texinfo C++ or Java. Interestingly enough, the technical strengths of Didier Verna the language bring serious drawbacks to its commu- Abstract nity of programmers (a phenomenon affecting all the dialects). These problems are known and have been Quickref is a global documentation project for the discussed many times [4, 7]. They may explain, at Common Lisp ecosystem. It creates reference manu- least partly, why in spite of its technical potential, als automatically by introspecting libraries and gen- the Lisp family of languages never really took over, erating corresponding documentation in Texinfo for- and probably never will. The situation can be sum- mat. The Texinfo files may subsequently be con- marized as follows: Lisp usually makes it so easy verted into PDF or HTML. Quickref is non-intrusive: to \hack" things away that every Lisper ends up de- software developers do not have anything to do to veloping his or her own solution, inevitably leading get their libraries documented by the system. to a paradox of choice. The result is a plethora of Quickref may be used to create a local website solutions for every single problem that every single documenting your current, partial, working environ- programmer faces. Most of the time, these solutions ment, but it is also able to document the whole work, but they are either half-baked or targeted Common Lisp ecosystem at once. The result is a to the author's specific needs and thus not general website containing almost two thousand reference enough. Furthermore, it is difficult to assert their manuals. Quickref provides a Docker image for an quality, and they are usually not (well) documented. easy recreation of this website, but a public version As this situation is well known, the commu- is also available and actively maintained. nity has been attempting to \consolidate" itself in Quickref constitutes an enormous and success- various ways. Several websites aggregate resources ful stress test for Texinfo. In this paper, we give related to the language or its usage (books, tuto- an overview of the design and architecture of the rials, implementations, development environments, system, describe the challenges and difficulties in applications, etc.). The Common Lisp Foundation generating valid Texinfo code automatically, and put (cl-foundation.org) provides technical, sometimes some emphasis on the currently remaining problems even financial, support and infrastructure for project and deficiencies. authors. Once a year, the European Lisp Sympo- sium (european-lisp-symposium.org) gathers the international community, open equally to researchers 1 Introduction and practitioners, newcomers and experts. Lisp is a high level, general purpose, multi-paradigm From a more technical standpoint, solving the programming language created in 1958 by John Mc- paradox of choice, that is, deciding on official solu- Carthy [2]. We owe to Lisp many of the programming tions for doing this or that, is much more problem- concepts that are still considered fundamental today atic | there is no such thing as an official authority (functional programming, garbage collection, inter- in the community. On the other hand, some libraries active development, etc.). Over the years, Lisp has do impose themselves as de facto standards. Two of evolved as a family of dialects (including Scheme, them are worth mentioning here. Most non-trivial Racket, and Clojure, to name a few) rather than as a Common Lisp packages today use ASDF for structur- single language. Another Lisp descendant of notable ing themselves (fig.1 has an example). ASDF allows importance is Common Lisp, a language targeting you to define your package architecture in terms of the industry, which was standardized in 1994 [5]. source files and directories, dependencies and other The Lisp family of languages is mostly known metadata. It automates the process of compiling and for two of its most prominent (and correlated) char- loading (dependencies included). The second one is acteristics: a minimalist syntax and a very high level Quicklisp (quicklisp.org). Quicklisp is both a cen- of expressiveness and extensibility. The root of the tral repository for Common Lisp libraries (not unlike latter, right from the early days, is the fact that CTAN) and a programmatic interface for it. With code and data are represented in the same way (a Quicklisp, downloading, installing, compiling and property known as homoiconicity [1, 3]). This makes loading a specific package on your machine (again, de- meta-programming not only possible but also trivial. pendencies included) essentially becomes a one-liner. Being a Lisp, Common Lisp not only maintains this One remaining problem is that of documenta- property, but also provides an unprecedented arsenal tion. Of course, it is impossible to force a library of paradigms making it much more expressive and author to properly document his or her work. One Quickref: Common Lisp reference documentation as a stress test for Texinfo 120 TUGboat, Volume 40 (2019), No. 2 (asdf:defsystem :net.didierverna.declt :long-name "Documentation Extractor from Common Lisp to Texinfo" :description "A reference manual generator for Common Lisp libraries" :author "Didier Verna" :mailto "[email protected]" :homepage "http://www.lrde.epita.fr/~didier/software/lisp/" :source-control "https://github.com/didierverna/declt" :license "BSD" ...) Figure 1: ASDF system definition excerpt could consider writing the manuals they miss for the 2.2 Documentation items third-party libraries they use, but this never happens Reference manuals generated by Quickref contain in practice. There is still something that we can do to information collected from various sources. First of mitigate the issue, however. Because Common Lisp all, many libraries provide a README file of some is highly reflexive, it is relatively straightforward to sort, which can make for a nice introductory chapter. retrieve the information necessary to automatically In addition to source files and dependencies, ASDF create and typeset reference manuals (as opposed to offers ways to specify project-related metadata in the user manuals). Several such projects exist already so-called system definition form. Figure 1 illustrates (remember the paradox of choice). In this paper this. Such information can be easily (programmati- we present our own, probably the most complete cally) retrieved and used. Next, Lisp itself has some Common Lisp documentation generator to date. built-in support for documentation, in the form of Enter Quickref. so-called docstrings. As their name suggests, doc- strings are (optional) documentation strings that may be attached to various language constructs such 2 Overview as functions, variables, methods and so on. Figure 2 Quickref is a global documentation project for the has an example. When available, docstrings greatly Common Lisp ecosystem. It generates reference man- contribute to the completeness of reference manu- uals for libraries available in Quicklisp automatically. als, and again, may be retrieved programmatically Quickref is non-intrusive, in the sense that software through a simple standard function call. developers do not have anything to do to get their li- braries documented by the system: mere availability (defmacro @defconstant (name &body body) in Quicklisp is the only requirement. In this sec- "Execute BODY within a @defvr Constant. tion, we provide a general overview of the system's NAME is escaped for Texinfo prior to rendering. features, design, and implementation. BODY should render on *standard-output*." `(@defvr "Constant" ,name ,@body)) 2.1 Features Figure 2: Common Lisp docstring example Quickref may be used to create a local website docu- menting your current, partial, working environment, As for the rest, the solution is less straightfor- but it is also able to document the whole Quicklisp ward. We want our reference manuals to advertise world at once, which means that almost two thousand as many software components as possible (functions, reference manuals are generated. Creating a local variables, classes, packages, etc.). In general there documentation website can be done in two different are two main strategies for collecting this informa- ways: either by using the provided Docker image tion. (the most convenient solution for an exhaustive web- Code walking. The first one, known as code walk- site), or directly via the programmatic interface, from ing, consists of statically analyzing the source code. within a running Lisp environment (when only the A code walker is usually at least as complicated as documentation for the local, partial, installation is the syntax of the target language, because it requires required). If you don't want to run Quickref yourself, a parser for it. Because of Lisp's minimalist syn- a public website is also provided and actively main- tax, using a code walker is a very tempting solution. tained at quickref.common-lisp.net. It always On the other hand, Lisp is extremely dynamic in contains the result of a full run of the system on the nature, meaning that many of the final program's latest Quicklisp distribution. components may not be directly visible in the source Didier Verna TUGboat, Volume 40 (2019), No. 2 121 Quicklisp foo/ Declt foo.texi Makeinfo foo.html Figure 3: Quickref pipeline ( Main thread, External Process) code. On top of that, programs making syntactic documentation can in turn be generated in many extensions to the language would not be directly different forms (notably PDF and HTML). Fi- parsable. In short, it is practically impossible to nally, it includes very convenient built-in anchor- collect all the required information by code walking ing, cross-referencing, and indexing capabilities. alone. Therefore, we do not use that approach. Quickref essentially runs this pipeline on the Introspection. Our preferred approach is by in- required libraries. Some important remarks need to trospection. Here, the idea is to actually compile be made about this process.