TUGboat, Volume 33 (2012), No. 2 199
Star TEX: The Next Generation Several ideas on how to modernize TEX already exist. Some of them have actually been implemented. Didier Verna In this paper, we present ours. The possible future Abstract that we would like to see happening for TEX is some- what different from the current direction(s) TEX’s While T X is unanimously praised for its typesetting E evolution has been following. In our view, modern- capabilities, it is also regularly blamed for its poor izing TEX can start with grounding it in an old yet programmatic offerings. A macro-expansion system very modern programming language: Common Lisp. is indeed far from the best choice in terms of general- Section 2 clarifies what our notion of a “better”, or purpose programming. Several solutions have been “more modern” TEX is. Section 3 on the next page proposed to modernize T X on the programming E presents several approaches sharing a similar goal. side. All of them currently involve a heterogeneous Section 4 on the following page justifies the choice approach in which T X is mixed with a full-blown pro- E of Common Lisp. Finally, section 5 on page 205 gramming language. This paper advocates another, outlines the projected final shape of the project. homogeneous approach in which TEX is first rewrit- ten in a modern language, Common Lisp, which 2 A better TEX serves both at the core of the program and at the The T X community has this particularity of being scripting level. All programmatic macros of T X are E E a mix of two populations: people coming from the hence rendered obsolete, as the underlying language world of typography, who love it for its beautiful itself can be used for user-level programming. typesetting, and people coming from the world of Prologue computer science, who appreciate it for its automa- tion and similarity to a programming language. It is TEX true that the way TEX works is much closer to that The [final] frontier. of a compiled language than a WYSIWYG editor. These are the voyages, In both worlds, TEX is unanimously acclaimed Of a software enterprise. for the quality of its typesetting. This shouldn’t Its continuing mission: be surprising, as it has always been TEX’s primary To explore new tokens, objective. The question of its programmatic capa- To seek out a new life, bilities, however, is much more arguable. People New forms of implementation. . . unfamiliar with programming in general easily ac- knowledge that TEX’s programmatic interface is not 1 Introduction trivial to use. For people coming from the world of In 2010, I asked Donald Knuth why he chose to computer science, it is even more obvious that TEX design and implement TEX as a macro-expansion is no match for a real programming language, partly system rather than as a full-blown, procedure-based, due to its macro-expansion nature. programming language. His answer was twofold: Let us recall that TEX was not originally meant 1. He wanted something relatively simple for his to be a programming language. To quote its author, secretary who was not a computer scientist. Donald Knuth [8]: I’m not going to design a programming lan- 2. The very limited computing resources at that guage; I want to have just a typesetting lan- time practically mandated the use of something guage. [. . . ] In some sense I put in many of much lighter than a complete programming lan- T X’s programming features only after kick- guage. E ing and screaming. The first part of the answer left me with a slight From this perspective, it seems natural to con- feeling of skepticism. It remains to be seen that T X E sider that a “better” T X, today, would essentially is simple to use. Although it probably is for simple E deliver the same typesetting quality from a more things, its programmatic capabilities are notoriously modern programmatic interface. More precisely: tricky. The second part of the answer, on the other hand, was both very convincing and arguably now • access to the typesetting subset of TEX’s primi- obsolete as well. Time has passed and the situation tives should be provided with a consistent syntax today is very different from what it was 30 years (in particular, no more need for \relax), ago. The available computing power has grown expo- • the programmatic API (\def, \if, etc.) should nentially, and so have our overall skills in language be dropped in favor of support from a real pro- design and implementation. gramming language,
Star TEX: The Next Generation 200 TUGboat, Volume 33 (2012), No. 2
• the system should remain simple to use, at least more integrated than the other alternatives is built for simple things, just as TEX is today, like this: a Lua interpreter is embedded in a more • the system should remain highly extensible and or less regular TEX, written in WEB and C. customizable, just as TEX is today, The idea that we suggest in this paper is that • and finally, preserving backward compatibility, another, fully integrated approach is also possible. although not considered mandatory, should also In this approach, another programming language be considered. would be used to completely rewrite TEX and pro- vide the desired programmatic layer at the same 3 Alternatives time. We are aware of at least one previous attempt 4 The idea of grounding TEX into a real program- at a fully integrated approach. NTS , the “New ming language is not new. Let us mention some Typesetting System” was supposed to be a complete such attempts that we are aware of. eval4tex1 and re-implementation of TEX in Java, but the project sTEXme2 both use Scheme (another dialect of Lisp). was never widely adopted. We don’t think that this PerlTEX[13, 15], as its name suggests, chooses Perl. project’s demise indicates in any way that the ap- QaTEX/PyTEX[4] use Python, and finally, also as proach is doomed in general. On the contrary, the its name suggests, LuaTEX3 employs Lua. remaining sections explain why we think that Com- These approaches, although motivated more or mon Lisp is a very good candidate for it. less by the same general idea, work in very different 4 Common Lisp: why? ways. Some of them wrap TEX in a programming Lisp is a very old language. It was invented by language by giving the language access to TEX’s in- ternals. Others wrap a programming language in John McCarthy in the late 1950s [9]. Contrary to what many people seem to think however, being old TEX by allowing TEX to execute code (LuaTEX be- longs to this category). Some do both. For instance, doesn’t imply being obsolete. In this case, it is a synonym for being mature and modern. When Lisp sTEXme ships with a extended Scheme engine that was invented, it was in fact way ahead of its time, to can access TEX’s internals, as well as a modified TEX engine that can evaluate Scheme code. the point that even its inventor hadn’t realized the extent of his creation. A sign of this is the recent Some of them allow authors to write TEX macros incorporation of features that Lisp already provided in a different language (PerlTEX lets you write TEX 50 years ago into so-called “modern” programming macros in Perl). Others, like QaTEX, aim at com- languages, such as C# with the addition of dynamic pletely getting rid of TEX macros so that all program- matic functionality is written in another language types, or C++ and Java with the addition of lambda (anonymous) functions. (Python, with PyTEX in that case). This particular case is closer to what we have in mind. 4.1 Standardization Finally, some approaches use a synchronous Common Lisp, in particular, was standardized in dual-process scheme in which both TEX and the programming language of choice run in parallel, com- 1994 [1] (it was in fact the first object-oriented lan- municating either via standard input/output redi- guage to get an ANSI standard). Remember how rection, or by file or socket I/O. That is the case of stability was important to Donald Knuth for TEX? This means that even across different implementa- sTEXme and PerlTEX. Others, like eval4tex use a multi-pass scheme instead. In a first pass, the “for- tions (there are half a dozen or so), that the core of eign” code is extracted and sent to the programming the language will never change, and in fact hasn’t for language. The programming language in question the last 20 years. Compare this to modern scripting languages such as Python or Ruby, for which the executes its code and sends it back to TEX. In the “standard” is essentially the current implementation final pass, TEX is left with only regular TEX macros to process. of the current version of the language written by the In spite of all these variations, it is worth stress- author of the language. . . ing that all these approaches have something in com- The Common Lisp standard is fairly comprehen- mon: they are heterogeneous. They involve both a sive: it includes not only the language core but a large programming language engine on one hand, and the library of functions. Of course, because the standard is 20 years old, it lacks several things that are con- original, though possibly modified, TEX engine on sidered important today (such as a multi-threading the other hand. Even LuaTEX which is somehow API). Every Common Lisp implementation provides 1 http://www.ccs.neu.edu/home/dorai/eval4tex/ its own version of non-standard features, but again, 2 http://stexme.sourceforge.net/ 3 http://www.luatex.org 4 http://nts.tug.org
Didier Verna TUGboat, Volume 33 (2012), No. 2 201
there are only half a dozen out there, and if one being a function call, now represents the list of 3 chooses to stick to only one of them, then one gets elements: the symbol + and the numbers 1 and 2. the same stability as for standardized features, or at Definitions To define a global variable, we can least backward-compatibility. write something like this: 4.2 General purpose vs. scripting (defvar devil-number 666) To define a function: Common Lisp also has this particularity of being both a full-blown, general purpose, industrial scale (defun dbl (x) (* 2 x)) programming language and a scripting or extension And that is the end of our Common Lisp crash language at the same time. This is something that course. With that knowledge, you know practically cannot be said of most modern languages out there all there is to know about the language in order to but is nevertheless crucial in the fully integrated use it for simple things. The rest is a matter of know- approach that we are advocating. This was certainly ing the names of standard (built-in) variables and not the case of Java, in the now dead NTS project. functions. In particular, this is certainly not more Common Lisp is indeed a full-blown, general pur- complicated than learning the basic and inconsistent pose, industrial scale programming language. It is TEX syntax (do you provide arguments in braces or multi-paradigm (functional, object-oriented, impera- inline with a final \relax to be on the safe side?), tive, etc.), highly extensible (both at the syntactic and it would also certainly be enough to write basic and semantic levels [12]), highly optimizable (no- LATEX documents like this: tably with static typing facilities [19, 20]) and has a (document-class article) plethora of libraries (such as Perl-compatible regu- ... lar expressions, database access, web infrastructure, (begin document) foreign function interfaces, etc.). Today, millions of (section "Title") lines of Common Lisp code are used in industrial ... applications all over the world. (end document) But Common Lisp is also a scripting language. 4.3 Built-in paradigms It is highly interactive (it comes with a REPL: a Read Eval Print Loop), highly dynamic (with fea- In a somewhat surprising way, Common Lisp pro- tures ranging from dynamic type checking to an em- vides programming paradigms, idioms or library- bedded JIT-compiler and debugger), highly reflexive based features that (LA)TEX also provide (or would (something crucial for extensibility and customizabil- like to provide). This means that such features are ity) and at the same time easy to learn (notably out already here for us and would not need to be re- of its minimalist syntax). implemented in a Lispy TEX. This section only This last point constitutes the beginning of our presents some of them; there are in fact many more. tour of the language. Remember that one of our objectives is to provide a system just as simple as 4.3.1 key=value pairs TEX, at least for simple things. Below is a one minute The success of key/value arguments in LATEX is pro- crash course on Lisp syntax. portional to their actual need: there are at least a dozen packages providing this functionality, each Literals Common Lisp provides literals such as and every one of them with its own pros and cons. numbers (1.254) or strings ("foobar"). Literals Common Lisp provides a clean and straightforward evaluate to themselves. implementation of this for free in its function call pro- Symbols Common Lisp provides symbols that can tocol (the so-called “lambda lists”). The following be used to name functions or variables (possibly at function takes one mandatory (positional) argument the same time). pi has the expected mathematical and two optional keyword arguments, which are in value. identity is the identity function. fact named, floating arguments: S-Expressions Compound expressions are written (defun include-graphics inside parentheses and denote function calls in prefix (file &key width height) ...) notation. (+ 1 2) represents the sum of 1 and 2. It can be called with just the mandatory argument: Quotation In Common Lisp, everything has a (include-graphics "image") value. If you want to prevent evaluation, put a quote in front of the expression. For instance, ’identity or with either or both keywords (prefixed with a ‘:’): is the symbol identity itself. ’(+ 1 2), instead of (include-graphics "image" :height
Star TEX: The Next Generation 202 TUGboat, Volume 33 (2012), No. 2
Keyword arguments can have default values which type ‘I’ and the correct spelling themselves may be dynamically computed. (e.g., ‘I\hbox’). Otherwise just continue, and I’ll forget about whatever was undefined. 4.3.2 Packages ? I\hrule LATEX implements the notion of packages, essentially a collection of macros stored in a file. The de-facto *\bye standard for this in Common Lisp is called ASDF5 (Another System Definition Facility). Common Lisp This kind of interaction pretty much resembles a REPL which Common Lisp, like every interactive systems resemble LATEX packages, only much more evolved. For instance, systems are composed of a language, provides out of the box. Where Com- hierarchy of files with customizable loading order, mon Lisp specifically comes into play is that every automatic dependency tracking and recompilation Common Lisp application may embed a interactive of obsolete object files (compare this to having only debugger for free, precisely useful for this kind of interpreted macros) and much more. error/recovery interaction. In Common Lisp, the programmer has the ability to implement his own 4.3.3 Namespaces recovery options (known as restarts in Lisp jargon) without unwinding the stack. This is different and A frequent rant about LATEX is the lack of name- much more powerful than regular catch/throw facili- spaces. Common Lisp provides a related concept ties. If you are not interested in using the full-blown called packages (not to be confused with LATEX pack- debugger, you can implement a function for catching ages). A package is a collection of symbols naming errors and listing the available recovery options in a functions, variables, or both. Packages have names bare 10 lines of code. through which you access their symbols. Packages may declare some symbols as public while the oth- 4.3.5 Dumping ers remain private. Suppose for instance that there Out of the historical concern for performance, T X is a package named ltx, that implements LAT X 2ε, E E has the ability to dump and reload so-called “format” and declares its function document-class as public. files, which saves a lot of parsing and interpretation From the outside, one would canonically refer to (cf. the \dump command). Given the increase in this function as ltx:document-class. On the other computing power over the last 30 years, the question hand, the need to reference all such LAT X symbols E of performance is admittedly much less critical than explicitly in a document would probably be cumber- it used to. Even today, though, performance is not a some. In such a situation, one may use use-package, concern that should be completely disregarded. For in which case all public symbols become directly ac- example, compiling a lengthy Beamer presentation cessible. Using the ltx package makes it possible to with lots of animations can still be annoyingly slow. call the function document-class implicitly, without Common Lisp happens to provide a dumping the package name prefix. feature out of the box. Although not part of the ANSI 4.3.4 Interactivity standard, all Lisp compilers provide it. In SBCL6 for instance, the function save-lisp-and-die dumps As you likely know, T X can be used in an interac- E the whole global state (stack excepted) of the current tive fashion. The following example shows a sample Lisp environment into a file that can later be quickly conversation between T X and a user who mistypes E reloaded with the --core command-line option. In- a macro name: stead of dumping a core image, it is also possible to didier(s003)% tex dump a fully functional standalone executable. This is TeX, Version 3.1415926 [...] Because the dumping mechanism is accessible **\relax to the end-user, interesting applications could be en- visioned with very little programming, such as mid- *\hule ! Undefined control sequence. document dumping. By outputting to in-memory <*> \hule strings instead of files, for example, a document au- thor could be given the possibility to dump in the ?H middle of a large document’s processing. The result- The control sequence at the end of the top ing facility would be quite similar to \includeonly — line of your error message was never \def’ed. except that the whole document would be typeset If you have misspelled it (e.g., ‘\hobx’), every time.
5 http://www.common-lisp.net/project/asdf 6 http://www.sbcl.org
Didier Verna TUGboat, Volume 33 (2012), No. 2 203
4.3.6 Performance equivalent to integers in other languages) and so is Let us tackle the problem of performance from a more the result of the multiplication. This is important general point of view. One frequent yet misinformed because there is no guarantee that the double of an argument against dynamic languages is: “they are integer remains the same-size integer. Consequently, slow”. From that point of view, it may seem odd in general, Lisp would need to allocate a bignum to to even begin to envision the reimplementation of a store the result. As it turns out, compiling this new version of program such as TEX in a dynamic language. One first and frequent misconception about in- the function leads to only 5 lines of machine code. teractive languages is that as soon as they provide The compiler is even clever enough to avoid using a REPL, they must be interpreted. This is in fact the integer multiplication operator, but a left shift not the case. Nowadays, many Common Lisp im- instruction instead. What we get in this context is plementations such as SBCL don’t even provide an in fact the behavior of a statically and weakly typed interpreter. Instead, the REPL has a JIT (Just In language such as C. Consequently, it should not be Time) compiler which compiles the expressions you surprising that the level of performance we get out type and only then executes them. To put this in per- of this is comparable to that of equivalent C code. Recent experimental studies have demonstrated that spective, compare the processes of interpreting TEX macros by expansion and executing Lisp functions this is indeed the case [19, 20]. compiled to machine code. . . This particular example is also a nice illustration Yet, starting with the assumption that perfor- of what we meant earlier by saying that Common mance should indeed be a concern (this is not even Lisp is both a full-blown, industrial scale, general necessarily the case), the argument of slowness may purpose programming language, and a scripting lan- still make some sense in specific cases. For exam- guage at the same time. When working at the script- ple, it is obvious that performing type checking at ing level, the first version of dbl is quick and good run-time instead of at compile-time will induce a enough. When working in the core of your applica- cost in execution speed. In general however, this tion however, you appreciate it when the language argument, as-is, is meaningless. For starters, let us provides a lot of tools (optimization ones notably) not confuse “dynamic language” with “dynamically to adjust your code to your specific needs. typed language”. A dynamic language gives you a 4.4 Extensibility and customizability lot of run-time expressive power, but that doesn’t necessarily mean that you have to use all of it, or Another aspect of the language well worth its own sec- that it is impossible to optimize things away. tion is its level of extensibility (adding new behavior) Let us continue on type checking in order to and customizability (modifying existing behavior). illustrate this. Look again at the definition for our We know how important this is in the (LA)TEX world, “double” function: which is a complicated ball of intermixed threads all interacting with each other [21], which wouldn’t be (defun dbl (x) (* 2 x)) possible without the level of intercession that TEX This function will no doubt be relatively slow, be- macros offer. Similarly, at least part of the success cause Common Lisp has a lot of things to do at of LuaTEX is due to its ability to provide access to run-time. For starters, it needs to check that x is TEX’s internals, so it seems that there is also a lot indeed a number. Next, the multiplication needs to of interest in this area. be polymorphic because you don’t double an integer the same way you double a float or a complex. That 4.4.1 Homoiconicity and reflection is in fact not the whole story, but we will stop here. Once again, Common Lisp is here to help. We men- On the other hand, consider now the following tioned earlier how the root of extensibility and cus- version: tomizability in Common Lisp is its highly reflexive (defun dbl (x) nature. Reflection is usually decoupled into introspec- (declare (optimize (speed 3) (safety 0)) tion (the ability to examine yourself) and intercession (type fixnum x) (the ability to modify yourself). (the fixnum (* 2 x))) In Lisp, reflection is supported in the most direct In this function, we request that the compiler op- and simple way one could think of. Remember the timizes for speed instead of safety. The result is expression (+ 1 2), with or without evaluation? As that the compiler will “trust” us and bypass all dy- we said before, this expression can either represent namic checks. Next, we actually provide static type a call to the function “sum” with the arguments 1 information. x is declared to be a fixnum (roughly and 2, or the list of three elements: the symbol + and
Star TEX: The Next Generation 204 TUGboat, Volume 33 (2012), No. 2
the numbers 1 and 2. What this really means is that Finally, reflexivity in Common Lisp goes as far every piece of code, if not evaluated, can be seen as a as allowing both introspection and intercession at the piece of data, and hence can be manipulated at will. level of package internals. In most other languages, it In fact, every piece of Lisp code is represented as a is simply not possible to access the so-called “private“ list, which happens to be a user-level data structure. parts of a namespace, class, or whatever encapsula- This property of a programming language is known tion scheme is supported. In Common Lisp, remem- as homoiconicity [5, 11]. ber that a public symbol is accessed by prefixing Another important distinction in this notion is its name with the name of the package and a colon structural vs. behavioral reflection [10, 17]. While separator, for example: ltx:document-class. structural reflection deals with providing a way to It turns out that it is just as easy to both intro- reify a program, behavioral reflection deals with ac- spect and intercede a package’s internals. One just cessing the language itself. Lisp is one of the very needs to use a double colon instead of a single one. few languages to provide both kinds of reflection to This is not unlike the @ character convention used by some extent, as we’ll now discuss. LATEX, along with the macros \makeatletter and \makeatother. The double colon really is a warning 4.4.2 Structural reflexivity that you are prying on private property, but nothing This section gives only a couple of examples of struc- technically prevents you from doing so. tural reflexivity, again, to demonstrate how some 4.4.3 Behavioral reflexivity well-known T X idioms map to Common Lisp in a E Lisp goes even further by providing some level of straightforward way. behavioral reflection as well. The functional nature of Lisp implies that func- Lisp macros (functions executed at compile- tions are first-class citizens in the language [3, 18]. time) provide a form of intercession at the compiler In general, this means that functions can be used level, allowing one to program language modifica- like any other object in the language. In particular, tions in the language itself (what [16] calls a “homo- this means that functions can be modified, stored in geneous meta-programming system”, as opposed to variables, etc. C++ templates for instance, which are heterogeneous: Storing a functional value in a variable will be a different language). useful in order to implement a variant of the function CLOS [2, 6], the Common Lisp Object System which needs to call the original one at some point. is written in itself, on top of a so-called Meta-Object This is equivalent to the following common TEX Protocol, known as the MOP [14, 7]. Using the CLOS idiom: MOP permits intercession at the object system level, \let\oldfoo\foo allowing the programmer to modify the semantics of \def\foo{... \oldfoo ...} the object-oriented layer. Defining a function several times is simply equiv- Finally, it is also possible to extend the Lisp alent to overriding the previous definition(s). This syntax, which is a form of intercession at the parser behavior matches that of \def or (more or less) level, allowing to modify the language’s syntax di- \renewcommand. It is in fact more powerful for at rectly. This is the only concrete example that we will least two reasons: provide in this section, although a striking one. We assume that the reader is familiar with TEX’s dou- 1. Since Common Lisp has a proper notion of scope ble superscript syntax, allowing to denote characters (and in fact provides both dynamic and lexi- that are not normally printable. cal scoping; something that very few other lan- Suppose that we are given a function called guages, can do), function or variable redefinition ^^-reader which performs TEX’s double-superscript can be performed at different scoping levels, not syntax to character conversion (this function is 10 only local or global. lines long). The following code effectively installs the 2. Because of its interactive nature, there is no corresponding syntax in the Common Lisp reader: real distinction between functions defined in (make-dispatch-macro-character #\^) a core image or executable and those defined (set-dispatch-macro-character #\^ #\^ in the REPL and that consequently, they can #’|^^-reader|) be redefined in the exact same way. Consider The first line informs the Lisp reader that the ^ what this really means for a minute: one could character is to be treated in a special way (do you redefine any function in the TEX program just see a relationship to active characters and catcodes?). as easily as any TEX macro. . . The second line informs the Lisp reader that if two
Didier Verna TUGboat, Volume 33 (2012), No. 2 205
such characters are encountered in a row, then, the set some variable to a specific value (but see regular parser should stop and pass control to our section 5.1.1): user-provided function. We can now verify that this (setf baselineskip extended syntax works: #g(b :plus x :minus y)) CL-USER> ^^M • Obviously, every TEX primitive command be- #\Return comes a Lisp function. Again, the point here is CL-USER> ^^00 to both simplify the syntax and make it more #\Nul consistent at the same time (no more \relax!). This particular example is a bit simplified, but it Here are a couple of examples: conveys the idea. By modifying the way the Lisp (input file) parser behaves, we are able to modify the language (hbox material) itself, not only the program we are executing. And (hbox material :to dim) again, we are not very far from TEX’s notion of active (hbox material :spread dim) characters. The reader familiar with TEX will notice im- 5 Common Lisp: how? mediately that the arguments in the last two examples are in reverse order, compared to the Section 2 on page 199 listed five objectives for a mod- regular T X versions. If this is really too much ern reimplementation of T X. Section 4 on page 200 E E to get accustomed to, variants are easy to im- demonstrated how Common Lisp can help fulfill three plement: of these goals: providing real programming capabil- ities, extensibility and customizability, all of this (hbox-to dim material) while maintaining a relative ease of use. (hbox-spread dim material) This section is devoted to the last two objectives, Such syntactic variants are usually implemented namely providing a more modern and consistent API with Lisp macros, evaluated at compile-time, so while at the same time (although not mandatory) that there is no additional run-time cost. maintaining backward compatibility. In actuality, 5.1.1 Lisp-2 this section provides a more concrete view of the project itself. Although very little has been imple- Let us go back to the baselineskip assignment mented already, the project does have a name: TiCL example for a minute: (the acronym for “TEX in Common Lisp”). (setf baselineskip
Star TEX: The Next Generation 206 TUGboat, Volume 33 (2012), No. 2
Figure 1: TiCL architecture
5.2 Backward compatibility expression. Since this syntax is invalid in TEX, it The programmatic interface to the typesetting part of cannot break any existing document. Below is an example illustrating this idea (taken TEX described in the previous section is what we call from [15]). The Lisp function ast is used to define a “procedural TEX”. Along with the actual typesetting T X macro \asts outputting a specified number of engine, this mostly corresponds to TEX’s stomach E and bowels. On top of that, it is in fact possible asterisks. to design either completely new typesetting systems \documentclass{article} or programmatic versions of Plain TEX, LATEX, etc. entirely in Lisp. \newcommand\asts{} In order to maintain backward compatibility, we __(defun ast (n) also need to implement “traditional” T X (still in (format nil "\\renewcommand\\asts{~A}" E (make-string n :initial-element #\*))) Lisp), that is, the surface layer consisting of both the macro versions of Lispified TEX primitives, and the \begin{document} rest of TEX’s programming API. This is mostly lo- __(ast 10) cated in TEX’s mouth and eyes. Figure 1 depicts this \asts architecture (disclaimer: this is an overly simplified, \end{document} extremely naive view; see section 5.3 for details). As For the curious, our current implementation of TEX’s mentioned earlier, the whole point of this architecture eyes in Common Lisp is roughly 200 lines. The being implemented in Lisp, in terms of extensibility support for the double subscript syntax (including and customizability, is that the user of TiCL can ba- parsing it, reading the Lisp code, evaluating it and sically interfere at every level of the system, whether inserting the result back into the regular TEX stream) by adding personal functions, rewriting built-in func- amounts to only 16 lines, that is, around 8% of the tions or even internal typesetting algorithms. total. Another advantage of this fully integrated ap- proach is that it is actually quite simple to provide 5.3 Expected problems “mixed” functionality, that is, using Common Lisp After all those “would” and “could”, let us get to a code directly in an otherwise regular T X source file. E more pessimistic (realistic?) view of the project. This The only requirement is an escape syntax allowing section sheds some darkness on the too-bright picture Common Lisp to take over interpretation of Lisp we have drawn of TiCL until now. As mentioned code, and insert the result back into the regular T X E earlier, TiCL is in fact pretty much only an idea character stream. A prototype for this has already at present. If this project ever comes to fruition, a been implemented. It simply consists in a Common number of problems are expected. Lisp implementation of TEX’s eyes with an additional bit of syntax: the appearance of two consecutive and 5.3.1 A huge task equal subscript characters in a regular TEX source file will trigger the evaluation of the subsequent Lisp Completely reimplementing TEX is a huge task. This is probably one of the reasons all alternative projects
Didier Verna TUGboat, Volume 33 (2012), No. 2 207
(NTS excepted) chose the hybrid approach instead of comprehensive programming language with scripting the fully integrated one. One thing that may help is capabilities makes this problem even worse because the existence of foreign function interfaces, notably we have more than a couple of “macros” to control for the C language. The project could be developed access to. We have a whole stack of language fea- gradually by linking to the WEB/C implementation tures to control, such as file I/O, operating system of the not-yet-reimplemented parts. interfaces, etc. Currently, we are not aware of any sandboxing library already available for Lisp. 5.3.2 Compatibility 5.3.6 Intercession Another question to consider is whether the TEX- incompatible part would eventually be accepted by This is in a similar vein as the sandboxing problem. the (or a new) community. This question is in fact In [21], we were complaining about (rejoicing in?) pertinent for LATEX3 as well and we don’t have an the huge intercession mess that the LATEX world is. answer. The existence of a compatibility mode with Well, now you should really be afraid because we are pluggable Lisp, as described in section 5.2 on the fac- going to make things worse! Remember that any Lisp ing page, would probably help getting people accus- executable granting scripting privileges to the end- tomed to the benefits of using Lisp, while remaining user effectively provides write-access to the whole in a reassuring context of traditional TEX. executable. This surely makes it trivial to extend or customize the system. Whether this is a good thing 5.3.3 TEX’s organs for the system in the long run, there is no way to Figure 1 on the preceding page was already pointed tell. After all, LATEX is alive and kicking, in spite of out to be overly simplified. In reality, we know that (because of?) its high intercession capabilities. . . the TEX organs don’t constitute a pipeline. Some levels from “down below” do affect the upper stages 6 Conclusion which makes things much more complicated. In the In this paper, we advocated a “homogeneous” ap- long run, this may imply that it would be impossi- proach to TEX modernization in which TEX itself is ble to have new programmatic typesetting systems first rewritten in a programming language serving (accessing “procedural TEX” directly) work in con- both at the core of the program and at the scripting junction with traditional TEX. Currently, we don’t level. All programmatic macros of TEX are hence know for sure, although we are aware of the fact that rendered obsolete, as the underlying language itself this was one of the reasons the NTS project failed. can be used for user-level programming. In a rather puzzling way, our notion of “modernity” consists 5.3.4 Mixed mode of reimplementing a 30-year-old language using a Along those same lines, “mixed” mode, that is, the 50-year-old one! ability to mix regular TEX macros with Lisp code is We demonstrated why we think that Common expected to be tricky. If we want to provide more Lisp is a very good choice for doing this. Common interaction than just the double subscript syntax, Lisp is both a scripting language and a full-blown, for example, the ability to define TEX macros in industrial scale, general-purpose programming lan- Lisp with Lisp access to the macro’s arguments, we guage at the same time. Common Lisp also provides are bound to encounter issues related to the time of several paradigms or idioms that are actually quite expansion. close to features that TEX either provides as well, or This problem is in fact not related to Lisp at would like to provide. all, but rather to any approach aiming to provide Will the TiCL project ever come to birth? That the ability to define TEX macros in another language. is not sure at that point, as it will require a tremen- Section 4 of [15] describes these kinds of problems dous effort. We would like to see it happening, of in more detail. course. Amongst all the current alternatives, LuaTEX seems to be the only one vigorously alive and gaining 5.3.5 Sandboxing momentum. As with any scriptable application, the security prob- Of course, the other project that needs to be lem is an important one. How much programmatic mentioned is LATEX3. We only do so in an anecdotal access to the application itself, or its surrounding en- fashion because it does not make use of a real pro- vironment, are we willing to give the end-user? Some gramming language, but continues in the tradition versions of TEX are equipped with means to address of building directly on top of TEX. Still, LATEX3 is this issue (cf. the -shell-escape command-line op- not experimental anymore and is light years ahead tion and the \write18 command). Using a true, of what LATEX 2ε is. In particular, it is much better
Star TEX: The Next Generation 208 TUGboat, Volume 33 (2012), No. 2 than its ancestor in terms of syntax consistency and 1960. Online version at http://www-formal. programmatic capabilities. Nevertheless, since it is stanford.edu/jmc/recursive.html. still restricted to TEX’s macro-expansion world, ev- [10] Patty Maes. Concepts and experiments in ery time a new programming paradigm is needed, it computational reflection. In OOPSLA. ACM, will have to be implemented manually. December 1987. To sum up, the “big TEX modernization plan” [11] M. Douglas McIlroy. Macro instruction extensions currently seems to follow two different paths: stay- of compiler languages. Commun. ACM, 3:214–220, ing strictly in the TEX area or creating hybrids of April 1960. TEX and another real programming language. The [12] Marjan Mernik, editor. Formal and Practical approach we would like to suggest with TiCL is a Aspects of Domain Specific Languages: Recent third one: a homogeneous approach in which the im- Developments, chapter 1. IGI Global, 2012. plementation language of TEX is also the one which [13] Andrew Mertz and William Slough. Programming serves at the scripting level. Will this project see the with PerlTEX. TUGboat, 28(3):354–362, 2007. light of day? Can these three approaches co-exist in [14] Andreas Paepcke. User-level language crafting — the long term? Only the future will tell. . . Introducing the CLOS metaobject protocol. In Andreas Paepcke, editor, Object-Oriented Epilogue Programming: The CLOS Perspective, chapter 3, These were the voyages, pages 65–99. MIT Press, 1993. Downloadable Of a software enterprise. version at http://infolab.stanford.edu/ Its continuing mission: ~paepcke/shared-documents/mopintro.ps. To explore new tokens, [15] Scott Pakin. PerlTEX: Defining LATEX macros To seek out a new life, using Perl. TUGboat, 25(2):150–159, 2004. New forms of implementation. [16] Tim Sheard. Accomplishments and research To \textbf{go}, challenges in meta-programming. In Walid Taha, editor, Semantics, Applications, and Where no TEX has gone before! Implementation of Program Generation, volume References 2196 of Lecture Notes in Computer Science, pages 2–44. Springer, 2001. [1] Common Lisp. American National Standard: Programming Language. ANSI X3.226:1994 [17] Brian C. Smith. Reflection and semantics in (R1999), 1994. Lisp. In Symposium on Principles of Programming Languages, pages 23–35. ACM, 1984. [2] Daniel G. Bobrow, Linda G. DeMichiel, [18] J.E. Stoy and Christopher Strachey. OS6 — Richard P. Gabriel, Sonya E. Keene, Gregor An experimental operating system for a small Kiczales, and David A. Moon. Common Lisp computer. Part 2: Input/output and filing Object System specification. ACM SIGPLAN system. The Computer Journal, 15(3):195–203, Notices, 23(SI):1–142, 1988. 1972. [3] Rod Burstall. Christopher Strachey — [19] Didier Verna. Beating C in scientific computing Understanding programming languages. Higher applications. In Third European Lisp Workshop at Order Symbolic Computation, 13(1-2):51–55, 2000. Ecoop, Nantes, France, July 2006. [4] Jonathan Fine. T X forever! In Proceedings E [20] Didier Verna. CLOS efficiency:instantiation. In EuroT X, pages 140–149, Pont-`a-Mousson, E International Lisp Conference, pages 76–90, MIT, France, 2005. DANTE e.V. Cambridge, Massachusetts, USA, March 2009. [5] Alan C. Kay. The Reactive Engine. PhD thesis, Association of Lisp Users. University of Hamburg, 1969. [21] Didier Verna. Classes, styles, conflicts: [6] Sonya E. Keene. Object-Oriented Programming in The biological realm of LATEX. TUGboat, Common Lisp: A Programmer’s Guide to CLOS. 31(2):162–172, 2010. http://tug.org/TUGboat/ Addison-Wesley, 1989. tb31-2/tb98verna.pdf. [7] Gregor J. Kiczales, Jim des Rivi`eres,and Daniel G. Bobrow. The Art of the Metaobject Didier Verna Protocol. MIT Press, Cambridge, MA, 1991. EPITA / LRDE [8] Donald E. Knuth. Digital Typography. CSLI 14-16 rue Voltaire Lecture Notes. CSLI, September 1998. 94276 Le Kremlin-BicˆetreCedex [9] John MacCarthy. Recursive functions of symbolic France expressions and their computation by machine, didier (at) lrde dot epita dot fr part I. Communications of the ACM, 3:184–195, http://www.lrde.epita.fr/~didier
Didier Verna