Tugboat, Volume 35 (2014), No. 1 99 Macro Memories, 1964–2013 David
TUGboat, Volume 35 (2014), No. 1 99
Macro memories, 1964–2013 I assume most readers know what macros are, but just in case: Typically one gives a name to a string David Walden of text, e.g., “\define\Macroname{Textstring}”; Contents then each time “\Macroname” is found (“called”) in the input text stream, it is replaced by “Textstring”. 1 McIlroy’s 1960 ACM paper 99 The macro definition can involve the additional 2 Some prior history of macros 100 substitution of text specified when the macro defini- 3 Strachey’s General Purpose Macrogenerator 101 tion is called. For example, 4 Midas macro processor 101 \define\Name#1{His name is #1} 5 More study and use of macro processors defines a macro named “Name” where “#1” indicates (and language extension capabilities) 102 a parameter to be substituted for when the macro is 6 Midas, macros, and the called. The macro might be called with “John” as ARPANET IMP program 103 the substitution text, as in the following 7 Ratfor and Infomail 104 \Name{John} 8 TEX, macros for typesetting 104 which would result in 9 M4 106 His name is John 10 Reflections 107 In addition to the “#1” indicating where a substitu- Introduction tion is to take place in the macro definition when In the summer of 2013, I was looking at a 1973 listing the macro is called, in this example the “#1” imme- of the ARPANET IMP (Interface Message Processor) diately after the macro name indicates there is one program1 which makes extensive use of macros. This such substitutable parameter. If there were more caused me to muse about the various macro proces- than one such parameter, a list such as “#1#2#3” sors I have used over the past 50 years which, in turn, would appear after the macro name in the definition, led to this note. specifying three such parameters. This note is not a thorough study, extensive Donald Knuth in the index to Volume 1 of The Art of tutorial, or comprehensive bibliography about macro Computer Programming gives this succinct definition processors. Such descriptions have already been pro- relating to macros: “Macro instruction: Specification vided by, for instance, Peter Brown, Martin Campbell- of a pattern of instructions and/or pseudo-operators Kelly, John Metzner, and Peter Wegner in their that may be used repeatedly within a program.” longer presentations of the topic.2,3,4,5 Instead, the 1 McIlroy’s 1960 ACM paper macro technology thread I follow herein is guided by the order in which I used or studied the various macro I’m pretty sure that while I was still in college at San processors. I hope this is usefully representative of Francisco State (1962–64) and using an IBM 1620 the scope of macro processor technology. computer, I had no concept of macros. The IBM I have three reasons for writing this note. (1) I 7094 at MIT Lincoln Laboratory, my first employer haven’t seen much new written about macro pro- after college (starting in June 1964), may have had cessors in recent years (other than what is on the a macro assembly capability, but I don’t think I ever web); thus, it is perhaps time for a new paper on used it. this sometimes under-appreciated topic. (2) Com- Probably my first contact with the concept of puter professionals and computing historians who macros was when an older colleague at Lincoln Lab have come to their fields only in the last decade or gave me his back issues of the Communications of two may not know much about macros, and this the ACM (and I joined the ACM myself to get future is a chance to share my fondness for and perspec- issues of the CACM). In one of these back issues, tive on macros. The citations in the endnotes also I read the article by Doug McIlroy on macros for may be a useful starting point for further study of compiler languages.6 macros, and maybe these notes will rekindle mem- McIlroy has the following example of defining a ories for other long-time computing people like me macro (although I am using an equal sign where he about some of their own experiences. (3) For (LA)TEX used an identity symbol): users who may not be computer programmers or fa- ADD, A, B, C = FETCH, A miliar with other macro processor systems and who ADD, B accomplish impressive things using TEX macros, this STORE, C note sketches the long technical history of which TEX where ADD7 is the macro name, and A, B, and C are macros are a part. the names of macro arguments to be filled in at the
Macro memories, 1964–2013 100 TUGboat, Volume 35 (2014), No. 1
time of the macro call, and the three lines of code Apparently Bell Labs was a particular hotbed are what is substituted. Thus, McIlroy shows this of macro activity in those early days. In a memorial macro being called with the sequence note for Douglas Eastwood,8 Doug McIlroy recounts: ADD, X, Y, Z On joining the Bell Labs math department, I resulting in the following: was given an office next to Doug Eastwood’s. Soon after, George Mealy . . . suggested to a FETCH, X ADD, Y small group of us that a macro-instruction STORE, Z facility be added to our assembler . . . This idea caught the fancy of us two Dougs, and This style of macro definition uses symbolic names set the course of our research for some time for the substitutable parameters, which can be useful to come. We split the job in half: Eastwood in remembering what one is doing with long macro took care of defining macros; McIlroy handled definitions. However, it is also a bit more complicated the expansion of macro calls. to implement such symbolic macro parameter names The macro system we built enabled truly compared with using special codes such as “#1”. astonishing applications. Macros took hold McIlroy’s 1960 paper goes on to show examples in the Labs’ most important project, elec- of macros in an ALGOL-like language and to explain tronic switching systems, in an elaborated the benefits of various features of macro processors. form that served as their primary program- For instance, ming language for a couple of decades. macro exchange(x,y;z) := Once macros had been incorporated, the begin assembler was processing code written whole- begin integer x,y,z; sale by machine (i.e., by the assembler itself) z:=y; y:=x; rather than retail by people. This stressed the x:=z; assembler in ways that had never been seen be- end exchange x and y fore. The size of its vocabulary jumped from end exchange about 100 different instructions to that plus defines a macro which, if called with an unlimited number of newly defined ones. The real size of programs jumped because one exchange(r1,ss3) human-written line of code was often short- results in hand for many, many machine-written lines. begin integer r1,ss3,.gen001 And the number of symbolic names in a pro- .gen001:=r1; gram jumped, because macros could invent ss3:=r1; new names like crazy. r1:=.gen001; By the way, Rosen’s book7 also had a paper (pp. 535– end exchange r1 and ss3 559) by George Mealy that touched on macros: “A Note that a special temporary register, .gen001, was Generalized Assembly System (Excerpts)”. created to replace z which was defined following the semicolon in the parameter list. 2 Some prior history of macros McIlroy’s paper also has a summary list of “sa- McIlroy’s paper also hints at some of the history of lient features” [the comments below in square brack- macro processors including half a dozen references9 ets are my notes on McIlroy’s list]: to prior macro processors; they were becoming fairly 1. definitions may contain macro calls widespread by the early 1960s. Lots of people were 2. parenthetical notation for compounding calls thinking about macros and macro processing by 1960. [e.g., so arguments to macro calls can include In section 6 of his paper, McIlroy says, multiple items separated by commas] . . . Conditional macros were devised indepen- 3. conditional assembly dently by several persons beside the author in the past year. In particular, A. Perlis pointed 4. created symbols [e.g., so labels or local variable out that algorithms for algebraic translation names in the body of a macro definition are could be expressed in terms of conditional unique for each call of the macro] macros. Some uses of nested definitions were 5. definitions may contain definition schemata discovered by the author; their first imple- 6. repetition over a list [see the example in the mentation was by J. Benett also of Bell Tele- discussion of Midas in section 4] phone Laboratories. Repetition over a list is
David Walden TUGboat, Volume 35 (2014), No. 1 101
due to V. Vyssotsky. Perlis also noted that possible preprocessor for any other language or as a macro compiling may be done by routines to a stand-alone string processor. large degree independent of ground language. Here is a simple definition in GPM: One existing macro compiler, MICA (Haigh), §DEF,REFORMATNAME,
Macro memories, 1964–2013 102 TUGboat, Volume 35 (2014), No. 1
part of the Midas assembler for the PDP-1. PDP-1 arguments — the Midas commands IRP (indefinite Midas had its origins in MIT’s TX-0 computer, all repeat over a list of arguments) and IRPC (indefinite the way back to MACRO on the TX-0. repeat over a string of characters). In our version of MACRO was an assembler with a macro proces- Midas, the IRP command might have been used as sor capability written by Jack Dennis for the TX-0. follows: I don’t know of a manual for TX-0 macros; however, IRP A, (W1, W2, W3) MACRO was later released by DEC with its PDP-1 (ADD A computer, and Jack Dennis states17 that he wrote ) 18 the manual for that. expanding to When I asked Jack Dennis about predecessor ADD W1 technology to MACRO, he mentioned McIlroy’s pa- ADD W2 per (which was published after MACRO was available ADD W3 on the TX-0 in 1959, so perhaps Jack saw a draft and in another example or preprint). Of his MACRO, Jack said,19 “Doug’s macro processor was of the string substitution sort IRP X,Y,(A,Q,B,R,C,T) . . . Mine was different: it permitted a user to give a (CLA X name to a sequence of assembly instructions, with in- STO Y ) teger parameters that would be added to instructions to create modified addresses. (Thus the essential expanding to mechanism was one’s complement binary arithmetic CLA A instead of string concatenation.)” STO Q Next on the TX-0 came Midas, which was de- CLA B rived by Robert Saunders from TX-0 MACRO. Then, STO R TX-0 Midas was moved to the PDP-1.20 The TX-0 CLA C Midas memo21 is dated November 1962 which sug- STO T gests that the PDP-1 Midas was up and running The same thing could have been accomplished sometime in 1963, as the Midas manual for the PDP- using the command to repeat for each character in a 1 says it was ported from the TX-0 where it had been string of characters: running for about a year. IRPC X,Y,(AQBRCT) In any case, the PDP-1 editing, assembling, and (CLA X debugging set of programs was probably the best set STO Y of interactive program development and debugging ) tools that were available for a mini-computer in the I’m pretty sure that the following also worked mid-1960s. Therefore, four of us using a Univac 1219 in our version of Midas:23 computer at Lincoln Lab decided to reimplement MACRO ADDTHEM X,Y,Z 22 these PDP-1 tools for our 1219. (ENTAL X Midas for the PDP-1 and our version for the IRP W,(Z) Univac 1219 had macro processor definition and call (ADD W formats that were similar to those in the tutorial part ) of McIlroy’s paper, e.g., “MACRO NAME ARG1, ARG2 STRAL Y (string)” to define a macro and “NAME X,Y” to call ) it with X and Y to be substituted for ARG1 and ARG2 when called with ADDTHEM A,B,(C,D,E) resulted in in the macro definition. For example, ENTAL A MACRO MOVE X,Y ADD C (ENTAL X ADD D STRAL Y) ADD E The above when called with STRAL B All in all, this macro effort was a significant piece of MOVE K,L computing and programming education for me.24 resulted in ENTAL K 5 More study and use of macro processors STRAL L (and language extension capabilities) Midas also had what I now think of as map In September 1967, I moved to Bolt Beranek and commands, i.e., apply some function over a list of Newman Inc. in Cambridge, MA, where I had access
David Walden TUGboat, Volume 35 (2014), No. 1 103
to the company’s PDP-1d time sharing system. I COMPLEXADD, A, B, C, D, E, F = FETCH, A immediately began extensive use of the macro facility ADD, C built into the editing program TECO.25 STORE, E TECO22 was an early, very powerful, text edi- FETCH B tor with a macro capability using the same type of ADD D keystrokes one used for editing. One could type a list STORE F of keystrokes to do some complex editing function with the obvious substitution when called with but delay evaluation and instead save the sequence COMPLEXADD, U, V, W, X, Y, Z of keystrokes (i.e., defining a macro) and then later give a few keystrokes to execute the saved string From macros as a way of extending languages it of keystrokes (i.e., calling or executing the macro). was a short step to the idea of extensible high-level TECO macros typically looked very cryptic. People languages. From 1966–1968 I took computer sci- also played games with what complicated things they ence courses at MIT as a part-time graduate student could do with TECO macros, e.g., calculating digits and eventually did Master’s thesis work (never com- of pi, implementing Lisp, etc.26 pleted) on a capability for extending a high level Also early on at BBN, as a weekend hack, I language. Unfortunately, I have lost the complete transcribed the Algol-like listing of Strachey’s GPM draft of the thesis report (and I never finished the system from his 1965 paper into PDP-1 Midas assem- accompanying program). However, I am sure that bly language and made it run. This was easy to do my thesis literature research and thinking influenced given Strachey’s complete description of the system. the next project I had at BBN. In 1967 to 1968, with the assignment to think I also investigated the TRAC language.27,28 TRAC about a programming language for what became was presented by Calvin Mooers as a text processing BBN’s PROPHET system (a tool to help medicinal language; but to my mind, TRAC was not so different chemists and research pharmacologists), I looked from a macro processor in the way it defines and deeply into extensible languages. I studied and re- manipulates strings.29 ported on McIlroy’s ideas6 and the ideas and imple- The basic TRAC operation is a call to a built-in mentations of several other researchers.30 Eventually, function introduced by a pound sign, e.g., as a proof of concept, I translated James Bell’s Pro- #(function-name,arg1,arg2,...) teus from the Fortran implementation in his thesis Two of the built-in functions are define string (ds) into PDP-10 assembly language. I turned this effort and call (cl), as follows: over to Fred Webb, who eventually replaced what I #(ds,greeting,Hello World) had done with a fresh implementation of an extensi- and ble language named PARSEC.31 PARSEC was used #(cl,greeting) in various versions (and later as the basis for RPL for the RS/1 system) by a multitude of people for resulting in replacement of the call by the string many years. “Hello World”. This note on macros is not the place to go more Another built-in TRAC function, ss, specifies deeply into extensible languages. The references in the strings for which a substitution is to be made at note 30 are a decent introduction to the state of the call time. Thus, art circa 1968. For the state of the art a decade #(ds,greeting,Hello, name) later, John Metzner’s graded bibliography on macro #(ss,greeting,name) systems and extensible languages is relevant.5 creates a macro with one call-time argument, such that 6 Midas, macros, and the #(cl,greeting,Hello Dave) ARPANET IMP program results in the text string “Hello, Dave”. At BBN I was part of the small team that in 1969 There are other built-in functions for string com- developed the ARPANET packet switch.1,32 We de- parison, and so on. veloped the packet switch software for a modified By this time, I was pretty fascinated by program- Honeywell 516 computer using the PDP-1d Midas ming languages and macros, in particular the idea assembler, using lots of macros, etc., to adapt Midas of extensions to programming languages. to know about the 516’s instruction set and paged Macros have often been used as a form of lan- memory environment. Bernie Cosell primarily con- guage extension. For instance, complex add might structed this hairy set of conversation macros. Our be defined, using McIlroy’s notation, as three person software team (Cosell, Will Crowther,
Macro memories, 1964–2013 104 TUGboat, Volume 35 (2014), No. 1
and me) also used Midas macros to facilitate devel- If this is called with \Greeting{Dave}, it results in opment of the packet switching algorithms to run on the 516. All this may be seen in a listing of the Hi Dave! I hope you’re well. IMP program available on the web.33 We also used The text “#1” tells the macro definition processor Midas macros to generate a concordance for the IMP (a) that there is one argument, and (b) where the program34 as well as to reduce the probability of call-time argument is to be substituted in the body of writing time-sharing bugs.35 A contemporary ver- the macro. If the macro definition allowed three ar- sion of this Midas macro assembler written in Perl guments, for instance, the sequence “#1#2#3” would by James Markevitch is also available for study.36 appear after the macro name and before the open curly bracket of the definition. 7 Ratfor and Infomail The TEX macro processor is enormously pow- The next project on which I saw something like a erful and flexible, in its unique way, and a compre- macro processor was an effort in the early 1980s to hensively documented piece of software.39,40,41,42 develop a commercial email system that would run Massive programs have been (and continue to be) on a variety of vendor platforms, e.g., DEC VAX, written in its macro language. For example, LATEX is IBM CICS, IBM 360, and Unix on the BBN C/70. For implemented entirely with TEX macros, as are other portability we decided to implement our email sys- variants or supersets of TEX (called “formats” in tem (called InfoMail) in Fortran, for which compilers TEX jargon) as well as thousands of LATEX “packages” already existed for the target platforms. However, to which extend or modify the capabilities of LATEX. avoid having to actually write Fortran code, we devel- One modest-size example of the use of TEX macros 37 oped the system using Ratfor (Rational Fortran). to extend LATEX can be found on pages 6–10 of “The Ratfor was not actually a macro processor but rather bibtext Style Option”.43 a programming language that acted as a preprocessor Personally, I tend to do my LATEX extensions us- to emit a Fortran program that could be compiled ing packages that other people have already written, by a standard Fortran compiler. Nonetheless, that although sometimes I make minor modifications to seems to me to be quite a lot like what a macro pro- existing packages. More commonly I use TEX macros cessor does. Also, Chapter 8 of the Software Tools (or a LATEX variation) to replicate small snippets of book (see previous note) describes the implementa- LATEX code which are used repeatedly — for consis- tion in Ratfor of a macro processor (based on the tency, and also to easily allow later changes of such macro processor for the programming language C). snippets as I figure out what I actually want them to Also during my BBN years between 1967 and 1995, I do. In 2004 I published an example of one such use 44 briefly used the C language, which includes a macro of a TEX macro; I encourage readers to take a look processor,38 which optionally can be used indepen- at it as it describes (far from completely) various dently of the rest of C. But from 1982 on I did no real ways TEX macros are defined and can be called. computer programming and thus didn’t track what Historically, there has been an interesting set of was happening in the world of macro processors. pressures around TEX’s macro capability. Originally Donald Knuth included only enough macro capability 8 TEX, macros for typesetting to implement his typesetting interface. However, he was persuaded by early users to expand that macro People who use (LA)TEX macros may already know much of what is in this section. However, some of it capability. That expanded capability allowed users to may be new to some readers. construct pretty much any logic they wanted on top of TEX (although often such add-on logic was awk- After retirement from BBN in 1995, I had started ward to code using macro-type string manipulations). using (LA)TEX in place of a word processor such as Mi- On the one hand, TEX and its macro-implemented crosoft Word, particularly for documents that were derivatives have always been very popular and there more than a one-page, one-time letter. This brought have been non-stop macro-based additions for over me in contact with the very sophisticated and com- 30 years. On the other hand, users despair at how plex macro processor embedded in the TEX type- annoying coding using macros is, moan about “why setting system. I have now been using this system Knuth couldn’t have included a real programming and its macro processor for typesetting for nearly 20 language within TEX”, and otherwise cast aspersions years, the longest in my life I have used any single on TEX’s macro capability. macro processor. Over the years various attempts have been made Here is an example of a TEX macro. to link TEX to a “real” programming language, typi- \def\Greeting#1{Hi #1! I hope you’re well.} cally invoking the programming language from TEX
David Walden TUGboat, Volume 35 (2014), No. 1 105
or the reverse; however, none of these efforts have call arguments can be specified with attribute-name/ come into widespread use. Over the past few years, value pairs. There are explicit commands in TEX however, a small group of TEX hackers have accom- creating local or global definitions, as well as various plished an apparently successful merger of the Lua other definition variations, such as delayed defini- programming language and TEX, called LuaTEX.45 tion and delayed execution of macro calls. TEX has LuaTEX maintains TEX’s macro processor (there is a rich rather than minimal set of conditional and really no way, and no reason, to get rid of it while arithmetic capabilities (some related only to position keeping TEX). Thus, the full power of the TEX macro in typesetting a page). There are also ways to pass processor is available for the many situations in which information between macros and, more generally, to it is the best tool, and the Lua language is available hold things to be used later during long, complicated for things which can be done much more easily in a sequences of evaluation and computation. These ca- procedural language. pabilities allow big programs to be written in the The history of the TEX macro processor partially macro language, and thus TEX also has a capability explains the above. Knuth has made the point that to trace the flow of macro definition and execution. he was designing a typesetting system that he didn’t TEX has another unusual capability that is some- want to make too fancy, i.e., by including a high times used with macros, although it is a capability level language. He has also noted that when he was more closely related to lexical analysis than to macro designing TEX he created some primitive typesetting definitions and calls; this is the TEX “category code” operations and then created a set of macros for the feature. TEX turns its input sequence of charac- more complete typesetting environment he wanted. ters into a list of tokens. A token is either a single He expanded the original macro capability when character with a category code (catcode) or a con- fellow Stanford professor Terry Winograd wanted trol sequence. For instance (using an example from to do fancier things with macros. Knuth’s idea was Knuth’s The TEXbook), the input “{\hskip 36 pt}” that TEX and its macro capability provided a facility is tokenized into with which different people could develop their own typesetting user interfaces, and this has happened { hskip 3 6 p t } to some extent, e.g., LATEX, ConTEXt, etc. where the opening brace is given the catcode of 1 It is perhaps worth discussing a few of the things (begin group), hskip gets no catcode because it is a that make the TEX macro capability different from, control sequence, 3 and 6 get catcodes of 12 (other), for example, the capability of GPM. the space after 36 gets catcode 10 (space), p and t GPM has a simple and unchanging definition and get catcodes of 11 (letter), and the closing brace gets calling syntax, as was described in Section 3. Macro catcode 2 (end group). (There are 16 such category definitions can include other macro definitions, and codes in all.) macros can have recursive calls (and without going The next step in the TEX engine decides what to back to study the GPM paper carefully, I assume that do with these different tokens, e.g., put the numbers the scope of macro definitions happens in the natural together into a numeric value and the letters together and obvious way). The definition associated with a into a unit of measurement, execute the primitive macro name can be “looked at” without evaluating hskip command, and so on. The backslash has a the definition; the definition associated with a macro catcode of zero indicating an escape character, thus name can be assigned to a different macro name; and telling TEX that the following letters (five in this there is a capability for converting numbers between case) form a control sequence; the space after the binary and decimal formats and for doing binary command name delimits the end of the name and arithmetic. No limit is specified for the number doesn’t otherwise count as a space. of arguments a macro definition and call can have. TEX also has the capability of changing which Finally, GPM is a stand-alone program; it processes character(s) have which catcode(s). For instance, the its input, and it is up to the user what happens next dollar sign (by default having catcode 3, indicating with its output. a shift to math mode) could be given the escape The TEX macro capability can do all those character catcode, and the backslash could be given things; but it cannot be used independent of TEX, the math shift catcode. This capability is routinely and in its straightforward form its macro definitions used in TEX program libraries to define macro names and calls are limited to nine arguments. TEX also internal to a program in the library which users has a way of defining a macro call to have a much of the library cannot use when (normally) defining more free-form format, and with some programming their own macro names. Basically, the entire input (or use of an appropriate TEX macro package) macro language of TEX can be changed.
Macro memories, 1964–2013 106 TUGboat, Volume 35 (2014), No. 1
The typeface design system, METAFONT, that that were relevant to the chapters of a book I co- Knuth developed in parallel with TEX has a more authored.51 I found the m4 macro processor.52 powerful macro capability (my memory is that he The following first entry in an HTML table:53 says somewhere that this was because he assumed more sophisticated people would be using META- FONT than T X). E was created with the following m4 macro definition When Knuth rewrote T X and METAFONT us- E that outputs HTML markup: ing literate programming techniques and targeting 46 define(‘_te’,‘ the Pascal programming language, he included a
David Walden TUGboat, Volume 35 (2014), No. 1 107
convert the O-code output of his BCPL compiler macro processors tend to operate at compile time, into machine language)57 sometimes in ways that increase runtime efficiency, • as an intermediate step in the work flow (for in- there probably hasn’t been as much interest in op- stance, a Perl program processes an input driver timizing macro processor performance as there has file to generate m4 macro calls, the macro calls been with programming languages. generate LATEX markup, and the LATEX engine From the point of view of the professional pro- generates PDF files from the LATEX markup). grammer (versus historian), macro processors are just another development tool, and they are avail- 10 Reflections able as preprocessors or embedded in whatever pro- General reflections. A computing historian who gramming language or other development tools the is a potential reader of this note might ask, “Why programmer is using. Details about how a macro should I care about macro processors?” There are processor works may annoy but probably won’t de- a couple of answers to this. First, macro proces- ter such working programmers. They just use what sors played an important role in the history of pro- is available as best they can, and don’t write the gramming languages. They were used with early theoretical articles. Practical use of macros also assemblers, before higher level languages became tends to be awkward, and describing their use can widespread. In fact, they allowed assembly language be messy. Thus, it is difficult in a short paper to users to create higher level language constructs by describe serious uses of macros. grouping together useful sequences of assembly lan- On the other hand, people who might be reluc- guage instructions under a single name, for exam- tant to use a “real” programming language may use ple, complex-add. As higher level languages came macros (e.g., editor macros) in quite sophisticated into use (e.g., Fortran, ALGOL), the usefulness of ways. In the multi-chapter interview of Knuth in his macro processors was carried over into higher level Companion to the Selected Papers of Donald Knuth languages (e.g., to create yet higher level linguis- (page 161),58 Knuth says of his wife, “I don’t think tic constructs). Soon stand-alone macro processors she will ever enjoy programming. She is good at were created that could act as a preprocessor for any creating macros for a text editor, sometimes impress- programming language rather than being embedded ing me with subtle tricks that I didn’t think of, but within a particular programming language. Second, macro writing is quite different from creation of what both embedded and stand-alone macro processors we call ‘recursive procedures’ or even ‘while loops.’ ” continue in widespread use today. We also find macro Macro processors are also not as tractable a topic processors embedded in the user interface of many for historical or theoretical writing as high level lan- computer tools other than programming languages, guages perhaps are. While many macro processors for instance, as part of text editors, operating system may be for relatively straightforwardly extending a shells, makefiles, and other applications and software programming language or aiding in a development packages (one example is Actions in Photoshop — task, some macro processors (such as m4 and the repeatable sequences of Photoshop commands). macro capability in TEX) have a big set of capa- The reader might next ask, “If macro processors bilities for dealing with sophisticated programming have been important in the history of programming, situations — they are set up to write big complicated why isn’t more written about them?” I think the programs, just as regular higher level languages are, answer to that question may have to do with the but often in more clumsy-to-use ways. There is a relative simplicity of many macro processors, which 160-page manual to describe all the capabilities of substitute one character string for another. That is the stand-alone m4 macro processor. In addition to not an area that requires much research, and gener- Knuth’s own writings,39,41 Eijkhout’s book40 has ally useful implementation methods have been well more than 50 pages on the extensive macro capabil- known since the 1960s or earlier; also, it is not too ities embedded in TEX, and Stephan von Bechtols- hard to do an ad hoc implementation of a macro heim wrote (perhaps excessively) a 650-page book59 processor as part of some piece of software one is on using TEX macros. (There is probably an interest- writing. Being mostly implemented as string manip- ing paper to be written comparing the issues inherent ulators, macro processors don’t have to have all of in these powerful macro processors with similar issues the debatable research topics that a full program- in regular procedural programming languages.) ming language has (side effects, call by name, scope, The immediately preceding discussion brings to etc. — although the sophisticated macro processors mind the question, “Where on the spectrum of pro- such as m4 and TEX have parallels for these sorts of gramming languages do we put macro processors?” programming-language-philosophy topics). Also, as
Macro memories, 1964–2013 108 TUGboat, Volume 35 (2014), No. 1
Early on they were used as ways to extend as- sor and using one for production work includ- semblers and compilers. Quickly they were used ing retargeting Midas to a new computer and to implement programming languages (e.g., WISP putting hints in a real-time program to reduce and SNOBOL). They were used to simulate different the probability of time-sharing bugs. “computers” (e.g., BCPL’s compiler of O-code, the • TECO and other editor macros were to reduce assembly language of the Honeywell 516 IMP com- editing keystrokes, especially when regular ex- puter). I personally have used them to define and pression searches and replacements are needed. process little special purpose markup languages (as • Ratfor and C were for programming efficiency. described in the m4 and TEX sections above). • TEX macros are for production typesetting, cre- Many macro processors are Turing-complete and ating extensions to LATEX (e.g., the style for a in this sense can compute anything a typical higher particular book), and as another sophisticated level language can compute. However, they are dis- macro processor to study. tinct from many higher level language compilers (dis- • m4 use has been to create new markup languages counting macro capabilities they might have, or other with which to target document source files to compile-time evaluation capabilities, as languages different output media. which execute interpretively often have, e.g., Lisp, In addition to assembly languages for different com- Perl) in that much computation in macro processors puters, Fortran for various mathematical projects, inherently goes on at “compile” time. This can make and the languages mentioned above, I briefly pro- their syntax and semantics harder to specify. grammed in Lisp and BCPL years ago. Lisp has In Knuth’s paper “The Genesis of Attribute a macro capability and I may have used it, but, if Grammars”, reprinted in his Selected Papers on Com- so, I don’t remember anything about it.61 While 60 puter Languages, Knuth explains (pages 434–435) writing these notes, I have learned that a version of that he didn’t use an attribute grammar to specify Strachey’s GPM existed for BCPL, known as BGPM. the semantics of TEX because he hadn’t been able to In recent years I have used Perl for several large think of any good way to define TEX precisely except and many small projects. Apparently Perl has a through the implementation using Pascal. He also capability for redefining parts of the Perl language couldn’t think of how “to define any other language (the equivalent of macros and more), but it confused for general-purpose typesetting that would have an me when I tried to read about it and am not likely easily defined semantics, without making it a lot less to try it. I suppose I can just use m4, which I already useful than TEX. The same remarks also apply to know, as a preprocessor for Perl; but that doesn’t METAFONT . ‘Macros’ are the main difficulty. As far stop me from wishing that a conventional macro as I know, macros are indispensable but inherently capability was built into Perl. difficult to define. And when we also add a capabil- ity to change the interpretation of input characters, Acknowledgments dynamically, we get into a realm for which clean Chuck Niessen helped me remember the capabili- formalisms seem impossible.” ties of the Univac 1219 macro assembler we wrote. Ralph Alter and Will Crowther responded to queries Personal reflections. Macro overview books, such relating to macro capabilities on Lincoln Laboratory as those by Brown and Campbell-Kelly, try to cate- computers. Stan Mazor confirmed my memory about gorize the uses of macro processors, for example, to whether or not we used macros in the early 1960s extend a programming language, to allow late bind- on the IBM 1620 at San Francisco State College. ing of variables to values, to communicate with the Tom Van Vleck reminded me of what languages with operating system, to implement desirable-but-non- macro processors were available at MIT in the late existent machine instructions, to insert debugging CTSS and early Multics era. Jim Wood reminded code into a program, and to simply abbreviate long me of some things about PARSEC. Bill Aspray and strings of text. Below is a short list of the ways I feel Martin Campbell-Kelly provided guidance on how I have made use of the macro processors described this paper might be developed into a more traditional in this note. academic computing history paper. Bernie Cosell • The macro processor described in McIlroy’s pa- pointed out half a dozen subtle issues or possibili- per, Strachey’s GPM, and Mooers’ TRAC were ties in the paper. Ralph Muha gave me his copy of for the study of programming languages, under- Gimpel’s SNOBOL4 algorithms book. Most helpfully, standing of how macro processors work, and for Karl Berry read drafts of this note, noted typos, coding practice. made suggestions for improvement, and provided ad- • Midas was about implementing a macro proces- ditional insight about and references for a number
David Walden TUGboat, Volume 35 (2014), No. 1 109
of macro processors, particularly TEX. Karl, along for the Study of Language and Information, Stanford with Barbara Beeton, also edited the final draft of University, 2003. the paper to ready it for publication. 11 Page 6 in the volume of Knuth’s selected papers. 12 Page 42 of the selected papers volume. Notes 13 Martin Campbell-Kelly told me [email of 2013-11-23], 1 http://walden-family.com/impcode/imp-code.pdf “Maurice Wilkes developed a macro-based system called 2 [Macro overview] P. J. Brown, Macro Processors and WISP for list processing (http://ai.eecs.umich.edu/ people / conway / CSE / M.V.Wilkes / M.V.Wilkes-Tech. Techniques For Portable Software, John Wiley & Sons, 2 1974. Memo.63.5.pdf) . . . Peter Brown was Wilkes’ PhD stu- dent.” 3 [Macro overview] Martin Campbell-Kelly, An Introduc- 14 [GPM ] C. Strachey, A General Purpose Macrogenera- tion to Macros (Computer monographs, 21), MacDonald tor, The Computer Journal, vol. 8 no. 3, 1965, pp. 225– & Co., London, 1974. 241. 4 [Macro overview] Peter Wegner, Programming Lan- 15 See Section 1.2, Historical References, at http://www. guages, Information Structures, and Machine Organiza- gnu.org/software/m4/manual/ tion, McGraw-Hill Book Company, 1968, section 2.6 and 16 section 3.1–3.4, pp. 130–180. [GPM ] Another implementation of GPM can be found in Section 8.8 of James F. Gimpel’s book Algorithms 5 [Macro overview] John R. Metzner, A graded bibliog- in SNOBOL4, John Wiley & Sons, 1976. (SNOBOL4 it- raphy on macro systems and extensible languages, ACM self was implemented using a collection of macros which SIGPLAN Notices, Volume 14 Issue 1, January 1979, create a virtual machine for the purpose of language pp. 57–64. portability: Ralph E. Griswold, The Macro Implemen- 6 [McIlroy’s model] M.D. McIlroy, Macroinstruction Ex- tation of SNOBOL4, W.H. Freeman and Company, San tensions for Compiler Languages, Communications of the Francisco, 1972. SNOBOL4 is sometimes known as “Macro ACM, vol. 3 no. 4, 1960, pp. 214–220. SNOBOL”.) 7 In 1967 McIlroy’s paper was reprinted in the now classic 17 Email of November 16, 2013. book Programming Systems and Languages, edited by 18 [Midas] MACRO Assembly Program for Programmed Saul Rosen, McGraw-Hill, New York, 1967. For this paper Data Processor-1 (PDP-1), Digital Equipment Corpora- I was looking at the reprint of Rosen’s book. In the reprint tion, Maynard, MA, 1963, http://bitsavers.informatik. McIlroy doesn’t say anything about the macro name in uni-stuttgart.de/pdf/dec/pdp1/PDP-1_Macro.pdf this example also being used within the definition but 19 Email of October 28, 2013. not apparently as a recursive call of the macro, i.e., the 3- 20 argument call can be distinguished from the 1-argument [Midas] http://ia601609.us.archive.org/9/items/ instruction. bitsavers_mitrlepdp1_1535627/PDP-1_MIDAS.pdf 21 [Midas] 8 [Macros at Bell Labs] May 4, 2009, http://ia601601.us.archive.org/11/items/ bitsavers_mittx0memo_2363951/ http://deememorial.blogspot.com/2009/05/ M-5001-39_MIDAS_Nov62.pdf doug-mcilroy-recalls-bell-labs.html 22 Ralph Alter wrote a version of TECO, Dan Murphy’s 9 [Other macro systems] M. Barnett, Macro-directive paper Tape Editing and Correcting Program: Dan Mur- Approach to High Speed Computing, Solid State Physics phy, The Beginnings of TECO, IEEE Annals of the History Research Group, MIT, Cambridge, MA, 1959; D.E. East- of Computing, 31(4), October–December 2009, pp. 225– wood and M.D. McIlroy, Macro Compiler Modifications 241. Will Crowther wrote a version of Alan Kotok’s DDT of SAP, Bell Telephone Laboratories Computation Cen- debugging program. Chuck Niessen and I wrote a version ter, 1959; I.D. Greenwald, Handling of Macro Instruc- of Robert Saunder’s macro assembler: Chuck wrote the tions, Communications of the ACM, vol. 2 no. 11, 1959, basic assembler and I wrote the macro-processor. pp. 21–22; M. Haigh, Users Specification of MICA, SHARE 23 User’s Organization for IBM 709 Electronic Data Pro- I have the line printer assembly listing and my hand- cessing Machine, SHARE Secretary Distribution SSD-61, written flow chart for this macro processor. It is tempting C-1462, 1959, pp. 16-63; A.J. Perlis, Official Notice on to try to make this macro processor run again on a 1219 ALGOL Language, Communications of the ACM, vol. 1 emulator. no. 12, 1958, pp. 8–33; A.J. Perlis, Quarterly Report of 24 This was the first program in which I used stacks the Computation Center, Carnegie Institute of Technol- and for which I thought about recursion in a profound ogy, October. 1969; Remington-Rand Univac Division, way. I think this was also the first time I wrote code for Univac Generalized Programming, Philadelphia, 1957. managing a symbol table. 25 10 [Other macro systems] This article was originally pub- I don’t remember using this capability in TECO at lished in Volume 7 of the Encyclopedia and Computer Lincoln Lab. Science and Technology, published by Michel Dekker in 26 I won’t bother with further mention of macros in other 1977. The article is reprinted as chapter 1 of Donald E. editors (or macros in various job control languages, shells, Knuth, Selected Papers on Computer Languages, Center or makefiles) in the rest of this note.
Macro memories, 1964–2013 110 TUGboat, Volume 35 (2014), No. 1
27 43 [TRAC ] Calvin Mooers and Peter Deutsch, TRAC:A [TEX ] http://mirror.ctan.org/macros/latex209/ Text Handling Language, Proceedings of the 20th ACM contrib/biblist/biblist.pdf 44 National Conference, 1965, pp. 229-246. [TEX ] http://tug.org/TUGboat/tb25-2/ 28 As part of my investigation, I visited TRAC creators tb81walden.pdf 45 Calvin Mooers at his Cambridge office (of the Rockford [TEX ] http://www.luatex.org/ Research Institute) and Peter Deutsch at the Cambridge 46 [TEX ] Donald E. Knuth, Literate Programming, home of his parents. http://literateprogramming.com/knuthweb.pdf 29 2 Brown’s book in fact classifies TRAC as a macro 47 Email of December 5, 2013, from Karl Berry. processor. 48 Private communication, January 11, 2014; the foot- 30 [Extensible languages] Niklaus Wirth, On Certain Basic notes in the quotation are from the author of the present Concepts of Programming Languages, Technical Report paper, not from Knuth. No. CS 65, Computer Science Department, Stanford Uni- 49 D. V. Schorre, META-II: A Syntax-oriented Compiler versity, May 1, 1967; B.A. Galler and A.J. Perlis, A Writing Language, D. V. Schorre, Proceedings of the ACM, Proposal for Definitions in ALGOL, Communications of 19th ACM National Conference, ACM, New York, 1964, the ACM, vol. 10 no. 4, April 1967, pp. 204–219; T.E. http://ibm-1401.info/Meta-II-schorre.pdf Cheatham, Jr., A. Fischer, and P. Jorrand, On the Basis 50 for ELF — An extensible language facility, AFIPS Fall Edgar T. Irons, A syntax-directed compiler for ALGOL Joint Computer Conference, 1968, pp. 937–948; J.V. Gar- 60, Communications of the ACM, vol. 4, 1961, pp. 51– wick, J.R. Bell, and L.D. Krider, The GPL Language, 55; Edgar T. Irons, The structure and use of a syntax- Programming Technology Report TER-05, Control Data directed compiler, Annual Review of Automatic Program- Corporation, Palo Alto, CA; Thomas A. Standish, A Data ming 3, 1962, pp. 207–227. Definition Facility for Programming Languages, PhD the- 51 http://walden-family.com/4prim/ sis, Carnegie Institute of Technology, 1967; James Rich- 52 [M4] http://www.gnu.org/software/m4/manual/ ard Bell, The Design of a Minimal Expandable Program- 53 The full table is at http://walden-family.com/ ming Language, PhD thesis, Stanford University, 1968. 4prim/archive/issues-list.htm 31 [Extensible languages] F. Webb, The PROPHET Sys- 54 [M4] http://en.wikipedia.org/wiki/ tem: An Overview of the PARSEC Implementation, BBN M4_(computer_language) Report 2319, September 1, 1972; PARSEC User’s Manual, 55 While m4 was originally developed by Brian Kernighan Bolt Beranek & Newman, Cambridge, MA, December and Dennis Ritchie in 1977 and released as part of AT&T 1972. Unix, the GNU version mentioned here was a complete 32 F.E. Heart et al., The Interface Message Processor rewrite by Ren´eSeindal with continuing updates by many for the ARPA Computer Network, AFIPS Conference others. The GNU m4 manual’s history section15 has a good Proceedings 36, June 1970, pp. 551–567. bit of additional history. 33 http://walden-family.com/impcode/ 56 http://tug.org/interviews/ c-listing-ps.txt 57 Martin Richards, Christopher Strachey and the Cam- 34 http://walden-family.com/impcode/ bridge CPL Compiler, Higher-Order and Symbolic Com- d-concordance.pdf putation (a special Christopher Strachey memorial issue), 35 http://walden-family.com/impcode/ 13, 2000, pp. 85–88. detect-interrupt-bugs.pdf 58 Donald E. Knuth, Companion to the Selected Papers 36 [Midas] http://walden-family.com/impcode/ of Donald Knuth, Center for the Study of Language and midas516.txt Information, Stanford University, 2012. 59 37 [Ratfor] Kernighan, B. and Plauger, P., Software Tools, [TEX ] Stephan von Bechtolsheim, TEX in Practice, Addison-Wesley, 1976. Volume III: Tokens, Macros, Springer-Verlag, 1993. 60 38 [C preprocessor] http:/ /gcc.gnu.org/onlinedocs/ Donald E. Knuth, Selected Papers on Computer Lan- cpp/ guages, Center for the Study of Language and Informa- 39 tion, Stanford University, 2003. [TEX ] Donald Knuth, The TEXbook, Addison-Wesley, 61 Tim Hart at MIT added a macro capability to Lisp in 1986, particularly chapter 20. 1963. Macros in Warren Teitelman’s BBN Lisp, which I 40 [TEX ] Victor Eijkhout, TEX by Topic, Addison-Wesley, used, were perhaps more well developed. In some sense 1991, particularly chapters 11–14, http://mirror.ctan. the original Lisp interpreter functioned a bit like a macro org/info/texbytopic processor. 41 [TEX ] Donald Knuth, Computers & Typesetting, Vol- ume B, TEX: The Program, Addison-Wesley, 1986. David Walden 42 http://walden-family.com [TEX ] Donald E. Knuth, Digital Typography, Center for the Study of Language and Information, Stanford University, 1999.
David Walden