some text
In ConTEXt MkII, the version that deals with pdf-
<<
\the\count0
everything, but it is a step forward. Also, compared Here we cannot be sure that a cell does not con- to TEX and MetaPost, grouping is done differently: tain a nested table, which is why we need to define there is a local prefix that makes variables (and the mapping as follows: functions are variables too) local in modules, func- \defineXMLnested[table]{\bTABLE} {\eTABLE} tions, conditions, loops, etc. The Lua code in this \defineXMLnested[tr] {\bTR} {\eTR} article contains such locals. \defineXMLnested[td] {\bTD} {\eTD} 7 An example: XML The \defineXMLnested macro is rather messy because it has to collect snippets and keep track of In practice most users will use a macro package and the nesting level, but users don’t see that code, they so, if a user sees T X, he or she sees a user interface, E just need to know when to use what macro. Once it not the code behind it. As such, they will also not works, it keeps working. encounter the code written in Lua that handles, for Unfortunately mappings from source to style instance, fonts or node list manipulations. If a user are never that simple in real life. We usually need sees Lua, it will most probably be in processing ac- to collect, filter and relocate data. Of course this tual data. Therefore, in this section I will give an ex- can be done before feeding the source to T X, but ample of two ways to deal with XML: one more suit- E MkII provides a few mechanisms for that too. For able for traditional T X, and one inspired by Lua. It E instance, to reverse the order you can do this: demonstrates how the availability of Lua can result in different solutions for the same problem. Whatever 7.1 MkII: stream-based processing Someone T X and X T X, we use a stream-based XML parser, E E E \defineXMLenvironment[article] written in T X. Each < and & triggers a macro that E {\defineXMLsave[author]} then parses the tag and/or entity. This method is {\blank author: \XMLflush{author}} quite efficient in terms of memory but the associ- author ated code is not simple because it has to deal with This will save the content of the element and article attributes, namespaces and nesting. flush it when the end tag is seen. So, given The user interface is not that complex, but in- previous definitions, we will get the title, some text volves quite a few commands. Take for instance the and then the author. You may argue that instead we should use for instance XSLT but even then a following XML snippet: mapping is needed from the XML to TEX, and it’s a matter of taste where the burden is put. Because ConT Xt also wants to support stan- Whatever E these do a good job in most cases, the code associ- ated with the solutions has never been satisfying. Supporting XML this way is doable, and Con- When using ConTEXt commands, we can imag- ine the following definitions: TEXt has used this method for many years in fairly complex situations. However, now that we have Lua \defineXMLenvironment[document] available, it is possible to see if some things can be {\starttext} {\stoptext} \defineXMLargument [title] done more simply (or differently). {\section} 7.2 MkIV: tree-based processing \defineXMLenvironment[p] {\ignorespaces}{\par} After some experimenting I decided to write a full When attributes have to be dealt with, for in- blown XML parser in Lua, but contrary to the
TUGBOAT
Volume 29, Number 3 / 2008 TUG 2008 Conference Proceedings
TUG 2008 350 Conference program, delegates, and sponsors th 352 Peter Flynn / TUG 2008: TEX’s 30 birthday LATEX 356 Niall Mansfield / How to develop your own document class — our experience Software & Tools 362 Jonathan Kew / TEXworks: Lowering the barrier to entry 365 J´erˆome Laurens / Direct and reverse synchronization with SyncTEX 372 Joachim Schrod / Xindy revisited: Multi-lingual index creation for the UTF-8 age 380 Taco Hoekwater / MetaPost developments: MPlib project report 383 Hans Hagen / The TEX–Lua mix 376 Joe McCool / A newbie’s experiences with Lilypond, Lilypond-book, LATEX and Perl 392 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab / docx2tex: Word 2007 to TEX 401 Jean-Michel Hufflen / Languages for bibliography styles
Dreamboat 413 Chris Rowley / Vistas for TEX: liberate the typography! (Part I) Fonts 418 Dave Crossland / Why didn’t METAFONT catch on? 421 Karel P´ıˇska / Creating cuneiform fonts with MetaType1 and FontForge 426 Ulrik Vieth / Do we need a ‘Cork’ math font encoding? 435 Ameer Sherif and Hossam Fahmy / Meta-designing parameterized Arabic fonts for AlQalam
Graphics 444 Manjusha Joshi / Smart ways of drawing PSTricks figures 446 Hans Hagen / The MetaPost library and LuaTEX Philology 454 Mojca Miklavec and Arthur Reutenauer / Putting the Cork back in the bottle — Improving Unicode support in TEX 458 Stanislav Jan Sarmanˇ / Writing Gregg Shorthand with METAFONT and LATEX Macros 462 Hans Hagen / The LuaTEX way: \framed Electronic Documents 464 Ross Moore / Advanced features for publishing mathematics, in PDF and on the Web 474 John Plaice, Blanca Mancilla and Chris Rowley / Multidimensional text 480 Manjusha Joshi / Data mining: Role of TEX files Abstracts 482 Abstracts (Fine, Hagen, Henkel, Hoekwater, Høgholm, K¨uster, Mancilla et al., Mittelbach, Peter, Rahilly et al., Rhatigan, Veytsman & Akhmadeeva, Veytsman) News 485 Calendar 486 TUG 2009 announcement
Advertisements 487 TEX consulting and production services TUG Business 484 TUG institutional members 488 TUG 2009 election TEX Users Group Board of Directors TUGboat (ISSN 0896-3207) is published by the Donald Knuth, Grand Wizard of TEX-arcana † ∗ TEX Users Group. Karl Berry, President Kaja Christiansen∗, Vice President Memberships and Subscriptions David Walden∗, Treasurer 2008 dues for individual members are as follows: ∗ Susan DeMeritt , Secretary Ordinary members: $85. Barbara Beeton Students/Seniors: $45. Jon Breitenbucher The discounted rate of $45 is also available to Steve Grathwohl citizens of countries with modest economies, as Jim Hefferon detailed on our web site. Klaus H¨oppner Membership in the T X Users Group is for the E Dick Koch calendar year, and includes all issues of TUGboat Ross Moore for the year in which membership begins or is Arthur Ogawa renewed, as well as software distributions and other Steve Peter benefits. Individual membership is open only to Cheryl Ponchin named individuals, and carries with it such rights TUG Philip Taylor and responsibilities as voting in elections. For † TUG Raymond Goucher, Founding Executive Director membership information, visit the web site. † TUGboat Hermann Zapf, Wizard of Fonts Also, (non-voting) subscriptions are ∗ available to organizations and others wishing to member of executive committee †honorary receive TUGboat in a name other than that of an individual. The subscription rate is $95 per year, See http://tug.org/board.html for a roster of including air mail delivery. all past (and present) board members, and other official positions. Institutional Membership Institutional membership is a means of showing con- Addresses Electronic Mail (Internet) tinuing interest in and support for both TEX and TEX Users Group the TEX Users Group, as well as providing a dis- P. O. Box 2311 General correspondence, counted group rate and other benefits. For further Portland, OR 97208-2311 membership, subscriptions: information, see http://tug.org/instmem.html or U.S.A. [email protected] contact the TUG office. Telephone Submissions to TUGboat, +1 503 223-9994 letters to the Editor: TEX is a trademark of the American Mathematical [email protected] Society. Fax Technical support for +1 206 203-3960 TEX users: Copyright c 2008 TEX Users Group. Copyright to individual articles within this publication [email protected] remains with their authors, so the articles may not be Web Contact the Board reproduced, distributed or translated without the authors’ permission. http://tug.org/ of Directors: For the editorial and other material not ascribed to http://tug.org/TUGboat [email protected] a particular author, permission is granted to make and distribute verbatim copies without royalty, in any medium, Have a suggestion? Problems not resolved? provided the copyright notice and this permission notice are The TUG Board wants to hear from you: preserved. Please email [email protected]. Permission is also granted to make, copy and distribute translations of such editorial material into another language, except that the TEX Users Group must approve translations [printing date: October 2008] of this permission notice itself. Lacking such approval, the original English permission notice must be included. Printed in U.S.A. The Communications of the TEX Users Group
Volume 29, Number 3, 2008 TUG 2008 Conference Proceedings TEX Users Group Board of Directors TUGboat (ISSN 0896-3207) is published by the Donald Knuth, Grand Wizard of TEX-arcana † ∗ TEX Users Group. Karl Berry, President Kaja Christiansen∗, Vice President Memberships and Subscriptions David Walden∗, Treasurer 2008 dues for individual members are as follows: ∗ Susan DeMeritt , Secretary Ordinary members: $85. Barbara Beeton Students/Seniors: $45. Jon Breitenbucher The discounted rate of $45 is also available to Steve Grathwohl citizens of countries with modest economies, as Jim Hefferon detailed on our web site. Klaus H¨oppner Membership in the T X Users Group is for the E Dick Koch calendar year, and includes all issues of TUGboat Ross Moore for the year in which membership begins or is Arthur Ogawa renewed, as well as software distributions and other Steve Peter benefits. Individual membership is open only to Cheryl Ponchin named individuals, and carries with it such rights Philip Taylor and responsibilities as voting in TUG elections. For Raymond Goucher, Founding Executive Director † membership information, visit the TUG web site. † TUGboat Hermann Zapf, Wizard of Fonts Also, (non-voting) subscriptions are ∗ available to organizations and others wishing to member of executive committee †honorary receive TUGboat in a name other than that of an individual. The subscription rate is $95 per year, See http://tug.org/board.html for a roster of including air mail delivery. all past (and present) board members, and other official positions. Institutional Membership Institutional membership is a means of showing con- Addresses Electronic Mail (Internet) tinuing interest in and support for both TEX and TEX Users Group the TEX Users Group, as well as providing a dis- P. O. Box 2311 General correspondence, counted group rate and other benefits. For further Portland, OR 97208-2311 membership, subscriptions: information, see http://tug.org/instmem.html or U.S.A. [email protected] contact the TUG office. Telephone Submissions to TUGboat, +1 503 223-9994 letters to the Editor: TEX is a trademark of the American Mathematical [email protected] Society. Fax Technical support for +1 206 203-3960 TEX users: Copyright c 2008 TEX Users Group. Copyright to individual articles within this publication [email protected] remains with their authors, so the articles may not be Web Contact the Board reproduced, distributed or translated without the authors’ permission. http://tug.org/ of Directors: For the editorial and other material not ascribed to http://tug.org/TUGboat [email protected] a particular author, permission is granted to make and distribute verbatim copies without royalty, in any medium, Have a suggestion? Problems not resolved? provided the copyright notice and this permission notice are The TUG Board wants to hear from you: preserved. Please email [email protected]. Permission is also granted to make, copy and distribute translations of such editorial material into another language, except that the TEX Users Group must approve translations [printing date: October 2008] of this permission notice itself. Lacking such approval, the original English permission notice must be included. Printed in U.S.A. TUG 2008 Proceedings University College Cork Ireland July 21–24, 2007
COMMUNICATIONS OF THE TEX USERS GROUP TUGBOAT EDITOR BARBARA BEETON PROCEEDINGS EDITOR KARL BERRY
VOLUME 29, NUMBER 3 • 2008 PORTLAND • OREGON • U.S.A. TUG 2008—TEX’s 30th birthday July 20–24, 2008 University College Cork, Ireland Sponsors TEX Users Group DANTE e.V. Human Factors Research Group, UCC Carleton Production Centre Malcolm Clark River Valley Technologies Thanks to the sponsors, and to all the speakers, teachers, and participants, without whom there would be no conference. The session chairs and speakers deserve special recognition for adhering to a extra-tight schedule this year. Special thanks also to Kaveh Bazargan for the video recording, and Duane Bibby for the (as always) excellent drawing. Conference committee Anita Schwartz Peter Flynn Robin Laakso Karl Berry Participants Leila Akhmadeeva, Bashkir State Medical Univ., Russia J´erˆome Laurens, Universit´ede Bourgogne Edd Barrett, TEX Live / OpenBSD Dag Langmyhr, University of Oslo Kaveh Bazargan, River Valley Technologies Olga Lapko, Moscow, Russia Nelson Beebe, University of Utah Blanca Mancilla, University of New South Wales Barbara Beeton, American Mathematical Society Niall Mansfield, UIT Cambridge Ltd, UK Mih´aly Bicz´o, E¨otv¨os Lor´and University, Hungary Patricia Masinyana, UNISA, South Africa A Johannes Braams, L TEX Project Joe McCool, Southern Regional College, UK Jennifer Claudio, St. Lawrence Academy Stephen McCullagh, Dublin Inst. for Advanced Studies A David Crossland, Wimborne, UK Frank Mittelbach, L TEX Project Sue DeMeritt, Center for Communications Research Ross Moore, Macquarie University, Australia Christine Detig, Net & Publication Consultance GmbH Winfried Neugebauer, Bremen, Germany Michael Doob, University of Manitoba Manuel P´egouri´e-Gonnard, TEX Live Hossam Fahmy, Cairo University, Egypt Steve Peter, Pragmatic Programmers Jonathan Fine, The Open University, UK Karel P´ıˇska, Academy of Sciences, Czech Republic Peter Flynn, University College Cork, Ireland John Plaice, University of New South Wales Ralf Gaertner, M¨unchen, Germany Kriszti´an P´ocza, E¨otv¨os Lor´and University, Hungary Michel Goossens, CERN Cheryl Ponchin, Institute for Defense Analyses Steve Grathwohl, Duke University Press Toby Rahilly, University of New South Wales Adelheid Grob, Universit¨at Ulm Arthur Reutenauer, GUTenberg Hans Hagen, Pragma ADE Daniel Rhatigan, The University of Reading Hartmut Henkel, von Hoerner & Sulger GmbH David Roderick, Carlisle, UK A A Morten Høgholm, L TEX Project Chris Rowley, L TEX Project Taco Hoekwater, Elvenkind BV Stanislav Sarman,ˇ Clausthal Univ. of Tech., Germany Klaus H¨oppner, DANTE e.V. Volker RW Schaa, DANTE e.V. Jean-Michel Hufflen, Universit´ede Franche-Comt´e Joachim Schrod, Net & Publication Consultance GmbH Jelle Huisman, SIL International Torsten Schuetze, M¨oglingen, Germany Manjusha Joshi, Bhaskaracharya Institute in Herbert Schulz, Naperville, Illinois Mathematics, India Anita Schwartz, University of Delaware Jonathan Kew, Thame, UK Heidi Sestrich, Carnegie Mellon University Timothy Kew, Thame, UK Martin Sievers, Trier, Germany Peter Knaggs, Bournemouth, UK Linas Stonys, VTEX Thomas Koch, K¨oln, Germany Sigitas Tolusis, VTEX Harald K¨onig, Balingen, Germany Eva van Deventer, UNISA, South Africa Reinhard Kotucha, Hannover, Germany Marc van Dongen, University College Cork, Ireland Valentinas Kriauciukas, VTEX Boris Veytsman, George Mason University Martha Kummerer, University of Notre Dame GS Vidhya, River Valley Technologies Johannes K¨uster, typoma GmbH Ulrik Vieth, Stuttgart, Germany Robin Laakso, TEX Users Group Alan Wetmore, US Army Research Laboratory TUG 2008—program
Monday, July 21 9:00 Peter Flynn, opening 9:15 Frank Mittelbach, Windows of opportunity: A (biased) personal history of two decades of LATEX development — are there lessons to be learned? 10:00 Steve Peter, A pragmatic toolchain: TEX and friends and friends of friends 10:30 break 10:45 Niall Mansfield, How to develop your own document class — our experience 11:15 Joe McCool, A newbie’s experiences with Lilypond, Lilypond-book, LATEX, and Perl 11:45 Jonathan Fine, Why we need LATEX3 12:30 lunch 2:15 Kriszti´an P´ocza, Mih´aly Bicz´o& Zolt´an Porkol´ab, docx2tex: Word 2007 to TEX 2:45 Jonathan Kew, TEXworks: Lowering the barrier to entry 3:15 break 3:30 Manjusha Joshi, Data mining: Role of TEX files? 4:00 Taco Hoekwater, LuaTEX: What has been done, what will be done 4:30 Hans Hagen, LuaTEX: The TEX-Lua mix reception
Tuesday, July 22 9:00 Mojca Miklavec & Arthur Reutenauer, Putting the Cork back in the bottle: Improving Unicode support in TEX extensions 9:30 Joachim Schrod, xindy revisited — multilingual index creation for the UTF-8 age 10:00 Ulrik Vieth, Do we need a Cork math font encoding? 10:30 break 10:45 Daniel Rhatigan, Three typefaces for mathematics 11:15 Johannes K¨uster, Minion Math: The design of a new math font family 11:45 Karel P´ıˇska, Creating cuneiform fonts with MetaType1 and FontForge 12:15 lunch 1:30 Ameer Sherif & Hossam Fahmy, Meta-designing parameterized Arabic fonts for AlQalam 2:00 Stanislav Sarman,ˇ Writing Gregg Shorthand with LATEX and Metafont 2:30 Dave Crossland, Why didn’t Metafont catch on? 3:00 break 3:15 John Plaice, Blanca Mancilla & Toby Rahilly, Multidimensional text 3:45 Blanca Mancilla, John Plaice & Toby Rahilly, Multiple simultaneous galleys: A simpler model for electronic documents 4:15 Toby Rahilly, John Plaice & Blanca Mancilla, Parallel typesetting
Wednesday, July 23 9:15 Manjusha Joshi, Smart ways of drawing PSTricks figures 9:45 Boris Veytsman & Leila Akhmadeeva, Medical pedigrees with TEX and PSTricks: New advances and challenges 10:30 break 10:45 Jonathan Fine, MathTran and TEX as a web service 11:15 Ross Moore, Advanced features for publishing mathematics, in PDF and on the Web 11:45 Morten Høgholm, The galley module, or How I Learned to Stop Worrying and Love the Whatsit 12:15 J´erˆome Laurens, Direct and reverse synchronization with SyncTEX 12:45 lunch Afternoon excursions, coaches expected to leave around 2pm. 7pm banquet
Thursday, July 24 9:00 Taco Hoekwater, MPlib: The project, the library and the future 9:30 Hartmut Henkel, Image handling in LuaTEX 9:30 Hans Hagen, MPlib: An example of integration 10:30 break 10:45 Hans Hagen, surprise LuaTEX talk 11:15 Jean-Michel Hufflen, Languages for bibliography styles 11:45 Boris Veytsman, Observations of a TEXnician for hire 12:15 Anita Schwartz, closing 12:30 lunch TUG 2008: TEX’s 30th birthday
Peter Flynn University College Cork Ireland http://tug.org/tug2008
’Twas the night before TUGconf and all through the Stone for the gift of gab, especially when it comes to No computer was stirring, not even my mouse. [house our passion for TEX. However, many of us did enjoy The bags were all stuffed and in boxes for care sharing these passionate discussions over Jameson, In hopes that the delegates soon would be there. Guinness and Beamish at the end of each long day. Attendees were nestled all snug in their planes The banquet provided a relaxing evening to enjoy a While visions of typesetting danced in their brains. nice dinner along with the opportunity to learn and enjoy the talents of our user group members outside I dare say someone with more poetic license can of TEX. make a better shot at it, but by the night before A lot has changed since we hosted the 1990 the 2008 Cork meeting the bags were indeed all meeting at Cork: fonts, encodings, packages, ver- ready, thanks to Anita and Tyler Schwartz, Arthur sions, features, systems, and people (some of us are Reutenauer and Karel P´ıˇska who gave up their eve- older and wiser; some of us just older!). Many of ning to sort T-shirts, mugs, programs, and the as- these changes were evident in the presentations, and sorted bits and pieces while registering the early ar- it was good to see so much new work being done. rivals. We finally headed for a much-needed beer, Many of us have had to fend off the “oh, that old and bumped into my son and his girlfriend in the thing” response to mentioning TEX, and perhaps we beer-garden (just to show that Cork is actually a don’t shout loudly enough about all the shiny new village of 200,000 people). features we get to see at conferences. The workshops (PSTricks and LAT X) were very E And a lot hasn’t changed: I had occasion to dig well attended, almost over capacity in one of the out some of the files I used back around 1990, and by rooms, and in addition to the expected content they changing \documentstyle to \documentclass and covered a lot of the vital but informal tips and hints fixing a couple of package names, they worked fine; that you only get in face-to-face tuition. a tribute to the stability of the underlying design The “Luck of the Irish” brought us excellent and the work of the maintainers and developers. weather and wonderful presenters. Everyone did a Perhaps embarrassingly, some of what hasn’t fantastic job adhering to the schedule and adjusting changed still haunts us, although we’d never tell to all the last minute changes. Thanks to the session that to the users of InDesign or QuarkXPress or chairs, Cheryl Ponchin, Anita Schwartz, myself and FrameMaker or (gasp) Word or OpenOffice. L X our renowned TUG office manager, Robin Laakso. Y is great, but we still don’t have an editing inter- As the local organiser I couldn’t get to as many face that non-technical writers can use. X TE EX is of the papers as I wanted to, but meeting old and wonderful but font installation is still a pain. The new faces is one of the benefits of conferences, and MiKTEX-derived package managers are cool, but not I always get a lot out of hearing what people have yet universal. And we still have people using \bf been doing and what they are using TEX for. Video and \it after all these years. recordings of many of the talks were made by the Next year we’re meeting at Notre Dame, so we stalwart Kaveh Bazargan, so if you could not be have ten months to make some more good changes. http: present, or for a reminder if you were, visit And they promise to have visitor wireless access, //www.river-valley.tv/conferences/tug2008. which UCC didn’t, no matter how loudly I screamed. The excursions were full of history and beautiful See you there! sites. Many would agree that most of us attending TUG conferences do not need to kiss the Blarney
352 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting At the reception. Around the front table: Adelheid Grob, Hans Hagen, Hartmut Henkel, Hans Hagen, Arthur Reutenauer, Taco Hoekwater, Jean-Michel Hufflen, Harald K¨onig, Dag Langmyhr, Reinhard Kotucha, Hartmut Henkel, (back of) Michael Doob, Nelson Beebe, Johannes Braams. Volker Schaa. Behind Hartmut: Leila Akhmadeeva, Olga Lapko, and Boris Veytsman. Behind Reinhard: Heidi Sestrich, Alan Wetmore.
Jonathan Fine. Hossam Fahmy.
Kaveh Bazargan, recording and listening. Slide from Daniel Rhatigan’s talk.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 353 Klaus H¨oppner, wielding camera. At the English Market. Kneeling: Reinhard Kotucha, Manuel P´egori´e-Gonnard, Dave Crossland. Standing: Heidi Sestrich, Kaveh Bazargan, Alan Wetmore, Martha Kummerer, Barbara Beeton, Manjusha Joshi
Dave Crossland, wielding camera.
The cliffs of Moher. Steve Peter, outside the Jameson distillery.
354 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Taco Hoekwater, inside the Beamish brewery.
Manjusha Joshi, winner of the drawing.
th TEX’s 30 birthday cake.
Morten Høgholm and family: Evguenia, David, Abigail.
Michael Doob, Barbara Beeton, Frank Mittelbach.
Photos courtesy of Jennifer Claudio, Morten Høgholm, Robin Laakso, Steve Peter, and Main quad. Ulrik Vieth.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 355 How to develop your own document class — our experience
Niall Mansfield UIT Cambridge Ltd. PO Box 145 Cambridge, England tug08 (at) uit dot co dot uk Abstract
We recently started re-using LATEX for large documents — professional computing books — and had to convert an old (1987) LATEX 2.09 custom class to work with LATEX 2ε. We first tried converting it to a stand-alone .cls file, which the documentation seemed to suggest is the thing to do, but we failed miserably. We then tried the alternative approach of writing an “add-on” .sty file for the standard book.cls. This was straightforward, and much easier than expected. The resulting style is much shorter, and we can use most standard packages to add extra features with no effort. This paper describes our experience and the lessons and techniques we learned, which we hope will encourage more people to write their own styles or classes.
1 Where we started from Years ago I wrote a book The Joy of X [1], about the X window system. It was in an unusual format called STOP [5], as enhanced by Weiss [6], summa- rized graphically in Figure 1. In 2008 I wanted to write another book in the same format [2, 3]. It has several interesting features that make it excellent for technical books, although those details are not rele- vant here. Suffice it to say that STOP required us to change how parts, chapters, sections and sub-sections are handled, and to provide extra sectional units at the beginning and end of each chapter. We also had to provide a summary table of contents, and for each Figure 1: A STOP sub-section or “module” chapter a per-chapter table of contents (TOC) on the first page of the chapter, and use PostScript fonts, which in 1987 was a non-trivial task. Back in 1987 a colleague of mine, Paul Davis, (Another reason for thinking this was that at least very kindly wrote the necessary style file for this one major publisher seems to have gone this route.) format, and it worked very well. However, in the In fact the same document continues “There are two major types of class: those like article, report meantime the world had moved on from LATEX 2.09 or letter, which are free-standing; and those which to LATEX 2ε. The challenge was to provide the func- are extensions or variations of other classes — for tionality of the old style, but under LATEX 2ε. example, the proc document class, which is built on 2 First attempt — failure the article document class.” What I ought to have Where do you start when developing a new style or done is started work on an “extension” or “variation” class, but I didn’t realize it. class? The document LATEX 2ε for class and package .sty A ε writers says: So I tried to convert the old to LTEX 2 , and failed miserably. (This wasn’t surprising, be- if the commands could be used with any doc- A cause LTEX 2.09 style files consisted of plain TEX ument class, then make them a package; and code, which I have always found very difficult to if not, then make them a class. understand.) The end-product was something that I took this to mean “We should write a class”. I almost worked, but had lots of small bugs, and when wrongly went one step further, and thought it also I fixed one problem, the change caused new problems meant we should start our own class from scratch. elsewhere.
356 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting How to develop your own document class — our experience
Another, equally large, disadvantage of this ap- space). These don’t do anything new, but they proach was that even if it had worked, the effort all take the same number of arguments in the to maintain it would have been huge. None of the same order; if a particular variant doesn’t actu- standard LATEX 2ε packages would have worked, so if ally need them all, we can just leave the irrele- we needed any changes — even “simple” things like vant ones empty. This makes it easy to change changing the page size — we’d have had to code them a figure from “no caption” to “side caption” or by hand ourselves. to “normal caption” with a couple of keystrokes. Lesson A: whether you call your file a class All these macros are in uit-figures.sty. or a style doesn’t matter much—it’s just a • Where we make substantial changes, e.g. to matter of a name. What is important is not the sectioning mechanism or to the format of to start from scratch, but to build, as far as page headings, we include it directly in our file possible, using existing code. uitbook.sty. At this point we almost gave up and considered To cheer ourselves up after previous failures, using Quark Xpress or InDesign. But luckily someone we did all the easy bits first. Those included the noticed the minitoc package, which looked like it convenience macros mentioned above, and the dozens might give us exactly what we needed for our per- of \RequirePackage calls to the packages that we chapter TOC, if only we could use it. We decided to needed: try again. caption chngpage color colortbl 3 Second attempt — a short blind alley courier crop endnotes fancyvrb We threw away everything we had and started again framed geometry graphicx helvet from scratch. We tried book.cls plus minitoc. This hhline ifthen latexsym layout addressed one of our most difficult requirements — makeidx mathpazo mcaption minitoc the per-chapter TOC —and did it so well that we nextpage paralist relsize showidx were encouraged to persevere, thank goodness. sidecap ulem url wrapfig We copied the book.cls file as uitbook.cls, At this point things were looking good. We had a and started adding our own modifications to this. style that worked. However, several STOP-specific After a few days this became messy, especially when features were still missing, so that’s what we had to bug-fixing: it wasn’t obvious which was our code implement next. (where the bugs were likely to be) and which was the The document LATEX 2ε for class and package original code. writers describes the boilerplate for a class or pack- Lesson B: in LATEX, the way to modify stan- age — analogous to telling a C programmer that he dard code is usually not to modify the original needs a main() function, and how to use #include file. Instead, extract just the piece that you statements. What it doesn’t tell you is how the want to change, save it as something.sty and standard classes work, and the common techniques modify just that little file. Then do they use. In the following sections we’ll explain the \usepackage{something.sty}. techniques that we came across. 4 Third attempt — success! 5 The hard bits 1 — over-riding existing functionality So we started again, leaving book.cls unchanged, and created our own file uitbook.sty to contain all We needed to change the TOC entry for Parts. This our changes. The convention we settled on is: is handled in the \l@part function in book.cls. • If something is just a convenience — e.g. a macro We copied this function to our uitbook.sty, and that is merely a shorthand to save typing but modified it there. The change involved was only a doesn’t add any new functionality — we create single line—to use a different font, and insert the a small .sty for it, and then \RequirePackage word “Part” — but it illustrates a couple of important that. In this way we can re-use the same conve- points: nience tools with other classes. Lesson C: copying a piece of standard code For example, we defined about 12 macros for in LATEX, and changing your own version of including graphics or verbatim examples of pro- it is a bit like over-riding a method in object- gram code, with or without captions, and with oriented programming. Everything that you captions in the usual place below the figure or haven’t changed continues to work as before, alternatively beside the figure (to save vertical but as soon as the relevant function (macro)
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 357 Niall Mansfield
\l@part, you can change how the TOC entries for your Parts are printed. The hook mechanism and the “over-riding functionality” technique above are more or less the same thing. Hooks are fundamental to how LATEX packages work: they let the package over-ride the standard operation with something different. As an example, consider the shorttoc.sty package, which is useful if you want a one-page summary table of contents before the main TOC, for example. The package contains only about 40 lines of code, and in essence, all it does is call the standard table of contents, having first redefined the variable \c@tocdepth to a Figure 2: How LAT X hooks work E small value to show only the top levels of contents. In effect, shorttoc.sty is using all the standard table-of-contents macros as hooks, although it hasn’t is called — \l@part in our example — the new changed any of them. code is used instead of the old (Figure 2). Lesson E: Lesson D: you don’t have to understand ev- hooks aren’t documented (as far erything to make a change to something rela- as we’ve been able to see). In fact they can tively small. As long as you change as little as never be exhaustively documented, because possible, you probably won’t break anything any package author can just copy any function \l@part else. In our case, even though \l@part con- (as we did with earlier) and over- tains lots of complex stuff, we were confident ride it with their own code, thus using that that our minor format changes would work, function as a hook. In real life, the only way because we didn’t modify anything else. you can determine the important hooks is by looking at the important packages, to see By the way, many functions or macros in the which functions they over-ride. package and class files are defined using the TEX Lesson F: when you copy a chunk of stan- primitive \def, instead of LATEX’s \newcommand. If you redefine an existing command with \def, you dard code, change the absolute minimum you don’t get an error, unlike \newcommand’s behavior. can get away with. The reason is you don’t really know what parts of it might be used as 6 The hard bits 2 — LATEX hooks hooks, or what might happen if you remove a call from it to some other hook. Resist the The first clue we got about how LATEX2 packages work with class files, i.e. how they modify their be- temptation to tidy or “improve” the code. havior, was reading ltsect.dtx — the documented 7 The hard bits 3 — adding extra A version of the sectioning code in core LTEX2. It has functionality a comment: “Why not combine \@sect and \@xsect and save doing the same test twice? It is not possible (This section describes a technique that you will often to change this now as these have become hooks!” come across, and which you might find useful.) What’s a hook? In the Emacs programmable Let’s say you want to change some function so editor, hooks are used to customize the editor’s be- that it continues to do exactly what it does at present, havior. For example, before-save-hook is a list but does something extra in addition, i.e. the new of Lisp functions that should be run just before a is a superset of the old. Here’s a real but slightly file is saved. By default the list is empty, but by weird example. The author of a book [4] was using adding your own functions to the list, you can have superscripts in his index for a special purpose. We Emacs perform any special actions you want, such as needed a list of the superscripts, and it was difficult checking the file into a version control system as well to get this from the source files. Using the TEX as saving it, etc. Emacs provides about 200 hooks, primitive \let, you can assign a whole function to letting you customize most aspects of its behavior. a new variable, and then call the same old function but with the new name. We used this as follows: In LATEX a hook is slightly different. It’s a named function or macro that some other part of the \let\origsuper=\textsuperscript system is going to call. For example, in Section 5 we \renewcommand{\textsuperscript}[1] used \l@part as a hook. As we saw, by redefining {\origsuper{XXX(#1)XXX}}
358 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting How to develop your own document class — our experience
This “saves” the original \textsuperscript defini- 9 The hard bits 5 — indirection in tion as \origsuper (Figure 3). Then it redefines macro names \textsuperscript, to call the original, unaltered (Again, this section describes a common technique function, but with a modified argument, so that the that you need to understand, although you might superscripted text is still superscripted, but is sur- not use it often yourself.) rounded by the strings XXX(...)XXX, which we can 1 The TEX commands \csname ... \endcsname then easily search for. let you construct a “control sequence” name, i.e. a macro, programatically and then invoke it. The following is equivalent to \textbf{fat cat}: \newcommand{\mymac}{textbf} \csname \mymac \endcsname{fat cat} The first line defines the variable mymac to be the string textbf, and the second line uses the variable to construct a macro name and invoke it, passing the argument ‘fat cat’ to it. Being able to invoke a function or macro programatically like this, instead of having to hard-code its literal name in your .sty file, makes it possible to handle many similar but slightly different cases compactly and with little duplication of common code. The sectioning mechanism uses this technique frequently, to construct names of variables or func- tions related to the level of the current “sectional 2 unit” (SU), such as the macros \l@part, \l@chapter, \l@section, etc. We’ll look at this in more detail Figure 3: Adding extra functionality to a macro in the next section, but for now, here’s a simple but artificial example of how it works. We define a macro \T, whose first argument is the style in which 8 The hard bits 4 — plain TEX’s syntax its second argument is to be printed: In 1987 I found plain TEX incomprehensible, and \newcommand{\T}[2] A nothing has changed. Using LTEX is non-trivial, but {\csname text#1\endcsname{#2}} it’s powerful and the results are more than worth Make stuff \T{bf}{heavy} the effort required. For me, the same is not true of or \T{it}{slanty}. The end. plain TEX: it’s too low level, and too complex. Its syntax is weird. Instead of helping you do what you This produces: heavy know you need to do, the syntax gets in your way Make stuff or slanty. The end. and makes things hard for you. (As an example, we 10 The hard bits 6 — changing sectioning recently found an “off by one” error in a standard package. To fix it, all that was needed was to change The most difficult thing we had to do was change how sectioning works. (We had to do this because ‘if X > Y’ to ‘if X >= Y’, but plain TEX doesn’t let you express things like that, so we had to get our STOP format has to print both section- and sub-section headings on sub-sections.) someone more experienced in plain TEX to change the code to do the equivalent.) For a beginner, sectioning is difficult in three separate ways: So, while plain TEX is wonderful and is the foun- dation on which LATEX is built, it’s not for everyone. 1. There are many functions involved: sections, (Or, it’s for almost no-one?) subsections and lower are defined in terms of Our feeble “solution” to this problem is to avoid \@startsection, which then uses \@sect (or it, and when that’s not possible, to copy code from \@ssect if it’s a “starred” sectioning command, packages that work, and hope that LuaTEX (www. which in turn calls \xsect); all these are com- luatex.org) will eventually make it easier to code plex, and written in plain TEX, which makes life complex or low-level macros. difficult.
1 catdvi file.dvi | tr -s " \t" "\n" | 2 A sectional unit is a part, chapter, section, subsection, fgrep ’XXX(’ | sort -u etc.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 359 Niall Mansfield
The way we got over this was by document- ing the functions. This is a work in progress, so we’ve made the rudimentary documentation available on our Web site (uit.co.uk/latex). 2. Sectioning uses indirection a lot. Because the same functions (\@startsection, etc.) handle many different levels of sectioning, they use in- direction to refer to various parameters for the Figure 4: Macro processor — output processing SU being operated on. For example: • The counters \c@part, \c@chapter, \c@section, \c@subsection, ... hold the This starts to make sense only when you realize number of the respective SU. that plain TEX behaves as a classical, stack-oriented, • The macros \thepart, \thechapter, macro processor (which also typesets!). Initially you \thesection, . . . specify how the respec- can consider the input stack to contain the whole tive counter is formatted. E.g. book.cls input file. The processor reads input from the file, defines: i.e. removes it from the stack. It just copies the input \renewcommand \thesection to the output, unless it’s either a macro definition, {\thechapter.\@arabic\c@section} or a macro invocation, in which case it’s evaluated so that the numbering on a section will be and the result is pushed back onto the input stack, of the form “4.9”. to be re-processed. To make this concrete, let’s look • Similarly, the variables \l@part, at a few simple examples for the m4 macro processor. \l@chapter, \l@section, . . . are what are The following input has no macros or anything else used to create the table-of-contents entry special, so it’s copied to the output without change: for the respective SU. % echo ’Hello, world.’ | m4 The first argument to the \@startsection and Hello, world. \@ssect functions is the type of the current as shown in Figure 4. The slightly more complex: SU, and the functions use this to construct the relevant item they need, as in: define(‘showarg’, ‘my arg is $1’) \csname l@#1\endcsname A showarg(mouse) A This technique isn’t intrinsically difficult, but defines a simple macro that takes a single argument. until you’re aware of it, the sectioning mecha- Run it and see what you get: nism can appear incomprehensible. % m4 ex1.m4 3. Functions seem to do funny things with their A my arg is mouse A arguments. We cover this in the next section. Now let’s have one macro reference another indi- 11 The hard bits 7 — plain TEX really is a rectly: macro processor define(‘concat’, ‘$1$2’) The file book.cls defines: define(‘showarg’, ‘my arg is $1’) \newcommand\section B concat(quick, brown) B {\@startsection {section} ... C showarg(fox) C D concat(sho, warg)(jumps) D i.e. a \section is just a call to \@startsection with 6 arguments, the first of which is the type and run this: of the current SU, as we explained above. However % m4 example.m4 \@startsection then calls \@sect with 7 arguments, B quickbrown B even though \@sect is defined to take 8 arguments. C my arg is fox C And then you realize that \section was defined D my arg is jumps D to take no arguments of its own at all! What’s happening? Why isn’t \section defined to take The B and C lines are straightforward, but line D is some arguments, like this: tricky: concat(sho, warg) is read from the stack, leaving only: \newcommand\section[1] {...} (jumps) D since \section is always called with a name argu- But what we’ve just read — concat(sho, warg) — ment, as in \section{Thanks}? evaluates to showarg, so the string {showarg} is
360 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting How to develop your own document class — our experience
12 The results, and lessons learned Our original LATEX 2.09 style file had 1400 lines of code, excluding comments. Our new LATEX 2ε style file has 300 lines of code, excluding comments: 34 are \RequirePackages, 150 lines make up 54 \newcommands for convenience-type functions that we ought to have isolated in separate files had we been disciplined enough, and 15 lines make up 11 \newenvironments. The other large chunk of code is 35 lines for our modified version of \@sect. Figure 5: Macro result pushed back onto stack Not only is our new style file much shorter, it’s easier to understand and maintain, and is much more flexible than our old one. We can use most standard pushed back onto the stack (Figure 5). The top of packages to add extra features with no effort, because the stack now looks like: we still provide all the hooks that add-on packages showarg(jumps) D rely on. Moreover, the standard class and style files which is re-evaluated as a call to showarg with argu- have had years of debugging and are very robust ment jumps. In other words, (jumps) was left lying and reliable. By re-using as much as possible, and on the stack, and it was picked up as an argument minimizing the amount of code changed, we’ve ended to a macro in due course. up with a stable system. We’ve had almost no bugs, The same thing happens in LATEX. In the fol- and the ones we did have we were able to fix quickly lowing code: and cleanly. Foo \textbf{cat} bar. (A) 13 Thanks \newcommand{\Ttwo}[2] {\csname text#1 \endcsname{#2}} Lots of people very kindly helped us over the years, Foo \Ttwo{bf}{cat} bar. (B) with advice and pieces of code. These include, but are \newcommand{\Tone}[1] not limited to: Donald Arseneau, Barbara Beeton, {\csname text#1 \endcsname} Timothy Van Zandt, and lots of other patient and Foo \Tone{bf}{cat} bar. (C) helpful people on the [email protected] mailing list. Foo \newcommand{\Tzero}{textbf} References \csname \Tzero \endcsname{cat} bar. (D) [1] Niall Mansfield. The Joy of X. Addison-Wesley, each line produces the same output: 1986. Foo cat bar. (A) [2] Niall Mansfield. Practical TCP/IP — Designing, Foo cat bar. (B) using, and troubleshooting TCP/IP networks Foo cat bar. (C) on Linux and Windows (first edition). Foo cat bar. (D) Addison-Wesley, 2003. However, in lines (C) and (D) the string {cat} is not specified as an argument to the Txxx macro [3] Niall Mansfield. Practical TCP/IP — Designing, we defined—it’s left lying around, conveniently sur- using, and troubleshooting TCP/IP networks rounded by braces, and is picked up later. on Linux and Windows (second edition). UIT This technique is used in the sectioning code. Cambridge Ltd., 2008. When you write ‘\section{Thanks}’, the macro [4] Jan-Piet Mens. Alternative DNS Servers — \section is invoked with no arguments, leaving Choice and deployment, and optional SQL/ the string {Thanks} on the stack. \section calls LDAP back ends. UIT Cambridge Ltd., 2008. \@startsection, which calls \@sect with 7 argu- [5] J.R. Tracey, D.E. Rugh, and W.S. Starkey. ments; but \@sect needs 8 arguments, so it picks up STOP: Sequential Thematic Organization of {Thanks} from the top of the stack, so it’s happy, Publications. Hughes Aircraft Corporation, and we don’t get any errors. Fullerton, CA, 1965. [6] Edmond H. Weiss. How to Write a Usable User Manual. ISI Press, 1985.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 361 TEXworks: Lowering the barrier to entry
Jonathan Kew 21 Ireton Court Thame OX9 3EB England [email protected]
1 Introduction The standard TEXworks workflow will also be PDF-centric, using pdfT X and X T X as typeset- One of the most successful TEX interfaces in recent E E E years has been Dick Koch’s award-winning TeXShop ting engines and generating PDF documents as the on Mac OS X. I believe a large part of its success has default formatted output. Although it will still be been due to its relative simplicity, which has invited possible to configure a processing path based on new users to begin working with the system with- DVI, newcomers to the TEX world need not be con- out baffling them with options or cluttering their cerned with DVI at all, but can generally treat TEX screen with controls and buttons they don’t under- as a system that goes directly from marked-up text stand. Experienced users may prefer environments files to ready-to-use PDF documents. T Xworks includes an integrated PDF viewer, such as iTEXMac, AUCTEX (or on other platforms, E based on the Poppler library, so there is no need WinEDT, Kile, TEXmaker, or many others), with more advanced editing features and project man- to switch to an external program such as Acrobat, agement, but the simplicity of the TeXShop model xpdf, etc., to view the typeset output. The inte- has much to recommend it for the new or occasional grated viewer also allows it to support source ↔ user. preview synchronization (e.g., control-click within Besides the relatively “clean” interface, a sec- the source text to locate the corresponding position ond factor in TeXShop’s success is probably the use in the PDF, and vice versa). This capability is based on the “SyncT X” feature developed by J´erˆome Lau- of a PDF-centric workflow, with pdfTEX as the de- E fault typesetting engine. PDF is the de facto stan- rens, now integrated into the X TE EX and pdfTEX en- dard for fully-formatted pages; every user knows gines in TEX Live 2008, MiKTEX, and other current what a PDF file is and what they can do with it. distributions. Bypassing DVI reduces the apparent complexity of 2 Features for initial release the overall process, and so reduces the “intimida- tion factor” for a newcomer. But TeXShop is built Figure 1 shows the current TEXworks prototype run- on Mac OS X-specific technologies, and is available ning on Windows Vista. While this is not a finished only to Mac users. There does not seem to be an interface, it gives an impression of how the applica- equivalent tool available on other platforms; there tion will look. TEXworks version 0 will be an easy- to-install application offering: are many TEX editors and environments, but none with this particular focus. 1. Simple (non-intimidating—this is not emacs or The TEXworks project is an effort to build a vi!) GUI text editor with similar TEX environment (“front end”) that will be i. Unicode support using standard OpenType available for all today’s major desktop operating sys- fonts tems — in particular, MS Windows (XP and Vista), ii. multi-level undo/redo typical GNU/Linux distributions, and other X11- based systems, in addition to Mac OS X. iii. search & replace, with (optional) regular expressions as well as simple string match To achieve this, TEXworks is based on cross- platform, free and open source tools and libraries. In iv. comment/uncomment lines, etc. particular, the Qt toolkit was chosen for the qual- v. (LA)TEX syntax coloring ity of its cross-platform user interface capabilities, vi. TEX-aware spell checker with native “look and feel” for each platform being vii. auto-completion for easy insertion of com- a realistic target. Qt also provides a rich application mon commands framework, facilitating the relatively rapid develop- ment of a usable product. viii. templates to provide a starting point for common document types
362 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting TEXworks: Lowering the barrier to entry
Figure 1: A recent TEXworks build running on Windows Vista: source and preview windows, with the TeXShop-style magnifying glass in use.
2. Ability to run TEX on the current document to 3 Current status generate PDF: An early TEXworks prototype was demonstrated at i. extensible set of TEX commands (with com- the BachoTEX conference (April 2008). It became mon commands such as pdftex, pdflatex, more widely available (though still considered a pro- xelatex, context, etc. preconfigured) totype, not a finished release) when a version was ii. also support running BibTEX, Makeindex, posted in mid-July before the TUG 2008 conference. etc. The current code is available as source (easy to build on typical GNU/Linux systems), and as precompiled iii. any terminal output appears in a “console” binaries for Windows and Mac OS X. panel of the document window; automati- At this time, the application supports text edit- cally hidden if no errors occur ing and PDF viewer windows, and has the ability to iv. “root document” metadata so “Typeset” run a typesetting job and refresh the output view, works from an \included file etc. There is not yet any documentation, and many 3. Preview window to view the output: potential “power user” features do not yet exist, but i. anti-aliased PDF display it is a usable tool in its current state. In addition to Windows (XP and Vista), it runs on Mac OS X (see ii. automatically opens when T X finishes E figure 2) and GNU/Linux systems (figure 3). iii. auto-refresh when re-typesetting (stay at A few features remain to be implemented be- same page/view) fore a formal release of “version 0”, including “sin- iv. TeXShop-like “magnifying glass” feature to gle instance” behavior, and various options for win- examine detail in the preview dow positioning; appropriate installer packages for v. one-click re-typesetting from either source Mac OS X and Windows are also needed, to simplify or preview setup. More information may be found online via the vi. source ↔ preview synchronization based on TEXworks home page at http://texworks.org/. J´erˆome Laurens’ SyncTEX technology
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 363 Jonathan Kew
Figure 2: TEXworks running on Mac OS X: using the Figure 3: TEXworks running on a typical GNU/Linux Preferences dialog to configure a typesetting tool. system (Ubuntu).
4 Future plans • Dig in to the code, and submit patches to im- After the release of version 0, several major addi- plement your favorite missing features. tional features are planned; some ideas high on the • Write on-line help, documentation and tutorials priority list include: for newcomers to TEXworks and TEX. • intelligent handling of TEX errors • Review and enhance the command completion • assistance with graphics inclusion and format lists available for the integrated editor. conversions • Provide well-commented templates for various • text search and copy in the PDF preview types of documents. • support rich PDF features such as transitions, • Design icons for the toolbars, etc.; TEXworks embedded media (sound, video), annotations, has some nice icons from Qt and the Tango etc. project, but others are merely rough placehold- • customizable palettes of symbols, commands, ers. etc. • Use the Qt Linguist tool to localize the user • TEX documentation lookup/browser interface for your language. • interaction with external editors and other tools • Package TEXworks appropriately for your fa- • additional support for navigating in the source, vorite GNU/Linux or BSD distribution, or cre- e.g., “folding” sections of text, recognizing doc- ate an installer for Windows or Mac OS X. ument structure tags such as \section, etc. There is a TEXworks mailing list for questions I expect development priorities to be guided by and discussions related to the project; see http:// user feedback as well as developer interest, once the lists.tug.org/texworks/ to subscribe, for the list initial version 0 release is available. archives, etc. The T Xworks source itself is maintained in a 5 Invitation to participate E Google Code project at http://code.google.com/ TEXworks is a free and open source software project, p/texworks/. Resources available through this site and all are welcome to participate and contribute to include the Subversion source repository, precom- its development. This does not necessarily mean piled binaries for Windows and Mac OS X, and an writing code; many other roles are equally impor- issue tracker for bug reports and feature suggestions. tant. Some possible ways to participate include the following. 6 Thanks • Use the prototype for some real work, and give The TEXworks project arose out of discussions at feedback on what’s good, what’s bad, what’s several recent TEX Users Group meetings, and has broken: received generous support from TUG’s TEX develop- – if there’s a current binary download avail- ment fund and its contributors, and from UK-TUG. able for your platform, try that; Special thanks to Karl Berry for his encouragement – get the code and try building it on your and support, and to Dick Koch for showing us the platform; provide bug reports (and fixes!) potential of a clean, simple TEX environment for the for whatever problems show up. average user.
364 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Direct and reverse synchronization with SyncTEX
Jérôme Laurens Département de mathématiques Université de Bourgogne 21078 Dijon Cedex FRANCE jerome (dot) laurens (at) u-bourgogne (dot) fr http://itexmac.sourceforge.net/SyncTeX.html
Abstract
We present a new technology named SyncTEX used to synchronize between the TEX input and the output.
1 What is synchronization? Creating documents with the TEX typesetting system often requires two windows, one for entering the text, the other one for viewing the resulting output. In general, documents are too long to fit in the visible frame of a window on screen, and what is really visible is only some part of either the input or the output. We say that the input view and the output view are synchronized if they are displaying the “same” portion of the document. Forwards or direct synchronization is the ability, for an output viewer, to scroll the window to a part corresponding to a given location in the input file. Backwards or reverse synchronization is the ability, for a text Figure 1: Synchronization in iTEXMac2 based on editor, to scroll the text view to a part corresponding SyncTEX technology with text analysis. to a given location in the output file. Figure 1 is a screenshot illustrating SyncTEX supported in iTEXMac2, the TEX front end devel- oped by the author on Mac OS X. The top window is a text editor where an extract of the “Not so (or --synctex=1) and the same for xetex. With this A short introduction to LTEX 2ε” is displayed. The option, a new informational line is printed at the end word “lscommand” has been selected and the viewer of the command output and reads SyncTeX written window at the bottom automatically scrolled to the on foo.synctex.gz. The new (compressed) auxil- position of this word in the output, highlighting it iary file named foo.synctex.gz is used by appli- with a small red arrow. The grey background was cations for the synchronization process; this is the added afterwards in the bottom window for the sake SyncTEX output file. of visibility on printed media. Setting the synctex option to -1 creates an uncompressed foo.synctex auxiliary file, more suit- 2 What is SyncTEX? able for certain operations. Setting it to 0 definitively This is a new technology embedded in both pdfTEX prevents SyncTEX activation. and X TE EX, available in the 2008 TEX Live and cor- There is also an eponymous new TEX primitive responding MiKTEX distributions. When activated, that you can set to 1 for SyncTEX activation from the it gives both text editors and output viewers the nec- source file: \synctex=1. It can be used, for example, essary information to complete the synchronization to temporarily disable SyncTEX operations for some process. It will be available in LuaTEX soon. input file by properly using \synctex=0. This prim- In order to activate SyncTEX, there is a new itive has no effect if the -synctex=0 command line command line option: option was given, or if the first page has already been pdftex -synctex=1 foo.tex shipped out (it is then too late to activate SyncTEX).
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 365 Jérôme Laurens
3 Other synchronization technologies expansion only (in its head), and it does not know the corresponding output record until ship out time The commercial software Visual TEX available on Windows has had the PDF synchronization capabil- which occurs later (in its stomach). The problem is to force T X to remember the input record information ity embedded in its TEX engine since 1999. The E until ship out time. commercial software TEXtures available on Mac OS X has had embedded synchronization since 2000, but 4.2 Partial solutions using macros between the text source and the DVI output. Neither implementation is freely available to the public and The first idea, developed in the srcltx package, will not be considered in the remainder of this article. is to use the \special macro to keep track of the input record information until ship out time. By this Turning to the TEX macro level, Aleksander Si- method, it inserts in the text flow invisible material monic, the author of the WinEdt TEX shell on Win- dows, wrote before 1998 the srcltx macro package that dedicated DVI viewers can understand. The to enable synchronization between the text source main problem is that this invisible material is not and the DVI output. It is based on the powerful expected to be there and can alter significantly the \special command and was later integrated into line breaking mechanism or cause other packages to malfunction, which is extremely troublesome. the TEX engine as “source specials”. Heiko Oberdiek wrote vpe in 1999 where PDF technologies are used The second idea, developed in the pdfsync pack- for reverse synchronization from the PDF output age, is also to use macros, but in a different way to the text input. In 2003, the author wrote the because it is more difficult to manage PDF contents pdfsync package discussed in [5], [6] and [7], to al- than DVI contents. This package automatically adds low synchronization between the PDF output and in the input source some macros that act in two the text input, following ideas from Piero d’Ancona. steps. At macro expansion time, they write to an auxiliary file the input record information with a This was based on the use of pdfTEX commands sim- ilar to \special, with the same limitations, namely unique identifying tag. They also insert in the text an incompatibility with very useful packages and flow invisible material to prepare TEX to write the unwanted changes in the layout. output record information at ship out time, with None of these solutions is satisfying, being either exactly the same identifying tag. In this design, incomplete or unsafe, as we shall see. the problems concerning line breaking and package incompatibility remain. Moreover, the mapping be- 4 Solving the synchronization problem tween input and output records is not one to one, 4.1 Stating the problem which renders synchronization support very hard to implement for developers. The problem is to define a mapping between an input In these two different solutions, we see the in- record given by an input file name and a line number herent limits of synchronization using macros. More in that file, and an output record given by a page generally, we can say that those macros are active number and a location in that page of the output observers of the input records. In fact, by inserting file. The input record describes a certain part of the invisible material in the text flow they interact with input text whereas the output record describes the the typesetting process. On the contrary, SyncTEX corresponding location where this text appears in the is a neutral observer that never interacts with the output. The original TEX engine does not provide typesetting process. any facility for such a correspondence, except the debugging information used to report syntax errors 4.3 How SyncTEX works (we call it the current input record). More precisely, In fact, the only object that ever knows both the T X does not know at the same time both the input E input and output records is the TEX engine itself, records and their corresponding output records. In so it seems natural to embed some synchronization short, TEX parses each different line of the input technology into it. text file and expands the macros with its “eyes” and We first have to determine what kind of infor- “mouth” (according to [2], page 38), then it sends a mation is needed to achieve synchronization. For list of tokens into its “stomach”. In turn, the stomach that purpose, we follow [2] at page 63: “TEX makes creates lines of text, stacks them into paragraphs, complicated pages by starting with simple individ- and stacks the paragraphs into pages. Once a page ual characters and putting them together in larger is complete with the objects properly positioned, it units, and putting these together in still larger units, is shipped out to the output file. During this process, and so on. Conceptually, it’s a big paste-up job. TEX keeps the input record information until macro
366 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Direct and reverse synchronization with SyncTEX
The TEXnical terms used to describe such page con- process reaches in general a precision of ±1 line. struction are boxes and glue.” The key words are With additional technologies such as implemented in “characters”, “boxes” and “glue”. But since an in- iTEXMac2, we can even synchronize by words (see dividual character requires a considerable amount figure 1), essentially always finding an exact word of extra memory, only horizontal boxes and vertical correspondence between input and output. boxes are taken into account at first. For these boxes, The next improvements are a consequence of we ask TEX to store in memory, at creation time and the overall design of SyncTEX. Since synchronization during their entire lifetime, the current input record. is deeply embedded into the TEX engines, there is At ship out time, we ask TEX to report for each box no TEX macro involved in the process. As a straight- the stored file name, the stored line number, the forward consequence, there cannot be any incom- horizontal and vertical positions, and the dimensions patibility with macro packages. Moreover, no extra as computed during typesetting. This information invisible material is added to the text flow, thus en- will be available for synchronization: for example suring that the layout of the documents is exactly the when the user clicks on some location of the PDF same whether SyncTEX is activated or not. As a mat- document, we can really find out in which box the ter of fact, it is absolutely impossible to determine click occurred, and then deduce the corresponding if the output was created with SyncTEX activated file name and line number. We have here the design by examining its contents. Finally, no assumptions for a neutral observer. are made about external macros or output format, But if this new information is sufficient for lo- so that synchronization works the same for Plain, cating, it cannot be used for synchronization due LATEX or ConTEXt as well as DVI, XDV or PDF. to the way TEX processes files. In fact, boxes can Of course, all this needs extra memory and com- be created in TEX’s mouth where the current input putational time but this is in no way significant. In record is accurate, but in general, they are created fine, we can say that with SyncTEX, the synchroniza- in the stomach when breaking lines into paragraphs, tion has become safe and more precise. for text that was parsed a long time ago and no longer corresponds to the current input record. This 5 Limits and improvements is particularly obvious when a paragraph spans many It is indisputable that abandoning the use of macros lines of the input text: the line breaking mechanism and choosing an embedded design is a great advance is engaged after the last line is parsed, and every hor- for synchronization. But still it is not perfect! Some izontal box then created will refer to the last input aspects of the implementation are not complete due line number even if the contained material comes to a lack of time, but others will prevent us from from a previous input line. For that reason, we also reaching the ultimate synchronization comparable to ask TEX to store input records for glue items, be- wysiwyg (an acronym for “What You See Is What cause they are created in TEX’s mouth, when the You Get”) as discussed in [7]. current input record is still accurate. By combining boxes and glue management, we 5.1 The DVI to PDF filters have accurate information about positions in the When producing a PDF document from a DVI or XDV output and the correspondence with positions in the output, we apply a filter like dvitopdf or xdv2pdfmx. input file. In fact, things are slightly more compli- But those filters can alter the geometry of the output cated because of TEX internals: kern, glue and math by changing the magnification or the offset of the nodes are more or less equivalent at the engine level, top left corner of the text. In that case, the SyncTEX so SyncTEX must treat them similarly, but this is information, which is accurate for the DVI file, is not better for synchronization due to the supplemental accurate for the PDF file. This problem is solvable information provided. by post-processing the SyncTEX output file with the SyncTEX does other sorts of magic concerning new synctex command line tool available in the file name management, the magnification and offset, distributions, eg but these are implementation details. xdv2pdfmx -m MAG -x XXX -y YYY foo 4.4 The benefits of SyncTEX should be followed by Embedding the synchronization technology deeply synctex update -m MAG -x XXX -y YYY foo inside the TEX engine solves many problems and But this is not a good design. Instead, the post- improves the feature significantly. processing should be embedded into the various DVI The most visible improvements are connected to PDF filters so that no further user interaction is with accuracy: with SyncTEX, the synchronization required.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 367 Jérôme Laurens
5.2 Using TEX’s friends open and the author welcomes test files, suggestions Some documents are created with a complex typeset- and advice. ting flow where TEX is just one tool amongst others. 5.5 A question of design In such circumstances, the TEX file is automatically generated from possibly many original input files by The two preceding limitations are consequences of some processor. Then, synchronization should occur a conceptual default in the actual synchronization between the output and the original input files rather design. With SyncTEX, the TEX engine has been modified to export some observed information useful than the TEX file, which is just an intermediate step. A for synchronizers. The problem is that we are able to For example, a bibliography in LTEX is generally observe only what T X allows us to, and this is not built by BibTEX based on a bibliography database E with a bib file extension using an intermediate auxil- always the best information we would like to have. A It would be more comfortable and efficient if T X iary LTEX file with a bbl file extension. At present, E the synchronization occurs between the PDF output already provided synchronization facilities from the and the bbl file and not the original bib file, as one very beginning. In that case, all the macros packages would prefer. would have to be compatible with the synchroniza- tion code and not the opposite. That would require Improving SyncTEX to properly manage this situation is not extremely complicated: we first have more work for the package maintainers but would also prevent any kind of layout and compatibility to define a SyncTEX map file format for the mapping between the lines of the original input files and the problems due to special nodes. In a different approach, a supplemental step lines of the produced TEX files, then we have to provide facilities to merge this mapping information could be to store synchronization information for each individual character, thus increasing the mem- into the SyncTEX output file. Then the processor could produce the map file, and a supplemental step ory requirements of the engine in a way similar to in the typesetting flow would update the SyncT X how X TE EX handles multi-byte characters. This idea E ´ information with that map. was originally proposed by Hàn Thê Thành, but it Sometimes it might not be appropriate to simply was abandoned because the SyncTEX output file was bypass the intermediate file. In that case, the viewer unbearably huge. With the new design, this idea can should synchronize with the auxiliary file using a certainly be revisited with more success. text editor which in turn should synchronize with Anyway, further investigations into the arcana the original input file using the map file. of the TEX program would certainly lead to a better synchronization accuracy but if we want to avoid 5.3 Accuracy and the column number huge changes in TEX and keep compatibility with As described above, we only take into account whole existing macro packages, we must admit that we lines in the input files and jump from or to lines have almost reached some insuperable barrier. in the text. This can suffice for textual files, but 6 Implementation in TEX Live does not when mathematical formulas are involved — we would like to have a more precise position in Without entering into great detail, we explain how the input. Unfortunately, when parsing the input the implementation of SyncTEX is carefully designed to ease code maintenance and enhancements, as far files, the original TEX engine does not handle column positions at all. And it seems that adding support as possible. for this supplemental information might need a great 6.1 A segmented implementation amount of work, probably much greater than the eventual benefits. All the SyncTEX related code is gathered in only one directory named synctexdir, in which the code is 5.4 For non-Latin languages split into different source files. The separation is orga- nized in order to share the maximum amount of code SyncTEX has been designed with a Latin language between the different engines, and to clearly iden- background: it relies on the fact that TEX automat- ically creates kern and glue nodes at parse time to tify the different tasks involved in the information manage interword spacing. For languages that do not management. All in all, we end up with 14 different have a comparable notion of word, the synchroniza- change files. When building the binaries, the partial tion will not be sufficiently accurate and will most make file synctex.mk has the duty to manage which certainly need further investigations. This question is change file should apply to which engine and when.
368 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Direct and reverse synchronization with SyncTEX
6.2 An orthogonal implementation synchronization flow, the user asks the viewer to One of the key concepts in modern code design is sep- synchronize the text editor with a given location aration, whose purpose is to ease code management in a given page of an output file, the viewer asks and maintenance. WEB Pascal does not offer facilities the SyncTEX parser for the input file name and line for code separation, nevertheless, it is possible to number corresponding to the location; it then asks the text editor to display the named input file and build all the engines with or without the SyncTEX feature, as explained in the synctex.mk file. It will highlight the numbered line. be useful for developers whenever a problem is caused 7.3 The new synctex command line tool by the SyncTEX patches. There are cases when the inclusion of the parser li- 7 Which software supports SyncTEX brary is not possible or even improbable (consider Up to now, we have focused on the technological for example Adobe’s Acrobat reader). For such situ- aspects of synchronization, and we have described ations, the synctex command line tool is the alter- in detail the foundations. It is time to look at the native designed to allow synchronization. It is just a concrete implementations of synchronization with wrapper over the SyncTEX parser library that acts as different methods, because this feature is useless if an intermediate controller between a text editor and a viewer. The description of its usage is obtained via it is not adopted by applications. SyncTEX not only the command line interface running synctex help consists of changes to the TEX engine, but gives developers tools to easily support the technology. view for direct synchronization and synctex help edit for reverse synchronization. 7.1 The SyncTEX parser library Provided that the text editor and the viewer The main tool is a library written in C, whose pur- have some scripting facilities, here is how this tool pose is to help developers implement direct and re- should be used. For direct synchronization, the user verse synchronization in their application. It con- asks the text editor to synchronize the viewer with sists of one file named synctex_parser.c and its a given line in a given input file, the text editor for- header counterpart synctex_parser.h, meant to wards this file name and line number to the synctex be included as-is in application sources. Both are tool together with some scripting command to acti- vate the viewer, the synctex tool transparently asks available on the SyncTEX web site [4]. The source file takes care of all the ancillary work concerning the SyncTEX parser for the page number and loca- tion corresponding to the information it has received, SyncTEX information management and the header file contains all the necessary help for an optimal then it asks the viewer to proceed with the help of usage of the programming interface. the scripting command. For reverse synchronization, the user asks the At this writing, TEXworks (presented by J. Kew in [1]), Sumatra PDF on the Windows platform, viewer to synchronize the text editor with a given location in a given page of an output file, the viewer and Skim and iTEXMac2 on Mac OS X, all support forwards this information to the synctex tool to- SyncTEX by including this parser library. For other gether with some scripting command to activate the applications, TEX users are encouraged to send a feature request to the developers of their favorite text editor, the synctex tool transparently asks the PDF or DVI viewer. SyncTEX parser for the input file name and line num- ber corresponding to the information it has received, 7.2 Remark about the document viewer then it asks the text editor to proceed according to It should be noticed that the tricky part of direct and the received scripting command. reverse synchronization should be handled by the Before this tool was available, developers had no solution other than directly parsing the contents of viewer only. The SyncTEX parser library is meant not for text editors but for viewers. In a normal the SyncTEX output file. This was generally made in direct synchronization flow, the user asks the text continuation of the implementation of pfdsync sup- editor to synchronize the viewer with a given line port. Comparatively, it is more comfortable to work in a given input file, the text editor forwards the with a .synctex file than a .pdfsync file because the file name and the line number to the viewer, the new syntax is extremely clear and straightforward, consequently reverse engineering was unexpectedly viewer asks the SyncTEX parser for the page number and location corresponding to the information it has encouraged. But this practice should be abandoned received, then it scrolls its view to the expected page for two reasons: it is certainly not compatible with and highlights the location. In a normal reverse forthcoming enhancements of SyncTEX, and it gener- ally does not work when changing the magnification
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 369 Jérôme Laurens
and the offset in the DVI to PDF conversion, as dis- cussed above. In order to convince developers to prefer the synctex tool, the specifications of the SyncTEX output file are considered private and will not be widely published. More details concerning usage and implementa- Figure 2: TEX output embedded in HTML, detail of tion are available on the SyncTEX web site [4]. http://en.wikipedia.org/wiki/Special_functions 8 Applications (2008/08/11) There are a variety of ways to use the newly avail- able information in the SyncTEX output file. Some were considered while designing this feature, others suggested by people at the conference. No doubt this The interest is at least twofold. It can serve list is not exhaustive. debugging purposes for publishers who want to elab- orate complicated page layouts, and it can also serve 8.1 Better typesetting mechanisms pedagogical purposes during TEX training sessions. TEX is well known for its high quality page breaking For these reasons, this viewer will be available on mechanism, but the hardware constraints that were the SyncTEX web site [4] once it has been properly crucial 30 years ago imposed some choices and delib- factored for distribution. Unfortunately, this bene- erate barriers. The limitation in memory usage led fits Mac OS X users only, but adding this feature to to a page by page design, where memory is freed each the new cross-platform TEXworks will eventually be time a page is shipped out. In that situation, a page considered. breaking algorithm cannot perform optimization in 8.3 Embedding T X output into HTML or a document as a whole, but only on a small number E running text of consecutive pages. In order to have global optimization algorithms, In web pages, it is rather common to include mathe- one can keep everything in memory until the end of matical formulas as embedded images built with TEX, the TEX run, but that would require a big change in to compensate for the limitations of web browsers. the engine. From another standpoint, SyncTEX has The example given in figure 2 is particularly ugly, demonstrated that it is possible to trace geometrical not only because the size of the mathematical text information throughout the typesetting process. It is does not conform to the size of the running text, but clear that the information actually contained in the also because the base lines of the formulas and the SyncTEX output file is not suitable for typesetting running text are not properly aligned. In fact, the purpose because it was designed for synchronization included images contain no information concerning only. But with some additional adaptations, there is the base line, and this is where SyncTEX can come no doubt that SyncTEX can help in designing global into play. The synchronization information contains optimization algorithms for even better typesetting. the dimensions of each box containing a mathemati- cal formula, in particular its height and depth, hence 8.2 Debugging facilities the exact position of the base line. We just have to During his presentation at the conference (see [3]), raise the image by the proper amount to obtain a the author used a lightweight PDF viewer to demon- correct vertical alignment. strate SyncT X. This viewer was primarily designed E 9 Concluding remarks as a proof of concept and as such, was meant to remain private. But one of its features might be of 9.1 Synchronizing by word great interest to the TEX community, as suggested In iTEXMac2, synchronization is enhanced to at- by different people at the conference, namely the tain the precision of a word or even a character, by ability to display over the text all the boxes, either combining SyncTEX with some text analysis. This horizontal or vertical, created during the typesetting was rather easy to accomplish because iTEXMac2 process. As it happens, this feature was already im- manages both the text input and the PDF output, plemented in an unknown modest PDF viewer for and also because the PDF library on Mac OS X has Mac OS X (whose name I have unfortunately lost) by text facilities. But this does not mean that only an parsing the result of the \tracingall macro in the integrated TEX environment is able to reach such log file. a level of accuracy. It is in fact more a matter of communication between different applications.
370 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Direct and reverse synchronization with SyncTEX
In fact, the text editors and the viewers allow ago (not one year ago as claimed by the author dur- some inter-process communication through the com- ing his presentation [3]), but he could only take some mand line, for example mate -l LINE foo.tex asks time for coding this in summer 2007, probably under the TextMate text editor to edit foo.tex at line LINE. the friendly pressure of some users dissatisfied with Only the line number is used and this is perfectly the limits of pdfsync. Although his first attempt was suitable for programmers because in general, com- hardly usable and finally abandoned, it introduced pilers and debuggers only need line numbers. But the author to the minutiae of the Web2c implemen- TEX users are not programmers. In iTEXMac2 this tation of TEX. Initially, SyncTEX was targeted at kind of practice has been reconsidered for reverse pdfTEX but Jonathan Kew helped in adapting it to synchronization and the information passed to the X TE EX and also with the integration into TEX Live. text editor contains not only the file name and the This short review seems to indicate that tech- line number, but also an extract of text surrounding nologies like SyncTEX could easily have become avail- the character under the mouse. This supplemental able many years ago. One can attribute the delay information is the hint used by the text editor to to a lack of effort devoted to the human interface have a better focus on the synchronization target. of TEX, which is highly regrettable. With SyncTEX For direct synchronization, the same idea applies and and tools like TEXworks, first steps are made in the the PDF viewer is asked to highlight a location in a right direction, because TEX really deserves a good given page, with the help of a similar textual hint. human interface, not just a user interface. In order to achieve synchronization by word or character, text editors and viewers should use a 10 Acknowledgements textual hint, both as senders and receivers. Of course The author gratefully thanks Hàn Thê´ Thành with- this requires some coding effort because input text out whom this work would never have started and and output text are not exactly the same (due to line Jonathan Kew without whom this work would not breaking for example), but the expense is affordable have reached the present stage. He received impor- as soon as efforts are combined. Once again, users tant remarks and valuable help from members of the are encouraged to submit feature requests to the pdfTEX, X TE EX, iTEXMac2 and TEX Live develop- developers of their favorite tools. ment teams; thanks to all of them. By the way, the new synctex command line tool anticipates the use of a textual hint by editors References or viewers through its -h command line option. [1] Jonathan Kew. TEXworks: Lowering the barrier to entry. In this volume, pages 362–364. 9.2 An historical standpoint [2] D. Knuth. The TEXbook. Addison Wesley, 1983. Synchronization with SyncT X appears for the 30th E [3] Jérôme Laurens. SyncTEX presentation at anniversary of TEX; we can legitimately wonder why TUG 2008. http://www.river-valley. and whether such a long period of gestation was tv/conferences/tug2008/#0302-Jerome_ necessary. In order to explain this delay, let us Laurens. review the ingredients that made SyncT X possible. E [4] Jérôme Laurens. SyncT X web site. http: As in many situations of software design, a fa- E //itexmac.sourceforge.net/SyncTeX.html. vorable context comes concurrently from available technologies and available workers. Regarding tech- [5] Jérôme Laurens. iTEXMac, an integrated TEX nological aspects, we can say in a reduction not very environment for Mac OS X. In TEX, XML, and Digital Typography, volume 3130/2004 far from reality that SyncTEX is nothing but a clever of Lecture Notes in Computer Science, pages usage of the Web2C implementation of TEX. Of course, developing on Mac OS X was rather easy and 192–202. Springer Berlin / Heidelberg, 2004. very efficient, but any other environment would cer- [6] Jérôme Laurens. The TEX wrapper structure: tainly provide the same result at the price of more A basic TEX document model implemented th programming work. in iTEXMac. In EuroTEX 2005, 15 Annual Concerning people, the author has claimed since Meeting of European TEX Users, 2005. the beginning of pdfsync that some synchronization [7] Jérôme Laurens. Will TEX ever be wysiwyg or should definitely be embedded in the TEX engine, in the PDF synchronization story. The PracTEX the hope that someday, someone else would do it. Journal 2007(3), 2007. Hàn Thê´ Thành was aware of the problem three years
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 371 Xindy revisited: Multi-lingual index creation for the UTF-8 age
Joachim Schrod Net & Publication Consultance GmbH Kranichweg 1 63322 R¨odermark, Germany jschrod (at) acm dot org http://www.xindy.org/ Abstract Xindy is a flexible index processor for multi-lingual index creation. It handles 44 languages with several variants out of the box. In addition, some indexes demand special sort orders for names, locations or different target audiences; xindy can handle them as well. Raw index files may have several encodings beyond ASCII. In particular, LATEX’s standard output encoding is supported directly, as is X TE EX’s UTF-8 output. With the new xindy TUG30 release, support for Windows is added; previously xindy was available only for GNU/Linux, Mac OS X, and other Unix-like systems.
1 What is xindy? Locations are more than page numbers: Most Xindy is an index processor. Just like MakeIndex, index processors can work only with numbers, or it transforms raw index information into a sorted maybe sequences of numbers such as “2.12”. Going index, made available as document text with markup further, xindy features a generalized notion of struc- tured location references that can be book names, that may be processed by TEX to produce typeset book indexes. Unlike MakeIndex, it is multi-lingual music piece names, law paragraphs, URLs and other and supports UTF-8 encoding, both in the raw index references. You can index “Genesis 3:16” and “Ex- input and in the tagged document output. odus 3:16” and Genesis will be in front of Exodus Overall, xindy has five key features: since they are not alphabetically sorted names any more, but terms in an enumeration. 1. Internationalization is the most important fea- Such location references may be combined into ture at all and was originally xindy’s raison a range, such as 6, 7, 8, and 9 becoming “6–9”. Also d’ˆetre: the standard distribution knows how to well-known are range specifications in the humani- handle many languages and dialects correctly ties, such as 6f or 6ff. With xindy, location ranges out of the box. can also be formed over structured references, but 2. Markup normalization and encoding support is some knowledge about the domain of the reference the ability to handle markup in the index keys components must be available. in a transparent and consistent way, as well as xindy is configured with a declarative style lan- different encodings. guage, where declarations look like 3. Modular configuration enables the reusability (some-clause argument1 argument2 ...) of index configurations. For standard indexing A file with such declarations is called an xindy mod- tasks, LATEX users do not have to do much except ule, and an xindy run may use several of these mod- use available modules. ules. This allows making available predefined mod- 4. Location references go beyond page numbers. ules for common indexing tasks, e.g., the f- and Locations may also be book names, section num- ff-range designation illustrated above. Xindy decla- bers or names, URLs, etc. rations are also used to configure output markup. Last, but not least, xindy is the practical result 5. Highly configurable markup is another corner- in research about a theoretical model of index cre- stone. While this is usually not as important A ation. Even if one does not use xindy the program, for LTEX users, it comes in handy if one works the model itself can provide valuable input for the with other author systems. creation of future index creation programs. The focus of this paper is the current state of multi-lingual and encoding support that’s available 2 Multi-lingual sorting for xindy. The paper’s scope does not include other Sorting is a multi-layered process where characters features which I’ll mention just in passing: are first determined, placed into categories that are
372 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Xindy revisited: Multi-lingual index creation for the UTF-8 age
albanian greek norwegian To address that valid question, I’d like to present a belarusian gypsy polish few peculiarities that show why index sorting is more bulgarian hausa portuguese complex than just sorting a few strings and why an croatian hebrew romanian ISO standard on string ordering is a good start but czech hungarian russian not the end of ordering index entries. danish icelandic serbian Cultural peculiarities For some languages, in- dutch italian slovak dex sort order depends on context, or the term’s english klingon slovenian semantics. German is a good example for this com- esperanto kurdish spanish plexity: there are two sort orders in wide usage and estonian latin swedish they differ in the sorting the “umlauts”. These are finnish latvian turkish the vowels with two dots above: ¨a,¨o,and ¨u. french lithuanian ukrainian One sort order sorts them as if there were a general lower-sorbian upper-sorbian following ‘e’, i.e., ‘¨a’is sorted as ‘ae’, ‘¨o’is sorted as georgian macedonian vietnamese ‘oe’, and ‘¨u’ is sorted as ‘ue’. That is the official sort german mongolian order, and is defined in an official German standard, Table 1: Predefined languages in xindy DIN 5007. This sort order is used for indexes in publications for the domestic market, for an audience that knows German and is thus expected to know that these characters are true letters on their own sorted the same for now (collation classes), and sorted and not just vowels with accents. Such a domestic either left-to-right or right-to-left. If this results in audience is also expected to have learned the sort index entries that are sorted the same but are not order of umlauts in their first school year and will identical, characters are reclassified with different be able to cope with that cultural peculiarity. rules and sorted again to resolve the ambiguities. A second way of index sorting drops that idio- Words from most languages can be sorted with this syncratic German feature and sorts umlauts as if process. It is standardized as ISO/IEC 14651 (Inter- they were vowels with accents, i.e., ‘¨a’is sorted just national String Ordering). like ‘a’, and so on. This sort order is used in indexes of publications for an international market, or where 2.1 Predefined languages an international audience is expected to read this Xindy provides the ability to sort indexes in different publication regularly. Especially dictionaries and languages; 44 of them are already prepackaged in the phone books use this non-standard way of sorting; distribution and are listed in Table 1. For some of we want to give our foreign visitors a chance to look these languages there are multiple sorting definitions: up the phone number of any Mr. or Ms. M¨uller they e.g., German has two different kind of sort orders, want to visit. This sort order has no official name, colloquially called DIN and Duden sorting (more on but is colloquially known as phone book sorting or that later). “Duden sorting”, after the most important dictionary While the sorting of all predefined languages of the German language that uses this sort order. may be expressed in terms of the ISO standard 14651 Legacy rules Some special and non-obvious sort named above, xindy’s abilities go beyond that. The orders are so old that the reason behind them is standard language modules are usable for indexes not known (at least, not to me). An interesting where index entries all belong to one language or example is French, where additional complexity has where foreign terms are sorted as if they would be been introduced in some previous time when it comes local. But if one mixes several languages in one to sorting names with accented characters: when two index, e.g., in an author index, one is able to define words have the same letters but differ in accents, the the sort rules that should be used individually, just existence of accents decides the final sort order — but for this text. While this is some work, of course, backwards, from right to left! xindy at least makes it possible to create indexes for The most prominent example is the four words such real international works that go beyond mere multi-lingualism. cote cˆote cot´e cˆot´e In the first pass of sorting a French index, these 2.2 Complexity of index sorting four words are sorted the same. In the second One might ask if this paper doesn’t make a mountain pass, they are still sorted the same — the second out of a molehill, and what’s the big deal with all this pass sorts uppercase letters before lowercase letters. supposed complexity of index sorting and creation The third pass then sorts from right to left and puts
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 373 Joachim Schrod
non-accented letters before accented letters: German, French, or any other language is a good −−→ and necessary start, and many index processors stop cote at that. But it is not sufficient for production of ←−− actual indexes where sorting rules appear that are cˆote not covered by standards. cot´e For example, in author indexes some languages cˆot´e handle parts of nobility names differently, depending on whether they are part of the name or a true This finally results in the sort order shown in the peerage title. In registers of places, city names might table above. be sorted differently than spelled. Transliteration Character recognition Sometimes legacy repre- must be taken into account, just like combination of sentations in files on computers introduce complex- alphabets within one index. ities: In Spanish, ‘ch’ and ‘ll’ are one letter and This boils down to the requirement that project- sorted accordingly, whereas in all other European or document-specific sort rules are needed. While one languages they are two letters. Traditionally, these book might sort ‘St.’ as it is written, Gault Millau Spanish letters are represented by two characters in will need a different sort order for its register—they a file; an index sort processor has to recognize them sort it as ‘Saint’. MakeIndex introduced a way to as such. do that by specifying print and sort keys explicitly, There is also the problem of what to do when a as in \index{Saint Malo@St. Malo}: It demands character appears in an index that does not exist in from the author that this explicit specification must that language, e.g., there is no ‘w’ in Latin. Should be used every time that term appears. one be pragmatic and sort it like modern languages Xindy goes a step further and allows the user of the Roman family, between ‘v’ and ‘x’? Or should to specify sort orders in a separate style file that one place it somewhere in the non-letter group? The may be used just for one document or reused for order might very well depend on the target audience all documents in a project. It still allows using an and intent of the respective index. explicit sort and print key in your TEX document — In the TEX world, character recognition is ar- but experience has shown that it is much less error- guably less an issue for non-Latin scripts — TEX au- prone to declare it once in an external file for a whole thors are used to specifying their characters with group of index terms than to write it explicitly in exact encodings or transliteration. Especially the each occurrence of that group. rise of Unicode text editors and their enhanced in- put support for non-Latin scripts make identification 4 Examples for xindy style declarations of characters easier than the supposedly ASCII-like Let’s have a look at how such document-specific representation in traditional encodings. declarations are done. We demonstrate their use for Beyond Europe The examples so far were “just” two purposes: index entry normalization and entry about European languages. (Admittedly, because I sorting. know most about them . . . ) Some languages use pho- Markup normalization is the process to decide netic sorting where one needs additional information if two raw index entries denote the same term and about words that are used in the sort algorithm. This should be combined into one processed index entry, does not change the algorithm itself, but available i.e., if they should be merged. authoring systems often do not support that aspect Especially with TEX, it might be that the same at all. (xindy does not support it out-of-the-box term appears differently in the raw index. This is either, but it has the functionality to describe such caused by macro expansions, especially when one sorting in its language modules.) produces part of the index entries automatically. De- Other languages use aspects of glyphs such as pending on the state of TEX’s processing, macros in strokes or number of strokes for sorting. Diacritics the raw index are sometimes expanded and some- may or may not influence sorting; sometimes they are times not expanded. vowels, sometimes they just denote special emphasis Here comes into play a point where xindy dif- and can be ignored. fers from MakeIndex: it ignores TEX or LATEX tags (macro names and braces) by default. With xindy, 3 One sort order is not enough you can write \index{\textbf{term}} and this will For multi-lingual index creation it is not sufficient be the index entry “term”, \textbf and the braces to define sort orders for languages. Having defined will be ignored. (Such index entries are usually not a language module with the default sort order of
374 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Xindy revisited: Multi-lingual index creation for the UTF-8 age
input manually, but are generated by other macros.) You can use UTF-8 there with the inputenc package, So, how would one index METAFONT, written but that encoding doesn’t matter for index sorting. in the Metafont logo font? The MakeIndex way When the raw index arrives at xindy, that original would be to use \index{METAFONT@\MF} for every encoding is not visible any more; we see only LICR. to-be-indexed occurrence of METAFONT, and that And we just need ISO-8859 encodings for sorting way still works with xindy. But in addition, one can languages that are supported by LATEX’s standard use an xindy style file with the declaration setup, which mostly use European scripts. (merge-rule "\MF" "METAFONT") While this is appropriate and useful for Euro- and just use \index{\MF} in the document. No pean languages, it won’t help authors with docu- risk to add typos to one’s index entries; these index ments in Arabic, Hebrew, Asian, or African lan- guages. But they also won’t use LICR much any- entries will be sorted as ‘METAFONT’ but output as METAFONT. how and will probably be better served by new pro- grams like X T X or Omega/Aleph. For these users, Merge rules may influence whole index terms E E all language modules are supplied in a variant that or just parts of them. One can also use regular expressions to normalize large classes of raw terms. knows about UTF-8 encodings as output by X TE EX or Omega’s low-level output of (Unicode) characters. Sort specifications Sort orders are specified with If one has a raw index file that was produced by very similar declarations: these systems, one can use xindy; it will “just work”. (sort-rule "¨a""a~e") Looking beyond UTF-8 is still not necessary in tells xindy to place ‘¨a’between ‘a’ and ‘b’. (The the TEX world; we have no TEX engine that will special notion ‘~e’ means “at the end”; there is also output UTF-16 or even wider characters to a raw ‘~b’ for “at beginning”.) index file or expect such encodings in a processed This is the low-level way to specify sort orders, index file. That’s good, because xindy can’t handle and it is used to create special document- or project- UTF-16 input — yet. This will probably be an en- specific sort orders as mentioned above. It is possible hancement of one the next major releases and shall to create whole language sort-order modules with help to open up xindy’s appeal beyond TEX-based that method as well — we did so at the start of the authoring environments. xindy project. 6 Availability But then Thomas Henlich wrote make-rules, a preprocessor to create xindy language modules for Through release 2.3, xindy was available only for languages where sort order can be expressed with GNU/Linux and other Unix-like systems. At the the ISO 14651 concept. For that preprocessor, one time of writing, release 2.4 has been prepared which th describes alphabets and sort orders over collation is nicknamed the TUG30 release, to honor TUG’s 30 classes with multiple passes, and xindy modules with birthday. This release adds support for Windows sort rules as shown above are created as a result. (2000, XP, and Vista), thus widening the potential user base considerably. For now, installation of a 5 Encoding of raw index files— Perl system is needed to use xindy; this should not LICR and UTF-8 be a big obstacle. At the moment, the most often used encoding for raw The TUG30 release is available for download at index files is the LATEX output of \index commands. xindy’s Web site www.xindy.org. Currently, it is That encodes non-ASCII characters as macros; the there in source form; binary distributions for several representation is called LATEX Internal Character operating systems will be added as time permits. Representation or LICR, as described in section 7.11 While release 2.3 is included in TEX Live 2008, of The LATEX Companion, 2nd ed. Xindy knows with executables for nearly all the platforms except about LICR: xindy modules exist with merge rules to Windows, including Mac OS X, we were not able to recognize these character representations. A special finish the new release 2.4 in time to make it to the invocation command for LATEX, texindy, picks them DVD. Hopefully, it will become available via the up automatically, so authors have no need to think new on-line update mechanism soon. Eventually, full about them. support for xindy will be available in TEX Live 2009. At the moment, LICR is mapped to an ISO-8859 The best place to look for user documentation encoding that’s appropriate for the given language, about xindy is The LATEX Companion, 2nd ed., sec- and that encoding is then the base for xindy’s sort tion 11.3. Documentation on the Web site is techni- rules. Please note that this is completely unrelated cally more complete, but improvements of its organi- to the encoding used in the author’s LATEX document. zation and accessibility are high on our to-do list.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 375 A newbie’s experiences with Lilypond, Lilypond-book, LATEX and Perl
Joe McCool Southern Regional College, Ireland mccoolj (at) src dot ac dot uk http://benburb.demon.co.uk/apache2-default/joe.html
Abstract The author is an active Irish traditional musician. He is also a keen inland boater. He is having a lot of fun composing a book on “Traditional Music for Boaters”. In this paper he describes his successes and frustrations using Lilypond, Lilypond-book, LATEX and ABC musical notation. Lilypond and LATEX have a lot in common. Neither are WYSIWYG, neither demand GUI’s. Both compile simple flat files to produce beautiful graphical output. Lilypond’s original manifestations produced output directly for LATEX, but of late users writing books have been encouraged to use Lilypond-book. This looks for Lilypond code within LATEX source files and produces graphics and associated instructions which can then be processed by LATEX. Most joy has come from automating these processes via GNU/Linux and Perl.
1 What’s out there C:Franz Schubert By definition classical music has an inherent connec- O:Austria M:C| tion with books and text. Traditional music does not. L:1/8 Classical music is written down in scores. Histori- Q:1/4=160 cally, traditional music is not. It is largely an aural K:C medium and tunes are learnt by ear. Having said G2|"C"c2c2e2e2|"C"c4G2G2|"G"G3G dcBA|"G" that, musical scores do now have an important role G4 z2G2|"C"c2c2 e2e2|"C"c4G2c2|"G"B2AB to play. More and more young traditonal musicians .... are learning how to read scores. Printed material The K: field represents the tune’s key. All lines allows us to store pieces that would otherwise be lost below this contain the music. Lines above are header and pencil and paper is a useful aid to composition. fields, with X: representing an index for this partic- Musicians like to share pieces and the arrival ular tune in a file of other ABC’s. Notes in quotes of the personal computer has made this easier than represent accompaniment chords. ever. But, where years ago we popped manuscripts in At first sight ABC seems ideal for what I wish the post, we now need to share scores electronically. to do. It has a simple input format. It can output This has brought about a plethora of programs and A PostScript files that I can incorporate into LTEX systems that enable us to do so. documents. ABC programs are open source and, 1.1 ABC notation most importantly, there are huge collections of ABC source files available on the internet. In the 1980’s Chris Walshaw, then at the Univer- But, I have a few issues with ABC: sity of Cambridge, began writing out fragments of • there is a gross lack of standardisation. What folk/traditional tunes using letters to represent the standardisation exists is often ignored by the notes. This became gradually formalised into what 1 authors of ABC files. is known as the ABC “standard”. Numerous small A programs have appeared to convert ASCII files of • I need a system comparable to LTEX in terms 2 ABC to printed scores. There are also programs to of typesetting quality. ABC does not have the A convert ABC code to midi. fine grained control of LTEX. Here’s an example of an ABC file: • It is difficult to avoid clashing problems: note X: 1 heads clashing with bar numbers, or grace notes T:The trout clashing with accidentals.
1 www.walshaw.plus.com/abc • At the time I started my project, the ABC mail- 2 for example, moinejf.free.fr ing list seemed to vanish!
376 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting A newbie’s experiences with Lilypond, Lilypond-book, LATEX and Perl
1.2 Commercially available software 1.5 Example Lilypond file Under Microsoft Windows, several commercial pro- \version "2.11.33" grams are available for typesetting music. Finale,3 \header { Sibelius,4 and Cakewalk5 are well known. Notewor- composer = "Joe Mc Cool" thy Composer is available as shareware.6 title = "The Eight Lock" dedication = "" 1.3 MusicTEX and MusiXTEX } voicedefault = { Both of these, based on what I could find out about 7 \relative c’ them, appeared too complex for me. They did \clef treble not seem to have an active development or support \key g \major community. \time 4/4 \repeat volta 2 { 1.4 Lilypond \time 4/4 I rejected the commercial products and Notewor- \clef treble thy because they use proprietary file formats and d’4 d8 e d c b4 they rely on GUI interfaces. I am a traditional- .... ist in more ways than one. Long ago I realised the } \repeat volta 2 { power, elegance and beauty of plain ASCII files under 89 a’4 a8 a b c4. GNU/Linux. Hence my final choice of Lilypond. .... • Lilypond’s originators have objectives very simi- } lar to those of LATEX: “to print music in the best } traditions of classical engraving with minimum \include "../new.score.ly" fuss”. Notice the indentation of code, similar to pro- • Lilypond uses plain ASCII, not dissimilar to ABC. gramming languages. voicedefault is a musical object • Lilypond enjoys ongoing development. that will subsequently process as a score (see below), • Its documentation is excellent. or a midi file.10 • Very active user support via mailing lists. As my project gathered weight I got tired of • Very fast keying of source files: having to edit individual Lilypond files in order to – Note durations need stating once only. In change the overall look of my book, hence my use of the input a4 b, the notes a and b have the the include statement. new.score.ly consists of: same duration. \score{ – Notes can be raised an octave using a fol- << lowing ’ or lowered by a following ,. Lily- { \voicedefault pond also provides a relative mode in } which it will position notes on the scale in >> a common sense, reasonable fashion. \layout{ \relative c’’ { #(layout-set-staff-size 20) b c d c b c bes a } } } These notes will all be placed within the In order to change the overall look of all my scale, rather than climbing higher upward. pieces, for example to change the staff size, all I have • key transposition is easy: \transpose d e ... to do is edit the above. • The excitement and delight I found putting to- gether my first Lilypond scripts was matched 2 My approach only by that of my first LATEX scripts. Lilypond My approach is constrained by the following goals: A and LTEX really are first cousins! • Each page should contain an integer number of 3 www.finalemusic.com tunes. Classical musicians are happy (or at least 4 www.sibelius.com willing) to turn a page in the middle of a piece, 5 www.cakewalk.com 6 traditional musicians are not. www.noteworthysoftware.com 7 www.tex.ac.uk/cgi-bin/texfaq2html?label=music • Traditional music is often played in sets. Two 8 www.lilypond.org or three jigs will be played one after the other, 9 In rural Ireland we have a saying: “If you think a donkey will do the job, use a horse!” 10 though midi support is regrettably not good in Lilypond.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 377 Joe McCool
or a group of hornpipes. When new tunes are Some text before a musical snippet. added to the collection, they must be kept as à j ¹ close together as possible to other members of ¹ ¹ ¹ their set, ideally on the same page, or on the Another snippet: same spread. à j ¹ ¹ ¹ ¹ • Brief texts and footnotes must appear on the Some more text. page of the tune to which they refer. Figure 1: small.pdf • Index entries must have the form ‘name : type : page number’. This reveals the page number Here the file small.tex constitutes the final and the tune type for each entry. output. This is then processsed by LATEX in the • The build process should produce midi files. normal way, with the result shown in Fig. 1. Lilypond-book creates a graphic for each line of 3 Combining Lilypond and LATEX music. It also creates a graphic for the whole snip- pet — in our small example, 2b589ef505-1.eps and Here is a simple example of a LATEX file (small.ly) containing Lilypond code: 2b589ef505.eps. My project has currently gath- ered about 200 tunes and the number of small files \documentclass{article} in OUTPUT hovers around 3500! \begin{document} Contents of OUTPUT/small.tex: \noindent Some text before a musical snippet.\\ \documentclass{article} \begin[quote,fragment]{lilypond} \usepackage{graphics} { \begin{document} c’ e’ g’ e’ \noindent } Some text before a musical snippet.\\ \end{lilypond} {% Another snippet:\\ \parindent 0pt% \begin[quote,fragment]{lilypond} \ifx\preLilyPondExample \undefined% { \relax% f’ g’ a’ b’ \else% } \preLilyPondExample% \end{lilypond} \fi% Some more text.\\ \def\lilypondbook{}% \end{document} \input lily-2b589ef505-systems.tex% This is processed by the command line: \ifx\postLilyPondExample \undefined% \relax% lilypond-book -f latex --psfonts \else% --output OUTPUT small.tex \postLilyPondExample% \fi% And in the OUTPUT directory Lilypond creates } the following files: Another snippet:\\ lily-2b589ef505-1.eps {% lily-2b589ef505.eps \parindent 0pt% lily-2b589ef505.ly \ifx\preLilyPondExample \undefined% lily-2b589ef505-systems.tex \relax% lily-2b589ef505-systems.texi \else% lily-2b589ef505.txt \preLilyPondExample% lily-eb070afcf7-1.eps \fi% lily-eb070afcf7.eps \def\lilypondbook{}% lily-eb070afcf7.ly \input lily-eb070afcf7-systems.tex% lily-eb070afcf7-systems.tex \ifx\postLilyPondExample \undefined% lily-eb070afcf7-systems.texi \relax% lily-eb070afcf7.txt \else% small.dep \postLilyPondExample% small.tex \fi% snippet-map.ly } snippet-names Some more text.\\ tmpMQ9ShM.aux \end{document}
378 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting A newbie’s experiences with Lilypond, Lilypond-book, LATEX and Perl
If LATEX can fit the snippet into a page in its but lilypond-book complained about not being able entirety it uses the whole graphic, otherwise it uses to find files. It is just not that clever. It is not able individual lines, placing some of the lines on the fol- to process my \newcommand. lowing page. This breaks my first requirement: pages should contain only an integer number of snippets. 6 Source collections Ironically the largest collections of traditional music 4 Overcoming the first limitation from all over the world are held in ABC files and there My first approach to this problem was to wrap the are quite a few search engines tuned specifically for 12 snippet in a Figure environment: searching ABC sites. There is also a Python script \begin{figure} available that converts ABC to Lilypond (abc2ly). .... Again, possibly because of the lack of ABC stan- lilypond code dards, abc2ly does not produce very tidy code and .... sometimes gets the repeats plain wrong. It is often \end{figure} brought to its knees by idiosyncratic ABC. An integer number of tunes then appeared on a 7 And then there was Perl page, but I found that the positioning of the graphics A was inconsistent, particularly at the end of chapters. Perl is ideal for processing text. Both LTEX and Google reported that lots of people had suffered from Lilypond are text based, so the marriage is obvi- this same problem, but I could find no solutions. ous. My collecting of ABC files and their subsequent Indeed the suggestion was that I abandon LATEX placement in my book is now almost completely au- altogether and use only lilypond and a particular tomatic: 11 stylesheet. • ABC file arrives in target folder (often via email) I also tried using the standard utility grep to • A Perl daemon: find a relation between my LAT X file, the Lilypond E 1. makes a backup file and the lilypond-book-generated eps files. I in- tended then to use conventional \includegraphics 2. cleans up ABC code commands to position the graphics manually. This 3. creates an index entry proved too cumbersome. 4. merges text to precede or follow this item My final code ended up as: 5. runs abc2ly \noindent 6. adds name of lily file to compilation list \begin{minipage}{\columnwidth} • a make invocation puts together book version \index{mytune:reels} \lilypondfile{mytune.ly}\\\\ I think of this process as resembling a trout: the \include{mytune.tex} Perl daemon watches for an ABC file arriving as a \end{minipage} result of an Internet search — just as a trout watches Here mytune.tex contains notes pertaining to for minnows! It is not perfect, but I have good error this particular piece. Wrapped within the minipage, reporting in place and mistakes are easily fixed by it was guaranteed to appear on the same page. It hand. might also contain footnotes. When sufficient new tunes have been added to This example also shows that if the Lilypond the repository, another Perl script employs lilypond- script is long, it can be stored in a file and referred to book, latex, dvips and ps2pdf to produce the final with \lilypondfile; lilypond-book then processes copy. its argument. 8 Subversion 5 Clever includes Small changes are made to this project daily and Ideally I would have liked code such as: sometimes the editing is done using machines on different sites. A small change can have a disastrous \newcommand{\lily}[1]{ effect on the end product. For this reason the whole \lilypondfile{#1}{#1}} 13 project is controlled using the Subversion version ..... \lily{lilys/my.tune.ly} control system.
11 lsr.dsi.unimi.it/LSR/Item?id=368 12 for example, trillian.mit.edu/~jc/cgi/abc/tunefind 13 subversion.tigris.org
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 379 MetaPost developments: MPlib project report
Taco Hoekwater Elvenkind BV Dordrecht, The Netherlands http://tug.org/metapost Abstract The initial stage of the MPlib project has resulted in a library that can be em- bedded in external programs such as LuaTEX, and that is also the core of the mpost program. This paper presents the current state of affairs, the conversion process of the MetaPost source code, and the application interface to the library.
1 MPlib project goals in most other cases it simply aborts. Finally, the The MPlib project is a logical consequence of the whole process of installing the program is compli- transfer of MetaPost development from its author cated: a fair bit of the TEX Live development tree John Hobby to the MetaPost development team. It is needed to compile the executable at all. originates from a desire to update MetaPost for use 2 Solutions in the 21st century. The first thing that needed to be done to make that happen was updating the pro- Many of the problems mentioned above are a side- gram source code and infrastructure to be closer to effect of the age of the source code: the source is METAFONT today’s programming standards. largely based on , and therefore written These days, programs are often written in the in Pascal WEB. And the bits that are not in Pascal form of shared libraries, with a small frontend appli- WEB are an amalgam of C code borrowed from other cation. When written in this way, a program can not projects, most notably pdfTEX. only be used as a standalone program, but can also Not wanting to lose the literate programming easily and efficiently be (re)used as a plugin inside documentation, we had only one practical way to other programs, or turned into a multi-user system proceed: using the CWEB system. CWEB has the service. same functionality that Pascal WEB has, except that With current MetaPost, such alternate uses are it uses C as the programming language instead of impossible because of the internals of the code. For Pascal, and it has some extensions so that it does example, MetaPost uses many internal global vari- not get in the way of the ‘normal’ C build system. ables. This is a problem because when two users Using CWEB, a single programming language would be accessing a ‘MetaPost’ library at the same now replaces all of the old Pascal and C code. The time, they would alter each other’s variables. For code has been restructured into a C library, the label another example, MetaPost has static memory allo- generator makempx has been integrated, and compi- cation: it requests all the computer memory it will lation now depends only on the ctangle program ever use right at startup. It never bothers to free and the normal system C compiler, so that a simple that memory, because it counts on the operating Autoconf script can be used for configuration of the system to clean up automatically after it exits. And build process. one file example: MetaPost not only opens files at 3 Code restructuring will, but it also writes to and, even more problem- atically, reads from the terminal directly. Whereas converting the C code of the font inclu- A large part of updating MetaPost is therefore sion and label processing subsystems to CWEB was fixing all these issues. But while doing this, there a fairly straightforward process, converting the Pas- are other weirdnesses to take care of at the same cal WEB core of MetaPost was a more elaborate time. undertaking. The present subsystem for typesetting labels In the first stage, the Pascal code within the (btex ... etex) is pretty complicated, requiring an WEB underwent an automatic rough conversion into array of external programs to be installed on top C code. Afterwards, the generated C code was man- of the normal mpost executable. And from a sys- ually cleaned up so that it compiled properly using tem viewpoint, the error handling of MetaPost is ctangle. This part took roughly one month, and not very good: it often needs user interaction, and the end result was an executable that was ‘just like’
380 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting MetaPost developments: MPlib project report
Pascal MetaPost, but using CWEB as source lan- • an interpreter that can repeatedly execute in- guage instead of Pascal WEB. dividual string chunks, with redirected I/O and After that was done, the real work started: all errors treated as if nonstopmode is in effect. • All of the global variables were collected into For the string-based interpreter, there are a few a C structure that represents an ‘instance’ of extra functions: MetaPost. • the runtime data can be fetched; this comprises • The Pascal string pool was stripped away so the logging information and the internal data that it is now used only for run-time generated structure representation of any generated fig- strings. Most internal functions now use normal ures, C strings. • the instance’s error state can be queried, • The PostScript backend was isolated from the core of the program so that other backends can • the userdata pointer can be retrieved, be added much more easily. • some statistics can be gathered, • All of the exported functions now use a C name- • PostScript can be generated from the image space. output, • Where it was feasible to do so, MPlib uses dy- • and some glyph information can be retrieved; namic memory allocation. There are a few ex- this is useful if you want to create a backend ceptions that will be mentioned later. yourself. • All input and output now uses function pointers that can be configured through the program- 4.1 Examples ming interface. Here is a minimalistic but complete example that • The MPlib library never calls exit() itself but uses the mpost emulation method in C code: instead returns to the calling program with a #include
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 381 Taco Hoekwater
l = mp:execute(chunk) The final MPlib 1.100 release will be released if l and l.fig then later in 2008, and the MPlib-based distribution will print (l.fig[1]:postscript()) replace the Pascal MetaPost distribution from that end point forward. mp:finish() After this release, work on the TODO list will end continue. Items already on the wishlist: 5 Using the command-line program • Start using dynamic memory allocation for the On the command line very little has changed. The remaining statically allocated items: the main executable mpost still exists. Now it is merely a memory, the number of hash entries, the num- thin wrapper that is much like the C code example ber of simultaneously active macro parameters, shown earlier, except with a few hundred more lines and the maximum allowed input nesting levels. because it has to set up the command-line properly. • An extension is being planned under the work- As mentioned already, the makempx functional- ing name ‘MegaPost’ that will extend the range ity has also been converted into a small library that and precision of the internal data types. is used by mpost to emulate the old label creation • In the future, we want to use MPlib to gener- system. The programs makempx, dvitomp, mpto, ate (OpenType) fonts. This requires support and dmp have been merged into this library and no from the core engine like overlap detection and longer exist as separate programs. For backward calculation of pen envelopes. compatibility, a user-supplied external label gener- • Error strategies are planned so that the be- ation program will be called if the mpxcommand haviour of the string-chunk based interface can environment variable is set, but normally mpost sets be configured properly. up the MPlib library to use the new embedded code. • There are desires to expand the API. For in- In the normal case, the only external program stance, it would be nice if applications were able that will be run is the actual typesetter (TEX or to use the equation solver directly. Troff). The command-line of mpost is extended to allow the specification of which typesetter to use. 7 Acknowledgements and contact
6 Planning and TODO The MPlib project could not have been done without funding by the worldwide TEX user groups, in par- Most development took place at the beginning of ticular: DANTE, TUG India, TUG, NTG, CSTUG, 2008, after which we entered a period of extensive and GUST. A big thank you goes to all of you for testing. This way we were relatively confident that giving us the opportunity to work on this project. the first version of the library was basically usable The general contact information for MetaPost from the start. and MPlib has not changed: The first beta release (1.091) was presented at • Web site and portal: the TUG 2008 conference. The distribution contains http://tug.org/metapost the MPlib library source, the code for the ‘mpost’ frontend, code for the Lua bindings, and the C and • User mailing list: Lua API documentation. http://lists.tug.org/metapost • Source code and bug tracker: http://foundry.supelec.fr/projects/metapost
382 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting The TEX–Lua mix
Hans Hagen Pragma ADE http://pragma-ade.com
Abstract
An introduction to the combination of TEX and the scripting language Lua.
1 Introduction ideas behind LuaTEX and the relationship between Lua and T X and how it presents itself to users. I The idea of embedding Lua into TEX originates in E some experiments with Lua embedded in the SciTE will not discuss the interface itself, which consists of editor. You can add functionality to this editor by quite a number of functions (organized in pseudo- loading Lua scripts. This is accomplished by a li- libraries) and the mechanisms used to access and brary that gives access to the internals of the editing replace internals (we call them callbacks). component. 2 TEX vs. Lua The first integration of Lua in pdfTEX was rel- T X is a macro language. Everything boils down atively simple: from TEX one could call out to Lua E to either allowing stepwise expansion or explicitly and from Lua one could print to TEX. My first ap- plication was converting math written in a calcula- preventing it. There are no real control features, like loops; tail recursion is a key concept. There tor syntax to TEX. Following experiments dealt with MetaPost. At this point integration meant as lit- are only a few accessible data structures, such as tle as: having some scripting language as an ad- numbers, dimensions, glue, token lists and boxes. dition to the macro language. But, even in this What happens inside TEX is controlled by variables, early stage further possibilities were explored, for mostly hidden from view, and optimized within the instance in manipulating the final output (i.e. the constraints of 30 years ago. PDF code). The first versions of what by then was The original idea behind TEX was that an au- thor would write a specific collection of macros for already called LuaTEX provided access to some in- ternals, like counter and dimension registers and the each publication, but increasing popularity among dimensions of boxes. non-programmers quickly resulted in distributed col- lections of macros, called macro packages. They Boosted by the Oriental TEX project, the team started exploring more fundamental possibilities: started small but grew and grew and by now have hooks in the input/output, tokenization, fonts and become pretty large. In these packages there are nodelists. This was followed by opening up hyphen- macros dealing with fonts, structure, page layout, ation, breaking lines into paragraphs and building graphic inclusion, etc. There is also code dealing ligatures. At that point we not only had access to with user interfaces, process control, conversion and much of that code looks out of place: the lack of some internals but also could influence the way TEX operates. control features and string manipulation is solved After that, an excursion was made to MPlib, by mimicking other languages, the unavailability of which fulfilled a long standing wish for a more nat- a float datatype is compensated by misusing dimen- sion registers, and you can find provisions to force ural integration of MetaPost into TEX. At that point or inhibit expansion all over the place. we ended up with mixtures of TEX, Lua and Meta- Post code. TEX is a powerful typographical programming As of mid-2008 we still need to open up more language but lacks some of the handy features of scripting languages. Handy in the sense that you will of TEX, like page building, math, alignments and need them when you want to go beyond the original the backend. Eventually LuaTEX will be nicely split up in components, rewritten in C, and we may even purpose of the system. Lua is a powerful script- end up with Lua gluing together the components ing language, but knows nothing of typesetting. To some extent it resembles the language that T X was that make up the TEX engine. At that point the E written in: Pascal. And, since Lua is meant for em- interoperation between TEX and Lua may be even richer than it is now. bedding and extending existing systems, it makes In the next sections I will discuss some of the sense to bring Lua into TEX. How do they compare? Let’s give some examples.
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 383 Hans Hagen
About the simplest example of using Lua in TEX hash slots in original TEX were more than enough for is the following: typesetting a book, but in huge collections of macros \directlua { tex.print(math.sqrt(10)) } they are not all that much. For that reason many macro packages use obscure names to hide their pri- This kind of application is probably what most vate registers from users and instead of allocating users will want and use, if they use Lua at all. How- new ones with meaningful names, existing ones are ever, we can go further than that. shared. It is therefore not completely fair to compare 3 Loops TEX code with Lua code: in Lua we have plenty of memory and the only limitations are those imposed In T X a loop can be implemented as in the plain E by modern computers. format (editorial line breaks, but with original com- In Lua, a loop looks like this: ment): for i=1,10 do \def\loop#1\repeat{\def\body{#1}\iterate} ... \def\iterate{\body\let\next\iterate end \else\let\next\relax\fi\next} \let\repeat=\fi % this makes \loop..\if..\repeat But while in the TEX example, the content di- % skippable rectly ends up in the input stream, in Lua we need This is then used as: to do that explicitly, so in fact we will have: \newcount \mycounter \mycounter=1 for i=1,10 do \loop tex.print("...") ... end \advance\mycounter 1 And, in order to execute this code snippet, in \ifnum\mycounter < 11 LuaTEX we will do: \repeat \directlua 0 { The definition shows a bit how TEX program- for i=1,10 do ming works. Of course such definitions can be tex.print("...") wrapped in macros, like: end \forloop{1}{10}{1}{some action} } and this is what often happens in more complex So, eventually we will end up with more code macro packages. In order to use such control loops than just Lua code, but still the loop itself looks without side effects, the macro writer needs to take quite readable and more complex loops are possible: measures to permit, for instance, nested usage and \directlua 0 { avoid clashes between local variables (counters or local t, n = { }, 0 macros) and user-defined ones. Above we used a while true do counter in the condition, but in practice expressions local r = math.random(1,10) if not t[r] then will be more complex and this is not that trivial to t[r], n = true, n+1 implement. tex.print(r) The original definition of the iterator can be if n == 10 then break end written a bit more efficiently: end \def\iterate{\body \expandafter\iterate \fi} end And indeed, in macro packages you will find } many such expansion control primitives being used, This will typeset the numbers 1 to 10 in ran- which does not make reading macros easier. domized order. Implementing a random number Now, get me right, this does not make TEX less generator in pure TEX takes a fair amount of code powerful, it’s just that the language is focused on and keeping track of already defined numbers in typesetting and not on general purpose program- macros can be done with macros, but neither of these ming, and in principle users can do without that: is very efficient. documents can be preprocessed using another lan- 4 Basic typesetting guage, and document specific styles can be used. We have to keep in mind that TEX was writ- I already stressed that TEX is a typographical pro- ten in a time when resources in terms of memory gramming language and as such some things in TEX and CPU cycles were far less abundant than they are are easier than in Lua, given some access to inter- now. The 255 registers per class and (about) 3000 nals:
384 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The TEX–Lua mix
\setbox0=\hbox{x}\the\wd0 end In Lua we can do this as follows: end } \directlua 0 { local n = node.new(’glyph’) And still it’s not okay, since TEX inserts kerns n.font = font.current() between characters (depending on the font) and glue n.char = string.byte(’x’) between words, and doing that all in Lua takes more tex.box[0] = node.hpack(n) code. So, it will be clear that although we will use tex.print(tex.wd[0]/65536 .. "pt") Lua to implement advanced features, TEX itself still } has quite a lot of work to do. One pitfall here is that T X rounds the number E 5 Typesetting stylistic variations differently than Lua. Both implementations can be wrapped in a macro resp. function: In the following examples we show code, but it is \def\measured#1{\setbox0=\hbox{#1}\the\wd0\relax} not of production quality. It just demonstrates a new way of dealing with text in T X. Now we get: E Occasionally a design demands that at some \measured{x} place the first character of each word should be up- The same macro using Lua looks as follows: percase, or that the first word of a paragraph should \directlua 0 { be in small caps, or that each first line of a paragraph function measure(chr) has to be in dark blue. When using traditional TEX local n = node.new(’glyph’) the user then has to fall back on parsing the data n.font = font.current() stream, and preferably you should then start such n.char = string.byte(chr) a sentence with a command that can pick up the tex.box[0] = node.hpack(n) text. For accentless languages like English this is tex.print(tex.wd[0]/65536 .. "pt") quite doable but as soon as commands (for instance end dealing with accents) enter the stream this process } \def\measured#1{\directlua0{measure("#1")}} becomes quite hairy. The next code shows how ConT Xt MkII defines In both cases, special tricks are needed if you E the \Word and \Words macros that capitalize the want to pass for instance a # character to the T X E first characters of a word or words. The spaces are implementation, or a " to Lua; namely, using \# in really important here because they signal the end of the first case, and Lua’s “long strings” marked with a word. double square brackets in the second. This example is somewhat misleading. Imagine \def\doWord#1% {\bgroup\the\everyuppercase\uppercase{#1}% that we want to pass more than one character. The \egroup} TEX variant is already suited for that, but the Lua function will now look like: \def\Word#1% \directlua 0 { {\doWord#1} function measure(str) if str == "" then \def\doprocesswords#1 #2\od tex.print("0pt") {\doifsomething{#1}{\processword{#1} % space! else \doprocesswords#2 \od}} local head, tail = nil, nil for chr in str:gmatch(".") do \def\processwords#1% local n = node.new(’glyph’) {\doprocesswords#1 \od\unskip} n.font = font.current() n.char = string.byte(chr) \let\processword\relax if not head then head = n \def\Words else {\let\processword\Word \processwords} tail.next = n The code here is not that complex. We split off end each word and feed it to a macro that picks up the tail = n first token (hopefully a character) which is then fed end tex.box[0] = node.hpack(head) into the \uppercase primitive. This assumes that tex.print(tex.wd[0]/65536 .. "pt") for each character a corresponding uppercase vari- ant is defined using the \uccode primitive. Excep-
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 385 Hans Hagen tions can be dealt with by assigning relevant code to in MkIV this function looks (slightly simplified) as the token register \everyuppercase. However, such follows: macros are far from robust. What happens if the function cases.process(namespace,attribute,head) text is generated and not input as is? What hap- local done, actions = false, cases.actions pens with commands in the stream that do some- for start in node.traverse_id(glyph,head) do thing with the following tokens? local attr = has_attribute(start,attribute) A Lua-based solution could look as follows: if attr and attr > 0 then unset_attribute(start,attribute) \def\Words#1{\directlua 0 local action = actions[attr] for s in unicode.utf8.gmatch("#1", "([^ ])") do if action then tex.sprint(string.upper( local _, ok = action(start) s:sub(1,1)) .. s:sub(2)) done = done and ok end end } end But there is no real advantage here, apart from end the fact that less code is needed. We still operate on return head, done the input and therefore we need to look to a different end kind of solution: operating on the node list. Here we check attributes (these are set on the function CapitalizeWords(head) TEX side) and we have all kind of actions that can local done = false be applied, depending on the value of the attribute. local glyph = node.id("glyph") Here the function that does the actual uppercasing for start in node.traverse_id(glyph,head) do is defined somewhere else. The cases table provides local prev, next = start.prev, start.next us a namespace; such namespaces need to be coor- if prev and prev.id == kern dinated by macro package writers. and prev.subtype == 0 then prev = prev.prev This approach means that the macro code looks end completely different; in pseudo code: if next and next.id == kern \def\Words#1{{
386 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The TEX–Lua mix
start.char = uc points decisions have to be made. Nothing is perfect, return true neither is LuaTEX, but most users won’t notice it as end long as it extends TEX’s life and makes usage more end convenient. return false end 6 Groups Such code is really macro package dependent: Users of TEX and MetaPost will have noticed that LuaTEX provides only the means, not the solutions. both languages have their own grouping (scope) In ConTEXt we have collected information about model. In TEX grouping is focused on content: by characters in a data table in the characters name- grouping the macro writer (or author) can limit the space. There we have stored the uppercase codes scope to a specific part of the text or have certain (uccode). The fonts table, again ConTEXt specific, macros live within their own world. keeps track of all defined fonts and before we change .1. \bgroup .2. \egroup .1. the case, we make sure that this character is present in the font. Here id is the number by which Lua- Everything done at 2 is local unless explicitly told otherwise. This means that users can write TEX keeps track of the used fonts. Each glyph node carries such a reference. (and share) macros with a small chance of clashes. In this example, eventually we end up with more In MetaPost grouping is available too, but variables explicitly need to be saved. code than in TEX, but the solution is much more robust. Just imagine what would happen when in .1. begingroup; save p; path p; .2. endgroup .1. the TEX solution we would have: After using MetaPost for a while this feels quite \Words{\framed[offset=3pt]{hello world}} natural because an enforced local scope demands multiple return values which is not part of the macro It simply does not work. On the other hand, the language. Actually, this is another fundamental dif- Lua code never sees T X commands, it only sees the E ference between the languages: MetaPost has (a two words represented by glyph nodes and separated kind of) functions, which T X lacks. In MetaPost by glue. E you can write Of course, there is a danger when we start open- draw origin for i=1 upto 10: ..(i,sin(i)) endfor; ing TEX’s core features. Currently macro packages know what to expect, they know what TEX can and but also: cannot do, and macro writers have exploited every draw some(0) for i=1 upto 10: ..some(i) endfor; corner of TEX, even the darkest ones. while the dirty with tricks in The TEXbook had an educational purpose, those of users sometimes have obscene traits. If we vardef some (expr i) = if i > 4 : i = i - 4 fi ; just stick to the trickery introduced for parsing in- (i,sin(i)) put, converting this into that, doing some calcula- enddef ; tions, and the like, it will be clear that Lua is more The condition and assignment in no way inter- than welcome. It may hurt to throw away thou- fere with the loop where this function is called, as sands of lines of impressive code and replace it by a long as some value is returned (a pair in this case). few lines of Lua but that’s the price the user pays In TEX things work differently. Take this: for abusing TEX. Eventually ConTEXt MkIV will be \count0=1 a decent mix of Lua and TEX code, and hopefully the solutions programmed in those languages are as \message{\advance\count0 by 1 \the\count0} clean as possible. \the\count0 Of course we can discuss until eternity whether The terminal wil show: Lua is the best choice. Taco, Hartmut and I are \advance \count 0 by 1 1 pretty confident that it is, and in the couple of years At the end the counter still has the value 1. that we have been working on LuaTEX nothing has There are quite a few situations like this, for in- proved us wrong yet. We can fantasize about con- stance when data such as a table of contents has cepts, only to find out that they are impossible to to be written to a file. You cannot write macros implement or hard to agree on; we just go ahead us- where such calculations are done, hidden away, and ing trial and error. We can talk over and over how only the result is seen. opening up should be done, which is what the team The nice thing about the way Lua is presented does in a nicely closed and efficient loop, but at some to the user is that it permits the following:
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 387 Hans Hagen
\count0=1 stance a reference to this section, things quickly start \message{\directlua0{% looking more complex. Also, users need to know tex.count[0] = tex.count[0] + 1}% what definitions to use in situations like this: \the\count0}
| first | ...last |
| left | ...right |
some text
dards like MathML, there are some more mechanismssome more
but these are hidden from the user. And although388 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The TEX–Lua mix stream-based approach, this time the whole tree is \xmlfirst{#1}{/one/(alpha|beta)/two/text()} loaded in memory. Although this uses more mem- which returns the content of the first element that ory than a streaming solution, in practice the differ- satisfies one of the paths (nested tree): ence is not significant because often in MkII we also /one/alpha/two needed to store whole chunks. /one/beta/two Loading XML files in memory is very fast and \xml- once it is done we can have access to the elements There is a whole bunch of commands like text that filter content and pipe it into T X. These in a way similar to XPath. We can selectively pipe E are calling Lua functions. This is no manual, so we data to TEX and manipulate content using TEX or Lua. In most cases this is faster than the stream- will not discuss them here. However, it is important based method. An interesting fact is that we can do to realize that we have to associate setups (consider them free formatted macros) with at least one ele- this without linking to existing XML libraries, and as a result we are pretty independent. ment in order to get started. Also, XML inclusions So how does this look from the perspective of have to be dealt with before assigning the setups. the user? Say that we have the simple article defin- These are simple one-line commands. You can also ition stored in demo.xml. assign defaults to elements, which saves some work. Because we can use Lua to access the tree and manipulate content, we can now implement parts of some text
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 389 Hans Hagen
Just to give you an idea, in the module that der article, print the content to implements the parser you will find definitions that TEX and return control to TEX. match the function calls in the above expressions. tex: Tell Lua to filter the paragraphs i.e. xml.functions.find = string.find skip title and author; since the b xml.functions.upper = string.upper element has no associated setup (or xml.functions.number = tonumber whatever) it is just serialized. So much for the different approaches. It’s up to lua: Filter the requested elements and the user what method to use: stream-based MkII, return control to TEX. tree-based MkIV, or a mixture. tex: Ask Lua to filter author. lua: Pipe author’s content to TEX. 8 TEX–Lua in conversation tex: We’re done. The main reason for taking XML as an example of lua: We’re done. mixing TEX and Lua is in that it can be a bit mind- This is a very simple case. In my daily work I am boggling if you start thinking of what happens be- dealing with rather extensive and complex educa- hind the scenes. Say that we have tional documents where in one source there is text, math, graphics, all kind of fancy stuff, questions and some bold text
390 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The TEX–Lua mix
example is support for OpenType fonts: Lua- spacing and display math splitting. TEX itself does not support this format di- The boundaries between these categories are not set rectly, but provides ways to feed TEX with the in stone. For instance, support for image inclu- relevant information. Support for OpenType sion and MPlib in ConTEXt MkIV sits between cat- features demands manipulating node lists. egories 3 and 4. Categories 3 and 4, and probably Knowledge of internals is a requirement. Ad- also 2, are normally the domain of macro package vanced spacing and language specific features writers and more advanced users who contribute to are made possible by node list manipulations macro packages. Because a macro package has to and attributes. The alternative \Words macro provide some stability it is not a good idea to let is an example of this. users mess around with all those internals, due to 4. Users can replace existing TEX functional- potential interference. On the other hand, normally ity. In MkIV there are numerous examples of users operate on top of a kernel using some kind of this, for instance all file I/O is written in Lua, API, and history has proved that macro packages are including reading from zip files and remote stable enough for this. locations. Loading and defining fonts is also Sometime around 2010 the team expects Lua- under Lua control. At some point MkIV will TEX to be feature complete and stable. By that time provide dedicated splitters for multicolumn I can probably provide a more detailed categoriza- typesetting and probably also better display tion.
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 391 docx2tex: Word 2007 to TEX
Kriszti´an P´ocza E¨otv¨os Lor´and University, Faculty of Informatics, Department of Programming Languages and Compilers, P´azm´any P´eter s´et´any 1/C. H-1117, Budapest, Hungary kpocza (at) kpocza dot net http://kpocza.net/
Mih´aly Bicz´o E¨otv¨os Lor´and University, Faculty of Informatics, Department of Programming Languages and Compilers, P´azm´any P´eter s´et´any 1/C. H-1117, Budapest, Hungary mihaly.biczo (at) t-online dot hu http://avalon.inf.elte.hu/personal/hdbiczo/
Zolt´an Porkol´ab E¨otv¨os Lor´and University, Faculty of Informatics, Department of Programming Languages and Compilers, P´azm´any P´eter s´et´any 1/C. H-1117, Budapest, Hungary gsd (at) elte dot hu http://gsd.web.elte.hu/
Abstract Docx2tex is a small command line tool to support users of Word 2007 to publish documents when typography is important or only papers produced by TEX are accepted. Behind the scenes, docx2tex uses common technologies to interpret Word 2007 OOXML format without utilizing the API of Word 2007. Docx2tex is published as a free and open source utility that is accessible and extensible by everyone. The source code and the binary executable of the application can be downloaded from http://codeplex.com/docx2tex/. This paper was originally written in Word 2007 and later converted to TEX using docx2tex.
1 Introduction rox and joined Microsoft where he created Microsoft There are two general methods to produce human Word [2, 3], the first and still the most popular word readable and printable digital documents: processor. Word is capable of producing simple and also complex documents, including those with many 1. Using a WYSIWYG word processor mathematical symbols. Another important feature 2. Using a typesetting system of Word is Track Changes that supports team work. Each of them has its own advantages and disadvan- Using Track Changes any of the team members can tages; therefore each of them has many use cases modify the document while these modifications are where one is better than the other. tracked and can be accepted or refused by the team WYSIWYG [1] is an acronym for the term What leader. You See Is What You Get that originates from the Typesetting is the process of putting characters late ’70s. WYSIWYG editors are usually favored by of different types in their correct place on the paper everyday computer users whose aim is to produce or screen. Before electronic typesetting systems be- good-looking documents in a fast and straightfor- came widely used, printed materials had been pro- ward way exploiting the rich formatting capabilities duced by compositors who worked by hand or us- of such systems. WYSIWYG editors and word pro- ing special machines. The aim of typesetting sys- cessors ensure that the printed version of the docu- tems is to create high-quality output of materials ment will be the same as the document that is vis- that may contain complex mathematical formulae ible on the screen during editing. The first WYSI- and complex figures. Similarly, electronic typeset- WYG word processor called Bravo was created at ting systems follow this goal and produce high qual- Xerox by Charles Simonyi, who is the inventor of ity, device independent output. The most popular intentional programming. In 1981 Simonyi left Xe-
392 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting docx2tex: Word 2007 to TEX
typesetting system is TEX [4] created by Donald E. also saving them in TEX format. It is a free and Knuth. TEX is mainly used by researchers and in- open source application; however, it interprets the dividuals whose aim is to achieve the best quality binary data of Word documents. output without sacrificing platform or device inde- Rtf2latex2e [8] is the most recent solution that pendence. The users of TEX use a special and ex- translates .rtf files to TEX. It is a free and open tensible DSL (Domain Specific Language) that was source application. designed to solve complex typesetting problems, pro- duce books containing hundreds of pages, and more. 3 Technology There is a big gap between these systems be- In this section we will enumerate and then briefly cause each tries to satisfy different demands, namely: review the technologies that are used in docx2tex produce common documents quickly even in a group and show how they cooperate. setting vs. achieve the best quality and typographi- The technologies used are the following: cally correct printout. To bridge this difference there 1. Office Open XML (ECMA 376 Standard [9], re- are both commercial and non-commercial tools that cently approved as an ISO Standard), the de- support conversion from Word or other WYSIWYG fault format of Word 2007. In brief: OOXML. formats to TEX (and back). The first direction, con- 2. Microsoft .NET 3.0 (CLI is ECMA 335 Standard verting from WYSIWYG (Word) formats to TEX, has [10] and ISO/IEC 23271:2006 Standard [11]). perhaps more frequent usage because many users edit the original text in Word for the sake of sim- 3. ImageMagick [12] to convert images. plicity and efficiency, and later convert it to TEX by OOXML files are simply XML and media files com- hand in order to ensure quality. pressed using ZIP. Docx2tex uses Microsoft .NET The problems with present conversion applica- 3.0 to open and unzip OOXML Word 2007 (.docx tions include the following: extension) documents. Microsoft .NET 3.0 has some 1. many of them are available only as proprietary special classes in the System.IO.Packaging name- tools; space that facilitate opening and unzipping OOXML 2. thus they have limitations (running times or files and also provide an abstraction to represent the page limit) when not purchased; included XML and media files as packages. The op- erations performed by this component are described 3. they support only the old, binary Word or Rich by the object line called OOXML depackaging in Fig- Text document format (.doc, .rtf); and/or ure 1. 4. they use the Word’s COM API to process docu- The most important component of docx2tex is ments, which makes them complex. the Core XML Engine that implements the basic In this paper we present an open source and free so- conversion from XML files to TEX. The Core En- lution that is capable of handling the new and open gine is responsible for reading and processing the Word 2007 .docx format natively by using standard XML data of the OOXML documents that is served technologies without using the COM API of Word by the OOXML depackaging component. The Core and without even installing Word. In this article Engine identifies parts of the OOXML document and the current features are presented along with further processes the contents of these parts (paragraphs, development directions. runs, tables, image references, numberings, . . . ). It is not responsible for processing parts of the OOXML 2 Existing solutions document that are available through a relation. For It has always been challenging to convert propri- those, docx2tex has a set of internal helper func- etary, binary or any other document formats to TEX. tions that are responsible for driving the process- Because Word is the most common editor, many ing of related entities such as image conversion, spe- tools try to convert from Word documents. One of cial styling and resolving the properties of numbered these tools is the proprietary Word2TeX [5], which lists. makes Word capable of saving documents in TEX When an image reference is found in the XML, format. This tool is embedded into Word, has an ImageMagick is called to produce EPS files from the evaluation period and can be purchased in different original image files. The resulting EPS files can be license packages. A similarly featured tool named embedded easily into TEX documents. Word-to-Latex [6] does not provide sources, though We support the exact output produced by Word it is available at no cost. 2007; other output variations saved by third party Another possibility is to use OpenOffice.org [7], applications that may differ from the ECMA 376 which is capable of reading Word documents and Standard are not supported.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 393 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab
Figure 1: UML sequence diagram
In the next sections the structure of OOXML 4.1 Supported features will be briefly discussed; first, let us review the idea Docx2tex supports the following features of Word of runs.A run is a piece of text which also has some 2007 and TEX: style specification. Runs are placed and removed 1. Normal text dynamically while the Word document is edited. A sentence or even a word can be divided into more 2. Italic, bold, underlined, stroked, small capitals, than one run with the same style. The component ... called TeXizer is responsible to join runs having the 3. Left, right, center aligned text same style to a simple run in the outgoing TEX code 4. Headings and sections, three levels and break the source line length at some predefined 5. Verbatim text value (default is 72). 6. Style mapping The previous description is illustrated by the 7. Simple tables UML sequence diagram in Figure 1. 8. Line and page breaks 4 Features of docx2tex 9. Numbered and bulleted lists In this section we list the supported and the unsup- 10. Multilevel lists and continuous numbered lists ported features of docx2tex. 11. Figure, table and listing captions 12. Cross references to captions and headings
394 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting docx2tex: Word 2007 to TEX
13. Image conversion from various formats (includ- PS C:\Phd\conf\2008_4_tex\example.docx> ing .png, .jpeg, .emf, etc.) to .eps ls -Recu |% {$_.FullName.SubString(30)} 14. Substitution of special characters (e.g. \, #, {, example.docx\customXml }, [, ], %, &, ˜, . . . ) example.docx\docProps 15. Text boxes example.docx\word 16. Basic math formulae, Word Equations support example.docx\_rels docx2tex supports normal and special text styles example.docx\[Content_Types].xml and also text alignments. We support heading styles example.docx\customXml\_rels Heading1, Heading2, and Heading3 that convert to example.docx\customXml\item1.xml \section, \subsection, and \subsubsection re- example.docx\customXml\itemProps1.xml spectively. Word does not support verbatim text example.docx\docProps\app.xml while TEX does. To work around this deficiency, example.docx\docProps\core.xml text marked with Verbatim style is converted to ver- example.docx\word\media batim text surrounded by \begin{verbatim} and example.docx\word\theme \end{verbatim}. There are many cases where we example.docx\word\_rels are required to use different styles for headings or example.docx\word\document.xml even verbatim. example.docx\word\fontTable.xml Only simple, left-aligned tables are supported. example.docx\word\numbering.xml Both numbered and bulleted lists are supported, in- example.docx\word\settings.xml cluding mixing and nesting. Continuous lists are example.docx\word\styles.xml also supported using the \setcounter, \enumi, and example.docx\word\webSettings.xml \theenumi commands. Figure, table and listings example.docx\word\media\image1.jpeg captions are recognized and we support referencing example.docx\word\theme\theme1.xml them; likewise with heading references. Image ref- example.docx\word\_rels\document.xml.rels erences are resolved and the images (mainly .png example.docx\_rels\.rels and .jpeg) embedded in the OOXML documents are The most important part is the document.xml converted to .eps. The width and height properties file that contains the document itself and references are queried and the same properties are used in the to external items. The numbering.xml file specifies resulting TEX documents. Some special TEX char- the style of the numbered or bulleted lists contained acters are also resolved and escaped in the resulting in document.xml. The styles.xml file specifies in- TEX document. Text found in Text Boxes of Word formation about the styles used in the document. documents are also processed and inserted in place Under the media subdirectory the embedded images of the resulting TEX document. can be found (image1.jpeg in our example). 4.2 Unsupported features 5.1.1 Structure of the document Docx2tex has only basic Word Equations (mathe- The text in the document.xml file is grouped into matical formulae) support at the time of publishing paragraphs. Every segment of the document is a this paper. We plan to add more support for Word paragraph (normal text, heading texts, images, etc.) 2007 Equations and Drawings that can be converted except for some special elements like tables. Para- to TEX mathematical formulas and xfigs respectively. graphs are further divided into runs.A run is a Both of them are described in XML format therefore piece of text that also has some style specification. our standard solution can be extended without intro- ducing other technologies. 5.2 Text conversion 5 A complex example The most fundamental feature of tools like docx2tex is the ability to interpret text runs with many basic In this section we will show a complex example bro- styling properties and convert them to TEX format. ken into significant parts that introduces the most Consider the following example sentence: This is a important features of docx2tex. sentence that with contains text different formatting. 5.1 The structure of the OOXML ZIP This sentence looks like the following in OOXML: package
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 395 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab
This is a
396 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting docx2tex: Word 2007 to TEX
the mapping file example.paraStyleName). The
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 397 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab
file that is processed by docx2tex. The above ele- ment is part of a complex multilevel numbered list: \newcounter{numberedCntA}
398 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting docx2tex: Word 2007 to TEX
Figure 2: Workflow
The second author accepts or rejects the changes ing our solution more robust and usable. of the first author and also makes his own changes We presented the main features of docx2tex, re- using the Track Changes function of Word. When lated primarily to text processing and formatting, the authors agree that the quality of the article is structuring the document, handling images, tables acceptable, they convert it to TEX using docx2tex and references. There are some important features and apply special formatting required for the partic- that we consider worth implementing in the future: ular conference or journal. Now the article can be 1. More Equations support submitted. 2. Embedded vector graphical drawings This workflow is illustrated in Figure 2. 3. Configuration settings As can be seen we exploited the strengths of 4. Optional font and coloring support both the WYSIWYG Word 2007 system to support 5. Documentation effective team work and the typesetting TEX system 6. Automated installer to produce the best quality printout. The conversion Multicolumn document handling and templat- between the file formats they use was performed us- ing may also worth considering. ing docx2tex. Note that currently docx2tex is able We published the application as free and open to do rough conversion and cannot apply special source software under the terms of the BSD [13] li- commands and styles. cense, so anybody can use it royalty free and can Readers not familiar with the Track Changes add new features to the current feature set. function should consider Figure 3. The source code and the binary of the appli- 7 Conclusion and further work cation can be downloaded from http://codeplex. com/docx2tex/. If the reader would like to partici- In this article we introduced a tool called docx2tex pate in the development of docx2tex, please contact that is dedicated to producing TEX documents from the authors. Word 2007 OOXML documents. The main advan- tage of this solution over classical methods is that References we process the bare XML content of OOXML pack- [1] Wikipedia article on What You See Is What ages instead of processing binary files or exploiting You Get. http://en.wikipedia.org/wiki/ the capabilities of the COM API of Word, thus mak- WYSIWYG
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 399 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab
Figure 3: Track Changes function of Word
[2] Wikipedia article on Microsoft Word. http: [9] ECMA 376 OOXML Standard. http://www. //en.wikipedia.org/wiki/Microsoft_ ecma-international.org/publications/ Office_Word standards/Ecma-376.htm [3] Product page of Microsoft Word. http: [10] ECMA 335 .NET CLI Standard. http://www. //office.microsoft.com/en-us/word/ ecma-international.org/publications/ FX100487981033.aspx standards/Ecma-335.htm [4] Wikipedia article on TEX. http://en. [11] Link to ISO/IEC 23271:2006 Standard. wikipedia.org/wiki/TeX http://standards.iso.org/ittf/ [5] Product page of Word2Tex. http://www. PubliclyAvailableStandards/index.html chikrii.com/ [12] Project page of ImageMagick. http: [6] Product page of Word-to-LaTeX. http:// //www.imagemagick.org/ kebrt.webz.cz/programs/word-to-latex/ [13] Article on BSD license. http://en. [7] Product page of OpenOffice.org. http: wikipedia.org/wiki/BSD_license //www.openoffice.org/ [8] Project page of rtf2latex2e. http:// sourceforge.net/projects/rtf2latex2e/
400 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
Jean-Michel Hufflen LIFC (EA CNRS 4269) University of Franche-Comté 16, route de Gray 25030 Besançon Cedex, France hufflen (at) lifc dot univ-fcomte dot fr http://lifc.univ-fcomte.fr/~hufflen Abstract BibTEX is the most commonly used bibliography processor in conjunction with LATEX. To put bibliography styles into action, it uses a stack-based language written with postfixed notation. Recently, other approaches have been proposed: some use a script programming language for designing bibliography styles, e.g., Perl in Bibulus; some are based on converters to XML texts and use XSLT for bibliography styles; a recent proposal — the biblatex package — consists of using LATEX commands to control the layout of generated references, and more. We propose a comparative study of these approaches and show which programming styles are encouraged, from a point of view related to methodology. Finally, we explain how this study has influenced the design of MlBibTEX, our multilingual reimplementation of BibTEX. Keywords Bibliographies, bibliography styles, BibTEX, software quality, bst, BibTEX++, cl-bibtex, MlBibTEX, packages natbib, jurabib, and latexbib,Tib, XML, XSLT, nbst, Perl, DSSSL.
1 Introduction impossible — nor an absolute comparison of several Three decades ago,1 some programming languages programming languages — which would not be of in- were designed to be universal, that is, to serve all terest — but a comparative study of languages used purposes. All of these languages—e.g., PL/1 [25], to develop bibliography styles. BibTEX [42] is the Ada [2] — have failed to be accepted as filling this bibliography processor most commonly used in con- A role. Nowadays, only the C programming language junction with the LTEX typesetting engine [40], so A [30] is still used for a very wide range of applications. most of the bibliography styles used for LTEX texts Present-day programming languages are very diverse are written using bst, the stack-based language of and put different paradigms into action: procedural BibTEX[40, § 13.6]. But other proposals exist, based programming, object-oriented programming, func- on other programming paradigms, and this article tional programming, process-oriented programming, aims to study the advantages and drawbacks of these logic programming, . . . In addition, most of these approaches. We will not discuss the typographical present languages are specialised, that is, designed conventions ruling the typesetting of bibliographies — for particular purposes, even if they are not formally readers interested in this topic can consult manuals limited to only one class of applications: two good ex- like [5, § 10], [6, §§ 15.54–15.76], [16, pp. 53–54]— amples are Java [28] and C# [39], originally designed but are interested only in the development of bibli- for client-server applications. But, if you are building ography styles — from scratch or derived from other a standalone application using the object-oriented existing styles — and the expressive power of lan- paradigm and are especially interested in the effi- guages used to do that. ciency of the resulting program, it is well-known that In Section 2, we recall the main quality factors of a better choice is C++ [47], even if code generated software, and show which factors are interesting from by Java and C# compilers have greatly improved a point of view related to bibliography styles. Then since their first versions. BibTEX is studied in Section 3, including some mod- The purpose of this article is neither a compari- ern use of this program. Other approaches are based son of all programming languages—which would be on XML,2 as shown in Section 4. This experience
1 2 That is, at the time of TEX’s first version. . . Let us recall EXtensible Markup Language. Readers interested in an that we are celebrating TEX’s 30th birthday. introductory book to this formalism can refer to [44].
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 401 Jean-Michel Hufflen
of dealing with several ways to develop bibliogra- 2.2 Tasks of a bibliography processor phy styles has influenced the design of MlBibTEX— Given citation keys —stored in an .aux file when a for ‘MultiLingual BibT X’, our multilingual reim- E source text is processed by LATEX [40, Fig. 12.1]—a plementation of BibTEX [18]: we explain that in bibliography processor searches bibliography data- Section 5. Reading this article requires only a ba- base files for resources associated with these keys, sic knowledge of BibTEX and a little experience performs a sort operation on bibliographical items,5 about bibliography styles; we think that our exam- and arranges them according to a bibliography style, 3 ples should be understandable, even if readers do the result being a source file for a ‘References’ section, not know thoroughly the languages used throughout suitable for a word processor. So does BibTEX. this article. Roughly speaking, a bibliography has to do two 2 Criteria kinds of tasks: Of course, this section does not aim to replace a • some are related to ‘pure’ programming, e.g., textbook about software quality, we just make precise sorting bibliographical items, while the terminology we use. Then we explain how these • others are related to put markup, in order for notions are applied to bibliography styles. the word processor to be able to typeset the bibliography of a printed work. 2.1 General point of view The extendability of such a tool concerns these The main reference for the terminology used in soft- two kinds of tasks. On the one hand, we should be ware quality is the beginning of [38], as recalled in able to add a new relation order for sorting biblio- most works within this topic. [38, Ch. 1] clearly graphical items, since these lexicographical orders distinguishes external quality factors, that may be are language-dependent [24]. On the other hand, detected by users of a product, and internal factors, we should be able to build a new bibliography style, that are only perceptible to computer professionals. according to a publisher’s specification. This style Here are the main external quality factors: may be developed from scratch if we do not find a correctness the ability of software products to ex- suitable existing style. Or we can get it by introduc- actly perform their tasks, as defined by the re- ing some changes to another style, i.e., reusing some quirements and specification; parts of the previous style. In addition, finding the robustness the ability of software systems to work parts that have to be changed is related to the notion even in abnormal conditions; of modular understandability. Of course, building extendability the ease with which products may a new bibliography style is not an end-user’s task, be adapted to changes of specifications; but it should be possible by people other than the reusability the ability of products may be com- bibliography processor’s developers. bined with others; Another notion is related to extending a bibli- others being efficiency, portability, verifiability, in- ography processor: improving it so that it is usable tegrity, ease of use, etc. Internal quality factors in- with more word processors. If we consider the for- clude modularity, legibility, maintainability, etc. The mats built on TEX[34], LATEX is still widely used, but factors related to modularity are studied more pre- more and more people are interested in alternatives, cisely in [38, Ch. 2], they include: such as ConTEXt [13]. Likewise, some new typeset modular decomposability the ability for a prob- engines, such as X TE EX [32] or LuaTEX [14], should lem to be decomposed into subproblems; be taken into account. In addition, it should be pos- modular composability the ability for modules sible to put the contents of a bibliography database to be combined freely with each other; file on a Web page, that is, to express the informa- tion about these items using the HTML language.6 modular understandability each module can be A last example is given by RTF:7 at first glance, separately understood by a human reader; deriving bibliographies using the internal markup modular continuity a small change in a problem language of Microsoft Word may seem strange, but specification results in a change of just a module 4 or few modules. 5 . . . unless the bibliography style is unsorted, that is, the 3 Complete texts may be downloaded from MlBibTEX’s order of items is the order of first citations. In practice, most home page: http://lifc.univ-fcomte.fr/~hufflen/texts/ bibliography styles are ‘sorted’. mlbibtex/hc-styles/. 6 HyperText Markup Language. [41] is a good introduc- 4 This terminology is related to mathematical analysis: a tion to this language. function is continuous if a small change in the argument will 7 Rich Text Format. A good introductory book to this yield a small change in the result. markup language is [4].
402 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
@STRING{srd = {Stephen Reeder Donaldson}} FUNCTION {format.title} { title empty$ @BOOK{donaldson1993, { "" } AUTHOR = srd, { title "t" change.case$ } TITLE = {The Gap into Power: A Dark and if$ Hungry God Arises}, } PUBLISHER = {HarperCollins}, SERIES = {The Gap}, FUNCTION {new.sentence.checkb} NUMBER = 3, { empty$ YEAR = 1993} swap$ empty$ and @BOOK{donaldson1993a, ’skip$ EDITOR = srd, ’new.sentence TITLE = {Strange Dreams}, if$ PUBLISHER = {Bantam-Spectra}, } YEAR = 1993} @BOOK{murphy-mullaney2007, Figure 2: Two functions from BibTEX’s plain style. AUTHOR = {Warren Murphy and James Mullaney}, TITLE = {Choke Hold}, Extending BibTEX, however, may be very te- PUBLISHER = {Tor}, dious, especially for functionalities related to pro- ADDRESS = {New-York}, gramming. For example, the only way to control the SERIES = {The New Destroyer}, SORT command consists of using the entry variable NUMBER = 2, NOTE = {The original series has been sort.key$ [40, Table 13.7]. Some workarounds may created by Richard Sapir and allow the definition of sort procedures according to Warren Murphy}, lexicographic orders for natural languages other than YEAR = 2007, English, but with great difficulty. Developing bibli- MONTH = nov} ography styles for word processors other than LATEX has been done, but only for formatters built on TEX, Figure 1: Bibliographical entries in the .bib format. e.g., ConTEXt [17]. In other cases, this task may be difficult since some features related to TEX are hard- such a strategy may cause Word end-users to discover wired in some built-in functions of BibTEX, e.g., the use of ‘~’ for an unbreakable space character is in progressively the tools related to TEX. the specification of the format.name$ function [23]. 3 BibTEX As an example, there is a converter from .bib format 3.1 Basic use to HTML: BibTEX2HTML [9]. It uses BibTEX, but most of this translator is not written using BibTEX’s Bib A How to use TEX in conjunction with LTEX is language, but in Objective CAML,9 a strongly typed explained in [40, § 12.1.3], and the .bib format, used functional programming language including object- within bibliography database files, is detailed in [40, oriented features [37]. Using such a tool—as well § 13.2]; an example is given in Figure 1. As men- as the bibliography styles developed for ConTEXt’s tioned above, bibliography styles are written in a texts [17] — is possible only if end-users do not put stack-based language using postfixed notation. As LATEX commands inside the values associated with an example, Figure 2 gives two functions used within BibTEX’s fields. the plain style of BibTEX. We think that the continuity of the bibliogra- BibTEX is indisputably correct8 and robust: as phy styles written using the bst language is average. far as we have used it, the bibliographies it derives Introducing some changes concerning the layout of have satisfactory layout, at least for bibliographies fragments is easy, e.g., short-circuiting case changes of English-language documents. In addition, it has for a title, as shown in [40, § 13.6.3], as well as chang- never crashed during our usage of it, even when ing the style of a string by using a command like dealing with syntactically incorrect .bib files. ‘\emph{...}’. That is due to the fact that inserting 8 When the word ‘correct’ is used in software engineering, additional strings before or after the contents of a it is related to the existence of a formal specification — i.e., a field is easy if this information is at the stack’s top mathematical description — of the behaviour, and the program and has not been popped yet by means of the write$ should have been proved correct w.r.t. this specification. Here we adopt a more basic and intuitive sense: the program’s results are what is expected by end-users. 9 Categorical Abstract Machine Language.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 403 Jean-Michel Hufflen
\bibitem[{Murphy\jbbtasep Mullaney\jbdy {2007}}% {}% {{0}{}{book}{2007}{}{}{}{}% {New-York\bpubaddr {} Tor\bibbdsep {} 2007}}% {{Choke Hold}% {}{}{2}{}{}{}{}{}}% ]{murphy-mullaney2007} \jbbibargs {\bibnf {Murphy} {Warren} {W.} {} {}\Bibbtasep \bibnf {Mullaney} {James} {J.} {} {}} {Warren MurphyJames Mullaney} {aus} {\bibtfont {Choke Hold}\bibatsep\ \apyformat {New-York\bpubaddr {} Tor\bibbdsep {} \novname\ 2007} \numberandseries {2}{The New Destroyer Series} \jbnote {1} {The original series has been created by Richard Sapir and Warren Murphy} } {\bibhowcited} \jbendnote {The original series has been created by Richard Sapir and Warren Murphy} \jbdoitem {{Murphy}{Warren}{W.}{}{}; {Mullaney}{James}{J.}{}{}} {} {} \bibAnnoteFile {murphy-mullaney2007}
Figure 3: BibTEX’s output as used by the jurabib package.
function. For the same reason, adding a closing punc- in an additional package. Citation and formatting tuation sign is easy; a shorthand example to do that functions can be customised by redefining these com- is the add.period$ function. Often handling a new mands. In other words, we can say that BibTEX field is easy, too [40, § 13.6.3]. On the other hand, delegates the layout of bibliographies to these com- changing the order of appearance of fields may be mands. tedious. In addition, it is well-known that there is no 3.2.1 Interface packages modularity within the bst language: each style is a Figure 3 gives an example of using the jurabib bib- monolithic file. If you develop a new style from an liography style. The LATEX commands provided by existing one, you just copy the .bst file onto a new the jurabib package can be redefined like any LATEX file, and apply your changes. Of course, doing such command, although the best method is to use the a task requires good ‘modular understanding’ of the \jurabibsetup command, as shown in [40, § 12.5.1]. functions belonging to the ‘old’ style. Sometimes, A similar approach is used within the amsxport bibli- that is easy — cf. the format.title function given ography style [8] and the bibliography styles usable in Figure 2 — while other times, understanding the with ConTEXt [17]. role of a function is possible only if you know the stack’s state — cf. the new.sentence.checkb func- This modus operandi is taken to extremes by tion in the same figure.10 the biblatex package [36]. In such cases, BibTEX 3.2 Task delegation is used only to search bibliography database files and sort references. The advantage: end-users can Originally, all the predefined bibliography styles pro- customise the layout of bibliographies without any A vided by BibTEX’s generated ‘pure’ LTEX texts, in knowledge of the bst language. But BibT X still A E the sense that only basic LTEX commands were used: remains used to sort references, and this task is not the \bibitem command, the thebibliography en- easily customisable, as mentioned above. vironment [40, § 12.1.2], and some additional com- mands for word capitalisation or emphasis. No addi- 3.2.2 Tib tional package was required when derived bibliogra- In fact, this notion of task delegation already existed A phies were processed by LTEX. in Tib[1], a bibliography processor initially designed This situation has changed when the author- for use with Plain TEX, although it can also be used date system was implemented by the natbib package with LATEX. An example of a bibliography style file and the bibliography styles associated with it [40, used by Tib is given in Figure 4: it consists of some § 12.3.2]. Progressively, other bibliography styles Tib commands — e.g., ‘f’ for ‘citations as footnotes’— have been released, working as follows: BibTEX’s followed by some definitions of T X commands for A E output is marked up with LTEX commands defined typesetting citation references and bibliographies’ 10 items. That is, Tib delegates a bibliography’s layout This function is used when the decision of beginning a new sentence within a reference depends on the presence of to these commands. Let us recall that the bibliogra- two fields within an entry. phy database files searched by this processor do not
404 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
# # standard footnote format (latex) # # if titles are desired in loc. cit. references, see note in stdftl.ttx # # include word-definition file (journals and publishers) I TMACLIB amsabb.ttz f footnotes L useibidandloccit C0 empty citation string O for multiple citations use ordering of reference file
%The lines below are copied verbatim into the output document as TeX commands. %First the file Macros.ttx is \input with Macros and default settings. %The control string \TMACLIB is just a path. %The \footnote macro is from LaTeX % \input \TMACLIB stdftl.ttx %macros for formatting reference list \Refstda\Citesuper %set general formats for reference list and citations \def\Lcitemark{\footnotemark}\def\Rcitemark{} \def\Citecomma{$^,$\footnotemark} \def\LAcitemark{\addtocounter{footnote}{1}\arabic{footnote}} \def\RAcitemark{} \def\LIcitemark#1\RIcitemark{\def\Ztest{ }\def\Zstr{#1}}
Figure 4: The footl.tib file.
%A |srd| ers to compile bst styles, and annotate or extend %T The Gap into Power: A Dark and Hungry God the result. As far as we know, they are not widely Arises used. If we consider a style already written in bst %P HarperCollins and to be adapted, this approach allows more am- %S The Gap %N 3 bitious changes. However, they do not propose a %D 1993 new methodology for designing such styles, so tak- ing maximum advantage of the target languages is %E |srd| difficult for style designers. %T Strange Dreams ++ %I Bantam-Spectra 3.3.1 BibTEX %D 1993 BibTEX++ [31] allows a bst style file to be com- %A Warren Murphy piled into Java classes [28]. As an example, the %A James Mullaney new.sentence.checkb function (cf. Fig. 2) is com- ... piled into the Java function new_sentence_checkb %O November 2007. The original series... given in Figure 6. BibTEX++ can also run native bib- The ‘srd’ abbreviation should be defined by means of liography styles developed in Java, from scratch or de- the following Tib command: rived from the compilation of ‘old’ styles. Other func- D srd Stephen Reeder Donaldson tionalities, such as the production of references for programs other than LAT X, can be implemented by Figure 5: Entries using the Refer format. E means of plug-ins. There are six steps in BibTEX++’s process: for example, parsing a .bst file is the fourth use the .bib format, but rather the Refer format,11 one. After each step, there is a hook, as a callback an example being given in Figure 5. that allows this process to be customised.
3.3 Extending bst 3.3.2 cl-bibtex The following works allow bibliography style writ- cl-bibtex [35] is based on ANSI12 Common Lisp [11]. It includes an interpreter for the bst language, and 11 The pybibliographer program can be used as a converter from the .bib format to the Refer format: see [40, § 13.4.5] for more details. 12 American National Standards Institute.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 405 Jean-Michel Hufflen
public void new_sentence_checkb(String s0,
406 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 407 Jean-Michel Hufflen
Figure 11: Example of a DSSSL stylesheet.
408 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
(define (process-and-closing-period) (let ((the-string (string-trim-right (data (current-node))))) ; Get the contents and leave trailing ; space characters. (literal (if (check-for-closing-sign? the-string) ; Checking if the-string ends with ‘.’, ‘?’, or ‘!’. the-string (string-append the-string period-string))))) (define (author-xor-editor node-list) (let ((author-node-list (select-elements node-list "author")) (editor-node-list (select-elements node-list "editor"))) (make sequence (cond ((node-list-empty? author-node-list) (if (node-list-empty? editor-node-list) (error "Neither author, nor editor!") (make sequence (process-node-list editor-node-list) (literal ", editor.")))) ((node-list-empty? editor-node-list) (make sequence (process-node-list author-node-list) (if (check-for-closing-sign? (string-trim-right (data author-node-list))) (empty-sosofo) (literal period-string)))) (else (error "Both author and editor!"))) (literal " "))))
Figure 12: Some auxiliary functions implemented in DSSSL.
(define (b-if$ sxml-mlbiblio-tree current-entry-plus) ;; sxml-biblio-tree is the complete tree of all the entries to be processed, current-entry-plus the annotated ;; tree of the current entry. (let* ((i2 ((b-bst-stack-pv ’pop))) ; “Else” part. (i1 ((b-bst-stack-pv ’pop))) ; “Then” part. (i0 ((b-bst-stack-pv ’pop)))) ; Condition. (if (integer? i0) (b-process-sequence (if (positive? i0) i1 i2) sxml-mlbiblio-tree current-entry-plus) (begin ((msg-manager ’bst-type-conflict) ’if$ i0) #t))))
Figure 13: Implementing if$ within MlBibTEX’s compatibility mode.
but the example we show illustrates how a functional Expressions like E or E 0 consist of assembling programming language can implement a bibliography literals by means of the make form, using types pre- style. More examples can be found in [10, § 7.5]. defined in DSSSL: paragraph, sequence, . . . The Figure 11 gives some excerpts of a stylesheet that generic type of such results is called sosofo20 w.r.t. displays the items of a bibliography by labelling them DSSSL’s terminology. with their own keys. The core expression language Figure 12 illustrates this style of programming of DSSSL is a side-effect free subset of Scheme. As by showing some specific details: how to implement shown in Figure 11, processing elements uses pattern- BibTEX’s add.period$ function, and the switch be- matching: tween author and editor elements for a book. This 21 (element name E) stylesheet can be run by the jade program; as shown in [10, § 7.5.2], the T X-like typeset engine (element (name 0 name) E 0) E able to process such results is JadeT X. E name E the expression specifies how to process the Fragments of DSSL stylesheets can be organised name element, unless this element is a child of the 0 into libraries, so this language is modular. Most of E element, in which case the 0 expression applies. the implementations of it are robust, efficient, but The choice of the accurate expression is launched by functions such as process-matching-children, 20 Specification Of a Sequence Of Flow Objects. 21 process-children, and process-node-list. James Clark’s Awesome DSSSL Engine.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 409 Jean-Michel Hufflen
they are neither extensible, nor easy to use, because X E we have to make precise a predefined backend. For T
example, if jade is used to process the complete Bib XSLT DSSSL Bibulus stylesheet given in Figures 11 & 12, the possible Correctness ✔ backends are tex (resp. rtf), in which case the result Robustness ✔ ✔ ✔ is to be processed by JadeTEX (resp. Microsoft Word Extendability ✘ ✔ ✘ ✔/✘ or OpenOffice). Deriving texts directly processable Reusability ✔/✘ A by LTEX or ConTEXt is impossible. Modularity poor ✔ ✔ ✔ ✔a ✔ ✔ 5 The application to MlBibTEX Continuity average Efficiency ✔ ✔ When we designed MlBibT X’s present version, we E Ease of use ✔ average ✘ ✘ had had much experience in programming DSSSL and a XSLT stylesheets. We thought that a language close . . . except for multilingual features, in XSLT 1.0. DSSSL to would provide more expressive power for Table 1: Languages for bibliography styles: synthesis. developing, but would be accessible only for program- mers. A language close to XSLT is better from this point of view, provided that an extension mechanism language. Analogous combinations exist, the most is given for operations related to pure programming, widely used are a logic programming language, like e.g., the definition of new relation orders [24]. In Prolog,22 called within a C (or similar) program. addition, performing some operations may be more This modus operandi allows programmers to use a difficult than in bst, e.g., the add.period$ function- very specialised language only when it is suitable. ality. There is an analogous example within TEX’s world: The only solution is to provide an initial library LuaTEX. Functionalities related to typesetting are legible from a point of view related to methodology performed by commands built into TEX, whereas [19]. A compatibility mode is needed in order to ease other functions are implemented by means of the the transition between old and new bibliography Lua language [26]. So TEX is used as the wonder- styles [20] — Figure 13 shows how the if$ function is ful typesetting engine that it is, and functionalities implemented within this mode, in comparison with difficult to implement with TEX’s language23 are del- the implementation of cl-bibtex, given in Figure 7. egated to a more traditional programming language. This progressively led us to the architecture described BibT X is still a powerful bibliography proces- in [22]. E sor, but the main way to extend it easily concerns Bib We can be objective about Ml TEX only with the layout of the bibliographies. That was sufficient difficulty. However, several points seem to us to some years ago, but not now with the use of Unicode, confirm our choices. First, XSLT has succeeded as a new processors like X TE EX, and new languages like XML language able to deal with texts, much more so HTML. than DSSSL with SGML texts. Second, the need for a classical programming language: using the whole 7 Acknowledgements expressive power of Scheme — and not a subset as in I have been able to write this article because I have DSSSL — allowed us to program efficiently, by using had much occasion to become familiar with the lan- advanced features of Scheme. Third, our experience guages and applications mentioned above. For ex- with ConTEXt [21] seems to confirm the extendability ample, some years ago, a colleague asked me to help of our tool. him with a DSSSL program because I knew Scheme: 6 Conclusion that was my first contact with this language and SGML, before XML came out. Another time, a friend Table 1 summarises our experience with the lan- who used Plain TEX asked me a question about Tib, guages we have described above. Of course, this although I had merely heard of the name of this synthesis is not as objective as benchmarks would program, and I discovered it in this way. So I was have been. It is just a study of the effort we have thinking about all these people when I was writ- made for developing bibliography styles, and a pro- ing this article, and I am grateful to them. Many fessional view of the results we have found. thanks to Karl Berry: as usual, he is a conscientious To end, let us make a last remark about what proofreader and ‘figure-positioner’. is done in MlBibTEX: the separation of function- alities related to programming, written in Scheme, 22 A good introductory book to this language is [7]. and specifications of layout, given in an XSLT-like 23 Some examples can be found in [45].
410 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Languages for bibliography styles
References TEXnische Komödie, Bd. 4/2003, S. 39–63. November 2003. [1] James C. Alexander: T ib: aTEX Bibliographic Preprocessor. Version 2.2, see [16] Hart’s Rules for Composers and Readers at CTAN:biblio/tib/tibdoc.tex. 1989. the University Press. Oxford University Press. [2] ANSI: The Programming Language Ada 39th edition. 1999. Reference Manual. Technical Report [17] Taco Hoekwater: “The Bibliographic ANSI/MIL-STD-1815A-1983, American Module for ConTEXt”. In: EuroTEX 2001, pp. National Standard Institute, Inc. LNCS No. 61–73. Kerkrade (the Netherlands). September 155, Springer-Verlag. 1983. 2001. [3] Neil Bradley: The Concise SGML [18] Jean-Michel Hufflen: “MlBibTEX’s Version Companion. Addison-Wesley. 1997. 1.3”. TUGboat, Vol. 24, no. 2, pp. 249–262. [4] Sean M. Burke: RTF Pocket Guide. O’Reilly. July 2003. July 2003. [19] Jean-Michel Hufflen: “Bibliography Styles [5] Judith Butcher: Copy-Editing. The Easier with MlBibTEX”. In: Proc. EuroTEX Cambridge Handbook for Editors, Authors, 2005, pp. 179–192. Pont-à Mousson, France. Publishers. 3rd edition. Cambridge University March 2005. Press. 1992. [20] Jean-Michel Hufflen: “BibTEX, MlBibTEX [6] The Chicago Manual of Style. The University and Bibliography Styles”. Biuletyn GUST, of Chicago Press. The 14th edition of a manual Vol. 23, pp. 76–80. In BachoTEX 2006 of style revised and expanded. 1993. conference. April 2006. [7] William F. Clocksin and Christopher S. [21] Jean-Michel Hufflen: “MlBibTEX Meets Mellish: Programming in Prolog. 5th edition. ConTEXt”. TUGboat, Vol. 27, no. 1, pp. 76–82. Springer-Verlag. 2003. EuroTEX 2006 proceedings, Debrecen, Hungary. July 2006. [8] Michael Downes: “The amsrefs LATEX Package and the amsxport BibTEX Style”. [22] Jean-Michel Hufflen: “MlBibTEX TUGboat, Vol. 21, no. 3, pp. 201–209. Architecture”. ArsTEXnica, Vol. 2, pp. 54–59. September 2000. In GUIT 2006 meeting. October 2006. [9] Jean-Christophe Filliâtre and Claude [23] Jean-Michel Hufflen: “Names in BibTEX TUG Marché: The BIBTEX2HTML Home Page. and MlBibTEX”. boat, Vol. 27, no. 2, June 2006. http://www.lri.fr/~filliatr/ pp. 243–253. TUG 2006 proceedings, bibtex2html/. Marrakesh, Morocco. November 2006. [10] Michel Goossens and Sebastian Rahtz, [24] Jean-Michel Hufflen: “Managing Order with Eitan M. Gurari, Ross Moore, and Relations in MlBibTEX”. TUGboat, Vol. 29, Robert S. Sutor: The LATEX Web Companion. no. 1, pp. 101–108. EuroBachoTEX 2007 Addison-Wesley Longmann, Inc., Reading, proceedings. 2007. Massachusetts. May 1999. [25] IBM System 360: PL/1 Reference Manual. [11] Paul Graham: ANSI Common Lisp. Series March 1968. in Artificial Intelligence. Prentice Hall, [26] Roberto Ierusalimschy: Programming in Englewood Cliffs, New Jersey. 1996. Lua. 2nd edition. Lua.org. March 2006. [12] Vidar Bronken Gundersen and [27] International Standard ISO/IEC Zeger W. Hendrikse: BIBTEX as 10179:1996(E): DSSSL. 1996. XML Markup. January 2007. http: [28] Java Technology. March 2008. //bibtexml.sourceforge.net. http://java.sun.com. [13] Hans Hagen: ConTEXt, the Manual. [29] Richard Kelsey, William D. Clinger, November 2001. http://www.pragma-ade. Jonathan A. Rees, Harold Abelson, com/general/manuals/cont-enp.pdf. Norman I. Adams iv, David H. Bartley, [14] Hans Hagen: “LuaTEX: Howling to the Gary Brooks, R. Kent Dybvig, Daniel P. Moon”. Biuletyn Polskiej Grupy Użytkowników Friedman, Robert Halstead, Chris Systemu TEX, Vol. 23, pp. 63–68. April 2006. Hanson, Christopher T. Haynes, [15] Harald Harders: „Mehrsprachige Eugene Edmund Kohlbecker, Jr, Donald Literaturverzeichnisse: Anwendung und Oxley, Kent M. Pitman, Guillermo J. Erweiterung des Pakets babelbib“. Die Rozas, Guy Lewis Steele, Jr, Gerald Jay
TUGboat, Volume 29 (2008), No. 3—Proceedings of the 2008 Annual Meeting 411 Jean-Michel Hufflen
Sussman and Mitchell Wand: “Revised5 [41] Chuck Musciano and Bill Kennedy: HTML Report on the Algorithmic Language Scheme”. & XHTML: The Definitive Guide. 5th edition. HOSC, Vol. 11, no. 1, pp. 7–105. August 1998. O’Reilly & Associates, Inc. August 2002. [30] Brian W. Kernighan and Dennis M. [42] Oren Patashnik: BIBTEXing. February 1988. Ritchie: The C Programming Language. 2nd Part of the BibTEX distribution. edition. Prentice Hall. 1988. [43] Chris Putnam: Bibliography Conversion [31] Ronan Keryell: “BibTEX++: Towards Utilities. February 2005. http://www.scripps. Higher-Order BibTEXing”. In: EuroTEX 2003, edu/~cdputnam/software/bibutils/ p. 143. ENSTB. June 2003. bibutils.html. [32] Jonathan Kew: “X TE EX in TEX Live and [44] Erik T. Ray: Learning XML. O’Reilly beyond”. TUGboat, Vol. 29, no. 1, pp. 146–150. & Associates, Inc. January 2001. EuroBachoTEX 2007 proceedings. 2007. [45] Denis B. Roegel : « Anatomie d’une [33] Oleg E. Kiselyov: XML and Scheme. macro ». Cahiers GUTenberg, Vol. 31, p. 19–27. September 2005. http://okmij.org/ftp/ Décembre 1998. Scheme/xml.html. [46] Bob Stayton: DocBook—XSL. The Complete [34] Donald Ervin Knuth: Computers Guide. 3rd edition. Sagehill Enterprises. & Typesetting. Vol. A: The TEXbook. February 2005. Addison-Wesley Publishing Company, [47] Bjarne Stroustrup: The C++ Programming Reading, Massachusetts. 1984. Language. 2nd edition. Addison-Wesley [35] Matthias Köppe: A BIBTEX System Publishing Company, Inc., Reading, in Common Lisp. January 2003. http: Massachusetts. 1991. //www.nongnu.org/cl-bibtex. [48] The Unicode Consortium: The Unicode [36] Philipp Lehman: The biblatex Package. Standard Version 5.0. Addison-Wesley. Programmable Bibliographies and Citations. November 2006. Version 0.7 (beta). December 2007. [49] W3C: XSL Transformations (XSLT). http://www.ctan.org/tex-archive/macros/ Version 1.0. W3C Recommendation. Edited latex/exptl/biblatex/doc/biblatex.pdf. by James Clark. November 1999. http: [37] Xavier Leroy, Damien Doligez, Jacques //www.w3.org/TR/1999/REC-xslt-19991116. Garrigue, Didier Rémy and Jéróme [50] W3C: XSL Transformations (XSLT). Vouillon: The Objective Caml System. Version 2.0. W3C Recommendation. Edited Release 0.9. Documentation and User’s by Michael H. Kay. January 2007. http:// Manual. 2004. http://caml.inria.fr/pub/ www.w3.org/TR/2007/WD-xslt20-20070123. docs/manual-ocaml/index.html. [51] Larry Wall, Tom Christiansen and Jon [38] Bertrand Meyer: Object-Oriented Software Orwant: Programming Perl. 3rd edition. Construction. Series in Computer Science. O’Reilly & Associates, Inc. July 2000. Prentice Hall International. 1988. [52] Thomas Widman: “Bibulus—a Perl XML [39] Microsoft Corporation: Microsoft C# Replacement for BibTEX”. In: EuroTEX 2003, Specifications. Microsoft Press. 2001. pp. 137–141. ENSTB. June 2003. [40] Frank Mittelbach and Michel Goossens, with Johannes Braams, David Carlisle, Chris A. Rowley, Christine Detig and Joachim Schrod: The LATEX Companion. 2nd edition. Addison-Wesley Publishing Company, Reading, Massachusetts. August 2004.
412 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Vistas for TEX: liberate the typography! (Part I) Chris Rowley Semantic Maths for All! Project Department of Mathematics and Statistics Faculty of Mathematics, Computing and Technology Open University, London, UK c.a.rowley (at) open dot ac dot uk Abstract This is a polemic in favour of liberating the core typesetting structures and algorithms around which TEX is built from the monolithic superstructure of the mini-dinosaur of a program called tex and its more or less modernised and approachable derivatives such as xetex and luatex. Although the high-level aims of the programme of activity advocated here have a lot in common with those of the very exciting and active LuaTEX project, the route I propose seems to me to be very different. The major ambition of the latter project is to embed (something similar to) the whole of the current TEX system within a vastly more complex monolith [sic] of an application which will presumably be well-adapted to the formatting needs of oriental languages. To this monolith are now being added many well-oriented intrusions [sic] into but a single instance of that ancient bedrock of TEX! Of course, luatex promises to provide a spectacularly sophisticated and highly hackable system that will eventually enable a great evolutionary radiation of species within the phylum of automated document processing; hence the importance and fascination, for me at least, of the developmental path of the LuaTEX project. Pursuing the paleontological metaphor well beyond its point of total and painful collapse, my plan can be thought of as providing many tools that can be easily dispersed in such a way that TEX’s clever genes can influence (for the good) far more aspects of the evolution of automated typesetting: all this abundance being more speedily and robustly achieved due to not being held back by the decision to build all future systems on a perfectly preserved and complete digestive system from a fossilised ancestral TEXosaur. I am here also making a plea to the Grand Technical Wizards of TUG to widen support from their development fund’s treasure chest to encompass projects that are designed to spread TEX’s influence and presence throughout the fertile modern world of document processing via its algorithms alone, without the dead weight of its monolithic, programmatic paradigm and the many somewhat dated aspects of its detailed software design. Adding topicality and an even longer time-base to the metaphors, please can we have plentiful levels of international funding to support an actual Big Bang to get the elementary particles of TEX spread throughout the typesetting universe, rather than funding only an engineering wonder (for interesting but small-scale experiments to find new TEX-like particles) such as the LHC: LuaTEX’s Hard-problems Cruncher, a ‘shining star in the East’ which I fear may spin-off some big black-holes to trap even the most energetic of us mortal, western programmers.
1 Introduction deep discussions, through examples, of the most im- This could easily have been the shortest genuine portant and most difficult part of doing this in a practical and useful way: the provision of good ex- paper in this, or any, TEX Users Group proceedings. All it needs to say is: please support Free, Open ternal interfaces. But we shall begin in Section 2, with some introductory remarks (not all strictly per- and Reusable Algorithms from TEX (yes folks, the FORAT Campaign starts here!). tinent) concerning current TEX’s use and misuse However, I will attempt, in this and subsequent of its formatting subsystems. The paper concludes papers, to expand on some examples of what I mean with a note on non-TEX math formatters, being an introduction to further study in this area. by liberating TEX’s formatting algorithms in the form, for example, of C++ libraries, embeddable The whole is permeated by a sincere plea for JavaScript or similar reusable artefacts. greatly increased support, of all types and by the In Section 3 I shall also provide more or less many TUGs and all individuals in the TEX com-
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 413 Chris Rowley
munity, for this outward-looking work so crucial to formatters. Such output could then be used by other ensuring a very long life for both the great gifts that ‘higher-level’ formatters to choose the formatting arrived within TEX, and also for the inventive and most suited to their higher purposes. experimental spirit of its pioneers. May this paper I shall not pursue such valuable enhancements be the first of many that promote the creation of here as I am only asking for the Moon, not Mars, Really Useful Software from TEX’s Inner Engineering . . . this year; the benefits of this approach to the (the RUSTIE project). practical optimisation of formatted documents have been long known and much discussed since Frank 2 The structure of TEX Mittelbach and I [18] introduced them. The sum of the ‘document manipulation’ parts of 3 Two (of many) examples TEX can be well described as a text processing ap- plication whose originality and utility is provided by The two of TEX’s many embedded formatters about many specialised formatting engines. Many of these which I have chosen to say a little more here are not engines are barely recognisable as independent soft- necessarily either the most complex or the easiest to ware artefacts due to the programming techniques specify, but they are both central to TEX’s raison wisely chosen by Knuth to implement the system [6]. d’ˆetre. I have somewhat presumptuously chosen to (His methodology can these days be well described put these into the form of outline draft specifica- as the EO method: Extreme Optimisation, of both tions. I hope that they will start a process that leads space and time!) smoothly and quickly to full specifications and to the Each of these deeply embedded formatting en- production of partial but usable ‘proof-of-concept’ gines will, when it has been disentangled from the library implementations. For the maths formatter sticky mess of connections and optimisations that some more detailed work has been done (Section 3.3) holds it deep within TEX’s embrace, reveal itself as a that will soon form the contents of a funding bid for thing of beauty. Less appealing, each such newly dis- the project. entangled creature, whilst happy to be breathing free, will also be a very vulnerable beast as it will have no 3.1 The paragraph formatter communication interfaces through which to nurture This is just one possible route towards the expo- it. Thus its liberators will need to carefully craft sure of a TEX-like paragraph mechanism and it is protective interfaces in order to ensure the fledgling treated very briefly here. I hope it inspires others formatter’s viability in the real and exciting, but to help expand this to a full and carefully explained non-programmatic, world of 21st century documents. specification, to be published in this series. These hidden gems include formatters for words, Note that this outline description assumes the LR-boxes (with natural and many other width speci- existence of methods for formatting ‘LR-boxes’: these fications), split-able [sic] LR-boxes, leaders of various are ‘single lines’ of text with well-defined spacing of types, paragraphs, split-able [sic] paragraphs, justi- ‘words’. In turn that formatter will require a ‘word fied lines, formulas, superscripts, fractions, radicals formatter’, etc. I plan to provide a fuller explanation with bar, etc. of these ideas in a future paper. One aspect of these specialised formatters clearly The inputs to this formatter would be as follows. shows TEX’s ancient pedigree, from a time when data • The material to be formatted (see below). flow had kept to a minimum. This is their lack of • Parameter settings. Although it may not be flexibility, in the following sense. With the exception essential, it would be very useful to have the of using the concept of ‘unset glue’, TEX’s formatters ability to input values for all the parameters will always combine to produce, from a given input, that are used by TEX’s algorithm for forming a unique, fully specified boxful of formatted output and formatting a paragraph. Of course, many (often with a claim of its ‘optimal’ quality). of them will have sensible default values that As Frank Mittelbach and I have copiously ar- could be fixed or, in context, inherited. gued over the years, for a sophisticated document formatter a more useful product would be a reason- The material to be formatted would consist of the ably sized collection of possible formattings that are following. all ‘good enough’. To this collection could possibly – Pure text (e.g. Unicode strings). be added some ranking of their absolute quality but – Some ‘formatting information’ such as: even more useful would be a few descriptors or quan- – font selection hints tisations of how good each is, together with other – line-breaking hints information that may be of use to other co-operating – word-division information
414 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Vistas for TEX: liberate the typography! (Part I)
– even direct formatting instructions(!) from the TEX world (if not all worlds), we must nev- all with a precisely defined syntax. ertheless face the reality that TEX Does Math!! — – Pre-formatted material in the form of ‘in-line both within the TEX world but also, far more and boxes’. growing rapidly, outside it, including much that will never see the guts of a TEX processor. Hence if, over The basic liberated version of this formatter the next 10 years or so, real world ‘TEX for maths’ would return two types of material (this is slightly is not to lose all contact with ‘TEX the processor’, more than current TEX’s internal mechanism can then TEX’s maths formatter must get out there and be bothered to do, despite the information being strut its stuff wherever maths is being stuffed into available). Applications using it are free to ignore digital form. item 2. For the liberated form of this important format- 1. A formatting of the paragraph: the formatted ter it is more difficult to specify in suitably general ‘lines’ of the paragraph plus other information terms the nature of the material to be formatted; the about their layout in a well-defined format. obvious specification is ‘Presentation MathML con- 2. The number of lines, information about the taining Unicode strings’ [16] but it is not clear to me break-points used (e.g. their location in the in- at this stage whether this will always be sufficiently put string, whether a word was broken), an rich in information about the mathematical struc- evaluation of how ‘good’ each break-point is and ture of the notation to be typeset. However, a lot of the quality of the formatting of each line. of work has been done, and is continuing, on a wide Note that here I have not explicitly discussed range of uses of mathematical notation in comput- any of TEX’s sophisticated escape mechanisms that ers, together with their associated description and implement a small range of specialised extra activ- formatting requirements. Exposing this formatter ities, such as ‘vadjust material’, marks or ‘whatsit will greatly help many aspects of this research and nodes’. These are good examples of Knuth’s cleverly development effort, in particular the task of deter- ad hoc use of small and tightly integrated extensions mining what needs to be encoded in the input to to a basic algorithm in order to emulate the effect ensure high quality maths output. of fully modelling these varied aspects of the docu- The output will be material that can be used ment formatting process as separate modules. Such by an ‘LR-box’ or ‘paragraph’ formatter. features of TEX thus express a limited collection of The current understanding of the parametrisa- ideas from a wider class that is important to auto- tion needed for this task is also still somewhat em- mated document processing. Being ad hoc, Knuth’s pirical. It is known that standard TEX’s algorithm is efficient implementations typically use inappropri- severely under-parametrised for the tasks in which it ate models and hence have severe deficiencies. It is claims supremacy but, in contrast, many applications therefore probably not sensible to reproduce such will require only a far simpler parametrisation. escape mechanisms when providing our exposures of It is therefore desirable that the implementation The Good Things of TEXTM. of this algorithm should build in sensible default rules A more advanced version of this liberated for- for determining plausible values of all TEX’s ‘maths matter would allow the replacement of TEX’s algo- parameters’ from data as meagre as just the nominal rithm for finding break-points but this would entail text font size. However, this liberation must at least provision for whole new parametrisations of the pro- remove all of current TEX’s explicit overloading in cess and thus could decrease its immediate usability. the ‘standard parameter set’ in order to be more A more practical way of providing this formatter may generally usable. It may also be sensible to remove be to split it up into smaller modules that undertake some aspects of the parametrisation from the current distinct parts of the task, leaving the master para- dependence on the choice of fonts. graph formatter with only the two tasks of control- The following subsection contains further de- ling the whole process and of handling all external tails of this project; it is an extract from a fund- interfaces (which would need to be customisable). ing proposal currently being pursued for this task. Throughout, the phrase ‘Standard LATEX’ refers to a 3.2 The mathematics formatter well known and defined (mainly ad hoc at present) This formatter has turned out to be the central fea- subset of LATEX’s math mode, but using the full range ture in this final form of my diatribe; thus I shall of Unicode maths characters, for encoding mathe- warn you that it gets tediously detailed from here on. matical structures and glyphs. This is not a good Although many of you would like to follow the exam- phrase for this beast, thus it has also been dubbed ple of Sebastian Rahtz [3] and banish all mathematics ‘LoTEX’ [17] (as in the common multiple Lcm).
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 415 Chris Rowley
3.3 Liberating TEX: maths formatting tive, agent-supported browsing of large collections of A fact and the obvious question: Many current appli- pages that contain a lot of connectivity and mathe- cations, like the above specification, require a maths matical intelligence embedded in semantically rich encodings of mathematical notation [19]. formatter: why not make it TEX’s? Until quite recently the standard TEX formatter 3.3.1 Inputs for Computer Modern was the only such application that was both widely available and offered even rea- • Maths material encoded in a fully specified lan- guage that describes, at least, the presentational sonable typeset quality. However, bits of TEX were, back then, totally incompatible with other applica- structure (or, visual semantics) of the material. tions that used different font resource technologies or This currently would typically be either a pre- series. Thus there are now many applications that, cisely defined (and large) subset of P-MML (Pre- with varying degrees of success, attempt to emulate sentation MathML 3.0 [16]) or a syntactically the quality of T X without its restrictions on input precise subset of the currently used range of E A 1 and output. There are now also some serious rivals LTEX-related math-mode syntax (LoTEX). to TEX’s typeset quality for mathematics but they In order to support a more semantically ori- A also share two, at least, of TEX’s problems: being too ented and user-friendly form of LTEX input, it far embedded in widely used and sophisticated docu- may be wise to provide a preprocessor that ac- ment processing systems; and being closely linked to cepts precisely restricted uses of \newcommand. particular font series. • A modern ‘maths font’ resource (similar to Open- Whilst it may not be feasible to get the highest Type with the necessary ‘math tables’ as sup- quality formatting using a given maths font series plied with Microsoft’s Cambria Math font). without careful choice of the set of layout parameters (and their values) specific to that series [7, 8], it is 3.3.2 Outputs certainly possible to give the world TEX’s current al- This is not so easy to standardise. The high-level gorithm with some liberation of the parametrisation. specification is that it will consist of ‘glyphs plus rules’ For example, it is straightforward to remove from (plus, possibly, other simple graphical components its layout parameter set-up the many explicit (and and colour features) that are absolutely positioned 30-year old) overloadings and relationships that were in a local coordinate system relative to a reference necessary to Knuth’s giving us this wonderful gift, so or ‘base point’ that in turn can be used to position expertly tuned to a particular font series (Monotype and orient the output on a page. It will also contain Modern), itself about 100 years old![15] the necessary pointers to glyph rendering and paint More importantly, this quality can now be made resources, etc. available in a portable form so that it could be easily Since there is no widely accepted standard lan- linked into any application, such as browsers or office guage for precisely such output, it will be necessary suites, that need to render structured representa- to use a simple fixed internal representation akin tions of mathematics, particularly MathML 3.0 and to TEX’s h/vlist or DVI languages. However, unlike A ‘Standard LTEX’ or LoTEX. Post-liberation, this the current TEX paradigm, modules to convert this valuable treasure can be further enhanced by making internal language to a range of commonly needed it more configurable so as to allow extensions of its languages will also be included. Examples of outputs capabilities in the following two areas: the diversity are thus some type of PDF and SVG fragments and of 1.5-dimensional (or maybe fully 2-dimensional) other application-specific formats such as RTF or the text-based layouts that it can construct; the range internal renderable format of a web browser. of fonts, glyphs, colour resources and other printing marks that can be used in these constructs (and 4 Other mathematics formatters maybe built-in interactivity). Time has mitigated against extending this section For many modern applications it will be essential beyond these very brief comments on some impor- to provide (at least in a derivative version) optimisa- tant existing non-TEX maths formatters. So there is tion for fast parallel rendering of a large collection another paper or two waiting in the wings. of (typically small) maths fragments, possibly with For a long time there have been non-TEX maths different ‘quality control requirements’. formatters in general use, such as techexplorer [20]. A typical such application (from the trendy There are now a large number of these; here is a par- worlds of Web 2.0 and/or the Semantic Web) of tial list of those that are definitely ‘fit-to-purpose’, the near-medium future will be fast, highly interac- 1 There is also a far wider need for LoTEX, together with standard translations to/from P-MML.
416 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Vistas for TEX: liberate the typography! (Part I)
be that quick, simple and clear rendering of sim- http://www.springerlink.com/content/ ple maths in browsers, or high quality use of well- 57432v516731367n/ designed fonts for more classical paper presentation: [8] Peter Karow and Herman Zapf. Digital jsmath, Mathplayer for Internet Explorer, typography. Private communication, 2003. Gecko-Math (used in Firefox et al.), [9] John Plaice, Yannis Haralambous and Microsoft’s Rich Edit (in Office 2007), Chris Rowley. An extensible approach to MathEX (incorporates techexplorer), SWiM. high-quality multilingual typesetting. In How much the design of each of these systems RIDE-MLIM 2003, IEEE Computer Society uses, or is influenced by, Knuth’s algorithm and Press, 2003. layout rules ([5], Appendix G) has not I think been [10] Extensible Markup Language (XML). much studied; maybe the individual authors of the http://www.w3c.org/XML systems once knew the answers? [11] GtkMathView Home Page. Only Microsoft’s system [23, 2, 22] (as demon- http://helm.cs.unibo.it/mml-widget strated in my talk at San Diego) claims typograph- [12] X TE EX. http://scripts.sil.org/xetex ical quality that is better than that of T X; it is E [13] LATEX: A document preparation system. also the only one that is not intimately connected http://www.latex-project.org with MathML [16] although the two are reasonably [14] LuaT X. http://www.luatex.org friendly. The creators of this RichEdit system state E http: that it uses ‘TEX’s mathematical typography prin- [15] Monotype Modern: Description. ciples’ but they go on to remark that this task was //www.paratype.com/fstore/default.asp? nevertheless ‘considerably harder than any of us imag- fcode=871&search=Monotype+Modern ined it would be’, taking 15 years of elapsed time [16] Mathematical Markup Language (MathML) to complete (fortunately, throughout this time the Version 3.0, W3C Working Draft. then boss of the whole company took a keen per- http://www.w3.org/TR/MathML3 sonal interest in the whole project). They conclude [17] Chris Rowley and Stephen Watt. The need for that ‘mathematical typography is very intricate and a ‘Standard LATEX’. Private communication, varied’. July 2007. Although the associated product does not seem [18] Frank Mittelbach and Chris Rowley. The to be widely used, the GtkMathView project [11] pursuit of quality: How can automated has worked on and documented a lot of interesting typesetting achieve the highest standards of ideas and artefacts in this area. craft typography? In Electronic Publishing, pages 261–273, References Cambridge University Press, 1992. [1] Hermann Zapf. About micro-typography and [19] Christoph Lange and Michael Kohlhase. the hz-program. Electronic Publishing 6(3), SWiM: A Semantic Wiki for Mathematical pages 283–288, Wiley, 1993. Knowledge Management. http://cajun.cs.nott.ac.uk/compsci/epo/ Poster at http://kwarc.info/projects/ papers/epoddaui.html swim/pubs/poster-semwiki06.pdf [2] About Rich Edit Controls. [20] Don DeLand. From TEX to XML: The legacy http://msdn.microsoft.com/en-us/ of techexplorer and the future of math on the library/bb787873(VS.85).aspx Web. Abstract in TUGboat 28:3, page 369, [3] Sebastian P. Q. Rahtz. Banish Maths! Daily TUG, Proceedings of the 2007 Annual Meeting. personal communications, 1990–2005. [21] Unicode. http://www.unicode.org [4] D. E. Knuth and M. F. Plass. Breaking [22] Murray Sargent. Using RichEdit 6.0 for Math. paragraphs into lines. Software – Practice and http://blogs.msdn.com/murrays/archive/ Experience, 11(11), pages 1119–1184, 1981. 2007/10/28/using-richedit-6-0-for-math. [5] D. E. Knuth. TEX: The Program. aspx Addison-Wesley, 1986, 1993. [23] Ross Mills and John Hudson, Editors. [6] D. E. Knuth. The TEXbook. Addison-Wesley, Mathematical Typesetting: Typesetting 1986, 1993. Solutions from Microsoft. Glossy brochure distributed at TypeCon 2007, Seattle, [7] Richard Southall. Designing a new typeface August 1–5, 2007. with METAFONT, in TEX for Scientific Documentation, Springer 1986, pages 161–179.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 417 Why didn’t METAFONT catch on?
Dave Crossland University of Reading, UK dave (at) lab6 dot com http://www.understandinglimited.com
Abstract METAFONT is an algebraic programming language for describing the shapes of letters, designed and implemented by Knuth as part of the original TEX type- setting system. It was one of the earliest digital type design systems, and is completely capable of dealing with the letters of any writing system, has always been freely available, and is remarkably powerful. Yet it never caught on with type designers. Why?
“There are three kinds of people. Those that can is not abstract and logical, for the most part; it is count, and those that can’t.” visual, concrete, more emotional than logical. There is type in typography, but there is also Throughout the long history of desktop pub- type in psychology: Personality type. lishing, people have generally not found TEX type- There are many ways of thinking about per- setting intuitive, preferring desktop publishing ap- sonality type [1] and the famous Myers–Briggs ty- plications with graphical user interfaces. Even for pology places importance on four attitudes. First, those who go deep enough into graphic design to ar- there is our preference for competition or coopera- rive at type design, METAFONT is almost entirely tion, or whether we tend to make decisions logically ignored — despite being freely available, completely or emotionally. Second, our use of language reveals capable, and remarkably powerful. I believe this the way we think, with some people preferring more issue of personality type is a primary reason why abstract language and others preferring more con- METAFONT has not caught on. crete language. Third is our attitude to time keep- Let’s consider type design divorced from the en- ing, which may be exploratory or scheduling, and gineering of font software for a moment. fourth is our orientation to socialising, where after Design happens at various scales at the same a party we may feel drained or energised. time. In type design, the lowest visible level is that Put together, these four preferences between of the letter, where you are dealing mainly with the two options yield 16 personality types. The book black shapes of the letter. It is obvious what those Please Understand Me 2 [2] puts them into a co- are, but there are also the ‘white’ shapes. If you are hesive system that groups the 16 types into four not sure what that means, imagine an image of a temperaments, fleshed out by labels and personified letter, and invert it so that the black becomes white by Greek gods: Epimethean ‘Guardians,’ Dionysian and the white becomes black. Now, look around ‘Artisans,’ Apollonian ‘Idealists,’ and Promethean you to find a letter printed large on something like ‘TEXXies’ ‘Rationalists.’ a poster or book cover. Looking at the letter, shift Such broad theories for how people differ prob- your awareness to the ‘negative space’ in and around ably can’t be taken too far, as ultimately people the letter, and bring these white shapes into percep- are all pretty much alike; “what one man can do, tual focus. It is hard to describe them, but they are another man can do.” But there is a common senti- there, and designing them is as important as design- ment that some of us are more abstract in our lan- ing the black shapes. guage and more logical in our thinking than others. The next level up is that of words. Here there Software is pretty abstract and logical, and peo- are not only the white shapes inside and around the ple who become immersed in the world of software letters, but those between the letters. At this level tend to be of a Promethean temperament. The ar- we can also see patterns in the black shapes across guments for software freedom especially have that letters; things that look similar, yet are not exactly kind of draw. TEX takes an abstract and logical ap- the same. proach to digital typography, from concept to usage, Consider the lowercase n and h. These contain and METAFONT is no exception. But graphic design several similar shapes, but looking closely, you will
418 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Why didn’t METAFONT catch on? see that there are slight differences. Balancing these been thought out; the TEX community provides a similarities and differences is a core part of the type clear example of this in the AMS Euler project [4], design process. There are strong patterns in some where a team of Stanford students attempted to sets of letters, weaker similarities in other sets, and digitise a new type design for mathematics that Zapf some letters that are less typical, yet still look like had drawn on paper; the developers tried both ap- they belong with the rest. The letter s is perhaps proaches and felt tracing outlines was most appro- the most different, and Knuth wrote an interesting priate. FontForge [5] is a vigorously developed free essay about the peculiarities of that letter [3]. software font editor application for working in this Finally, there is the level of paragraphs. When way today. a paragraph is set with a typeface, a different im- While not suitable for implementing existing pression of the letters emerges. This must be taken type designs, METAFONT’s abstract and logical na- into account at the other levels. Seen in this way, a ture makes it powerful for dealing with type at the type design is a collection of individual glyph shapes level of words. While initially slow, it speeds up that fits together cohesively at all levels. later stages of the design process, especially when We can now see clearly the subtle distinction covering very large character sets. I think it is ide- between a font and a typeface. The same typeface ally suited to developing new type designs where the can be implemented in a variety of typesetting tech- designer is not sure of the precise look that they are nologies — metal, software, even potato — with the trying to capture and want to experiment with a end result appearing the same. A font is a typeface variety of sweeping changes to their design. implemented in software. The term ‘software’ spans TEXworks [6] attempts to make TEX typeset- programs and data, and fonts are a peculiar kind of ting more visual and interactive. While still abstract software because they are both programs and data, and logical compared to desktop publishing applica- while normally the two have some separation. Ex- tions like Scribus [7], its user interface design and amples of programs within fonts are TrueType hints the SyncTEX technology [8] tightly interconnect the and OpenType layout features; these instruct the code and the document, making TEX more visual, computer to display the type in various ways. The interactive, concrete and emotional. data in a font is the glyph point data and metrics Today METAFONT source code is written, var- table data. ious programs are run to generate graphics, then There are generally two approaches to imple- another program is used to view them. These pro- menting typefaces in software. The ‘outline’ ap- grams may be METAFONT, or METAFONT and then proach involves drawing each letter by interactively mftrace [9], or METAPOST [10] with MetaType1 placing points along its outline. This attempts to [11]. All involve a whole long process that is similar be a direct facsimile of drawing letters on paper. to writing TEX documents in the traditional man- Interpolation between sets of outlines means this ner. But with TEXworks, the TEX source code is approach can handle the creation of large typeface rendered into a document in near real-time, so there families. is a very quick Boyd cycle [12] between adjusting the The ‘stroke’ approach is where each letter is typesetting code and seeing the document rendered. constructed by specifying points along the path of Perhaps if there was a graphical user interface a pen’s stroke, and the attributes of the pen’s nib to visualise METAFONT code in near real-time, type at those points. Archetypal pieces can be designed designers who feel writing code is unintuitive could and used like Lego blocks to construct whole glyphs, be more confident about doing so. The simple GUI with refinements made for the individual require- shown by Sherif & Fahmy in their Arabic design ments of each letter. With parametrisation to make work is an example of this [13]. It might even be the shared values of shapes easily adjustable, such feasible to have two-way interaction between code as widening stems or modifying serifs, this approach and rendering, as Inkscape [14] achieves for SVG. can handle a large typeface family in a cohesive and Perhaps then, METAFONT might catch on. powerful way. Today the outline approach is dominant be- Dave Crossland is an international public speaker cause it gives instant visual feedback and exacting on software freedom and fonts, runs a small busi- control; it is direct and visceral. This means design- ness doing type and information design and systems ing type at the level of individual letter shapes is administration, and is a committee member of UK- intuitive and a typeface emerges quickly. TUG. He is currently studying at the University It is especially suited to implementing exist- of Reading’s Department of Typography on the MA ing type designs where all the aspects have already Typeface Design programme.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 419 Dave Crossland
References [9] http://lilypond.org/mftrace/ [1] http://en.wikipedia.org/wiki/Category: [10] http://www.tug.org/metapost.html Personality_typologies [11] http://www.ctan.org/tex-archive/fonts/ [2] Keirsey, D. Please Understand Me II: utilities/metatype1/ Temperament, Character, Intelligence. 1998: [12] Osinga, F. Science, Strategy and War: Prometheus Nemesis. The Strategic Theory of John Boyd. 2006: [3] Knuth, D. Digital typography. 1999: CSLI. Routledge. [4] Knuth, D. & Zapf, H. AMS Euler: A New [13] Sherif, A. and Fahmy, H. Meta-designing Typeface for Mathematics. 1989: Scholarly parameterized Arabic fonts for AlQalam. In Publishing. this volume, 435–443. [5] http://fontforge.sourceforge.net [14] http://www.inkscape.org [6] http://tug.org/texworks/ [15] http://metafont.tutorial.free.fr [7] http://www.scribus.net [8] http://itexmac.sourceforge.net/SyncTeX. Copyright c 2008 Dave Crossland. Verbatim copy- html ing and redistribution of this entire article is per- mitted, provided this notice is preserved.
420 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Creating cuneiform fonts with MetaType1 and FontForge
Karel Píška Institute of Physics of the ASCR, v.v.i. CZ-182 21 Prague, Czech Republic piska (at) fzu dot cz Abstract A cuneiform font collection covering Akkadian, Ugaritic and Old Persian glyph subsets (about 600 signs) has been produced in two steps. With MetaType1 we generate intermediate Type 1 fonts, and then construct OpenType fonts using FontForge. We describe cuneiform design and the process of font development.
1 Introduction tional macros are defined in MetaType1 extensions I am interested in scripts, alphabets, writing sys- or may be appended by the user. For illustrations tems, and in fonts, their computer representation. see the examples below. An explanation of some Ten years ago I decided to create Type 1 fonts for technical details and techniques how to work with cuneiform, and last year, to extend them to Unicode MetaType1 can be found in the tutorial written by OpenType versions. In my older Type 1 version Klaus Höppner [7], which also includes a simple com- (1998/9) [4] the raw text was written ‘by hand’ and plete example and Makefile. then directly compiled into Type 1 with t1asm from 2.1 Font description in MetaType1 t1utils [10]. The glyph set consisted of several METAFONT METAPOST separate Type 1 components to cover Akkadian (ac- As usual with or the compi- cording to Labat) in a ‘Neo-Assyrian’ form (three lation is invoked by a main control file — naakc.mp: files), Ugaritic, and Old Persian. input fontbase; Three books served as principal sources: two use_emergency_turningnumber; Akkadian syllabaries edited by R. Labat [1] and input naak.mpe; maybeinput "naakc.mpd"; F. Thureau-Dangin [2], and the encyclopedia The maybeinput "naakc.mph"; World’s Writing Systems [3]. No scanning of pictures maybeinput "naug.mph"; or clay tablets was performed. The fonts are based maybeinput "naop.mph"; on a starting point — my simple design of wedges in beginfont three variant forms (see also Fig. 1, below): maybeinput "naak.mpg"; ¡¢£¤¥ ¡¢£¤¥ maybeinput "naug.mpg"; ¡¢£¤¥ ¡¢£¤¥ ¡¢£¤¥ maybeinput "naop.mpg"; endfont A ‘Academic’ B ‘Bold (Filled)’ C ‘Classic’ Global font parameters may be defined in a font Our aim is to use free and open source software header file — naakc.mph: to produce “open source fonts”. Thus, to create the % FONT INFORMATION fonts only non-proprietary tools have been employed: pf_info_familyname "NeoAssyrianClassicType1"; MetaType1; FontForge; t1utils, gawk; other stan- pf_info_weight "Medium"; dard Unix utilities such as bash, sed, sort, . . . In the pf_info_fontname "NeoAssyrianClassicType1"; following sections we will explain the process of cre- pf_info_version "002.001"; ating fonts and illustrate it with numerous examples. pf_info_author "Karel Piska at fzu.cz 2008"; pf_info_italicangle 0; 2 Producing Type 1 with MetaType1 pf_info_underline -100, 50; The MetaType1 package [6], developed by the au- pf_info_fixedpitch false; thors of Latin Modern, T X Gyre and other font pf_info_adl 750, 250, 0; E italic_shift:=0; collections (B. Jackowski, J. Nowacki, P. Strzelczyk): Internal glyph names and metric data can be • runs METAPOST (any available version) to pro- assigned as follows: duce eps files with outlines for all glyphs; • % INTRODUCE CHARS collects all the data into one Type 1 file. standard_introduce("ash.akk"); The information about the font and its glyphs standard_introduce("hal.akk"); is described in the METAPOST source files; addi- standard_introduce("mug.akk");
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 421 Karel Píška
standard_introduce("zadim.akk"); find_outlines(pa,pb)(pc); standard_introduce("ba.akk"); p[nw]:=pc1; standard_introduce("zu.akk"); Fill p[nw]; ..... nw:=nw+1; % METRICS enddef; wd._ash.akk=240; ht._ash.akk=160; dp._ash.akk=0; Complete glyphs — that is, cuneiform signs — wd._hal.akk=340; ht._hal.akk=160; dp._hal.akk=0; are demonstrated in the following examples. Hori- wd._mug.akk=380; ht._mug.akk=220; dp._mug.akk=0; wd._zadim.akk=460; ht._zadim.akk=220; zontal/vertical wedges, composites or their groups dp._zadim.akk=0; are also defined by macros; the arguments, for exam- wd._ba.akk=420; ht._ba.akk=240; dp._ba.akk=0; ple, denote their lengths and coordinates. wd._zu.akk=500; ht._zu.akk=240; dp._zu.akk=0; beginglyph(_mash.akk); ..... save p; path p[]; nw:=0; (We do not define the encoding in Type 1.) whv(0,120); standard_exact_hsbw("mash.akk"); 2.2 Glyph contours in MetaType1 endglyph; Simple (atomic) elements — single wedges are defined beginglyph(_sag.akk); by macros: save p; path p[]; nw:=0; def wh(expr l,x,y)= wh(240,0,160); r:=5; w:=40; b:=5; c:=20; d:=70; wh(160,80,80); z[nw] 0=(x+l,y+r); wv(240,220,240); z[nw] 0a=(x+d,y+r); z[nw] 1b=(x+d,y+r); wv(240,300,240); z[nw] 1=(x,y+w); whhv(400,120); z[nw] 1a=(x+b,y+c); z[nw] 2b=(x+b,y-c); standard_exact_hsbw("sag.akk"); z[nw] 2=(x,y-w); endglyph; z[nw] 2a=(x+d,y-r); z[nw] 3b=(x+d,y-r); z[nw] 3=(x+l,y-r); p[nw]=compose_path.z[nw](3); Fill p[nw]; nw:=nw+1; enddef; 2.3 Generating Type 1 An intermediate Type 1 is generated from scratch: FN=$1 # font file name MT1=$2 # MetaType1 direction mpost ’\generating:=0;’ input $FN.mp Another definition of a wedge — a single path: gawk -f $(MT1)/mp2pf.awk \ def pwh(expr l,x,y)= -v CD=$(MT1)/pfcommon.dat -v NAME=$FN (x+l,y+r)..controls(x+d,y+r)..(x,y+w) gawk -f $(MT1)/packsubr.awk -v LEV=5 \ ..controls(x+b,y+c)and(x+b,y-c)..(x,y-w) -v OUP=$FN.pn $FN.p ..controls(x+d,y-r)..(x+l,y-r)--cycle; t1asm -b $FN.pn $FN.pfb enddef; with some simplifications (intentional for cuneiform): In compound elements, the rendering of inter- glyph design is simple; no kerning pairs are needed; secting areas may depend on printer/viewer. There- the characters occupy independent boxes; no hyphen- fore, removing overlap in Type 1 (and probably ation; and no internal encoding in the intermediate also in OpenType) is required. We use the macro Type 1 is defined. Theoretically, we could use a find_outlines: common Type 1 with several external encoding vec- tors, but in practice, joining all the glyphs into one OpenType font is a better and simpler solution. 3 FontForge and producing OpenType Scripts in the FontForge scripting language read def whv(expr x,y)= Type 1, build data for OpenType (especially, define save pa,pb,pc; path pa,pb,pc; the encoding) and then generate OTF and TTF files. r:=5; w:=40; b:=5; c:=20; d:=70; Here is a table showing the Unicode areas with which pa:=pwh(200,x,y); pb=pwv(200,x+80,y+100); we are concerned:
422 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Creating cuneiform fonts with MetaType1 and FontForge
Definition of encoding (Unicode) Select("u12580");CopyReference(); Plane 0 Select("u1200A");Paste(); U+0020–U+007F ASCII block ... Plane 1 This operation may also be executed using a Cuneiform ranges “Standard” Unicode FontForge routine: U+10380–U+1039F Ugaritic U+103A0–U+103D7 Old Persian #!/usr/bin/fontforge # addref.pe U+12000–U+123FF Cuneiform signs # 1. font, 2. glyph point in container U+12500–U+1277F Neo-Assyrian glyph container # 3. reference point # addref.pe $fontname.sfd u12743 u12000 (temporary “Private Area”) Open($1);Select($2);CopyReference(); To begin, we introduce a new Unicode font: Select($3);Paste();Save($1); #!/usr/bin/fontforge # 1.sfd, 2.names, 3.pfb, 4.otf, 5.ttf and then: New();Reencode("UnicodeFull") addref.pe $FN u12743 u12000 SetFontNames($2,$2,$2) addref.pe $FN u1264E u12009 #SetFontOrder(3); # cubic addref.pe $FN u12580 u1200A #SetFontOrder(2); # quadratic ... ScaleToEm(250) Save($1) Generating OpenType itself completes step 2: ... we can generate both OTF and TTF. Then we copy glyphs from Type 1 to OpenType: # $4 is OTF, $5 is TTF we open and read a Type 1 font and access glyphs Open($1);Generate($4); # with options by name (in Type 1) and copy them to appropriate Open($1);Generate($5); # with options locations addressed by Unicode numbers: Unfortunately, the glyph repertoire does not Open($3);Select("ash.akk");Copy();Close();\ correspond to Unicode because, first, more than 300 Open($1);Select("u12501");Paste(); glyphs do not have Unicode code points, and, on Save($1);Close(); the other hand, my fonts cover only about 20% of Open($3);Select("hal.akk");Copy();Close();\ the Unicode Sumerian-Akkadian cuneiform range Open($1);Select("u12502");Paste(); (cuneiform signs and numeric signs). Save($1);Close(); In the final OpenType fonts, PostScript glyph Open($3);Select("mug.akk");Copy();Close();\ Open($1);Select("u12503");Paste(); names are omitted, the Akkadian glyph container Save($1);Close(); (OTF/a) contains all Neo-Assyrian glyphs (according ... to Labat), partly defined by references in Unicode A FontForge user command eliminates the repetition: cuneiform block (OTF/c). Here is a table showing some of the correspondences: #!/usr/bin/fontforge # copy.pe # SF source font, SG source glyph PostScript OTF/a OTF/c # DF destination font, DG dest. glyph ash.akk 12501 u12038 Open($1);Select($2);Copy();Close(); hal.akk 12502 u1212C Open($3);Select($4);Paste(); mug.akk 12503 u1222E Save($3); zadim.akk 12504 with references for the Neo-Assyrian block: ba.akk 12505 u12040 zu.akk 12506 u1236A # $SF is source font su.akk 12507 u122E2 # $SFD temporary font (internal) shun.akk 12508 ./copy.pe $SF "ash.akk" $SFD u12501 bal.akk 12509 u12044 ./copy.pe $SF "hal.akk" $SFD u12502 adII.akk 1250A ./copy.pe $SF "mug.akk" $SFD u12503 bulII.akk 1250B ... tar.akk 1250C u122FB The Neo-Assyrian glyphs are allocated in the an.akk 1250D u1202D container; existing glyphs are linked from the Cunei- ka.akk 1250F u12157 form range by references: 3.1 Hinting Select("u12743");CopyReference(); Select("u12000");Paste(); The OpenType output was “satisfactory” as auto- Select("u1264E");CopyReference(); hinted with FontForge (Fig. 1); no hinting instruc- Select("u12009");Paste(); tions are included in the TrueType fonts.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 423 Karel Píška
The two following examples show font usage. 1. Glyph index, numbers correspond to Labat [1]: a a a: 579/1 (579-NAc67) a’ aI: 397 (397-NAb141) á N aII: 334 (334-NAb78) à π aIII: 383 (383-NAb127) see pi a4 F aIIII: 579/2 (582-NAc70) see àm a7 M aVII: 589 (589-NAc77) see ha aa K aa: 579/6 (587-NAc75) ˘ ab « ab: 128 (128-NAa128) áb ¤ abII: 420 (420-NAb164) see lid ablal ablal: 525 (525-NAc13) ad ad: 145 (145-NAa145) ád adII: 10 (10-NAa10) 2. Sample text in Akkadian (with transliteration): aFnνFaæ>V[ 7aV ] \a\na \kur\nu\giIIII \a \qaq\qa\ri [\ \la \ta\a\ri \ ] a.na kur.nu.gi4 a qaq.qa.ri [ la ta.a.ri ] g* ȓ>8U [ ÔM ] \DETd\innana\dumu\miII \DETd\sin \uII\zu\un\shaII \ [\ \ish\kun\ ] dinnana.dumu.mí dsin ú.zu.un.šá [ iš.kun ] ÔMV* ȓ>8[ш] \ish\kun\ma \dumu\miII \DETd\sin \uII\zu\un[\sha] Figure 1: Wedge design in three variants with iš.kun.ma dumu.mí dsin ú.zu.un.[ša] hinting. aFDeÉeb èB7 \a\na \eII \e\tte\e \shu\bat \DETd\ir\kal\la d a.na é e.t.e.e šu.bat ir.kal.la 4 Support for (pdf)LATEX and X TE EX aFDшeVs!7a+> \a\na \eII \sha \e\ri\bu\shuII \la \a\ssu\uII Old and simple LATEX macros for Type 1 fonts and a.na é ša e.ri.bu.šú la a.s.u.ú (pdf)LATEX were modified for X E LATEX to bind sym- aFHçшa:U7aa \a\na \har\ra\ni \sha \a\lak\ta\shaII \la \ta\a\a\rat bolic glyph names using their Unicode numbers. a.na har.ra.ni ša a.lak.ta.šá la ta.a.a.rat ˘ \def\NAfontC#1{% xakkadian.sty aFDшeVs!µ>ν>H \a\na \eII \sha \e\ri\bu\shuII \zu\um\mu\uII \nu\uII\ra \font\NAC="[nacunc.ttf]" at #1pt a.na é ša e.ri.bu.šú zu.um.mu.ú nu.ú.ra } aÔsstνaBbνÉN \def\NAfont{\NAfontC{10}} % default font \a\shar \sahar\haII \bu\bu\us\su\nu \a\kal\shu\nu \tti\itt\ttu a.šar sahar.há bu.bu.us.su.nu a.kal.šu.nu t.i.it..t.u % ˘ ˘ \def\AKK{% Line 1: Akkadian text using the cuneiform font Line 2: The corresponding source input in the LATEX command {\AKK source } \NAfont% Line 3: Transliteration (dots and spaces added manually) \def\ash{{\NAC\char"12501}}% \let\dil\ash\let\tilIIII\ash\let\ttil\ash% 5 Conclusion \let\rum\ash\let\ruIII\ash\let\ina\ash% Both METAFONT and MetaType1 (=METAPOST) \let\asIII\ash\let\azIII\ash% are programmable. But METAFONT produces only \def\hal{{\NAC\char"12502}}\let\buluh\hal% bitmaps, while in MetaType1, we must not define \def\mug{{\NAC\char"12503}}% areas to fill or unfill with bitmap matrices which \let\muk\mug\let\muq\mug\let\puk\mug% would depend on the device (resolution, blacker and \def\zadim{{\NAC\char"12504}}% other parameters). Rather, we are restricted to out- \def\ba{{\NAC\char"12505}}% lines: \let\paII\ba% • \def\zu{{\NAC\char"12506}}% glyphs must be defined by closed curves, i.e. \let\ssuII\zu% sequences of splines; ... • we produce the Type 1 format directly; • the MetaType1 commands Fill/Unfill denote the output of curves in the PostScript Type 1
424 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Creating cuneiform fonts with MetaType1 and FontForge
representation with proper path direction and 6 Acknowledgements correct order of spline segments; I want to thank the authors of MetaType1, FontForge • final filling/unfilling is delegated to the Post- (G. Williams) and other developers and maintainers Script/PDF rasterization systems. of free and open source software. Between PFB (Type 1) and OTF (PS/CFF fla- vored OpenType) we can find only formal differences References in internal representation and organization; the math- [1] René Labat. Manuel d’épigraphie akkadienne. ematical outline curves and hints are identical. On Troisième édition. Imprimerie nationale, Paris, the other hand, OTF and TTF (TrueType flavored 1959. OpenType) may differ in approximation of curve [2] F. Thureau-Dangin. Le syllabaire accadien. segments, since the underlying representations use Librairie Orientaliste Paul Geuthner, Paris, cubic and quadratic polynomials, respectively. For 1926. our simple cuneiform design of wedges, though, a [3] The World’s Writing Systems. P. T. Daniels common approximation is workable. X LAT X can E E and W. Bright, eds. Oxford University Press, read all font formats: T X fonts, OTF, TTF, etc.; E New York–Oxford, 1996. OpenOffice 2.3 (on my computer) can work only with TTF. (I can say nothing about MS Word because I [4] Karel Píška. Fonts for Neo-Assyrian do not have this product.) Cuneiform. Proceedings of the EuroTEX MetaType1 and FontForge give the advantage Conference (Paperless TEX), Heidelberg, of programmability with open source data. In Font- Germany, September 20–24, 1999, Günter Forge, the interactive approach in glyph design is Partosch and Gerhard Wilhelms, eds. Gießen, dominant; theoretically we could define glyph out- Augsburg, 1999, ISSN 1438-9959, 142–154. lines in the FontForge scripting language but it would http://www-hep.fzu.cz/~piska/cuneiform. be very difficult and inefficient. METAFONT/META- html POST (MetaType1) are more flexible and modular: [5] Cuneiform script (Wikipedia). http://en. they allow for solving mathematical equations, com- wikipedia.org/wiki/Cuneiform_script mon processing and maintenance of related fonts, [6] Bogusław Jackowski, Janusz M. Nowacki, automatic calculation of parameters, and systematic Piotr Strzelczyk. Programming PostScript modifications. Type 1 fonts using MetaType1: Auditing, A typical task for MetaType1 is to combine a enhancing, creating. Proceedings of EuroTEX small number of components into many composite 2003, Brest, France, 24–27 June 2003. glyphs uniformly. This is common for “special kinds TUGboat 24:3, pp. 575–581, http://tug.org/ of fonts”: just as Latin Modern and TEX Gyre can TUGboat/Articles/tb24-3/jackowski.pdf; combine letters + accents, the cuneiform fonts can CTAN:fonts/utilities/metatype1; combine wedges; operations to produce composite ftp://bop.eps.gda.pl/pub/metatype1. glyphs can be defined and applied in a simple way, [7] Klaus Höppner. Creation of a PostScript and generation and maintenance can be repeated for Type 1 logo font with MetaType1. Proceedings numerous fonts. of XVII European TEX Conference, 2007. The older non-Unicode versions of cuneiform TUGboat 29:1, pp. 34–38, http://tug.org/ fonts have been already referenced in the subsection TUGboat/Articles/tb29-1/tb91hoeppner. “External links / Fonts” in http://en.wikipedia. pdf. org/wiki/Cuneiform_script (a web search for “cu- [8] George Williams. Font creation with neiform” should find it also). They have been already FontForge. EuroTEX 2003 Proceedings, used by scholars; e.g. for syllabaries and computer TUGboat 24:3, pp. 531–544, http://tug.org/ transliteration of sample texts for students. TUGboat/Articles/tb24-3/williams.pdf; Now I plan to finish and publish the new “Uni- http://fontforge.sourceforge.net. code” version, by extending the glyph repertoire [9] Free Software Foundation. GNU awk, to other glyphs and other shapes, corresponding to http://www.gnu.org/software/gawk. other languages and their historical period. Prelimi- nary experimental OpenType fonts are available on [10] Eddie Kohler. t1utils (Type 1 tools), my web site [11]. http://freshmeat.net/projects/t1utils. My final wish is that the MetaType1 package [11] http://www-hep.fzu.cz/~piska/cuneiform/ would be extended to “MetaOpenType” to produce opentype.html OpenType font formats directly.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 425 Do we need a ‘Cork’ math font encoding?
Ulrik Vieth Vaihinger Straße 69 70567 Stuttgart Germany ulrik dot vieth (at) arcor dot de
Abstract The city of Cork has become widely known in the TEX community, ever since it gave name to an encoding developed at the European TEX conference of 1990. The ‘Cork’ encoding, as it became known, was the first example of an 8-bit text font encoding that appeared after the release of TEX 3.0, and was later followed by a number of other encodings based on similar design principles. As of today, the ‘Cork’ encoding represents only one out of several possible choices of 8-bit subsets from a much larger repertoire of glyphs provided in fonts such as Latin Modern or TEX Gyre. Moreover, recent developments of new TEX engines are making it possible to take advantage of OpenType font technology directly, largely eliminating the need for 8-bit font encodings altogether. During the entire time since 1990 math fonts have always been lagging behind the developments in text fonts. While the need for new math font encodings was recognized early on and while several encoding proposals have been discussed, none of them ever reached production quality or became widely used. In this paper, we review the situation of math fonts as of 2008, especially in view of recent developments of Unicode and OpenType math fonts such as the STIX fonts or Cambria Math. In particular, we try to answer the question whether a ‘Cork’ math font encoding is still needed or whether Unicode and OpenType might eliminate the need for TEX-specific math font encodings.
1 History and development of text fonts was the first example of a standard 8-bit text font 1.1 The ‘Cork’ encoding and thus became the T1 encoding. While the ‘Cork’ encoding was certainly an im- When the 5th European TEX conference was held in portant achievement, it also introduced some novel Cork in the summer of 1990, the TEX community features that may have seemed like a good idea at was undergoing a major transition phase. TEX 3.0 that time but would be seen as shortcomings or had just been released that year, making it possible problems from today’s point of view, after nearly to switch from 7-bit to 8-bit font encodings and to two decades of experience with font encodings. support hyphenation for multiple languages. In retrospect, the ‘Cork’ encoding represents Since the ability to properly typeset and hy- a typical example of the TEX-specific way of doing phenate accented languages strongly depended on things of the early 1990s without much regard for overcoming the previous limitations, European T X E standards or technologies outside the TEX world. users wanted to take advantage of the new features Instead of following established standards, such and started to work on new font encodings [1, 2, 3]. as using ISO Latin 1 or 2 or some extended versions As a result, they came up with an encoding that for Western and Eastern European languages, the became widely known as the ‘Cork’ encoding, named ‘Cork’ encoding tried to support as many languages after the site of the conference [4]. as possible in a single font encoding, filling the 8-bit The informal encoding name ‘Cork’ stayed in A font table to the limit with accented characters at use for many years, even after LTEX 2ε and NFSS2 the expense of symbols. Since there was no more introduced a system of formal encoding names in room left in the font table, typesetting symbols at n 1993–94, assigning OT for 7-bit old text encodings, first had to be taken from the old 7-bit fonts, until n n T for 8-bit standard text encodings, and L for local a supplementary text symbol TS1 encoding [6] was or non-standard encodings [5]. The ‘Cork’ encoding introduced in 1995 to fill the gap.
426 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Do we need a ‘Cork’ math font encoding?
When it came to implementing the T1 and TS1 beyond the limits of 8-bit fonts towards Unicode and encodings for PostScript fonts, it turned out that the OpenType font technology. encodings were designed without taking into account the range of glyphs commonly available in standard 2.2 Unicode support in new TEX engines PostScript fonts. As we are approaching the TUG 2008 conference at Both font encodings could only be partially im- Cork, the TEX community is again undergoing a ma- plemented with glyphs from the real font, while the jor transition phase. While TEX itself remains frozen remaining slots either had to be faked with virtual and stable, a number of important developments fonts or remain unavailable. At the same time, none have been going on in recent years. of the encodings provided access to the full set of Starting with the development of PDFTEX since available glyphs from the real font. the late 1990s the use of PDF output and scalable PostScript or TrueType fonts has largely replaced 1.2 Alternatives to the ‘Cork’ encoding the use of DVI output and bitmap PK fonts. As an alternative to using the T1 and TS1 encodings Followed by the ongoing development of X TE EX for PostScript fonts, the TeXnANSI or LY1 encoding and LuaTEX in recent years the use of Unicode and was proposed [7], which was designed to provide ac- OpenType font technology is also starting to replace cess to the full range of commonly available symbols the use of 8-bit font encodings as well as traditional (similar to the TeXBase1 encoding), but also matched PostScript or TrueType font formats. the layout of the OT1 encoding in the lower half, so Putting everything together, the development that it could be used as drop-in replacement without of new fonts and new TEX engines in recent years any need for virtual fonts. has enabled the TEX community to catch up with In addition to that, a number of non-standard developments of font technology in the publishing encodings have come into use as local alternatives industry and to prepare for the future. to the ’Cork’ encoding, such as the Polish QX, the The only thing still missing (besides finishing Czech CS, and the Lithuanian L7X encoding, each of the ongoing development work) is the development them trying to provide better solutions for the needs of support for Unicode math in the new TEX engines of specific languages. and the development of OpenType math fonts for In summary, the ‘Cork’ encoding as the first Latin Modern and TEX Gyre. example of an 8-bit text encoding (T1) was not only followed by additional encodings based on the same 3 History and development of math fonts design principles for other languages (Tn ), but also When TEX was first developed in 1977–78, the 7-bit supplemented by a text symbol encoding (TS1) and font encodings for text fonts and math fonts were complemented by a variety of local or non-standard developed simultaneously, since both of them were encodings (LY1, QX, CS, etc.). needed for typesetting mathematical textbooks like As became clear over time, the original goal of The Art of Computer Programming. the ‘Cork’ encoding of providing a single standard When TEX 3.0 made it possible to switch from encoding for as many languages as possible couldn’t 7-bit to 8-bit font encodings, it was the text fonts possibly be achieved within the limits of 8-bit fonts, driving these new developments while the math fonts simply because there are far too many languages and remained largely unchanged. symbols to consider, even when limiting the scope As a result, the development of math fonts has to Latin and possibly Cyrillic or Greek. been lagging behind the corresponding text fonts for nearly two decades now, ever since the development 2 Recent developments of text fonts of the ‘Cork’ encoding started in 1990. 2.1 Unicode support in new TEX fonts In principle, a general need for new math fonts It was only in recent years that the development was recognized early on: When the first implementa- tions of ‘Cork’ encoded text fonts became available, of the Latin Modern [8, 9, 10] and TEX Gyre fonts [11, 12] has provided a consistent implementation for it was soon discovered that the new 8-bit text fonts all the many choices of encodings. couldn’t fully replace the old 7-bit text fonts with- As of today, the ‘Cork’ encoding represents only out resolving the inter-dependencies between text one out of several possible 8-bit subsets taken from and math fonts. In practice, however, nothing much a much larger repertoire of glyphs. The full set of happened since there was no pressing need. glyphs, however, can be accessed only when moving
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 427 Ulrik Vieth
3.1 The ‘Aston’ proposal In the end, the Unicode math efforts continued The first bit of progress was made in the summer of over several years until the symbols were accepted [19, A 20] and several more years until an implementation 1993, when the LTEX3 Project and some TEX users group sponsored a research student to work on math of a Unicode math font was commissioned [21] by font encodings for a few months. a consortium of scientific and technical publishers, As a result, a proposal for the general layout known as the STIX Project. of new 8-bit math font encodings was developed When the first beta-test release of the so-called and presented at TUG 1993 at Aston University [13]. STIX fonts [22] finally became available in late 2007, Unlike the ‘Cork’ encoding, which became widely nearly a decade had passed without making progress known, this ‘Aston’ proposal was known only to on math font encodings for TEX. some insiders and went largely unnoticed. While the STIX fonts provide all the building After only a few months of activity in 1993 the blocks of Unicode math symbols, they are still lacking project mailing list went silent and nothing further TEX support and may yet have to be repackaged in happened for several years, even after a detailed a different way to turn them into a usable font for A use with T X or other systems. report was published as a LTEX3 Project Report E [14]. Despite the progress on providing the Unicode math symbols, the question of how to encode all 3.2 The ‘newmath’ prototype the many Unicode math symbols in a set of 8-bit The next bit of progress was made in 1997–98, when font encodings for use with traditional TEX engines the ideas of the ‘Aston’ proposal were taken up again still remains unresolved. Most likely, only a subset and work on an implementation was started. of the most commonly used symbols could be made This time, instead of just discussing ideas or available in a set of 8-bit fonts, whereas the full range preparing research documents, the project focussed of symbols would be available only when moving to on developing a prototype implementation of new Unicode and OpenType font technology. math fonts for several font families using a mixture 4.2 OpenType math in MS Office 2007 of METAFONT and fontinst work [15]. When the results of the project were presented While the TEX community and the consortium of scientific publishers were patiently awaiting the re- at the EuroTEX 1998 conference [16], the project was making good progress, although the results were still lease of the STIX fonts before reconsidering the topic very preliminary and far from ready for production. of math font encodings, outside developments have Unfortunately, the project then came to a halt continued to move on. In particular, Microsoft has soon after the conference when other activities came moved ahead and has implemented its own support to the forefront and changed the scope and direction for Unicode math in Office 2007. of the project [17, 18]. They did so by adding support for math type- Before the conference, the goal of the project had setting in OpenType font technology [23, 24] and been to develop a set of 8-bit math font encodings by commissioning the design of the Cambria Math font as an implementation of an OpenType math for use with traditional TEX engines (within the constraints of 16 families of 256 glyphs) and also to font [25, 26, 27]. In addition, they have also adopted provide some example implementations by means of an input language called ‘linear math’ [28], which is reencoding and enhancing existing font sets. strongly based on TEX concepts. After the conference, that goal was set aside and While OpenType math is officially still consid- put on hold for an indefinite time by the efforts to ered experimental and not yet part of the OpenType bring math into Unicode. specification [29], it is already a de facto standard, not only because it has been deployed to millions 4 Recent developments of math fonts of installations of Office 2007, but also because it 4.1 Unicode math and the STIX fonts has already been adopted by other projects, such as the FontForge font editor [30] and independent font While the efforts to bring math into Unicode were designs such as Asana Math [31]. certainly very important, they also brought along a In addition, the next release of the STIX fonts lot of baggage in the form of a very large number of scheduled for the summer of 2008 is also expected additional symbols, making it much more work to to include support for OpenType math. provide a reasonably complete implementation and nearly impossible to encode all those symbols within the constraints of traditional TEX engines.
428 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Do we need a ‘Cork’ math font encoding?
4.3 OpenType math in new TEX engines In the case of OpenType math, Microsoft has At the time of writing, current development versions used its powers as one of the vendors controlling the specification to implement an extension of the of X TE EX have added some (limited) support for OpenType math, so it is already possible to use OpenType format and declare it as ‘experimental’ until they see fit to release it. Fortunately, Microsoft fonts such as Cambria Math in X TE EX [32], and this OpenType math support will soon become available was smart enough to borrow from the best examples of math typesetting technology when they designed to the TEX community at large with the upcoming OpenType math, so they chose T X as a model for release of TEX Live 2008. E many of the concepts of OpenType math. Most likely, LuaTEX will also be adding support for OpenType math eventually, so OpenType math 5.2 The details of OpenType math is likely to become a de facto standard in the TEX world as well, much as we have adopted other outside The OpenType MATH table One of the most developments in the past. distinctive features of an OpenType math font is the presence of a MATH table. This table contains a 4.4 OpenType math for new TEX fonts? number of global font metric parameters, much like Given these developments, the question posed in the the \fontdimen parameters of math fonts in TEX title of this paper about the need for new math font described in Appendix G of The TEXbook. encodings may soon become a non-issue. In a traditional TEX setup these parameters are If we decide to adopt Unicode and OpenType essential for typesetting math, controlling various aspects such as the spacing of elements such as big math font technology in new TEX engines and new fonts, the real question is no longer how to design operators, fractions, and indices [34, 35]. the layout of encoding tables but rather how to deal In an OpenType font the parameters of the with the technology of OpenType math fonts, as we MATH table have a similar role for typesetting math. will discuss in the following sections. From what is known, Microsoft apparently consulted with Don Knuth about the design of this table, so 5 Future developments in math fonts the result is not only similar to TEX, but even goes 5.1 Some background on OpenType math beyond TEX by adding new parameters for cases where hard-wired defaults are applied in TEX. The OpenType font format was developed jointly In the X TE EX implementation the parameters by Microsoft and Adobe, based on concepts adopted of the OpenType MATH table are mapped internally from the earlier TrueType and PostScript formats. to TEX’s \fontdimen parameters. In most cases this The overall structure of OpenType fonts shares the mapping is quite obvious and straight-forward, but extensible table structure of TrueType fonts, adding unfortunately there are also a few exceptions where support for different flavors of glyph descriptions in 1 some parameters in TEX do not have a direct cor- either PostScript CFF or TrueType format. respondence in OpenType. It is not clear, however, One of the most interesting points about Open- whether these omissions are just an oversight or a Type is the support for ‘advanced’ typographic fea- deliberate design decision in case a parameter was tures, supporting a considerable amount of intelli- deemed irrelevant or unnecessary. gence in the font, enabling complex manipulations Support for OpenType math in X TE EX still re- of glyph positioning or glyph substitutions. At the mains somewhat limited for precisely this reason; user level, many of these ‘advanced’ typographic fea- until the mapping problems are resolved, X TE EX tures can be controlled selectively by the activation has to rely on workarounds to extract the necessary of so-called OpenType feature tags. parameters from the OpenType MATH table. Despite its name, the OpenType font format At the time of writing, the extra parameters is not really open and remains a vendor-controlled introduced by OpenType generalizing the concepts specification, much like the previous TrueType and of TEX have been silently ignored. It is conceivable, PostScript font formats developed by these vendors. however, that future extensions of new TEX engines The official OpenType specification is published on might eventually start to use these parameters in the a Microsoft web site at [29], but that version may math typesetting algorithms as well. not necessarily reflect the latest developments. In the end, whatever technology is used to type- set OpenType math, it remains the responsibility of the font designer to set up the values of all the many 1 An extensive documentation of the OpenType format and its features as well as many other important font formats parameters affecting the quality of math typesetting. can be found in [33]. Unfortunately, for a non-technical designer such a
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 429 Ulrik Vieth
task feels like a burden, which is better left to a In a traditional TEX setup the math setup con- technical person as a font implementor. sists of a series of 8-bit fonts organized into families. For best results, it is essential to develop a good Typically, each font will contain one set of alphabets understanding of the significance of the parameters in a particular style and a selection of symbols filling and how they affect the quality of math typesetting. the remaining slots. In [35] we have presented a method for setting up In a Unicode setup the math setup will consist the values of metric parameters of math fonts in TEX. of only one big OpenType font, containing all the For OpenType math fonts, we would obviously have math symbols and operators in the relevant Unicode to reconsider this procedure. slots, as well as all the many styles of math alphabets Font metrics of math fonts Besides storing the assigned to slots starting at U+1D400. global font metric parameters, the OpenType MATH As a result, there will be several important con- table is also used to store additional glyph-specific ceptual implications to consider in the design and information such as italic corrections or kern pairs, implementation of OpenType math fonts, such as as well as information related to the placement of how to handle font switches of math alphabets, how math accents, superscripts and subscripts. to include the various sizes of big operators, delim- iters, or radicals, or how to include the optical sizes In a traditional TEX setup the font metrics of math fonts have rather peculiar properties, because of superscripts and subscripts. much of the glyph-specific information is encoded Handling of math alphabets In a traditional or hidden by overloading existing fields in the TFM TEX setup the letters of the Latin and Greek alpha- metrics in an unusual or non-intuitive way [36]. bets are subject to font switches between the various For example, the width in the TFM metrics is math families, usually containing a different style in not the real width of the glyph. Instead, it is used to each family (roman, italic, script, etc.). indicate the position where to attach the subscript. In a Unicode setup each style of math alphabets Similarly, the italic correction is used to indicate the has a different range of slots assigned to it, since offset between subscript and superscript. each style is assumed to convey a different meaning. As another example, fake kern pairs involving a When dealing with direct Unicode input, this skewchar are used to indicate how much the visual might not be a problem, but when dealing with center of the glyphs is skewed in order to determine traditional TEX input, quite a lot of setup may be the position where to attach a math accent. needed at the macro level to ensure that input such In OpenType math fonts all such peculiarities as \mathrm{a} or \mathit{a} or \mathbf{a} will will become obsolete, as the MATH table provides be translated to the appropriate Unicode slots. data structures to store all the glyph-specific metric An additional complication arises because the information in a much better way. In the case of Unicode code points assigned to the math alphabets indices, OpenType math has extended the concepts are non-contiguous for historical reasons [37]. While of TEX by defining ‘cut-ins’ at the corners on both most of the alphabetic letters are taken from one big sides of a glyph and not just to the right. block starting at U+1D400, a few letters which were Unfortunately, while the conceptual clarity of part of Unicode already before the introduction of OpenType math may be very welcome in principle, Unicode math have to be taken from another block it may cause an additional burden on font designers starting at U+2100. developing OpenType math fonts based on tradi- An example implementation of a LATEX macro tional TEX fonts (such as the Latin Modern fonts) package for X TE EX to support OpenType math is and trying to maintain metric compatibility. already available [32], and it shows how much setup In such cases it may be necessary to examine the is needed just to handle math alphabets. Fortunately, metrics of each glyph and to translate the original such a setup will be needed only once and will be metrics into appropriate OpenType metrics. applicable for all Unicode math fonts, quite unlike Font encoding and organization The encoding the case of traditional TEX fonts where each set of of OpenType fonts is essentially defined by Unicode math fonts requires its own macro package. code points. Most likely, a typical OpenType math Handling of size variants Ever since the days font will include only a subset of Unicode limited to of DVI files and PK fonts, TEX users have been ac- the relevant ranges of math symbols and alphabets, customed to thinking of font encodings in terms of while the corresponding text font may contain a numeric slots in an encoding table, usually assuming bigger range of scripts. a 1:1 mapping between code points and glyphs.
430 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Do we need a ‘Cork’ math font encoding?
However, there have always been exceptions to activate them by a standard OpenType substitution this rule, most notably in the case of a math extension mechanism using a feature tag ssty=0 and ssty=1, font, where special TFM features were used to set up not much different from the standard substitutions a linked list from one code point to a series of next- for small caps or oldstyle figures in text fonts. larger glyph variants representing different sizes of It is important to note that the optical design operators, delimiters, radicals, or accents, optionally variants intended for use in first and second level followed by an extensible version. scripts, using proportions adjusted for smaller sizes, In a traditional TEX font each glyph variant has are nevertheless provided at the basic design size a slot by itself in the font encoding, even if it was and subsequently scaled down using a scaling factor addressed only indirectly. defined in the OpenType MATH table. In an OpenType font, however, the font encod- If an OpenType math font lacks optical design ing is determined by Unicode code points, so the variants for script sizes and does not support the additional glyph variants representing different sizes ssty feature tag, a scaled-down version of the basic cannot be addressed directly by Unicode code points design size will be used automatically. The same will and have to remain unencoded, potentially mapped also apply to non-alphabetic symbols. to the Unicode private use area, if needed. Use of OpenType feature tags Besides using While the conceptual ideas of vertical and hori- OpenType feature tags for specific purposes in math zontal variants and constructions in the OpenType fonts, most professional OpenType text fonts also use MATH table are very similar to the concepts of char- feature tags for other purposes, such as for selecting lists and extensible recipes in TEX font metrics, it is small caps or switching between oldstyle and lining interesting to note that OpenType has generalized figures. Some OpenType fonts may provide a rich these concepts a little bit. set of features, such as a number of stylistic variants, While TEX supports extensible recipes only in initial and final forms, or optical sizes. a vertical context of big delimiters, OpenType also Ultimately, it remains to be seen how the use of supports horizontal extensible constructions, so it OpenType feature tags will influence the organization would be possible to define an extensible overbrace of OpenType fonts for TEX, such as Latin Modern or underbrace in the font, rather than at the macro or TEX Gyre, not just concerning new math fonts, level using straight line segments for the extensible but also existing text fonts. parts. In addition, the same concept could also be So far, the Latin Modern fonts have very closely applied to arbitrarily long arrows. followed the model of the Computer Modern fonts, Optical sizes for scripts In a traditional TEX providing separate fonts for each design size and each setup math fonts are organized into families, each font shape or variant. of them consisting of three fonts loaded at different While it might well be possible to eliminate design sizes representing text style and first and some variants by making extensive use of OpenType second level script style. feature tags, such as by embedding small caps into If a math font provides optical design sizes, such the roman fonts, implementing such a step would as in the case of traditional METAFONT fonts, these imply an important conceptual change and might fonts are typically loaded at sizes of 10 pt, 7 pt, 5 pt, cause unforeseen problems. each of them having different proportions adjusted Incorporating multiple design sizes into a single for improved readability at smaller sizes. font might have similar implications, but the effects If a math font doesn’t provide optical sizes, such might be less critical if they are limited to the well- as in the case of typical PostScript fonts, scaled-down controlled environment of math typesetting. versions of the 10 pt design size will have to make do, In the TEX Gyre fonts the situation is somewhat but in such cases it may be necessary to use bigger simpler, because these fonts are currently limited to sizes of first and second level scripts, such as 10 pt, the basic roman and italic fonts and do not have 7.6 pt, 6 pt, since the font may otherwise become too small caps variants or optical sizes. unreadable at such small sizes. Incorporating a potential addition of small caps In OpenType math the concept of optical sizes in TEX Gyre fonts by means of OpenType feature from TEX and METAFONT has been adopted as well, tags might well be possible without causing any in- but it is implemented in a different way, typical for compatible changes. Similarly, incorporating some OpenType fonts. Instead of loading multiple fonts expanded design variants with adjusted proportions at different sizes, OpenType math fonts incorporate for use in script sizes would also be conceivable when the multiple design variants in the same font and designing TEX Gyre math fonts.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 431 Ulrik Vieth
5.3 The impact of OpenType math testing and fine-tuning to achieve optimal placement As we have seen in the previous sections, Open- of math accents and indices. Type math fonts provide a way of embedding all the Finally, there will be the question of assembling relevant font-specific and glyph-specific information the many diverse elements that have to be integrated needed for high-quality math typesetting. in a comprehensive OpenType math font. So far, In many aspects, the concepts of OpenType the various styles of math alphabets and the various optical sizes of these alphabets have been designed math are very similar to TEX or go beyond TEX. However, the implementation of these concepts in as individual fonts, but in OpenType all of them OpenType fonts will be different in most cases. have to be combined in a single font. Moreover, the Given the adoption of OpenType math as a optical sizes will have to be set up as substitutions de facto standard and its likelihood of becoming an triggered by OpenType feature tags. official standard eventually, OpenType math seems 6 Summary and conclusions to be the best choice for future developments of new In this paper we have reviewed the work on math math fonts for use with new TEX engines. font encodings since 1990 and the current situation While X TE EX has already started to support of math fonts as of 2008, especially in view of re- OpenType math and LuaTEX is very likely to follow, adopting OpenType for the design of math fonts for cent developments in Unicode and OpenType font technology. In particular, we have looked in detail Latin Modern or TEX Gyre will take more time and will require developing a deeper understanding of the at the features of OpenType math in comparison to concepts and data structures. the well-known features of TEX’s math fonts. Most importantly, however, it will also require While OpenType math font technology looks rethinking many traditional assumptions about the very promising and seems to be the best choice for way fonts are organized. future developments of math fonts, it also presents Thus, while the topic of font encodings of math many challenges that will have to be met. fonts may ultimately become a non-issue, the topic While support for OpenType math in new TEX of font technology will certainly remain important. engines has already started to appear, the develop- ment of math fonts for Latin Modern or TEX Gyre 5.4 The challenges of OpenType math using this font technology will not be easy and will Developing a math font has never been an easy job, take considerable time. so attempting to develop a full-featured OpenType In the past, the TEX conference in Cork in 1990 was the starting point for major developments in text math font for Latin Modern or TEX Gyre certainly presents a major challenge to font designers or font fonts, which have ultimately led to the adoption of implementors for a number reasons. Unicode and OpenType font technology. First, such a math font will be really large, even Hopefully, the TEX conference at Cork in 2008 in comparison with text fonts, which already cover might become the starting point for major develop- a large range of Unicode.2 It will have to extend ments of math fonts in a similar way, except that this across multiple 16-bit planes to account for the slots time there will be no more need for a new encoding of the math alphabets starting at U+1D400, and it that could be named after the site of the conference. will also require a considerable number of unencoded Acknowledgements glyphs to account for the size variants of extensible glyphs and the optical variants of math alphabets. The author wishes to acknowledge feedback, sugges- Besides the size of the font, such a project will tions, and corrections from some of the developers also present many technical challenges in dealing of projects discussed in this review. with the technology of OpenType math fonts. A preprint of this paper has been circulated While setting up the font-specific parameters of on the Unicode math mailing list hosted at Google the OpenType MATH table is comparable to setting Groups [38] and future discussions on the topics of this paper are invited to be directed here. up the \fontdimen parameters of TEX’s math fonts, setting up the glyph-specific information will require References detailed attention to each glyph as well as extensive [1] Yannis Haralambous: TEX and Latin alphabet languages. TUGboat, 10(3):342–345, 1989. http://www.tug.org/TUGboat/Articles/tb10-3/ 2 In the example of the Cambria Math font, the math font tb25hara-latin.pdf is reported to have more than 2900 glyphs compared to nearly 1000 glyphs in the Cambria text font.
432 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Do we need a ‘Cork’ math font encoding?
[2] Nelson Beebe: Character set encoding. [12] Jerzy B. Ludwichowski, Bogusław Jackowski, TUGboat, 11(2):171–175, 1990. Janusz M. Nowacki: Five years after: Report http://www.tug.org/TUGboat/Articles/tb11-2/ on international TEX font projects. TUGboat, tb28beebe.pdf 29(1):25–26, 2008. Proceedings of the 17th [3] Janusz S. Bień: On standards for CM font European TEX Conference 2007, Bachotek, extensions. TUGboat, 11(2):175–183, 1990. Poland. http://www.tug.org/TUGboat/Articles/tb11-2/ http://www.tug.org/TUGboat/Articles/tb29-1/ tb28bien.pdf tb91ludwichowski-fonts.pdf [4] Michael Ferguson: Report on multilingual [13] Alan Jeffrey: Math font encodings: A activities. TUGboat, 11(4):514–516, 1990. workshop summary. TUGboat, 14(3):293–295, http://www.tug.org/TUGboat/Articles/tb11-4/ 1993. Proceedings of the TUG 1993 Conference, tb30ferguson.pdf Aston University, Birmingham, UK. http://www.tug.org/TUGboat/Articles/tb14-3/ [5] Frank Mittelbach, Robin Fairbairns, Werner tb40mathenc.pdf Lemberg: LAT X font encodings, 2006. E [14] Justin Ziegler: Technical report on math font http://www.ctan.org/tex-archive/macros/ encodings. LAT X3 Project Report, 1993. latex/doc/encguide.pdf E http://www.ctan.org/tex-archive/info/ [6] Jörg Knappen: The release 1.2 of the Cork ltx3pub/processed/l3d007.pdf encoded DC fonts and text companion fonts. [15] Math Font Group (MFG) web site, archives, TUGboat, 16(4):381–387, 1995. Reprint from papers, and mailing list. the Proceedings of the 9th European T X E http://www.tug.org/twg/mfg/ Conference 1995, Arnhem, The Netherlands. http://www.tug.org/twg/mfg/archive/ http://www.tug.org/TUGboat/Articles/tb16-4/ http://www.tug.org/twg/mfg/papers/ tb49knap.pdf http://www.tug.org/mailman/listinfo/ [7] Berthold K. P. Horn: The European Modern math-font-discuss fonts. TUGboat, 19(1):62–63, 1998 [16] Matthias Clasen, Ulrik Vieth: Towards a new http://www.tug.org/TUGboat/Articles/tb19-1/ A Math Font Encoding for (L )TEX. Cahiers tb58horn.pdf GUTenberg, 28–29:94–121, 1998. Proceedings [8] Bogusław Jackowski, Janusz M. Nowacki: of the 10th European TEX Conference 1998, Latin Modern: Enhancing Computer Modern St. Malo, France. with accents, accents, accents. TUGboat, http://www.gutenberg.eu.org/pub/GUTenberg/ 24(1):64–74, 2003. Proceedings of the publicationsPDF/28-29-clasen.pdf TUG 2003 Conference, Hawaii, USA. [17] Ulrik Vieth et al.: Summary of math http://www.tug.org/TUGboat/Articles/tb24-1/ font-related activities at EuroTEX 1998. jackowski.pdf MAPS, 20:243–246, 1998. [9] Bogusław Jackowski, Janusz M. Nowacki: http://www.ntg.nl/maps/20/36.pdf Latin Modern: How less means more. [18] Ulrik Vieth: What is the status of new math TUGboat, 27(0):171–178, 2006 Proceedings font encodings? Posting to mailing list, 2007. of the 15th European TEX Conference 2005, http://www.tug.org/pipermail/ Pont-à-Mousson, France. math-font-discuss/2007-May/000068.html http://www.tug.org/TUGboat/Articles/tb27-0/ [19] Barbara Beeton, Asmus Freytag, Murray jackowski.pdf Sargent III: Unicode Support for Mathematics. [10] Will Robertson: An exploration of the Latin Unicode Technical Report UTR#25. 2001. Modern fonts. TUGboat, 28(2):177-180, 2007. http://www.unicode.org/reports/tr25/ http://www.tug.org/TUGboat/Articles/tb28-2/ [20] Barbara Beeton: Unicode and math, a tb89robertson.pdf combination whose time has come — Finally! [11] Hans Hagen, Jerzy B. Ludwichowski, Volker TUGboat, 21(3):174–185, 2000. Proceedings of RW Schaa: The new font project: TEX Gyre. the TUG 2000 Conference, Oxford, UK. TUGboat, 27(2):250–253, 2006. Proceedings http://www.tug.org/TUGboat/Articles/tb21-3/ of the TUG 2006 Conference, Marrakesh, tb68beet.pdf Morocco. [21] Barbara Beeton: The STIX Project — From http://www.tug.org/TUGboat/Articles/tb27-2/ Unicode to fonts. TUGboat, 28(3):299–304, tb87hagen-gyre.pdf 2007. Proceedings of the TUG 2007 Conference,
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 433 Ulrik Vieth
San Diego, California, USA. [30] George Williams: FontForge. Math typesetting http://www.tug.org/TUGboat/Articles/tb28-3/ information. tb90beet.pdf http://fontforge.sourceforge.net/math.html [22] STIX Fonts Project: Web Site and Frequently [31] Apostolos Syropoulos: Asana Math. Asked Questions. www.ctan.org/tex-archive/fonts/Asana-Math/ http://www.stixfonts.org/ [32] Will Robertson: Experimental Unicode math http://www.stixfonts.org/STIXfaq.html typesetting: The unicode-math package. [23] Murray Sargent III: Math in Office Blog. http://scripts.sil.org/svn-public/xetex/ http://blogs.msdn.com/murrays/default.aspx TRUNK/texmf/source/xelatex/unicode-math/ [24] Murray Sargent III: High-quality editing and unicode-math.pdf display of mathematical text in Office 2007. [33] Yannis Haralambous: Fonts and Encodings. http://blogs.msdn.com/murrays/archive/2006/ O’Reilly Media, 2007. ISBN 0-596-10242-9 09/13/752206.aspx http://oreilly.com/catalog/9780596102425/ [25] Tiro Typeworks: Cambria Math Specimen. [34] Bogusław Jackowski: Appendix G Illuminated. http://www.tiro.nu/Articles/Cambria/ TUGboat, 27(1):83–90, 2006. Proceedings Cambria_Math_Basic_Spec_V1.pdf of the 16th European TEX Conference 2006, [26] John Hudson, Ross Mills: Mathematical Debrecen, Hungary. Typesetting: Mathematical and scientific http://www.tug.org/TUGboat/Articles/tb27-1/ typesetting solutions from Microsoft. tb86jackowski.pdf Promotional Booklet, Microsoft, 2006. [35] Ulrik Vieth: Understanding the æsthetics of http://www.tiro.com/projects/ math typesetting. Biuletyn GUST, 5–12, 2008. [27] Daniel Rhatigan: Three typefaces for Proceedings of the 16th BachoTEX Conference mathematics. The development of Times 2008, Bachotek, Poland. 4-line Mathematics, AMS Euler, and Cambria http://www.gust.org.pl/projects/e-foundry/ Math. Dissertation for the MA in typeface math-support/vieth2008.pdf design, University of Reading, 2007. [36] Ulrik Vieth: Math Typesetting in TEX: The http://www.typeculture.com/academic_ Good, the Bad, the Ugly. MAPS, 26:207–216, resource/articles_essays/pdfs/tc_article_ 2001. Proceedings of the 12th European TEX 47.pdf Conference 2001, Kerkrade, Netherlands. [28] Murray Sargent III: Unicode Nearly Plain http://www.ntg.nl/maps/26/27.pdf Text Encodings of Mathematics. Unicode [37] Unicode Consortium: Code Charts for Technical Note UTN#28, 2006. Symbols and Punctuation. http://www.unicode.org/notes/tn28/ http://www.unicode.org/charts/symbols.html [29] Microsoft Typography: OpenType [38] Google Groups: Unicode math for TEX. specification version 1.5. http://groups.google.com/group/unimath http://www.microsoft.com/typography/otspec/
434 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Meta-designing parameterized Arabic fonts for AlQalam
Ameer M. Sherif, Hossam A. H. Fahmy Electronics and Communications Department Faculty of Engineering, Cairo University, Egypt ameer dot sherif (at) gmail dot com, hfahmy (at) arith dot stanford dot edu http://arith.stanford.edu/~hfahmy
Abstract In this paper we discuss how parameterized Arabic letters are meta-designed using METAFONT and then used to form words. Parameterized Arabic fonts enable greater flexibility in joining glyphs together and rendering words with imperceptible junctions and smoother letter extensions. This work aims to produce written Arabic with quality close to that of calligraphers. Words produced using our parameterized font are compared to other widely used fonts in a subjective test and results are presented.
1 Introduction follows them. According to a survey we conducted, The Arabic script is used for a multitude of languages our design surpasses the widely used fonts. and is the second most widely used script in the world. Our work is not yet finished. In the future, we However, due to the inherent complexity [3, 6] of need to provide for the automatic placement of dots producing high quality fonts and typesetting engines, and diacritic marks and complete the rest of the the support for Arabic digital typography has been required shapes. very weak. 2 Strokes in Arabic glyphs OpenType is currently the de facto standard font technology. It has many features to support a wide The Arabic alphabet, although consisting of 28 dif- variety of scripts, yet has its limitations for Arabic [7]. ferent letters, depends on only 17 different skeletons. The most significant limitations are probably the The dots added above or below some of these skele- following two. tons are the means of differentiating one letter from another. For example the letters ˇg¯ım ( ) and h¯a’ 1. The concept of letter boxes connecting together ˘ ( ) have the same shape as the letter h¯a’ ( ), but via other boxes of elongation strokes is not . ˇg¯ım has a dot below, and h¯a’ has a dot above. When suitable for highest quality Arabic typesetting. we discuss a primitive we mention˘ its use in the group When connecting glyphs to one another, the of letters having the same skeleton, not individual junctions rarely fit perfectly because adjacent letters, and this further simplifies our designs. letter glyphs usually have different stroke direc- tions at the starting and ending points. 2.1 The Arabic measurement unit 2. The use of pre-stored glyphs for different liga- Over a thousand years ago, Ibn-Muqlah, one of the tures is limiting. The number of possible lig- early theorists of Arabic calligraphy, was probably atures is far greater than what can be made the first to make the choice of the nuqt¯a (Arabic for available. dot) as a measurement unit for letter˙ forms [3]. He In order to achieve an output quality close to chose it in order to have some fixed measurements that of Arabic calligraphers, we modeled [7] the pen between different letter forms. For example, in the META Naskh writing style, the height of ’alif is 4 nuqt¯as, nib and its movement to draw curves using - ˙ FONT. In this paper, we use the pen stroke macros and the width of an isolated n¯un is 3 nuqt¯as. The nuqt¯a or dot we refer to is that made by the˙ pen used that we have defined to meta-design the primitive ˙ glyphs needed for a good quality Arabic font. So to write the letter, i.e., it is not a constant like the pt. The horizontal width of the nuqt¯a in Naskh (where far, we are working with the Naskh writing style ˙ and we provide a fully dynamic and flexible design the pen is held at an inclination of 70 degrees to the leading to smooth junctions between letters. We also horizontal) is approximately equal to the diagonal of developed a simple algorithm to perform kerning a dot drawn by the pen as shown in Fig. 1. Since the dot is a square then the nuqt¯a width is slightly less in the case of letters that do not connect to what ˙
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 435 Ameer M. Sherif, Hossam A. H. Fahmy
than the pen width multiplied by the square root of 2. In our work we take it as 1.4 × pen width and in our METAFONT programs it is simply abbreviated as n.
Figure 2: Selecting points to define the path of the letter ’alif .
This’alif example shows that the minimum num- Figure 1: The nuqta as a measurement unit. ber of points to choose for any stroke is two, and their locations are at the endpoints of the stroke. These are the easiest points to select. Intermediate points are then chosen when curve parameters such 2.2 Stroke point selection as starting and ending directions and tensions are We define a calligrapher’s pen stroke as a continu- not enough to define the curve as needed for captur- ous movement of the pen. The location where the ing important letter features. Hence more points are calligrapher pauses defines the end of a stroke and usually needed in stroke segments with sharp bends the start of a new one. Thus, a circular path may be or in asymmetric strokes. considered as only one stroke because the start and end points are defined by the movement of the hand 2.3 Stroke point dependencies and not by the appearance. In our design, we model the direction of the stroke as The first step in the process of meta-designing it is drawn by the calligrapher, i.e., the stroke of the any primitive or letter is to select the points through letter ’alif is drawn from top to bottom. The points which the pen strokes pass. This is not an easy in our designs are numbered in order according to choice. When designing outline fonts, the solution is the pen direction. So for the letter’alif , the stroke usually to scan a handwritten letterform, digitize its begins at point 1 and ends at point 2. outline, then make the necessary modifications. We However, a calligrapher chooses the starting did not adopt this approach because Arabic letters point of the ’alif stroke depending on the location do not have fixed forms but rather depend on the of the base line. This means that point 1 is cho- calligrapher’s style. Since we are meta-designing, we sen relative to point 2, so we define 1 based on 2. are more concerned with how the letter is drawn and Since METAFONT is a declarative language, not an not just a single resulting shape. Hence, instead of imperative one, the two statements: z1 = z2 + 3; capturing the fine details of a specific instance of the and z2 = z1 - 3; evaluate exactly the same. Yet letter by one calligrapher, we wanted to capture the we try to make the dependencies propagate in the general features of the letter. To help us accomplish natural logical order, which then makes editing the this, we based our design on the works of multiple METAFONT glyph code an easier job; hence, the first calligraphers. expression is the better choice. The letter ’alif is shown in Fig. 2 with three different possibilities of point selection. The leftmost 3 Meta-designing Arabic letters glyph requires the explicit specification of the tan- Several characteristics of the letter shapes discovered gential angles at points 1 and 2. In the middle glyph, during our design process were not mentioned explic- just connecting the points 1–3–4–2 with a B´ezier itly in most calligraphy books. Calligraphers do not curve can produce the same curve without explicitly measure their strokes with precise rulers and their specifying any angles: z1..z3..z4..z2. descriptions are only approximate. Detailed features Theoretically, we can specify the path using an of the letters are embedded implicitly in their curves infinite number of points, but the fewer the points, as they learned them by practice. However, in our the better the design and the easier to parameterize design, we represent the stroke mathematically and it. Adding more points that also lie on the same path require accurate descriptions. The following sections can be done as in the rightmost glyph, but point 5 is show a couple of examples. redundant because the stroke is symmetric, and can We start by studying the letter shapes and not- be produced without explicitly specifying any angles ing the fine variations that might exist between ‘sim- or tension. ilar’ shapes. Then we select the stroke points, decide
436 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Meta-designing parameterized Arabic fonts for AlQalam
on the pen direction at each point based on the let- ter shape and stroke thickness, and finally draw the strokes using our qstroke macro [7]. Figure 4: The tail primitive. 3.1 The concept of primitives Meta-design enables us to break the forms of glyphs into smaller parts, which are refered to as primitives. Even in cases with kerning where the tail may These primitives may be whole letters or just parts collide with deep letters that follow it, many calligra- of letters that exist exactly as they are or with small phers raise the letter as a whole without modifying modifications in other letters. We can save design the tail’s shape. Fig. 5 shows an example of kerning time and increase meta-ness by reusing primitives. applied to letters with tails. We follow the same In Knuth’s work on his Computer Modern (CM) approach in our design and the tail requires no flexi- fonts [4], primitives were not explicitly defined as bility parameters. black boxes and then reused. The use of primitives requires more parameterization than what CM deals with. There, the use of primitives was worthy only in small and limited flexibility shapes such as serifs and arcs, for which Knuth wrote subroutines. The difference between his work and ours is that he pa- rameterized letters to get a large variety of fonts, Figure 5: Four consecutive tails in a word as written while we parameterize primitives to make the letters in the Qur’an [1], [26:148]. Notice the identical tails more flexible and better connected, not to produce despite the different vertical positioning. different fonts. Indeed, as mentioned earlier, our current focus is the Naskh writing style only. Listing 1 shows the code for the tail. The natural Our classification of Arabic primitives consists direction of drawing is from point 3 to 4 to 5. But in of three categories: fact, the stroke is only between 3 and 4; the last part • Type-1 primitives are used in many letters of the tail is called a sh¯a.zya and calligraphers usually without any modifications. outline it using the tip of the pen nib then fill it in. We use the METAFONT filldraw macro for that • Type-2 primitives are dynamic and change purpose. We use filldraw instead of fill in order shape slightly in different letters. to give thickness to the sh¯a.zya edge at point 5. Also • Type-3 primitives are also dynamic but much note the coordinate points dependencies: z4 depends more flexible. on z3 and z5 depends on z4 not z3. This makes 3.2 Type-1 primitives modification of the glyph much easier by separating the definition of the stroke segment and that of the This category includes diacritics and pen strokes sh¯a.zya, i.e., if we modify the stroke, the sh¯a.zya is common in many letters. The nuqt¯a, k¯af ’s sh¯ara, not affected, unlike if z5 was a function of z3. and the hamza are Type-1 primitives.˙ Fig. 3 shows the sh¯ara of the letter k¯af and the hamza. These glyphs are drawn using a pen with half the width z4 = z3 +(−1.7n, −2n); z z n, . n of the regular pen. Other diacritics like the short 5 = 4 +(−2 36 ); path raa body; vowels fatha and kasra are dynamic, and do change . raa body = z3{dir −95} . . tension 1.3 . . length and inclination angle. z4{dir −160}; qstroke(raa body, 85, 100, 0, 0); path shathya; shathya = (x4, top y4){dir −160} . . {dir 160}z {dir −38} . . tension 1.3 . . Figure 3: The sh¯ara of k¯af and the hamza. 5 (rt bot z4){dir 15} - - cycle; pickup pencircle scaled 1.2; The ‘tail’ primitive is another example of a Type- filldraw shathya; 1 primitive. It is used as the ending tail in letters like w¯aw, r¯a’, and z¯ay in both their isolated and Listing 1: METAFONT code for the tail primitive. ending forms. Fig. 4 shows the tail designed using METAFONT on the left and its use in w¯aw and r¯a’ The code first defines the points in relation to on the right. each other using the nuqt¯a (n) as the unit of measure- ˙
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 437 Ameer M. Sherif, Hossam A. H. Fahmy
ment. Then a path variable is created which holds the path definition of the stroked part of the tail. The qstroke macro [7] is used to draw the stroke. The second path defines the sh¯az.ya outline. 3.3 Type-2 primitives This class of primitives has few parameters that enable only slight variations in the primitive’s shape to facilitate its use in different letters. We will discuss two primitives of this type: the ‘w¯aw head’ and the ‘’alif stem’. 3.3.1 The w¯aw head primitive Figure 7: The w¯aw head primitive. This primitive is a circular glyph used in the starting and isolated forms of the letters w¯aw, f¯a’, and q¯af . (final form), and t¯a’ (all forms). Most calligraphers It consists of two parts: the head and the neck. In ˙ most calligraphy books, the head is described as describe the straight stroke in the l¯am, k¯af , and t¯a’ as being identical to the ’alif . Fig. 8 shows a being exactly the same in all three letters. However ˙ small differences exist between the heads due to the calligrapher’s description [5] stating that the form of connections with different letter skeletons. Fig. 6 the vertical stroke in the different letters is exactly shows the letters f¯a’ and q¯af as drawn in three the same as the ’alif . This is a crude approximation books. Note how the circular head does in fact look because there are differences in the thickness, cur- vature, inclination, and height (in case of the t¯a’) slightly different in both letters, yet none of these ˙ books mention that there are variations in the head. between the isolated’alif and the modified form used in other letters.
Figure 8: Approximate directions in calligraphy books.
Fig. 9 shows our design: on the far right the Figure 6: The letters f¯a’ and q¯af as drawn by isolated ’alif and to its left the modified ’alif that three calligraphers, from top to bottom: Afify [2], is used in l¯am and k¯af . The modified ’alif is thin- Mahmoud [5], and Zayed [8]. ner with less curvature near the middle, or in other words more tension, together with more overall in- Fig. 7 shows that the w¯aw head primitive con- clination. Listing 2 shows that they both have the sists of 2 strokes, one between points 1–2, the other same height and the thickness of the stem is achieved between points 2–3–4. We approximate the differ- by increasing the pen nib angle. ences between the w¯aw, f¯a’, and q¯af by altering the 3–4 segment. Thus the same primitive may be used for the three letters in their isolated form. This same primitive is used in their ending forms by mov- ing point 1 down and to the right, to connect to a preceding letter or kash¯ıda. 3.3.2 The ’alif stem primitive Figure 9: The ’alif primitve. The stem of the’alif (Fig. 2) is used in many letters: l¯am (all forms), k¯af (isolated and final forms), m¯ım
438 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Meta-designing parameterized Arabic fonts for AlQalam
% Description for isolated ’alif curve := −100; incline := 70; height := 4.5n; z2 = z1 + (0, −height); path saaq; saaq = z1{dir curve} . . tension 1.4 . . z2{dir curve}; qstroke(saaq, incline, incline, 0, 0); % Description for ’alif used in lam and kaaf curve := −95; incline := 79; height := 4.5n; Figure 10: The kasa primitive. z2 = z1 + (0.3n, −height); path saaq; saaq = z1{dir curve} . . tension 1.4 . . z2{dir curve}; qstroke(saaq, incline, incline − 3, 0, 0);
Listing 2: METAFONT code for the ’alif primitive.
3.4 Type-3 primitives Type-3 primitives are glyphs that have a wider dy- namic range, and greater flexibility. In this section, Figure 11: The letter n¯un shown with kasa widths of 3, 9, 10 and 13 nuqt¯as. we discuss the skeleton of the letter n¯un, called the ˙ kasa (Arabic for cup) and the kash¯ıda. Calligraphers often use the great flexibility of these primitives to justify lines. differing by 3 nuqt¯as. The parameter tatwil con- trols this length by˙ varying the distance between 3.4.1 Kasa primitive points 3 and 4, both the horizontal and vertical com- The body of the letter n¯un is used in the isolated ponents, as shown in this line of code: and ending forms of s¯ın, ˇs¯ın, s¯ad, d¯ad, l¯am, q¯af , . . z3 = z4+(1.74n, 0.116n)+(0.5tatwil, 0.025tatwil)∗n; and y¯a’. Fig. 10 shows the kasa in five letters. The kasa has two forms, short and extended. The short As tatwil increases, point 3 moves further from form is almost 3 nuqt¯as in width in the case of n¯un, point 4 both to the right and upward. This vertical one nuqt¯a longer in y¯a’˙ , and slightly shorter in l¯am. change helps maintain the curvature in the kash¯ıda. This difference˙ between the kasa of the l¯am and the If no vertical adjustment is made, longer kash¯ıdas n¯un is not well documented in calligraphy books, look like separate straight lines with a sharp corner where most calligraphers mention that both are the at their intersection with the surrounding letters. same and only few state that in the l¯am it is slightly Calligraphers, on the other hand, draw curved lines smaller. rather than straight ones producing aesthetically An important property of the kasa is that it can better shapes. For these reasons, in our definition of be extended to much larger widths. In its extended the stroke, the tangential direction at point 3 is left form, it can range from 9–13 nuqt¯as. Fig. 11 shows free depending on the distance between 3 and 4. We the short form together with three˙ instances of the will see in the next section how kash¯ıdas are adjusted longer form generated from the same METAFONT to join letters together smoothly. code. Note that its width can take any value be- tween 9 and 13, not just integer values, depending on line justification requirements. Also note how the starting senn (vertical stroke to the right) of the letter is shorter in extended forms. 3.4.2 Kashida primitive
Another very important primitive for justification, Figure 12: The initial form of the letter h. ¯a’ with two the kash¯ıda is used in almost all connected letters. As different kash¯ıda lengths. an illustrative example, Fig. 12 shows the letter h. ¯a’ in its initial form with two different kash¯ıda lengths,
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 439 Ameer M. Sherif, Hossam A. H. Fahmy
4 Forming words The combination of primitives to form larger entities is the final step towards producing Arabic script that is as cursively connected and flexible as calligraphers’ writings. The parameterization of the glyphs allows us to piece them together perfectly as if they were drawn with just one continuous stroke. Figure 13: Placing a kash¯ıda between the letters h. ¯a’ 4.1 Joining glyphs with kash¯ıdas and d¯al with different lengths: 2, 3, 5 and 7 nuqtas. The most widely used glyph to connect other letters is the kash¯ıda. In this section we will explain the mechanism we use in order to make the junction between letters as smooth as possible. In current Figure 14: The word Mohammed as an example of font standards, such as OpenType and TrueType, vertical placement (Thuluth writing style). kash¯ıdas are made into fixed glyphs with pre-defined lengths, and are substituted when needed between letters to give the feeling of extending the letter. But 4.2 Vertical placement of glyphs because that design for the kash¯ıda is static, as are the rest of the surrounding letters, they rarely join In written Arabic, the existence of some letter com- well. It is evident that the word produced is made binations may force the starting letter of a word to of different segments joined by merely placing them be shifted upwards in order to accommodate for the close to each other. ending letters to lay on the baseline of the writing. In our work, the kash¯ıda is dynamic and can A very simple example of that property is the name take continuous values, not just predefined or discrete Mohammed when written with ligatures, where the values. We believe that when a kash¯ıda is extended initial letter m¯ım is written well above the baseline, between any two letters, it does not belong to just as shown in Fig. 14. one of them; instead, it is a connection between them It is hence obvious that the starting letter’s both. This belief is the result of experimenting with vertical position is dependent on the word as a whole. different joining methods. It might then be thought that it is easier to draw the Let us take the problem of joining the two letters words starting from the left at the baseline and then move upwards while proceeding to the right. But h. ¯a’ and d¯al as an example to illustrate the kash¯ıda joining mechanism we have developed. The solution this has two problems: one is that the horizontal we propose is to pass the tatwil parameter to the positioning of the last letter depends on the position macros producing the two glyphs, and the kash¯ıda of the first letter on the right and on the length of the length is distributed between both glyphs. This word, and the second is that a left to right drawing enables us to fix the ends of the glyphs to be joined would be against the natural direction of writing and at one angle, which is along the x-axis in the Naskh may result in an unnatural appearance. style, since any kash¯ıda in that style must at one The solution is then to walk through the word point move in this direction before going up again. till its end and analyze each letter to know where to To accommodate long kash¯ıdas, each glyph ending position the beginning letter vertically, and then start point is moved further from its letter and slightly the actual writing at the right from that point going downwards. Long kash¯ıdas need more vertical space left. This process is what a calligrapher actually does in order to curve smoothly, sometimes pushing the before starting to write a word. So, for a combination letters of a word upwards. of letters, we benefit from the declarative nature of METAFONT Other than affecting the ending points, the pa- . The following rules are applied: rameter also affects the curve definition on both sides • The horizontal positioning starts from the right, by varying the tensions, while keeping the direction • the vertical positioning starts from the left at of the curves at the intersection along the negative the baseline, and x-axis (since the stroke is going from right to left). • writing starts from the right. The resulting word at many different kash¯ıda lengths is shown in Fig. 13. To illustrate this better, see Fig. 15. The lig- ature containing the letters s¯ın, ˇg¯ım, and w¯aw is traced from left-to-right as shown, going through points 1–2–3, the starting points of each glyph, until
440 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Meta-designing parameterized Arabic fonts for AlQalam
4.5 A final example This section describes a more illustrative example shown in Fig. 17. This example, showing four in- stances of the word sujud, demonstrates the many properties and benefits of our parameterized font. Figure 15: Tracing a word from left-to-right to know First, it shows flexibility in stretching and compress- the starting vertical position. ing words for line justification purposes. This flexi- bility is due to two capabilities of the font: dynamic length kash¯ıdas and glyph substitution. For a very small line spacing, the s¯ın is written on top of the h. ¯a’, and the kash¯ıda after the h. ¯a’ is almost zero. When more space is available, the kash¯ıda after the h. ¯a’ is stretched and the senn connecting the s¯ın and the h. ¯a’ is also made slightly longer. Further elongation is made possible by breaking the ligature Figure 16: Kerning: letters with ‘tails’ need special between s¯ın and h¯a’. And finally, the maximum attention. The first line shows our output to the right . and the current font technologies to the left. The last length is obtained by elongating the kash¯ıda between two lines show our output. the two letters. Theoretically, we could get more stretching of this kash¯ıda and even add another one after the h. ¯a’, but calligraphic rules ultimately limit the stretching. the vertical position of point 1 is known. The next step is to start writing the word starting from point 1. Isolated letters like d¯al are not taken into considera- tion, because they do not affect the preceding letters vertically. 4.3 Dealing with kerning To imitate what calligraphers do as in Fig. 5, we had to invent a special kerning algorithm. Fig. 16 shows our output. We raise the level of any ‘deep letter’ following a ‘tail letter’ to kern correctly. Any Figure 17: The word sujud written with different non-deep letter holds the level steady. The end of lengths ranging from 10.5 to 16.5 nuqtas. the word restores the baseline. 4.4 Word lengths 5 Testing the output In reality, word lengths are not selected by the cal- ligraphers word by word, but instead, are chosen 5.1 A GUI to simplify tests based on the justification requirements of a whole To facilitate the task of testing our ideas, we made a line. When a word length is decided according to the simple graphical user interface (GUI) to write words line it exists in, this length should be passed to a main using our METAFONT programs. This allows us to macro that calls the glyph macros in order to form make changes to parameters and draw words faster the word. This main macro should decide the length than having to edit the code manually. Fig. 18 shows of kash¯ıdas to be added depending on the minimum the block diagram describing the operation of the GUI length of each letter. For example, that the word and Fig. 19 shows the different window components under consideration is the one shown in Fig. 13, and of the GUI. that the desired total length of the word is 10 nuqt¯as. First, the user types in a word or sequence of In order to calculate the extension or the tatwil˙ words in the input word text box, then presses the parameter between the letters, it subtracts all the start button. This parses the input word(s) into a minimum lengths of the individual letters. In our string of letters, removing any space characters. Each example, the head of the h. ¯a’ is 4 nuqt¯as wide, and character can be selected from a list and its shape the base of the d¯al is 3 nuqt¯as, hence the˙ word has a determined from the letterform list at the bottom of minimum length of only 7 nuq˙ t¯as. In order to stretch the screen. Also a length extension can be input in it to 10, the added kash¯ıda is˙ 3 nuqt¯as wide. the length text box (default is zero). ˙
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 441 Ameer M. Sherif, Hossam A. H. Fahmy
exactly equivalent. We selected the 16 words to test three main features: • connections between letters, • extension/tatwil of letters, and • kerning. Fig. 21 shows the first page of the test. Figure 18: Block diagram describing the operations of the graphical user interface.
Figure 19: Screenshot of our simple GUI.
When the write button is pressed, the letter- forms selected and their extra lengths are written in files, then METAFONT executes the glyph programs. Finally, a DVI previewer opens the resulting output as seen in Fig. 20.
Figure 21: First page of the test where we ask the readers to rate their comfort.
Figure 20: Screenshot of the DVI previewer We then surveyed 29 Arabic readers (14 males displaying the output word. and 15 females) with ages ranging from 10 to 70 years. In the survey, readers were asked to evaluate and rate words in terms of their written quality and how 5.2 The survey comfortable they feel reading the words. A rating In order to test if readers are comfortable with the of 1 is given to low quality and uncomfortable words, way our parameterized font looks as compared to and a rating of 5 is given to words of high written other Naskh fonts, we made a list of 16 words, each quality and most comfort to the reader. written in four different Naskh fonts: This test methodology is known as the Mean • Simplified Arabic, Opinion Score (MOS), a subjective test often used to evaluate the perceived quality of digital media after • Traditional Arabic, compression or transmission in the field of digital • DecoType Naskh, and signal processing. The MOS is expressed as a single • AlQalam parameterized font. number in the range of 1 to 5, where 1 is lowest For a more reliable and unbiased test, the order of quality and 5 is highest quality. In our case, the MOS fonts used is varied in consecutive rows and all words is a numerical indication of the perceived quality of are set to approximately the same sizes although a written word. Finally, the total MOS is calculated nominal point sizes of the different fonts are not as the arithmetic mean of all the individual scores.
442 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Meta-designing parameterized Arabic fonts for AlQalam
Table 1 shows the MOS scores for individual words 6 Conclusion of each font and the total average. The work covered in this paper is just a small step towards the realization of a system to produce output Table 1: MOS results for each font. comparable to that created by Arabic calligraphers, and much more work is still needed. We need to com- plete our design and finish all the required shapes. There is a strong need for non-engineering re- search on the readability and legibility of the different kinds of Arabic fonts comparable to the many stud- ies conducted on Latin letters. It is important to categorize the different calligraphers’ writing styles as well as the regular computer typefaces: are they easy and fast to read in long texts, or not? Which are better to use in titles or other ‘isolated’ materials, and which are better for the running text? By presenting our effort, we hope to open the door for future exciting work from many researchers. References [1] The Holy Qur’an. King Fahd Complex for Printing the Holy Qur’an, Madinah, KSA, 1986. [2] Fawzy Salem Afify. ta‘leem al-khatt al-‘arabi The results clearly show an increased comfort [Teaching Arabic calligraphy]. Dar Ussama, with the parameterized font with respect to the other Tanta, Egypt, 1998. popularly used fonts. One feature clearly distinguish- [3] Mohamed Jamal Eddine Benatia, Mohamed ing the different fonts is kerning. This is very evident Elyaakoubi, and Azzeddine Lazrek. Arabic text in words 1, 3, 5, 6, 9, 11, 13, and 14 which had kern- justification. TUGboat, 27(2), January 2007. ing applied. The addition of smooth kash¯ıdas for extension also results in a large difference in scores [4] Donald E. Knuth. Computer Modern Typefaces, as with words 7, 10, and especially 12 which contains volume E of Computers and Typesetting. a long kash¯ıda. The results of word 15 show that the Addison-Wesley, Reading, MA, USA, 1986. use of complex ligatures is also an important feature [5] Mahdy Elsayyed Mahmoud. al-khatt al-‘arabi, for the comfort of readers. We believe that more dirasah tafsiliya muwassa‘a [Arabic calligraphy, work and collaboration with calligraphers can yield a broad detailed study]. Maktabat al-Qur’an, even better results. Cairo, Egypt, 1995. [6] Thomas Milo. Arabic script and typography: A brief historical overview. In John D. Berry, editor, Language Culture Type: International Type Design in the Age of Unicode, pages 112–127. Graphis, November 2002. [7] Ameer M. Sherif and Hossam A. H. Fahmy. Parameterized Arabic font development for AlQalam. TUGboat, 29(1), January 2008. [8] Ahmad Sabry Zayed. ahdath al-turuq leta‘leem al-khotot al-‘arabiya [New methods for learning Arabic calligraphy]. Maktabat ibn-Sina, Cairo, Egypt, 1990.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 443 Smart ways of drawing PSTricks figures
Manjusha Susheel Joshi Bhaskaracharya Institute of Mathematics Law College Road, Pune 411004, India manjusha dot joshi at gmail dot com www dot bprim dot org Abstract We present a method of using PSTricks in conjunction with free software packages for interactive drawing (Dr. Geo, Dia, Gnuplot, Xfig) to produce various types of figures, via exporting as TEX and modifying the result.
1 Drawing geometric figures In our institute, we arrange various training pro- grammes for International Maths Olympiad compe- tition, publish the journal Bona Mathematica, and produce other mathematical study materials. For all these we need to draw complex geometric figures. 1.1 Accuracy requirements All the mathematical material is in LATEX format. Faculties require .eps files to include in their ques- tion papers, notes, books and articles. In figures, the text used for labels, etc., must all be in the same font. The material is widely dis- tributed among students and others, so from the printing point of view, the figures should be very accurate. Figure 1: Drawing a figure in Dr. Geo We want to produce such exact figures while not spending excessive amounts of time on draw- A ing them. Also, these figures must be usable in the file is different from that in the LTEX file, and the sharpness of the figure may not be good. LATEX sources. We want to minimize issues such as license, cost, Of course we wish to avoid such problems. and support and therefore maximize availability of 1.3 Export figures as TEX the software. In general, use of free software is en- couraged in the institute. As an alternative to PostScript export, software such For drawing geometric figures, the free software as the above supports exporting figures in TEX for- package Dr. Geo (home page http://www.gnu.org/ mat. Advantages of this approach include: software/dr-geo) is well-suited for us. Figure 1 • The ‘Export as TEX file’ options generate the shows a typical session in progress. Other free soft- entire code for users. ware GUI drawing programs include Dia for flow- • The resulting fig.tex file has PSTricks code charts, among other purposes (http://www.gnome. for the figure, which is editable. org/projects/dia), Gnuplot for function plotting • Users may not know every PSTricks command, (http://www.gnuplot.info), and Xfig for general but they can easily modify the code in useful drawing (http://www.xfig.org). ways knowing only a small set of commands. • Users need not calculate (x, y) co-ordinates, as 1.2 Convert figures directly to PostScript? they are already generated according to the fig- If a figure file is generated by another software pro- ure. One important thing not to worry about. gram, converted to ps or eps by an external utility, and then inserted in the LATEX file, we have observed 1.4 Step by step that the font type and size of labels in the converted Here is our process, step by step.
444 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Smart ways of drawing PSTricks figures
Z1 b
W3 b Q1 b
P3 b b P2 O b G b N b N H bb b W2
Q2 2 b b
Z2 b b b Q3 b Z3
P1 W1
Figure 2: Nine point circle, drawn with modified PSTricks code
• Open a graphical tool for drawing geometric fig- and W3. (We intentionally left this discrepancy in ures, e.g., one of those mentioned above. Draw the figure for expository purposes.) the figure. Drawing sequence is an issue with opaque colors. • Label the objects at appropriate places, as near With transparent colors, the sequence typically does by as possible. Save. not matter. • Export as a TEX figure. 3 Example A • Open it in any editor or (L )TEX IDE to edit. Figure 2 also serves as a general example showing kile.sourceforge.net We often use Kile ( ). the success of our workflow. In the figure, nine red • Change all the labels in the figure to be in math points can be seen. These points are constructed mode, i.e., enclosed in $...$, especially labels using Dr. Geo, i.e., Dr. Geo was responsible for cor- of vertices. rectly finding these nine intersecting points. • If the positions of labels are not perfect, then With PSTricks, one can use different colours adjust the x and y coordinates as needed, in the and textures to highlight points and regions, and neighbourhood of the x and y which have been place labels at the appropriate co-ordinates and use calculated by Dr. Geo. the same font as LATEX file in which figure will be inserted. In this case, we wanted to use the figure 2 Sequence of drawing objects matters on the front page of a journal, and so we needed to Sometimes two objects will overlap, so that the ob- make it more colourful. ject drawn earlier becomes invisible. To see an ex- In a sense, Dr. Geo works here as a front end ample, refer to Figure 2, a ‘nine point circle’. for PSTricks. With the help of such a front end, Looking at the points of intersection at W1 and users can save considerable time in calculating the W3, we can see the construction clearly, namely, co-ordinates for the figure. Z2Z3 ⊥ Z1W1 and Z1Z2 ⊥ Z3W3. However, at To learn about the actual mathematics of the point W2, the red dot marker has obscured that nine point circle, see http://www.csm.astate.edu/ Z1Z3 ⊥ Z2W2. If the dot were drawn before the Ninept.html. lines, the lines would be visible, as they are at W1
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 445 The MetaPost library and LuaTEX
Hans Hagen Pragma ADE http://pragma-ade.com
Abstract
An introduction to the MetaPost library and its use in LuaTEX.
1 Introduction process images with a format to be loaded, or If MetaPost support had not been as tightly inte- to create such a format. This function returns grated into ConT Xt as it is, at least half of the a library object. E • projects Pragma ADE has been doing in the last Execute sequences of MetaPost commands, decade could not have been done at all. Take for in- using the object’s execute method. This re- stance backgrounds behind text or graphic markers turns a result. • alongside text (as seen here). These are probably the Check if the result is valid and (if it is okay) request the list of objects. Do whatever you most complex mechanisms in ConTEXt: positions are stored, and positional information is passed on want with them, most probably convert them to MetaPost, where intersections between the text to some output format. You can also request areas and the running text are converted into graph- a string representation of a graphic in Post- ics that are then positioned in the background of the Script format. text. Underlining of text (sometimes used in the edu- There is no need to close the library object. As long cational documents that we typeset) and change bars as there were no fatal errors, the library recovers well (in the margins) are implemented using the same and can stay alive during the entire LuaTEX run. mechanism because those are basically a background Support for MPlib depends on a few compo- with only one of the frame sides drawn. nents: integration, conversion and extensions. This You can probably imagine that a 300 page doc- article shows some of the code involved in supporting ument with several such graphics per page takes a the library. Let’s start with the conversion. while to process. A nice example of such integrated 2 Conversion graphics is the LuaTEX reference manual, that has an unique graphic at each page: a stylized image of The result of a MetaPost run traditionally is a a revolving moon. PostScript language description of the generated graphic(s). When PDF is needed, that PostScript code has to be converted to the target format. This includes embedded text as well as penshapes used for drawing. Here is an example graphic: Most of the running time integrating such graphics seemed to be caused by the mechanics of the process: starting the separate MetaPost interpreter and having to deal with a number of temporary files. Therefore our expectations were high with regards to integrating MetaPost more tightly into LuaTEX. Be- sides the speed gain, it also true that the simpler the Figure 1 process of using such use of graphics becomes, the more modern a TEX runs looks and the less prob- draw fullcircle lems new users will have with understanding how all scaled 2cm the processes cooperate. withpen pencircle xscaled 1mm yscaled .5mm This article will not discuss the application in- rotated 30 withcolor .75red ; terface of the MPlib library in detail; for that there is Notice how the pen is not a circle but a rotated the manual. In short, using the embedded MetaPost ellipse. Later on it will become clear what the con- interpreter in LuaTEX boils down to the following: sequences of that are for the conversion. • Open an instance using mplib.new, either to How does this output look in PostScript? In
446 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The MetaPost library and LuaTEX abridged form, it looks like this: rently lives in mlib-pdf.lua. %!PS-Adobe-3.0 EPSF-3.0 Now, with the embedded MetaPost library, con- %%BoundingBox: -30 -30 30 30 version goes still differently because now it is possi- %%HiResBoundingBox: -29.62 -29.283 29.62 29.283 ble to request the drawn result and associated infor- %%Creator: MetaPost 1.090 mation in the form of Lua tables. %%CreationDate: 2008.09.23:0939 figure={ %%Pages: 1 ["boundingbox"]={ % [preamble omitted] ["llx"]=-29.623992919922, %%Page: 1 1 ["lly"]=-29.283935546875, 0.75 0 0 R 2.55513 hlw rd 1 lj 10 ml ["urx"]=29.623992919922, q n 28.34645 0 m ["ury"]=29.283935546875, 28.34645 7.51828 25.35938 }, 14.72774 20.04356 20.04356 c ["objects"]={ 14.72774 25.35938 7.51828 { 28.34645 0 28.34645 c ["color"]={ 0.75, 0, 0 }, [...] ["linecap"]=1, [0.96077 0.5547 -0.27734 0.4804 0 0] t S Q ["linejoin"]=1, P ["miterlimit"]=10, %%EOF ["path"]={ The most prominent code here concerns the { path. The numbers in brackets define the transfor- ["left_x"]=28.346450805664, mation matrix for the pen we used. The PDF variant ["left_y"]=-7.5182800292969, looks as follows: ["right_x"]=28.346450805664, ["right_y"]=7.5182800292969, q ["x_coord"]=28.346450805664, 0.750 0.000 0.000 rg 0.750 0.000 0.000 RG ["y_coord"]=0, 10.000000 M }, 1 j ... 1 J }, 2.555120 w ["pen"]={ q { 0.960769 0.554701 -0.277351 ["left_x"]=2.4548797607422, 0.480387 0.000000 0.000000 cm ["left_y"]=1.4173278808594, 22.127960 -25.551051 m ["right_x"]=-0.70866394042969, 25.516390 -13.813203 26.433849 ["right_y"]=1.2274475097656, 0.135002 24.679994 13.225878 c ["x_coord"]=0, 22.926120 26.316745 18.644486 ["y_coord"]=0, 37.478783 12.775526 44.255644 c }, [...] ["type"]="elliptical", h S }, Q0 ["type"]="outline", g 0 G }, Q }, The operators don’t look much different from } the PostScript, which is mostly due to the fact that This means that instead of parsing PostScript in the PostScript code, the preamble defines short- output, we can operate on a proper datastructure cuts like c for curveto. Again, most code involves and get code like the following: the path. However, this time the numbers are differ- function convertgraphic(result) ent and the transformation comes before the path. if result then In the case of PDF output, we could use TEX local figures = result.fig itself to do the conversion: a generic converter is im- if figures then plemented in supp-pdf.tex, while a converter opti- for fig in ipairs(figures) do mized for ConTEXt MkII is defined in the files whose local llx, lly, urx, ury names start with meta-pdf. But in ConTEXt MkIV = unpack(fig:boundingbox()) we use Lua code for the conversion instead. Thanks if urx > llx then to Lua’s powerful Lpeg parsing library, this gives startgraphic(llx, lly, urx, ury) cleaner code and is also faster. This converter cur- for object in ipairs(fig:objects()) do
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 447 Hans Hagen
if object.type == "..." then else ... width = wx flushgraphic(...) end ... local sx, sy = p.left_x, p.right_y else local rx, ry = p.left_y, p.right_x ... local tx, ty = p.x_coord, p.y_coord end if width ~= 1 then end if width == 0 then finishgraphic() sx, sy = 1, 1 end else end rx, ry, sx, sy = rx/width, ry/width, end sx/width, sy/width end end end end Here result is what the library returns when if abs(sx) < eps then sx = eps end one or more graphics are processed. As you can de- if abs(sy) < eps then sy = eps end return sx, rx, ry, sy, tx, ty, width duce from this snippet, a result can contain multiple end figures. Each figure corresponds with a beginfig ... endfig. The graphic operators that the con- If sx and sy are 1, there is no need to transform verter generates (so-called PDF literals) have to be the path, otherwise a suitable transformation ma- encapsulated in a proper box so this is why we have: trix is calculated and returned. The function itself uses a few helpers that make the calculations even • startgraphic: start packaging the graphic more obscure. This kind of code is far from triv- • flushgraphic: pipe literals to T X E ial and as already mentioned, these basic algorithms • finishgraphic: finish packaging the graphic were derived from the MetaPost sources. Even so, It does not matter what number beginfig was these snippets demonstrate that interfacing using passed, the graphics come out in the natural order. Lua does not look that bad. A bit more than half a dozen different object In the actual MkIV code things look a bit dif- types are supported. The example MetaPost draw ferent because it does a bit more and uses optimized command above results in an outline object. This code. There you will also find the code dealing with object contains not only path information but also the actual transformation, of which these helpers are carries rendering data, like the color and the pen. just a portion. So, in the end we will flush code like 1 M which sets If you compare the PostScript and the PDF code the miterlimit to 1, or .5 g which sets the color you will notice that the paths looks different. This is to 50% gray, in addition to a path. because the use and application of a transformation Because objects are returned in a way that matrix in PDF is different from how it is handled closely resembles MetaPost’s internals, some extra in PostScript. In PDF more work is assumed to be work needs to be done in order to calculate paths done by the PDF generating application. This is with elliptical pens. An example of a helper func- why in both the TEX and the Lua based converters tion in somewhat simplified form is shown next: you will find transformation code and the library function pen_characteristics(object) follows the same pattern. In that respect PDF differs local p = object.pen[1] fundamentally from PostScript. local wx, wy, width In the TEX based converter there was the prob- if p.right_x == p.x_coord lem of keeping the needed calculations within TEX’s and p.left_y == p.y_coord then accuracy, which fortunately permits larger values wx = abs(p.left_x - p.x_coord) than MetaPost can produce. This plus the parsing wy = abs(p.right_y - p.y_coord) code resulted in a lot of T X code which is not that else -- pyth: sqrt(a^2 + b^2) E wx = pyth(p.left_x - p.x_coord, easy to follow. The Lua based parser is more read- p.right_x - p.x_coord) able, but since it also operates on PostScript code it wy = pyth(p.left_y - p.y_coord, too is kind of unnatural, but at least there are fewer p.right_y - p.y_coord) problems with keeping the calculations sane. The end MPlib based converter is definitely the cleanest and if wy/coord_range_x(object.path, wx) least sensitive to future changes in the PostScript >= wx/coord_range_y(object.path, wy) then output. Does this mean that there is no ugly code width = wy left? Alas, as we will see in the next section, dealing
448 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The MetaPost library and LuaTEX with extensions is still somewhat messy. In practice \stopreusableMPgraphic users will not be bothered with such issues, because \reuseMPgraphic{test} writing a converter is a one time job by macro pack- Because withcolor is clever enough to under- age writers. stand what color type it receives, this is ok, but how about: 3 Extensions \startreusableMPgraphic{test} In Metafun, which is the MetaPost format used with color c ; ConTEXt, a few extensions are provided, such as: c := \MPcolor{mycolor} ; • cmyk, spot and multitone colors fill fullcircle scaled 1cm withcolor c ; • including external graphics \stopreusableMPgraphic • linear and circular shades Here the color variable only accepts an RGB • texts converted to outlines color and because in ConTEXt there is mixed color • inserting arbitrary texts space support combined with automatic colorspace Until now, most of these extensions have been im- conversions, it doesn’t know in advance what type it plemented by using specially coded colors and by is going to get. By implementing color spaces other injecting so-called specials (think of them as com- than RGB using special colors (as before) such type ments) into the output. On one of our trips to a mismatches can be avoided. The two techniques (coding specials in colors TEX conference, we discussed ways to pass informa- tion along with paths and eventually we arrived at and pre/postscripts) cannot be combined because a associating text strings with paths as a simple and script is associated with a path and cannot be bound efficient solution. As a result, recently MetaPost was to a variable like c. So this again is an argument for extended by withprescript and withpostscript using special colors that remap onto CMYK spot or directives. For those who are unfamiliar with these multi-tone colors. new features, they are used as follows: Another area of extensions is text. In previous versions of ConT Xt the text processing was already draw fullcircle withprescript "hello" E withpostscript "world" ; isolated: text ended up in a separate file and was processed in a separate run. More recent versions of In the PostScript output these scripts end up ConTEXt use a more abstract model of boxes that before and after the path, but in the PDF converter are preprocessed before a run, which avoids the ex- they can be overloaded to implement extensions, and ternal run(s). In the new approach everything can that works reasonably well. However, at the moment be kept internal. The conversion even permits con- there cannot be multiple pre- and postscripts asso- structs like: ciated with a single path inside the MetaPost inter- nals. This means that for the moment, the scripts for i=1 upto 100 : draw btex oeps etex rotated i ; mechanism is only used for a few of the extensions. endfor ; Future versions of MPlib may provide more sophis- ticated methods for carrying information around. but since this construct is kind of obsolete (at least The MkIV conversion mechanism uses scripts in the library version of MetaPost) it is better to use: for graphic inclusion, shading and text processing for i=1 upto 100 : but unfortunately cannot use them for more ad- draw textext("cycle " & decimal i) rotated i ; vanced color support. endfor ; A nasty complication is that the color spaces in Internally a trial pass is done so that indeed MetaPost don’t cast, which means that one cannot 100 different texts will be drawn. The throughput assign any color to a color variable: each colorspace of texts is so high that in practice one will not even has its own type of variable. notice that this happens. color one ; one := (1,1,0) ; % correct Dealing with text is another example of using cmykcolor two ; two := (1,0,0,1) ; % correct Lpeg. The following snippet of code sheds some light one := two ; % error on how text in graphics is dealt with. Actually this is fill fullcircle scaled 1cm a variation on a previous implementation. That one withcolor .5[one,two] ; % error was slightly faster but looked more complex. It was In ConTEXt we use constructs like this: also not robust for complex texts defined in macros \startreusableMPgraphic{test} in a format. fill fullcircle scaled 1cm local P, S, V, Cs = lpeg.P, lpeg.S, lpeg.V, withcolor \MPcolor{mycolor} ; lpeg.Cs
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 449 Hans Hagen local btex = P("btex") if special then local etex = P(" etex") special(object.postscript,object) local vtex = P("verbatimtex") end local ttex = P("textext") end local gtex = P("graphictext") Here, metapost is just the namespace used by local spacing = S(" \n\r\t\v")^0 the converter. The prescript tag tf triggers a func- local dquote = P(’"’) tion: local found = false function metapost.specials.tf(specification, object) local function convert(str) tex.sprint(tex.ctxcatcodes, found = true format("\\MPLIBsettext{%s}{%s}", return "textext(\"" .. str .. "\")" metapost.textext_current,specification)) end if metapost.textext_current local function ditto(str) < metapost.textext_last then return "\" & ditto & \"" metapost.textext_current end = metapost.textext_current + 1 local function register() end found = true ... end end Again, you can forget about the details of this local parser = P { function. What’s important is that there is a call [1] = Cs((V(2)/register out to TEX that will process the text. Each snippet + V(3)/convert + 1)^0), gets the number of the box that holds the content. [2] = ttex + gtex, The macro that is called just puts stuff in a box: [3] = (btex + vtex) * spacing * Cs((dquote/ditto + (1-etex))^0) \def\MPLIBsettext#1#2% * etex, {\global\setbox#1\hbox{#2}} } In the next processing cycle of the MetaPost code, the textext macro does something different : function metapost.check_texts(str) found = false vardef textext(expr str) = return parser:match(str), found image ( end _tt_n_ := _tt_n_ + 1 ; draw unitsquare If you are unfamiliar with Lpeg it may take xscaled _tt_w_[_tt_n_] a while to see what happens here: we replace the yscaled (_tt_h_[_tt_n_] + _tt_d_[_tt_n_]) text between btex and etex by a call to textext, withprescript "ts" a macro. Special care is given to embedded double withpostscript decimal _tt_n_ ; quotes. ) When text is found, the graphic is processed enddef ; two times. The definition of textext is different for This time the (by then known) dimensions of each run. For the first run we have: the box storing the snippet are used. These are vardef textext(expr str) = stored in the _tt_w_, _tt_h_ and _tt_d_ arrays. image ( The arrays are defined by Lua using information draw unitsquare about the boxes, and passed to the library before withprescript "tf" the second run. The result from the second Meta- withpostscript str ; Post run is converted, and again the prescript is used ) as trigger: enddef ; function metapost.specials.ts(specification, After the first run the result is not really con- object,result) verted, just the outlines with the tf prescript are local op = object.path filtered. In the loop over the object there is code local first, second, fourth like: = op[1], op[2], op[4] local prescript = object.prescript local tx, ty if prescript then = first.x_coord, first.y_coord local special = metapost.specials[prescript] local sx, sy = second.x_coord - tx, fourth.y_coord - ty
450 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The MetaPost library and LuaTEX
local rx, ry result in boxes = second.y_coord - ty, fourth.x_coord - tx • lua: process the code again using the library if sx == 0 then sx = 0.00001 end but this time let the unitsquare be trans- if sy == 0 then sy = 0.00001 end formed according to the text dimensions metapost.flushfigure(result) • lua: convert the resulting graphic and replace tex.sprint(tex.ctxcatcodes,format( the transformed unitsquare by the boxes with "\\MPLIBgettext{%f}{%f}{%f}{%f}{%f}{%f}{%s}", text sx,rx,ry,sy,tx,ty, metapost.textext_current)) The processor itself is used in the MkIV graphic func- ... tion that takes care of the multiple passes mentioned end before. To give you an idea of how it works, here is At this point the converter is actually convert- how the main graphic processing function roughly ing the graphic and passing PDF literals to TEX. As looks. soon as it encounters a text, it flushes the PDF code local current_format, current_graphic collected so far and injects some TEX code. The TEX macro looks like: function metapost.graphic_base_pass(mpsformat,str, preamble) \def\MPLIBgettext#1#2#3#4#5#6#7% local prepared, done {\ctxlua{metapost.sxsy(\number\wd#7, = metapost.check_texts(str) \number\ht#7,\number\dp#7)}% metapost.textext_current \pdfliteral{q #1 #2 #3 #4 #5 #6 cm}% = metapost.first_box \vbox to \zeropoint{\vss\hbox to \zeropoint if done then {\scale[sx=\sx,sy=\sy]{\raise\dp#7\box#7}% current_format, current_graphic \hss}}% = mpsformat, prepared \pdfliteral{Q}} metapost.process(mpsformat, { Because text can be transformed, it needs to be preamble or "", scaled back to the right dimensions, using both the "beginfig(1); ", original box dimensions and the transformation of "_trial_run_ := true ;", the unitsquare associated with the text. prepared, local factor = 65536*(7200/7227) "endfig ;" -- helper for text }, true ) -- true means: trialrun function metapost.sxsy(wd,ht,dp) tex.sprint(tex.ctxcatcodes, commands.edef("sx",(wd ~= 0 and "\\ctxlua{metapost.graphic_extra_pass()}") 1/( wd /(factor))) or 0) else commands.edef("sy",(wd ~= 0 and metapost.process(mpsformat, { 1/((ht+dp)/(factor))) or 0) preamble or "", end "beginfig(1); ", "_trial_run_ := false ;", So, in fact there are the following two processing str, alternatives: "endfig ;" • tex: call a Lua function that processes the }) graphic end • lua: parse the MetaPost code for texts and end decide if two runs are needed function metapost.graphic_extra_pass() Now, if there was no text to be found, the continu- metapost.textext_current = metapost.first_box ation is: metapost.process(current_format, { • lua: process the code using the library "beginfig(0); ", • lua: convert the resulting graphic (if needed) "_trial_run_ := false ;", and check if texts are used table.concat(metapost.text_texts_data(), " ;\n"), Otherwise, the next steps are: current_graphic, • lua: process the code using the library "endfig ;" • lua: parse the resulting graphic for texts (in }) end the postscripts) and signal TEX to process these texts afterwards The box information is generated as follows: • tex: process the collected text and put the function metapost.text_texts_data()
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 451 Hans Hagen
local t, n = { }, 0 time in the future, the postponed calls to \ctxlua for i = metapost.first_box, metapost.last_box and the explicit \pdfliterals can and will be re- do placed by using direct node generation, but that re- n = n + 1 quires a rewrite of the internal LuaTEX support for if tex.box[i] then PDF literals. t[#t+1] = format( The snippets are part of the mlib-* files of "_tt_w_[%i]:=%f;_tt_h_[%i]:=%f;_tt_d_[%i]:=%f;", MkIV. These files are tagged as experimental and n,tex.wd[i]/factor, n,tex.ht[i]/factor, will stay that way for a while yet. This is shown n,tex.dp[i]/factor by the fact that by now we use a slightly different ) approach. else Summarizing the impact of MPlib on exten- break sions, we can conclude that some are done better end and some more or less the same. There are some end conceptual problems that prohibit using pre- and return t postscripts for everything (at least currently). end This is a typical example of accessing informa- 4 Integrating tion available inside TEX from Lua, in this case in- The largest impact of MPlib is processing graphics formation about boxes. at runtime. In MkII there are two methods: real The trial_run flag is used at the MetaPost runtime processing (each graphic triggered a call to end; in fact the textext macro looks as follows: MetaPost) and collective processing (between TEX vardef textext(expr str) = runs). The first method slows down the TEX run, the if _trial_run_ : second method generates a whole lot of intermediate % see first variant above PostScript files. In both cases there is a lot of file else : I/O involved. % see second variant above In MkIV, the integrated library is capable of fi processing thousands of graphics per second, includ- enddef ; ing conversion. The preliminary tests (which in- This trickery is not new. We have used it al- volved no extensions) involved graphics with 10 ran- ready in ConTEXt for some time, but until now the dom circles drawn with penshapes in random colors, multiple runs took way more time and from the per- and the throughput was around 2000 such graphics spective of the user this all looked much more com- per second on a 2.3 MHz Core Duo: plex. It may not be that obvious, but in the case of a trial run (for instance when texts are found), after the first processing stage, and during the parsing of the result, the commands that typeset the content will be printed to TEX. After processing, the com- In practice there will be more overhead involved mand to do an extra pass is printed to TEX also. So, than in the tests. For instance, in ConT Xt informa- once control is passed back to TEX, at some point E tion about the current state of T X has to be passed TEX itself will pass control back to Lua and do the E extra pass. on also: page dimensions, font information, typeset- The base function is called in: ting related parameters, preamble code, etc. The whole T X interface is written around one function metapost.graphic(mpsformat,str, E process function: preamble) local mpx = metapost.format(mpsformat metapost.graphic(metapost.format("metafun"), or "metafun") "mp code") metapost.graphic_base_pass(mpx,str,preamble) Optionally a preamble can be passed as the end third argument. This one function is used in sev- The metapost.format function is part of the eral other macros, like: mlib-run module. It loads the metafun format, pos- \startMPcode ... \stopMPcode sibly after (re)generating it. \startMPpage ... \stopMPpage Now, admittedly all this looks a bit messy, but \startuseMPgraphic{name} ... in pure TEX macros it would be even more so. Some- \stopuseMPgraphic
452 TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings The MetaPost library and LuaTEX
\startreusableMPgraphic{name}... In the styles that Pragma ADE uses internally, \stopreusableMPgraphic there is support for the generation of placeholders for \startuniqueMPgraphic {name}... missing graphics. These placeholders are MetaPost \stopuniqueMPgraphic graphics that have some 60 randomly scaled circles with randomized colors. The time involved in gen- \useMPgraphic{name} erating 50 such graphics is (on my machine) some \reuseMPgraphic{name} 14 seconds, while in LuaT X only half a second is \uniqueMPgraphic{name} E needed. The user interface is downward compatible: in MkIV the same top-level commands are provided as in MkII. However, the (previously required) config- uration macros and flags are obsolete. This time, the conclusion is that the impact on ConT Xt is immense: The code for embedding E Because LuaTEX needs more startup time and graphics is very clean, and the running time for deals with larger fonts resources, pdfTEX is gener- graphics inclusion is now negligible. Support for text ally faster, but now that we have MPlib, LuaTEX in graphics is more natural now, and takes no run- suddenly is the winner. time either (in MkII some parsing in TEX takes place, and if needed long lines are split; all this takes time).
TUGboat, Volume 29 (2008), No. 3 — TUG 2008 Conference Proceedings 453 Putting the Cork back in the bottle — Improving Unicode support in TEX Mojca Miklavec Faculty of Mathematics and Physics, University of Ljubljana Arthur Reutenauer GUTenberg, France http://tug.org/tex-hyphen Abstract Until recently, all of the hyphenation patterns available for different languages in TEX were using 8-bit font encodings, and were therefore not directly usable with UTF-8 TEX engines such as X TE EX and LuaTEX. When the former was included in TEX Live in 2007, Jonathan Kew, its author, devised a temporary way to use them with X TE EX as well as the “old” TEX engines. Last spring, we undertook to convert them to UTF-8, and make them usable with both sorts of TEX engines, thus staying backwardly compatible. The process uncovered a lot of idiosyncrasies in the pattern-loading mechanism for different languages, and we had to invent solutions to work around each of them.
1 Introduction 2 The new architecture Hyphenation is one of the most prominent features The core idea is that after converting the patterns of TEX, and since it is possible to adapt it to many to UTF-8, the patterns are embedded in a structure languages and writing systems, it should come as no that can make them loadable with both sorts of TEX surprise that there were so many patterns created engines, the ones with native UTF-8 support (X TE EX, so quickly for so many languages in the relatively LuaTEX) as well as the ones that support only 8-bit early days of TEX development. As a result, the files input.1 that are available often use old and dirty tricks, in The strategy for doing so was the following: for order to be usable with very old versions of TEX. In each language hlangi, the patterns are stored in a file particular, all of them used either 8-bit encodings or called hyph-hlangi.tex. These files contain only the accent macros (\’e, \v{z}, etc.); Unicode did not raw patterns, hyphenation exceptions, and comments. yet exist when most of these files were written. They are input by files called loadhyph-hlangi.tex. This was a problem when X TE EX was included This is where engine detection happens, such as this in TEX Live in 2007, since it expects UTF-8 input by code for Slovenian: default. Jonathan Kew, the X TE EX author, devised % Test whether we received one or two arguments a way of using the historical hyphenation patterns \def\testengine#1#2!{\def\secondarg{#2}} with both X T X and the older extensions of T X: % We are passed Tau (as in Taco or TEX, E E E % Tau-Epsilon-Chi), a 2-byte UTF-8 character for each pattern file hhyphi.tex, he wrote a file called \testengine T!\relax xu-hhyphi.tex that detects if it is run with X TE EX % Unicode-aware engines (such as XeTeX or LuaTeX) or not; in the latter case, it simply inputs hhyphi.tex % only see a single (2-byte) argument \ifx\secondarg\empty directly, and otherwise, it takes actions to convert all \message{UTF-8 Slovenian Hyphenation Patterns} the non-ASCII characters to UTF-8, and then inputs \else the pattern file. \message{EC Slovenian Hyphenation Patterns} To sum up, in TEX Live 2007, X TE EX used the \input conv-utf8-ec.tex \fi original patterns as the basis, and converted them \input hyph-sl.tex to UTF-8 on the fly. The only trick is to make T X look at the Uni- In the ConTEXt world, on the other hand, the E code character for the Greek capital Tau, in UTF-8 patterns had been converted to UTF-8 for a couple of years, and were converted back to 8-bit encodings by encoding: it uses two bytes, which are therefore read the macro package, depending on the font encoding. by 8-bit TEX engines as two different characters; thus In an attempt to go beyond that and to unify 1 A note on vocabulary: in this article, we use the word those approaches, we then decided to take over con- “engine” or “TEX engine” for extensions to the program TEX, versions for all the pattern files present in TEX Live in contrast to macro packages. We then refer to (TEX) en- gines with native UTF-8 support as “UTF-8 engines”, and to at that time (May 2008), for inclusion in the 2008 the others as “8-bit engines”, or sometimes “legacy engines”, TEX Live release. borrowing from Unicode lingo.
454 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Putting the Cork back in the bottle — Improving Unicode support in TEX
the macro \testengine sees two arguments. UTF-8 Here is a table of the encodings we support: engines, on the other hand, see a single character ConTEXt LATEX Comments (Greek capital Tau), thus a single argument before ec T1 “Cork” encoding the exclamation mark, and \secondarg is \empty. il2 latin2 ISO 8859-2 If we’re running a UTF-8 TEX engine, there is il3 latin3 ISO 8859-3 nothing to do but to input the file with the UTF-8 lmc lmc montex (Mongolian) patterns; but if we’re running an 8-bit engine, we qx qx Polish have to convert the UTF-8 byte sequences to a single t2a t2a Cyrillic byte in the appropriate encoding. For Slovenian, as for most European languages written in the Latin 2.1 Language tags: BCP 47 / RFC 4646 alphabet, it happens to be T1. This conversion is A word needs to be said about the language tags we taken care of by a file named conv-utf8-ec.tex in used. As a corollary to the completely new naming our scheme. Let’s show how it works with these three scheme for the pattern files and the files surround- 2 characters: ing them, we wanted to adopt a consistent naming • ‘č’ (UTF-8 h0xc4, 0x8di, T1 0xa3), policy for the languages, abandoning the original • ‘š’ (UTF-8 h0xc5, 0xa1i, T1 0xb2), names completely, because they were problematic • ‘ž’ (UTF-8 h0xc5, 0xbei, T1 0xba). in some places. Indeed, they used ad hoc names In order to convert the sequence h0xc4, 0x8di which had been chosen by very different people over to 0xa3, we make the byte 0xc4 active, and define it many years, without any attempt to be systematic; to output 0xa3 if its argument is 0x8d.3 The other this has led to awkward situations; for example, the sequences work in the same way, and the extracted name ukhyphen.tex for the British English patterns: content of conv-utf8-ec.tex is thus:4 while “UK” is easily recognized as the abbreviation \catcode"C4=\active for “United Kingdom”, it could also be the abbrevi- \catcode"C5=\active ation for “Ukrainian”, and unless one knows all the % names of the pattern files by heart, it is not possible \def^^c4#1{% to determine what language is covered by that file \ifx#1^^8d^^a3\else % U+010D \fi} from the name alone. % It was therefore clear that in order to name files \def^^c5#1{% that had to do with different languages, we had to \ifx#1^^a1^^b2\else % U+0161 use language codes, not country codes. But this was \ifx#1^^be^^ba\else % U+017E \fi\fi} not sufficient either, as can be seen from the example % ensure all the chars above have valid lccode’s: of British English, since it’s not a different language \lccode"A3="A3 % U+010D from English. \lccode"B2="B2 % U+0161 Upon investigation, it turned out that the only \lccode"BA="BA % U+017E standard able to distinguish all the patterns we had As the last comment says, we also need to set was the IETF “Best Current Practice” recommenda- non-zero \lccodes for the characters appearing in tion 47 (BCP 47), which is published as RFC docu- the pattern files, a task formerly carried out in the ments; currently, it’s RFC 4646.5 This addresses all pattern file itself. the language variants we needed to tag: The information for converting from UTF-8 to • Languages with variants across countries or re- the different font encodings has been retrieved from gions, like English. the encoding definition files for LATEX and ConTEXt, and gathered in files called henci.dat. The converter • Languages written in different scripts, like Ser- files are automatically generated with a Ruby script bian (Latin and Cyrillic). from that data. • Languages with different spelling conventions, 2 like Modern Greek (which underwent a reform The only non-ASCII characters in Slovenian. 3 The same method would work flawlessly if the sequence known as monotonic in 1982), and German (for contained three or more bytes — although this case doesn’t which a reform is currently happening, started arise in our patterns — since the number of bytes in a UTF-8 in 1996). sequence depends only on the value of the first byte. 4 5 Problems would happen if a T1 byte had been made In the past, it has been RFC 1766, then RFC 3066, active in that process, but for reasons inherent to the history and is currently being rewritten, with the working title RFC of TEX font encodings, as well as Unicode, this is never the 4646bis. RFC 4646 is available at ftp://ftp.rfc-editor. case for the characters used in the patterns, a fact the authors org/in-notes/rfc4646.txt, and the current working ver- consider a small miracle. The proof of this is much too long sion of RFC 4646bis (draft 17) at http://www.ietf.org/ to be given in this footnote, and is left to the reader. internet-drafts/draft-ietf-ltru-4646bis-17.txt.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 455 Mojca Miklavec and Arthur Reutenauer
A list of all the languages with their tags can guages, though (French, Danish and Latin), we used be found in appendix A. a spechyph-hlangi-ot1.tex file. 3 Dealing with the special cases 3.2 Multiple pattern sets for the There were so many special cases that one might say same language that the generic case was the special one! Another interesting issue was with Ukrainian and Russian, where different complications arose. 3.1 Pattern files designed for First, the pattern files were also devised for mul- multiple encodings tiple encodings, but in a different manner: here, the The first problem we encountered was with patterns encoding is selected by setting the control sequence that tried to accommodate both the OT1 and the T1 \Encoding before the pattern file is loaded. De- encoding in the same file. pending on the value of that macro, the appropriate The first language for which this had been done conversion file is then input, that works in the same was, historically, German, and the same scheme was way as our conv-utf8-henci.tex files. There is of subsequently adopted for French, Danish, and Latin. course a default value for \Encoding, which for both The idea is the following: in each of these languages, languages is T2A,6 the most widespread font encod- there are characters that are encoded at different ing for Russian and Ukrainian, and the one used in positions in OT1 and in T1; for German, it is the the pattern files; thus, no conversion is necessary if sharp s ‘ß’; for French, it is the character ‘œ’, etc. In \Encoding is kept to its default value. order to deal with that, each pattern that happened Then, both Russian and Ukrainian had several to contain one of these characters was duplicated pattern files, with different authors and/or hyphen- in the file, with intricate macros to ignore them ation rules (phonetic, etymological, etc.). Those were selectively, depending on the font encoding used. selected with a control sequence called \Pattern, by This would have been very awkward to repro- default as for Russian (by Aleksandr Lebedev), and duce in our architecture, if at all possible: it would mp for Ukrainian (by Maksym Polyakov). have meant that each word such as, say, “cœur” in Both those choices could, of course, be overrid- French would need to yield two different byte strings den only at format-building time, since the patterns in 8-bit mode, for OT1 and T1 (cˆˆ1bur and cˆˆf7ur, are frozen at that moment. respectively). We therefore decided to put the du- Finally, they used a special trick, implemented plicate patterns in a separate file called spechyph in file hypht2.tex, to enable hyphenation inside -hlangi-ot1.tex that is input only in legacy mode, words containing hyphens, similar to Bernd Raichle’s after the main file hyph-hlangi.tex. hypht1.tex for T1 fonts. The patterns packaged in this fashion should Those three features had to be addressed in very therefore behave in the same way as the historical different ways in our structure: while the first one files, enabling a few breakpoints with non-ASCII was irrelevant in UTF-8 mode, it would have implied characters in OT1 encoding. We would like to stress, fundamental changes in our loadhyph-hlangi.tex though, that OT1 is definitely not the way to go for files for 8-bit engines, since the implicit assumption these languages. We only supported this behaviour that any language uses exactly one 8-bit encoding for the sake of compatibility, but we doubt it is very would no longer be met. The second feature was eas- useful: if one uses OT1 for German or French, one ier to handle, but still demanded additional features would indeed have a few patterns with ‘ß’ or ‘œ’, in our loadhyph-hlangi.tex files. Finally, the third respectively, but many more patterns, with accented feature, although certainly very interesting, seemed characters, would be missed. In order to take full more fragile than what we felt was acceptable. advantage of the hyphenation patterns, one needs to Upon deliberation, we then decided to not in- use T1 fonts. clude those features in the UTF-8 patterns before It has to be noted that in addition, we ended TEX Live 2008 was out, but to still enable them in up not using the aforementioned approach in the legacy mode, in order to ensure backward compat- case of German, because we wanted to account for ibility. And thanks to subsequent discussions with the ongoing work to improve the German patterns; Vladimir Volovich, who devised the way the Russian thus, we decided to use the new patterns with the patterns were packaged, and inspired the Ukrainian UTF-8 engines, but not with the 8-bit engines, for ones, we could include a list of hyphenated compound compatibility reasons. In the latter case, we simply words which we put in files called exhyph-ru.tex include the original pattern file in T1 directly, with no conversion whatsoever. For the three other lan- 6 Actually t2a, lowercase.
456 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Putting the Cork back in the bottle — Improving Unicode support in TEX
and exhyph-uk.tex, respectively. The strategy we Among the immediate and practical problems used is thus: is, in particular:
• In UTF-8 mode, input the UTF-8 patterns, then 5.1 . . . for something completely different the ex- file. • In legacy mode, simply input the original pat- Babel would need to be enhanced in order to enable tern file directly. different “variants” for at least two languages. One is Norwegian, for which two written forms exist, known Therefore, the only feature missing, overall, in as “bokm al” and “nynorsk” (ISO 639-1 nb and nn, TEX Live 2008, is the ability to choose one’s favorite respectively).8 At the moment, Babel has only one patterns in UTF-8 mode: for each language, we only “Norwegian” language. The second is Serbian, which converted the default set of patterns to UTF-8. Set- can be written in both the Latin and the Cyrillic ting \Pattern will thus have no effect in this case, alphabets; these possible variants which are not yet but it will behave as before in 8-bit mode. Now that taken into account in Babel. TEX Live 2008 has been released we intend to change that behaviour soon, and to enable the full range of 6 Acknowledgements features that the original pattern files had. First and foremost, we wish to thank wholeheartedly It should also be noted that in TEX Live 2007, Karl Berry, who supported the project from the Bulgarian used the same pattern-loading mechanism, beginning and guided us with advice, as well as but that there was actually only one possible encod- Hans Hagen, Taco Hoekwater and Jonathan Kew, ing, and only one pattern file, so there was no real for their technical help, and, finally, Norbert Preining, choice, and it was therefore straightforward to adapt who went through the trouble of integrating the new the Bulgarian patterns to our new architecture. package into TEX Live. 4 TEX Live 2008 Appendix A List of supported languages The result of our work has been put on CTAN under ar Arabic la Latin the package name hyph-utf8, and is the basis for fa Farsi mn-cyrl Mongolian hyphenation support in T X Live 2008. We don’t eu Basque mn-cyrl-x-2a Mongolian (new E patterns) consider our work to be finished (see next section), bg Bulgarian no Norwegian and we welcome any discussion on our mailing-list cop Coptic nb Norwegian ([email protected]). We also have a home page Bokm al hr Croatian nn Norwegian at http://tug.org/tex-hyphen, to which readers Nynorsk are referred for more information. cs Czech zh-latn Chinese Pinyin The package has been released in the TDS layout, da Danish pl Polish nl Dutch pt Portuguese with the TEX files in tex/generic/hyph-utf8 and eo Esperanto ro Romanian subdirectories. The encoding data and Ruby scripts et Estonian ru Russian are available in source/generic/hyph-utf8. Some fi Finnish sr-latn Serbian, Latin language-specific documentation has been put in script fr French sr-cyrl Serbian, Cyrillic doc/generic/hyph-utf8. script de-1901 German, “old” sh-latn Serbo-Croatian, 5 Andnow... spelling Latin script de-1996 German, “new” sh-cyrl Serbo-Croatian, There still are tasks we would like to carry out: the spelling Cyrillic script hypht1.tex / hypht2.tex behaviour has already el-monoton Monotonic Greek sl Slovene been mentioned, and one of the authors has lots of el-polyton Polytonic Greek es Spanish grc Ancient Greek sv Swedish ideas on how to improve Unicode support yet more grc-x-ibycus Ancient Greek, tr Turkish in UTF-8 T X engines. Ibycus E encoding We appeal to pattern authors to make contact hu Hungarian en-gb British English with us in order to improve and enhance our package; is Icelandic en-us American many of them have already communicated with us, English to our greatest pleasure, and we’re confident that id Indonesian uk Ukrainian ia Interlingua hsb Upper Sorbian our effort will be understood by all the developers ga Irish cy Welsh dealing with language-related problems.7 it Italian
7 The acknowledgement section, had it been as long as the 8 authors would have wished it to be, would have more than The ISO standard also includes a code for “Norwegian”, doubled the size of this article. no, although this name is formally ambiguous.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 457 Writing Gregg Shorthand with METAFONT and LATEX
Stanislav Jan Sarmanˇ Computing Centre Clausthal University of Technology Erzstr. 51 38678 Clausthal Germany Sarman (at) rz dot tu-clausthal dot de http://www3.rz.tu-clausthal.de/~rzsjs/steno/Gregg.php
Abstract We present an online system, which converts English text into Gregg shorthand, a phonetic pen writing system used in the U.S. and Ireland.
Shorthand is defined [2] as a method of writing de- two forms of signs exist for s and th: the right (s) ➓ , rived from the spelling system and the script of a (th) ➧and the left (S) ✞, (Th) ✟, respectively. particular language. In order to approximate the Blended consonant signs: speed of speech: (ld) (nd) (rd) (tm) (tn) (td)(dd) • The graphic redundancy of written characters ❞ ♥ ➉ ➵ ➸ ➢ and ligatures derived from longhand letters is reduced. can be thought of as built-in ligatures. Vowel signs: • Silent letters are ignored, and complex ortho- for sound(s) as in graphic rules are simplified. The spelling only [a] ✂ at, art, ate ✖ serves as a reference system of what is largely ✴ ❿ phonetic writing. [e]/[i] pit, pet, peat (o) ① pot, port ➂ , pour ➃ • Abbreviations are utilized. ➹ ➚ ➾ Gregg shorthand was first published in 1888. In (u) tuck, took , tomb the following three sections we want to show how its Elementary diphthong (triphone) signs: for sounds as in inventor J. R. Gregg has applied the above general ✌ ❏ principles in the creation of his shorthand system [ai] high [aee] ☛ client ✩ and how the current system version, the centennial ❖ ✭ edition [7], can be implemented in METAFONT [5]. [iie] create When writing shorthand the curvilinear motion 1 Simplified writing of a pen in the slope of longhand traces ellipse or 1.1 The alphabet of Gregg Shorthand oval-based curves. We shape these elementary short- METAFONT 1 hand signs in as continuous curvature Signs of consonant phonemes written forward: splines with zero curvature at the end points. (k) (g) (r) (l) (n) (m) (t) (d) (th)(Th) ❳ ❇ ➇ ❛ ♠ ❢ ➟✮ ➧✟ Half-consonants h, w, prefixes and suffixes are realized as markers, e.g. a dot over ✴ [i] such as Signs of consonant phonemes written downward: in “he”; ❍ means preceding h; another dot under (p) (b) (f) (v) (sh) (ch) (jh) (s)(S) (along) the sign (t) ➟such as in ■ denotes the ❸ ✗ ✷ ➴ ➔ ✥ ❲ ➓ ✞ suffix -ing in “heating”. The signs of voiced consonant phonemes are larger 1.2 Metaform versions of their unvoiced opposites. As the “mini- mal pairs”2 of words with phonemes s vs. z, th vs. Gregg shorthand is an alphabetic system — written dh and sh vs. zh are very rare, there are only the words are combined together from elementary signs unvoiced versions of these signs. On the other hand by joining, so that in general the Gregg shorthand glyphs have the following metaform: 1 Phonemes are denoted in typewriter type, their signs { ({p,}C{,s}) | [+|-][{p,}V {,s}] }+ are parenthesized or bracketed. where V are circular vowels, C denotes (blended) 2 such as face vs. phase, sooth vs. soothe, mesher vs. mea- sure, pronunciations of which differ only in one place consonants and vowels u and o, optional p and s
458 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Writing Gregg Shorthand with METAFONT and LATEX
stand for markers of semiconsonants, prefixes or suf- κ1 κ1 κ κ fixes. In more detail: 0 0 V ::= a | e | i | ai | aee | iie (r) (k) (r)(k) C ::= b | d | dd ... o | u (2) p ::= h | w | ... over | under ... Figure 1: An example of a G continuous s ::= ing | ly | ... CC joining Examples: metaform he -[h,i,] ❍ 1.3.2 Joinings with circular signs heating -[h,i,](,t,ing) ■ The signs of consonants can be classified into seven need (n)-[i](d) categories. Looped vowels before or after a conso- needed (n)-[i](dd) nant are written within the curves or combined with needing (n)-[i](,d,ing) straight consonants as right curves, so that the fol- The metaform corresponds directly to the META- lowing VC, CV prototype joinings can occur: FONT program code, e.g.: ❳ ➇ ❢ ➟ ✥ ❸ ✷ (n)-[i](,d,ing) ↔ I(,n,); V(-1,,i,); C(,d,ing); i.e. the shorthand glyph of this particular character VC ✄ ✎ ✍ ✖ ✵ ✑ ☞ is initiated with (n), then the signs of right vowel CV ❨ ➈ ❤ ➡ ➣ ❹ ✸ -[i] and the sign of consonant d with suffix ing are appended. Observe the realization of diphthongs as VC or CV joinings, too: 1.3 Joining for sounds as in We distinguish between joinings where circular signs -[a](u) ❷ how ▲ are involved and joinings without them. (o)+[i] ⑤ toy ➪ -[i](u) ✶ few ✼ 1.3.1 CC Joinings All VC or CV joinings are done in such a man- Examples of CC joinings ordered by their connec- ner that both components have identical tangents (0) (2) tivity grade G –G follow: and curvature equal to zero at the connection point. continuous: ✲ (d)(n), ⑦ (u)(n), ❯ (t)(S) As an example of G(2) connecting of a consonant (with turning point!) sign with a vowel sign consider prepending the sign ✂ ❳ ✄ tangent continuous: ✷ + ❛ = ❂ (f)(l), of -[a] before (k) as in :
❸ + ➇ = ➄ (p)(r), ➹ + ❸ = ➘ (u)(p), 0 −1/r −α ❇ ➴ ❋ α r + = (g)(v) 0 0 r curvature continuous: ➇ + ❳ = ❒ (r)(k) (k) -[a] -[a](k) The first two types of joinings are handled by META- (2) FONT means; G(2) continuous connecting can be Figure 2: An example of a G continuous VC done only in special cases using Hermite interpo- joining lation for B´ezier splines, as follows. t If two endpoints zi, unit tangents di, didi = 1 Here the spline approximation of (k) is such and curvatures κi, i = 1, 2 are given, the control that the curvature equals zero at the end points; + − this is the case with other consonants, too. The points z0 = z0 + δ0d0, z1 = z1 − δ1d1 have to be determined from the following system of nonlinear spline approximation of right circle with radius r equations [3]: for the sign -[a] consists of 8 segments such that the curvature κ = 0 at the top point and κ ≈−1/r (d × d )δ = (a × d ) − 3/ κ δ2 (1) 0 1 0 1 2 1 1 at the other points. Let α be the angle of the tangent 3 2 (d0 × d1)δ1 = (d0 × a) − /2κ0δ0 (2) in the zeroth point of (k). Placing the -[a] circle at (a = z1 − z0). the point with distance r from the zeroth point of If the tangents are parallel, i.e. (d0×d1) = 0 and (k) and rotating it by the angle of −α a curvature the curvatures κi have opposite signs, the equations continuous joining is obtained. (1)–(2) are trivially solvable, so that G(2) joining is For a G(2) continuous spline approximation of possible (here after some kerning): the unit circle with curvature κ= 0 in one point we
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 459 Stanislav Jan Sarmanˇ use the traditional B´ezier spline circle segment ap- 2 Phonetic writing th proximation for the segments 1–7 (see the 7 seg- Gregg shorthand uses its own orthography, which 1 ment on the left) z( /2) = 1, so that δ0 = δ1 = must be acquired by the shorthand learner and in 4 4 /3 tan(θ/4) and κi = −(1 − sin (θ/4)). Using this a system such as ours, the pronunciation of a word th value as left point curvature of the 8 segment (see has to be known. We use the Unisyn multi-accent on the right) and demanding κ1 = 0, the Hermite lexicon4 with about 118,000 entries [4] comprising interpolation with the equations (1)–(2) for the un- British and American spelling variants. The fields knowns δ0 und δ1 can be solved for segments with ◦ of lexicon entries are: spelling, optional identifier, θ< 60 . part-of-speech, pronunciation, enriched orthography, and frequency. Examples are: + κ−1 = 0 owed;;VBD/VBN; { * ouw }> d >;{owe}>ed>;2588 z0 z1 z1 live;1;VB/VBP; { l * i v } ;{live};72417 live;2;JJ; { l * ae v } ;{live};72417} κ1 ≈−1 a z1 z0 Homonyma cases such as “live” above, in which − κ0 ≈−1 the pronunciation helps to identify word meaning z1 are much more rarer than the cases in which the
+ a use of pronunciation yields homophones resulting in z0 shorthand homographs. Consider the very frequent θ θ right, rite, wright, write: { r * ai t } with short- z0 ➏ κ0 ≈−1 hand glyph (r)+[ai](t) or the heterophones read;1;VB/NN/NNP/VBP; { r * ii d };{read};94567 ◦ Figure 3: 7th and 8th segment (θ = 45 ) of our unit read;2;VBN/VBD; { r * e d };{read};94567 right circle spline approximation both written as ➊ (r)+[e|i](d). Thus phonetic writing may speed up the shorthand recording, but For CVC joinings 7×7 cases are possible in the newly-created homographs complicate the deci- all — many of them can be transformed by reflection phering of written notes. and rotation into one another. All these joinings can The pronunciation of a word has to be trans- be made G(2) continuous. It must be decided before formed to the above defined metaform, e.g.: writing a right consonant, whether the loop is to be needed: { n * ii d }.> I7 d >⇒ (n)-[i](dd) written as a left or as a right loop, compare: needing: { n * ii d }.> i ng >⇒ (n)-[i](,d,ing) team (t)+[i](m) ➥ vs. ❧ (m)-[i](t) meet This major task consists of a number of trans- formations done in this order: CVC ❳ ➇ ❢ ➟ ✥ ❸ ✷ ❳ ❩ ❴ ❪ ❵ ✢ ❫ ❬ • Elimination of minor (redundant) vowels such as the schwa @r in ➇ ➎ ➋ ➐ ➒ ➍ ➑ ➌ fewer: { f y * iu }> @r r >. ❢ ❦ ♦ ✐ ❥ ✈ ✉ There is a general problem with the schwas (@). Which are to be ruled out and which of them are ➟ ➭ ➦ ➥ ➠ ➤ ➳ ➯ to be backtransformed5 to the spelling equiva- ✥ ✧ ✦ ★ ↔ ➛ ➙ ↕ lent? Consider data: { d * ee . t == @ } ⇒ (d)-[a](t]-[a] ❸ ❻ ❼ ➀ ➁ ❺ ❾ ❽ upon: { @ . p * o n } ⇒ (u)(p)(n) ✷ ✾ ✻ ✿ ❁ ❀ ✺ ✽ Both @ are backtransformed, but stressed o is cancelled. Technically speaking the shorthand glyphs are • Finding of prefixes (un-, re-, ...), suffixes (-ing, realized as -ly, . . . ), handling of semiconsonants h, w. • an array of G(0)–G(2) continuous METAFONT- • Creation of ligatures such as dd, parenthesizing paths and consonants and bracketing circular vowels. • an array of discontinous marker paths. • Finding the proper orientation of loops. Slanting (default 22.5◦) and tilting3 of charac- These tranformations are done through a series of ters is done at the time of their shipping. cascaded context dependent rewrite rules, realized 4 which itself is a part of a text-to-speech system 3 G(2) continuity is invariant under affine transformations. 5 by a lex program
460 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Writing Gregg Shorthand with METAFONT and LATEX by finite state transducers (FSTs). An example of are entered in the form latex (i.e. latex#1) and such a rule is latex#2, respectively. " I7" -> 0 || [ t | d ] " }.>" _ " d >" The task of the online conversion of given text which zeroes the string " I7" in the left context of to shorthand notes is done in the following four consonant t or d and the right context of "d >"; i.e.: steps: { n * ii d }.> I7 d > ⇒ { n * ii d }.> dd > 1. The input stream, stripped of LATEX commands, Effectively this rule creates the ligature (dd) and is tokenized.8 There are two kinds of tokens: can be thought of as the reverse of the phonetic rule • Punctuation marks, numbers and common " e" -> " I7" || [ t | d ] " }.>" _ " d >" words for which a metaform entry in the which determines the pronunciation of perfect tense abbreviation dictionary of brief forms and suffix > e d > in the left context of t or d: phrases exists and { n * ii d }.> I7 d > ⇐ { n * ii d }.> e d > • the other words. 3 Abbreviations 2. At first for a word of the latter category its The Gregg shorthand, like every other shorthand pronunciation has to be found in Unisyn lex- system since Tironian notes (43 B.C.), takes advan- icon [4]. From this its metaform is built by a tage of the statistics of speaking6 and defines about program coded as the tokenizer above in the 380 so called brief forms. Here are the most com- XEROX-FST tool xfst [1]. mon English words: 3. In a mf run for each of the tokens using its the and a to of was it in that metaform a shorthand glyph (i.e. a METAFONT ➫ ✏ ✡ ➺ ② ➬ ❚ ➩ character) is generated on-the-fly. (th) (nd) (a) (t)(o) (o) (u)(S) (t) (n) (th)-[a] 4. Then the text is set with LATEX, rendered in the Writing the most common bigrams together, e.g.: pipeline → dvips → gs → ppmtogif and sent of the in the to the on the to be from the server to the browser. ③ ❘ ➽ ⑥ ➻ PDF output with better resolution can be generated. Also a djvu-backend exists, which produces an anno- (o)(th) (n)(th) (t)(o)(th) (O)(n)(th) (t)(b) tated and searchable shorthand record. PostScript spares lifting the pen between words thus increasing Type 3 vector fonts can be obtained, too. shorthand speed. These bigrams are entries in the dictionary of References about 420 phrases as well as the following examples: ✔ [1] Kenneth R. Beesley and Lauri Karttunen. Finite as soon as possible: -[a](s)(n)(S)(p): State Morphology. CSLI Publications, Stanford, ◆ 2003. if you cannot: +[i](f)(u)(k)(n): [2] Florian Coulmas. Blackwell Encyclopedia of Writing Systems. Blackwell Publishers, 1999. if you can be: +[i](f)(u)(k)(b): ▼ [3] C. deBoor, K. H¨ollig, and M. Sabin. High accu- racy geometric hermite interpolation. Computer thank you for your order: (th)(U)(f)(u)(d): ➨ ❰ Aided Geometric Design, 4:269–278, 1987. you might have: (u)(m)-[ai](t)(v): [4] Susan Fitt. Unisyn multi-accent lexicon, The abbreviation dictionary is coded in lexc [1]. 2006. http://www.cstr.ed.ac.uk/projects/ unisyn/. 4 text2Gregg [5] Donald E. Knuth. The METAFONTbook, vol- Our text2Gregg software is an online system,7 which ume C of Computers and Typesetting. Addison- records input text as Gregg shorthand. LAT X input Wesley Publishing Company, Reading, Mass., E th notation is accepted. 5 edition, 1990. Homonyma variants such as [6] Stanislav Jan Sarman.ˇ DEK-Verkehrsschrift latex;1,rubber; { l * ee . t e k s } ❜ mit METAFONT und LATEX schreiben. Die latex;2,computing; { l * ee . t e k } ❝ TEXnische Kom¨odie, to be published. [7] Charles E. Zoubek. Gregg Shorthand Dictionary, Centennial Edition, Abridged Version. Glencoe 6 The 15 most frequent words in any text make up 25% Divison of Macmillan/McGraw-Hill, 1990. of it; the first 100, 60%, . . . . 7 see project URL and also DEK.php for the German short- 8 Care has to be taken for proper nouns, which are under- hand DEK counterpart [6] lined in Gregg shorthand.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 461 The LuaTEX way: \framed
Hans Hagen Pragma ADE http://pragma-ade.com Abstract
ConTEXt s \framed macro has many flexible options for typesetting a paragraph. This short note discusses its reimplementation in Lua for ConTEXt MkIV.
1 \framed and its width Here the outer \hsize was made a bit smaller. As you can see the frame is determined by the widest One of the more powerful commands in ConTEXt is \framed. You can pass quite a few parameters line. Because it was one of the first features we that control the spacing, alignment, backgrounds needed, the code in ConTEXt that is involved in de- and more. This command is used all over the place termining the maximum natural width is pretty old. (although often hidden from the user) which means It boils down to unboxing a \vbox and stepwise grab- that it also has to be quite stable. bing the last box, penalty, kern and skip. That is, Unfortunately, there is one nasty bit of code we unwind the box backwards. that is hard to get right. Calculating the height of a However, one cannot grab everything; or, in T X speak: there is only a limited number of box is not that complex: the height that TEX reports E is indeed the height. However, the width of box is \lastsomething commands. Special nodes, such as determined by the value of \hsize at the time of whatsits, cannot be grabbed and make the analyzer typesetting. The actual content can be smaller. In abort its analysis. There is no way that we can solve the \framed macro by default the width is calculated this in traditional TEX and in ConTEXt MkII. automatically. 2 \framed with LuaTEX \framed So how about LuaT X and ConT Xt MkIV? The [align=middle,width=fit] E E {Outbeyondtheethernetthespectrumspreads macro used in the \framed command is: \unknown} \doreshapeframedbox{dosomething this shows up as (taken from Casino Nation by with\box\framebox} Jackson Browne): In LuaTEX we can manipulate box content at Out beyond the ethernet the Lua level. Instead of providing a truckload of ex- the spectrum spreads . . . tra primitives (which would also introduce new data types at the TEX end) we delegate the job to Lua. Or take this quote (from A World Without Us \def\doreshapeframedbox by Alan Weisman): {\ctxlua{commands.doreshapeframedbox \hsize=.6\hsize (\number\framebox)}} \framed[align=middle,width=fit] Here \ctxlua is our reserved instance for Con- {\inputweisman} TEXt, and commands provides the namespace for This gives a multi-line paragraph: commands that we delegate to Lua (so, there are Since the mid-1990s, humans more of them). The amount of Lua code is far have taken an unprecedented smaller than the TEX code (which we will not show step in Earthly annals by here; it s in supp-box.tex if you want to see it). introducing not just exotic functioncommands.doreshapeframedbox(n) flora or fauna from one iftex.wd[n]~=0 then ecosystem into another, but localhpack=node.hpack actually inserting exotic genes localfree=node.free into the operating systems of localcopy=node.copy_list individual plants and animals, localnoflines,lastlinelength,width=0,0,0 where they re intended to do locallist=tex.box[n].list exactly the same thing: copy localdone=false themselves, over and over. forhinnode.traverse_id('hlist',list)do done=true
462 TUGboat, Volume 29 (2008), No. 3 TUG 2008 Conference Proceedings The LuaTEX way: \framed
local p = hpack(copy(h.list)) we repack the content again, this time permanently. lastlinelength = p.width Now we use the maximum encountered width which if lastlinelength > width then is forced by the keyword exactly. Because all glue is width = lastlinelength still present we automatically get the desired align- end ment. We create local shortcuts to some node func- p.list = nil tions which makes it run faster; keep in mind that free(p) this is a core function called many times in a regular end if done then ConTEXt job. if width ~= 0 then In looking at ConTEXt MkIV you will find quite for h in node.traverse_id('hlist',list) do a lot of Lua code and often it looks rather complex, if h.width ~= width then especially if you have no clue why it s needed. Think h.list = hpack(h.list,width,'exactly') of OpenType font handling which involves locating h.width = width fonts, loading and caching them, storing features and end later on applying them to node lists, etc. end However, once we are beyond the stage of de- end veloping all the code that is needed to support the tex.wd[n] = width basics, we will start doing the things that relate more end to the typesetting process itself, such as the previous -- we can also work with lastlinelength end code. One of the candidates for a similar Lua-based end solution is for instance column balancing. From the previous example code you can deduce that manipu- In the first loop we inspect all lines (nodes with lating the node lists from Lua can make that easier. type hlist) and repack them to their natural width Of course we ll be a few more years down the road with node.hpack. In the process we keep track of by then. the maximum natural width. In the second loop
TUGboat, Volume 29 (2008), No. 3 TUG 2008 Conference Proceedings 463 Advanced features for publishing mathematics, in PDF and on the Web
Ross Moore Mathematics Department, Macquarie University, Sydney, Australia [email protected] http://www.maths.mq.edu.au/staff/ross.html Abstract Increasingly, mathematical, scientific and technical information is being dis- tributed by electronic means, but having a high-quality paper printout remains important. We show here examples of techniques that are available for having both high-quality typesetting, in particular of mathematics, as well as useful navigation features and text extraction within electronic documents. For HTML, we show some aspects of the use of jsMath within webpages; e.g., for mathematics journals or conference abstracts. With PDF, as well as the usual bookmarks and internal hyperlinks for cross-references and citations, advanced features include: (i) metadata attachments; (ii) copy/paste and search- ing for mathematical symbols or the underlying TEX coding; (iii) pop-up images of (floating) figures and tables; (iv) mathematical symbols within bookmarks; (v) bookmarks for cross-referenced locations. A further feature, particularly useful with mathematics papers, is the ability to make batched searches of the American Math. Society’s MathSciNet database, allowing hyperlinks to be generated for most bibliography entries.
1 Introduction with sectioning commands and other environments. The nature of scientific publication is changing: it is Here such effects are presented in a graphical becoming increasingly common for articles to be ac- way, as figures with extensive captions. Some ex- cessed and read on-line, without the need for print- tra text explains aspects that are not immediately ing. However, many researchers still prefer to print apparent from seeing the images alone, along with out an article, having first obtained it as a PDF file, a short outline of how these were implemented in A say. Thus it is necessary to produce the PDFs in such LTEX, in terms of which (internal) macros needed a way as to cater for both online and printed for- to be patched, and what extra resources were used. mats. For online reading, one needs navigation aids The HTML examples in Section 3 show use of 1 that give quick and easy access to cross-references, Davide Cervone’s jsMath software, which can be citations, metadata, and such. However, with retro- linked to any website to provide proper typeset- born versions of books and journal articles, the ad- ting of mathematics, as well as high-quality print- dition of such aids must not have an effect on the ing. This solves many of the problems that ac- original pagination. Furthermore, for long-term dig- company other methods of displaying mathematics ital archiving, accurate metadata and links to access within webpages. cited materials become especially important. 2 PDF files of full articles There are effects that are possible with current web-based and PDF technologies, but which hith- The author has produced retro-born versions of all 2 erto have not been widely used with scientific ar- articles from a complete journal volume as part of a ticles. The effects were programmed for process- feasibility trial to move from print-only to online ac- cess. Bearing in mind the considerations mentioned ing with pdf-LATEX, along with extra packages and above, and with pdf-T X and Unicode support be- coding to adjust the output produced by LATEX in- E ternal macros. Almost no further adjustments were coming more widely available, much care was taken made to the body of files used to produce the original to incorporate navigational aids that are available printed version of each paper, apart from the imposi- tion of \label and \ref commands where they had 1 jsMath homepage: http://www.math.union.edu/~dpvc/ not formerly been used with cross-references. Where jsMath/ . 2 Bulletin of the Australian Mathematical Society, Vol- papers had been submitted using AMS-TEX, or even ume 72 (2005); freely available online at http://www.austms. Plain TEX, then some extra markup was needed also org.au/Bulletin .
464 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Advanced features for publishing mathematics, in PDF and on the Web with PDF documents. Several new techniques were use with mathematics. Similar support for the full developed to make these documents as useful as pos- set of Euler fonts is planned, and other symbol fonts sible to researchers. These are now described briefly. also can be supported. a. Copy and Paste of blocks of text which include b. Images of figures and tables which pop up (see mathematical symbols (see Figure 1). Figure 2) near the place in the text where the This is achieved by adding a character map (CMap) figure/table has been referenced. resource3 to each of the fonts that TEX uses, for This feature allows figures and tables to be viewed typesetting the text and the mathematics. Such re- without changing the PDF page that is displayed. sources do not affect the appearance of the type- It requires JavaScript5 (or ECMAScript) to be en- set material, but associate each (perhaps accented) abled within the PDF browser. If the toggled image letter and mathematical symbol with its Unicode pops up in a place that is inconvenient for further code-point. In this way symbols are given a unique reading, then it can be shifted to elsewhere on the identity which can be used for copy/paste to other page. With further developments of the PDF specifi- applications, and for searching within the PDF itself. cations and browser software, the means to move the Currently the actual result of a copy/paste action image could be redesigned to become more intuitive. may depend on the particular software being used; (Indeed, it would be nice if browsers had a ‘cross- that is, the PDF-browser used to view the article, reference spy-glass’ feature, providing a small-sized and the text-editor, or other software, into which view of a different part of the same PDF, in response the content is pasted. to clicks on the cross-reference anchors.6) The author has produced CMap resources for In the event that JavaScript has been disabled the following old-style TEX font encodings: OT1, in the PDF browser, so that the pop-up mechanism OML, OMS, OMX. Articles in the journal volumes1 won’t work, the toggle button should not appear also used symbols from the AMS fonts MSAM and and the underlying cross-reference hyperlink should MSBM, Euler Fraktur fonts, and a few other char- work as usual. Unfortunately, not all PDF browsers acters, so CMap resources have been made for their implement this properly.7 Even worse, Xpdf and ‘U’ (Unknown) encodings; namely files umsa.cmap, eVince, prior to the v0.8.3 release (4 June 2008) of umsb.cmap, ueuf.cmap, ueufb.cmap, ulasy.cmap, and the Poppler library, would not even load documents upzd.cmap (Zapf Dingbats) and upsy.cmap (Adobe built with pdfTEX containing form fields; when built Symbol). Resources have also been created for LY1 with more recent versions of the library, these now (Lucida) and LMR (Lucida Bright Math symbols) work as intended. encodings. Implementing this pop-up feature was done by In the case of the OML encoding, as used with altering the expansion of the standard LATEX macros the cmmi math-italic font family, the ordinary let- \figure and \table, thus changing the way that ters A, B,..., a, b,..., z are associated with “math float contents are handled, and of \ref for build- alphanumeric symbols” in Unicode Plane 1. Such ing the toggle buttons. The first run of the LATEX symbols can be seen as the M, P , x and L within job is largely unchanged, apart from recording the the TextEdit window in the middle image of Fig- number of floats encountered as the expansion of a ure 1. However, bold symbols from cmmib fonts macro \hasfloats within the .aux file, followed by use the same OML encoding. Thus CMap resources a macro \testforfloats. On the next LATEX run, have been constructed that are specific to the font as the .aux file is read \testforfloats is encoun- face and style, rather than just to the encoding. tered, triggering code that reads the number of floats Similarly files omlmit.cmap and omlbit.cmap, sup- and causes extra packages to be loaded, such as port the cmmi and cmmib font families respectively. pdftricks.sty, insdljs.sty for inserting JavaScript cod- Similarly there is OT1-encoded support for normal, ing into a PDF, and pdfpopup.sty which has coding italic, sans-serif (medium and bold) and typewriter alphabets used with mathematics. 5 See http://en.wikipedia.org/wiki/JavaScript . 6 ALATEX package, called mmap.sty, is now avail- In fact the Skim browser for Macintosh (Mac OS X) has such a feature, providing an extra window to be opened, fo- CTAN 4 .cmap able at . This package provides these cusing on that portion of a document surrounding the target files and coding that causes the CMap resources to of a cross-reference hyperlink. This tool could be improved by be included when the appropriate font is loaded for respecting any destination view that may have been specified within the PDF. 3 See Adobe CMap and CID specifications at http://www. 7 For example, Apple’s Preview browser, at least up to adobe.com/devnet/font/pdfs/5014.CIDFont_Spec.pdf . version 3.0.9, neither supports JavaScript, nor respects the 4 . . . in the directory location .../tex-archive/macros/ button flags. Hence not only do pop-up images fail to work, latex/contrib/mmap/. but also the underlying hyperlink cannot be accessed.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 465 Ross Moore
Figure 1: Copy/Paste of mathematical content: the middle image shows the result of pasting the contents that have been selected and copied, as shown in the upper image. The actual result of a copy/paste action may depend on the particular software being used; that is, the PDF-browser used to view the article. The lower image shows how searching for mathematical symbols is done, with suitably enabled PDF viewers.
466 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Advanced features for publishing mathematics, in PDF and on the Web
————
————
Figure 2: Pop-up figures & tables: the images show how when the mouse hovers over a reference to a figure, it highlights a button to toggle showing and hiding a floating image. This image is moveable in case its natural position is inconvenient; e.g., obscuring where you wish to read.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 467 Ross Moore for building buttons to show and hide the pop-up essary for a scientific document to include explicit images. programming which controls how its content be dis- Now when a float is encountered, the full con- played, but just have declarations of which browser- tents of the environment is written out into a file, supplied functions are to be used. The implemen- named according to the figure or table number, as tation presented here is mainly to demonstrate the part of a new LATEX job which includes a preamble usefulness and practicality of such a ‘pop-up’ feature loading most of the same packages as in the main for cross-referenced material, so that browser ven- job. This subsidiary job is run, using LATEX fol- dors might be encouraged to incorporate a similar lowed by dvips/ps2eps/epstopdf to get a single-page feature within their own publicly-available software. PDF with correct bounding box. The file is loaded However, there are certainly other, simpler uses for back into the main job using \pdfrefximage within pop-ups to show extra images that are not found a \setbox, and its PDF object reference number elsewhere among the usual pages of a document. is recorded for later use in placing the image on a c. Extended use of bookmarks (see Figure 3), ... JavaScript button field, when an appropriate \ref Use of bookmarks is quite common for the major occurs. The object reference is also written into the sections of a document; this is automatic when using .aux file as the expansion of a macro having its name \usepackage{hyperref} with a LATEX document. derived from the figure number. This allows the ref- This is here extended further to creating bookmarks erence to be used on the next run, with any \ref for definitions, Theorems, Lemmas, etc., and also command that occurs before the float has been en- for figures, tables, and some equation displays. To countered, and also allows checking to see whether avoid the bookmark window becoming too cluttered, the reference number has changed, in which case a only those equations that have actually been cross- message is written to the .log file warning that an- referenced within the document are given their own other LAT X run is required. Finally, the tokens for E bookmark.8 the float environment are recovered and processed Having such bookmarks means that there are normally. named destinations with the PDF at all the impor- On subsequent LAT X runs, the image files do E tant places for the structure and content of the doc- not need to be rebuilt, but are loaded from the PDF ument. Furthermore these names are available in a images created on an earlier run. If editing of the separate file, so potentially this can be used to con- main document source changes the order of floats, struct hyperlinks directly to these important places. the toggle buttons may become associated with the This could be extremely useful in the context of a wrong images. Simply delete those images from the digital archive. current directory; the correct ones should be gener- ated afresh on the next LATEX run. In case the con- For figures and tables, a meaningful string to tent of a float contains \ref and \eqref commands be the textual anchor in the ‘Outline’ window is ob- or citations, the subsidiary job that creates an im- tained as the first sentence in the caption. This is age also loads a copy of the .aux file from the main obtained by reading the caption from the .lof or .lot job, which copy was made at the end of the previ- file and parsing to locate the first full stop (‘.’). With ous run. It’s possible that cross-references have not Theorems, Lemmas, Propositions, etc., the anchor- fully stabilised, so simply delete any affected image; text uses the appropriate numbering, as seen in Fig- after two more runs it will have been regenerated ure 3. The limiting of bookmarks to only those refer- and included. enced is achieved by patching the \@setref internal If browser software had a ‘cross-reference spy- macro to implement a ‘memory’ of referenced labels. glass’ feature (as suggested in the first paragraph of A line is written into the .aux file; this defines a this item), then not only would there be no need for macro, with name derived from the label. The cod- the PDF to contain JavaScript coding for the extra ing for placing equation numbers is patched so that buttons, but also there would not be doubling-up of \df@tag now also places an anchor, and a bookmark the information contained in figure and table floats. for this anchor when the memory indicates that the Furthermore, such a pop-up-like feature would ‘just equation has been referenced — which is known on nd A work’ also for cross-references to section headings, the 2 and subsequent LTEX runs. numbered equations, etc., as well as to the floats, and perhaps also for citations and ‘back-references’ 8 It can be argued that if an equation is not referenced from the bibliography (see Figure 7, for example). then it doesn’t need an equation number. However, many articles have been written where the author has not followed This is surely the way that such a feature ought this maxim. For creation of bookmarks, this maxim has been to be implemented; ideally it should not be nec- programmed-in.
468 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Advanced features for publishing mathematics, in PDF and on the Web
Figure 3: Bookmarks: these three images, taken from different articles, illustrate various aspects of the automatically created ‘bookmarks’. These provide easy access to the important parts of the document, including front-matter as well as sections of the mathematical article itself; such as definitions, theorems, proofs, remarks and figures. The middle image shows that mathematical symbols can be used within bookmarks, while from the lower one it can be seen that a bookmark is not produced for every numbered equation, but only for those that are cross-referenced within the article itself.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 469 Ross Moore
Figure 5: Index page: containing links to the Australian Mathematical Society’s website, for accessing each of the articles from the same journal issue.
metadata in useful formats: (i) a BibTEX entry suit- able for adding to a .bib file; (ii) a short HTML- formatted entry, suitable for an RSS feed; (iii) an extensive metadata file in XML format, which in- cludes the complete bibliography of the article, as well as its own publishing details. Each of these metadata files is generated “on- Figure 4: Cover page: this includes hyperlinks to the-fly” by suitable T X macro coding, using the recover the embedded metadata attachments, in E information provided for typesetting the PDF doc- various formats. Also there are links to the websites of the Mathematical Reviews and Zentralblatt Math ument itself. Thus, barring mistakes in the various reviewing agencies, where this article has been format translations required for this, the metadata reviewed. The large button enclosing the title links is guaranteed to be consistent with what appears in directly to the start of the article proper. print.9 Access to these files is made easy by anchors on the cover page (see Figure 4) located above the white text of the article title and authors, which it- d. . . . including support for mathematical symbols self anchors a hyperlink to the start of the article and some super-/subscripts. proper. The identifying strings displayed in the bookmarks When available, database codes for reviews of Bib window allow for the full range of Unicode code- the article are included with the TEX and XML points. This means that mathematical symbols can metadata files. In this case, the cover page has fur- appear, which is most appropriate for figure cap- ther anchors, for hyperlinks which give direct access to the review at Math Reviews10 and Zentralblatt- tions, and mathematics is used with (sub-)section 11 titles and theorem names, say. Even though the MATH. The same typesetting run that produces the PDF also builds an HTML webpage for the ar- Unicode specification does not allow for complete 12 alphabets of super-scripted and sub-scripted letters, ticle. This presents all the metadata and has the an attempt is made to make good use of those that same hyperlinks, and more. Some format transla- are available. Some further work is needed on this tions are also required when building such webpages. aspect of bookmarks. Part of the metadata for an article is the con- text in which it has been published; namely, the e. Inclusion of attachments (see Figure 4), contain- complete journal issue. For this, we have chosen to ing metadata for the article, in various useful for- mats. 9 This is a useful feature for freshly published articles, The metadata for a scientific paper is important but is not really appropriate for a digital library or preprint for various purposes, not least of which is citation archive, which would presumably have its own database of metadata already prepared and checked for the documents within future works. By distributing a document that it serves. with its own metadata, common difficulties can be 10 A subscription to MathSciNet is required to make use of avoided, such as the incorrect spelling of an au- this hyperlink. 11 thor’s name, or wrong affiliation, etc., or just getting Unregistered users have reduced access to the features available at this site. the page numbers wrong. These PDFs come with 12 http://www.austms.org.au/Publ/Bulletin/V72P1/ text-file attachments containing various amounts of 721-5019-LoPo/index.shtml
470 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Advanced features for publishing mathematics, in PDF and on the Web
———— Figure 7: Hyperlinks to MathSciNet: the small raised M is the anchor for a hyperlink that connects to the American Mathematical Society’s website. It links to the review of this cited bibliography item. It is possible to have multiple such links, as can be seen in entry [9] in the middle image of Figure 6. ———— tion. Figure 7 shows an elegant solution that places the hyperlink anchors discreetly into the left mar- gin. The caret-accent character has been chosen to be suggestive of an upward link; that is, to the ma- terial preceding the bibliography, which is the main body content. The second problem, which is perhaps the more Figure 6: Back-reference hyperlinks: the middle difficult one, is that of how to automate the col- image indicates how the caret (ˆ) in the left margin is lection of data required to build hyperlinks to re- the anchor for an active hyperlink, which jumps to the viewing services, such as MathSciNet and Zentral- location where the particular reference has been cited. blatt-MATH, or other online archives of scientific ma- The upper image shows the resulting change of focus. As there can be several citations of the same item, the terial. For these PDFs the author made use of a carets are right-aligned, with up to eight in a row, as small program called bmref, which is essentially a 13 in the lower image. ‘batched’ version of MathSciNet’s Mref tool. Pro- vided by Patrick Ion (Associate Editor of Math- ematical Reviews, AMS), this short Perl program include the complete front-matter that would ap- sends a carefully constructed XML file as a query pear in the printed version of the journal. The im- to the MathSciNet database, as an HTTP ‘POST’ ages of Bookmarks in Figure 3 show the kind of to http://www.ams.org/batchmref . The result re- material that is included: Editors, Copyright State- turned is an XML file containing the same data, but ment, Information for Authors, etc. Of course the with extra fields added for (i) the number of matches Index page (see Figure 5) lists all other articles ap- found, (ii) the reference numbers (MR-number) of pearing within the same issue. For each article there found matches, and (iii) full bibliographic informa- is an active hyperlink, using the page number as tion, for each bibliographic item included in the orig- the visible anchor, that directs a web browser to inal submission. the public page at the Australian Mathematical So- This allows the MR-numbers to be obtained for ciety’s website where the article’s abstract can be all items in the bibliography (well, all those that read, and its metadata (including references) exam- have been reviewed), with a single submission. Each ined. Also, the name of the article itself is the an- MR-number is sufficient to build the desired hyper- chor for another hyperlink to the start of the article link. Of course this is not 100% reliable, and some proper. searching at MathSciNet can uncover MR-numbers f. Hyperlinking from the bibliography to the place for items that were not found in the automated within the text (i.e. back-referencing) where the search; but the bulk14 of the job is done automat- citation occurred (see Figure 6), and to reviews ically. TEX coding was developed to analyse the at MathSciNet (see Figure 7). author-supplied bibliography prior to constructing Including back-references is not new, nor is having the XML file for use with bmref. Taking advantage of hyperlinks within the bibliography, when such are supplied by the article’s author. However, for arti- 13 See http://www.ams.org/mathscinet-mref . 14 cles where the original printed version did not have Of 516 separate cited items from 51 papers, 67 were not found automatically; 19 of these were found with some man- these features, there is the problem of how to include ual searching. The remainder were to journals not covered at the extra information without upsetting the pagina- MathSciNet, or to unpublished theses, etc.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 471 Ross Moore
Figure 8: jsMath in webpages: the image at right illustrates the high quality and proper scaling of mathematical expressions within a webpage that uses the jsMath applet software. This contrasts starkly with the image at left, displaying poorer quality and lack of scalability in the static images, used with older web technologies.
TEX’s \write18 feature to run external commands data readily available which could be easily searched and await the reply, the whole process can be fully instead. Indeed then it might be possible to include automated and integrated with the typesetting run: links not just to reviews of an article, but to the ar- craft the XML file, send to MathSciNet, analyse the ticle itself. Some appropriate icon or symbol would result, extract the MR-numbers, then make these then be used to indicate the kind of information to available for creation of hyperlinks. After a success- be found at the target of the hyperlink. ful result of one such run, there is no need for the same tasks to be repeated on subsequent typesetting 3 HTML pages for abstracts, etc. runs. At the end of 2007, the Australian Mathematical The bibliographic information returned can be Society made a complete change in the hosting ar- requested to be in any of the usual formats: bibtex, rangements of its website,15 as well as a change in amsrefs, TeX, html or as an HTML hyperlink. Thus the publication arrangements for its journals. There this information could be used to check the biblio- were still several journal issues16 that were not cov- graphic details provided by the author, or could even ered by the new arrangements, for which PDFs were replace it altogether. This was not done with these still to be served from the Society’s site. This ne- tests, due to the desire to have the online content be cessitated the need for abstract pages which were the same as what was printed; however, if any fac- publicly available, having hyperlinks to the PDFs tual errors were noticed (such as incorrect Volume which are accessible only with a subscription. The or page numbers) then these were corrected. job of creating these pages fell to the current author, in the rˆole of web-editor for the Society. In the hope of finding capabilities comparable The same techniques that were used for the Bul- to bmref, the author also searched Zentralblatt-MATH letin PDFs were used for this task, only now there and other journal archives, for the availability of on- was no full article PDF being produced. Thus the line tools for batched searches — unfortunately with- principal outputs were the HTML pages and meta- out success. Searches for a single article could be data files. MR-numbers were obtained in the same automated, but with fuzzy-matching this would re- sult in multiple hits, requiring significant extra pro- 15 Australian Mathematical Society’s website: cessing to determine whether the sought-after article http://www.austms.org.au/ . 16 actually had been located at that site. J. Austral. Math. Soc., Vol. 83 (1) & (2): http: //www.austms.org.au/Publ/JAustMS/, and ANZIAM In the context of a digital library or preprint Journal, Vol. 49 (1) & (2): http://www.austms.org.au/ archive, there may well be a large database of meta- Publ/ANZIAM/ .
472 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Advanced features for publishing mathematics, in PDF and on the Web
Figure 9: Help, Options & fonts with jsMath: the image at right shows how to bring up the jsMath control panel, displaying the ready availability of options and help features. In the Options panel, shown at right, one sees the great flexibility in the way different fonts or images may be used to construct the mathematics. Best quality is obtained by downloading and installing the jsMath fonts, or using local Unicode fonts. way as described above, only now the reference data processing since less information needs to be down- was presented in a different way, as LATEX-formatted loaded from that site. Alternatively, a local Uni- .bbl files. There were other differences in the meta- code font could be used for similar speed gains, but data too, requiring some minor adaptations of cod- the resulting layout of complicated mathematical ex- ing used previously. pressions might not be as finely tuned as with the A significant addition however, was the choice jsMath TEX fonts. Now Copy/Paste and searching to use jsMath for the mathematical expressions that refer to the Unicode code-points for mathematical appeared in titles, abstracts and occasionally within symbols, whereas otherwise these operations would the names of cited papers. Davide Cervone’s jsMath use the position in traditional TEX encodings. software1 is a JavaScript applet that is effectively Conclusion a cut-down version of a TEX compiler that does proper typesetting of stand-alone17 mathematical Here we have described several advanced features expressions. The author had used this before with applicable in particular to the electronic publication the ≈ 5000 abstract submissions for the ICIAM0718 and presentation of mathematical papers, but which Congress, and had made several suggestions for bug- have much wider utility. The emphasis is more on fixes and other improvements that Cervone willingly the nature of these features rather than on any spe- implemented. As well as providing a better qual- cific means of implementation, since various forms of ity presentation of mathematics within webpages, implementation can be possible appropriate to the jsMath also gives proper printing using fonts rather particular circumstances of distribution and produc- than images, and solves the problem of rescaling tion. The existence of example documents2 is “proof the mathematics to suit a web-surfer’s choice of font of concept” that these are achievable with current size. Figure 8 displays this, by giving a comparison TEX software. Indeed these can serve as a test bed with a page produced using older methods. for some of the features where there are inconsis- A lot of help is readily available, as Figure 9 tent levels of support within current browsers; e.g., shows. The jsMath ‘Options’ panel allows various Copy/Paste and searching with regard to mathe- choices of fonts to use with the mathematics being matical symbols, handling of overlaid annotations, shown. Specially prepared TEX fonts can be down- spy-glass views, etc. loaded. When installed in the local OS, these can Hopefully, in the near future, some of these be used with pages from any website that employs ideas will become standard practice, with consistent jsMath. This not only gives the best possible quality support across browser software, for the benefit of of image (since it leverages font-rendering machinery academics and researchers in mathematics and of on the local operating system), but also speeds up the scientific community generally.
17 It doesn’t do full pages, nor handle counters, cross- referencing, citations, etc. 18 Browse at the ICIAM07 timetable: http://www. iciam07.ethz.ch/timetable/ .
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 473 Multidimensional text
John Plaice School of Computer Science and Engineering The University of New South Wales UNSW SYDNEY NSW 2052, Australia plaice (at) cse dot unsw dot edu dot au
Blanca Mancilla School of Computer Science and Engineering The University of New South Wales UNSW SYDNEY NSW 2052, Australia mancilla (at) cse dot unsw dot edu dot au
Chris Rowley Mathematics and Statistics The Open University in London 1-11 Hawley Crescent, Camden Town London NW1 8NP, UK c.a.rowley (at) open dot ac dot uk
Abstract The standard model of text, based on XML and Unicode, assumes that docu- ments are trees whose leaves are sequences of encoded characters. This model is too restrictive, making unnecessary assumptions about documents, text and the processing applied to these. We propose instead that text and documents be encoded as tuples, i.e., sets of dimension-value pairs. Those dimensions used for content are split into the property dimensions, which are named by elements of an unstructured set, and the indexing dimensions, which form a structured set, often enumerated. Using our approach allows natural solutions for a wide range of encoding requirements: encoding of documents at multiple levels of abstraction (glyph, character, word, stem-declension pair, compound word, etc.); encoding by linear, tree, DAG and multidimensional structures. Our model is upwardly compatible with existing approaches.
1 Introduction into the nature of text initiated by authors Plaice In this article, we present a new model for manip- and Rowley [5, 6]. The current standard computer ulating documents in which every structure is en- model of documents assumes that the structure of coded as a tuple, a set of dimension-value pairs. The a text is a tree—normally encoded using XML — simpler elements are ordinary tuples encoding basic whose leaves are sequences of Unicode characters information, while more complex elements encode and where the intermediate nodes contain sets of mappings from structured index sets towards sim- attribute-value pairs to define properties. The con- pler elements. clusion of the aforementioned research was that the The advantage of this new model is that it al- current model makes the assumption that text is lows documents to be encoded in many different simply something to be shuffled around, possibly ways, taking into account logical structure, visual chopped up for rendering purposes, but that it has structure and linguistic analysis. Furthermore, the no structure of its own; furthermore, the origin of proposed model is upwardly compatible with exist- the view of a document as some form of stream ing practice. of bytes can be traced directly back to the near- This model is one result of a research project simultaneous invention of the typewriter and the
474 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Multidimensional text telegraph. The relatively recent distinction between The structure of this paper is as follows. We “character” and “glyph”, where character is an ab- present a brief analysis of, and show the current lim- straction from glyph, still makes the assumption itations of, various existing models for the encoding that the visual presentation of text is the key, de- of texts and documents in §2. We follow with a pre- spite the fact that, for example, English, Amharic sentation of our new model (§3) using an extended and Chinese are normally encoded at, respectively, example. We then (§4) describe some features of its the letter, syllable and (sub)word levels. use to declined word-stem sequences, parse trees, hi- As large collections of documents are brought erarchical document structure, attribute-value ma- together, so that they can be searched, it is clear trices and tables. We conclude with a discussion of that this “standard” model leaves something to be future work. desired. Searching through a multilingual database of texts requires substantial linguistic support, and 2 Existing models here it becomes essential that the different languages We examine in this section three approaches to deal- be handled at similar levels. Moreover, in linguis- ing with text. Although not exhaustive, it does pro- tics, texts are often encoded using parse trees and vide us with indications of where a more complete attribute-valued matrices, often DAGs (directed acy- model should be heading. clic graphs) or digraphs (directed graphs). As for the structure of documents containing 2.1 XML documents text, the tree structure itself is not always appro- An XML document is typically encoded as a tree. In priate. Although it is true that a two-dimensional some sense, this is all there is to say, but a proper table can be encoded as a tree, this is only true understanding of XML requires examining not just by imposing ordering on its two dimensions (either the obvious tree structure, but also the structures of by rows or by columns); they are not naturally en- the nodes and the leaves in these trees. coded by a purely hierarchical structure but by its For each element in an XML document, there antithesis, a completely crossed structure. More- is a possibly empty list of attribute-value pairs, and over, two-dimensional tables are often used to visu- a possibly empty list of child elements. The leaves alize multidimensional data, whose encoding truly of the tree consist of PCDATA (Parsed Character requires a multidimensional data structure such as Data) or CDATA (Character Data), in both cases those investigated in a different context by one of sequences of characters from the Unicode or some the authors [1], not just to capture and simplify the other character set, with PCDATA being parsed by semantics, but to ensure an efficient and tractable the XML parser. storage mechanism. Therefore XML, often presented as a simple en- Importantly, what distinguishes the electronic coding, actually requires four data structures to de- document from all previous forms of document is scribe a document: that it is recreated every time that it is read, lis- • the tree; tened to, studied or processed. As a result, with a • the list of elements; slight change of parameters, it can be recreated dif- • the attribute-value list; and ferently from any previous occasion. The standard • the character sequence. model completely breaks down for these kinds of situations: it must resort to programming the docu- In addition, there are arbitrary restrictions on at- ment and, in so doing, loses the possibility of having tributes which limit their usability: attributes can- the document being properly indexed for searchabil- not contain multiple values, nor can they contain ity. (See [4] for further discussion.) tree structures. The tuple structure that we are proposing al- Our model includes such XML tree-based docu- lows us, as shall be shown below, to define a num- ments but it can also handle non-hierarchical struc- ber of different structures, including ordered streams tures that do not necessarily have any natural XML and trees, DAGs and multidimensional structures. encoding. For example, a sentence in a document can be en- 2.2 Linguistics AVMs coded as a sequence of characters, as a parse tree with words as leaves, or as a “feature structure” In linguistics, it is common to model written lan- from linguistics known as an attribute-valued ma- guage using structures consisting of a tree and an as- trix. Indeed, it could easily be encoded as all of the sociated attribute-value structure [3, p. 16]. For ex- above, with appropriate link structures connecting ample, here is the parse tree for the sentence “Mary the components. seems to sleep.”:
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 475 John Plaice, Blanca Mancilla and Chris Rowley
[S1: [ NP2: Mary Assignments Examinations VP1:[V1: seems Grade VP3: [ Aux3: to Ass1 Ass2 Ass3 Midterm Final VP3:[V3: sleep ]]]]] 1991 Winter 85 80 60 75 75 The associated structure, called an attribute-value Spring 80 65 75 60 70 70 matrix (AVM), is presented below. Fall 80 85 55 80 75 Note: The subscripts in the parse tree correspond 1992 Winter 85 80 75 75 to the nodes in the AVM. Node 2 appears twice 70 in the AVM, being the subject of both node 1 and Spring 80 80 75 75 node 3. Fall 75 70 65 60 80 70 It appears that, in 1991 alone, Assignment 3 was pers 3rd agr identical across the three terms but for 1992, we subj 2 "num sg # can see no such simple explanation of the larger box pred mary since it seems to amalgamate this assignment with an examination. 1 subj 2 In this example and using their notation, the comp 3 pred sleep three dimensions and their value sets are as follows: tense none pred seem Year = {1991, 1992} tense pres Term = {Winter, Spring, Fall} Because of such shared structures, AVMs are Assignments · Ass1, often considered to be DAGs. Howewer, it is possible Assignments · Ass2, to have cyclical structures in an AVM: Consider the Assignments · Ass3, Marks = noun phrase “the man that I saw”, whose parse tree Examinations · Midterm, is here: Examinations · Final, Grade [ NP1: [ Det: the N’: [ N: man A small change of syntax would transform this into S’2: [ Comp: that an example of our model. S: [ NP: I VP: [ V: saw ]]]]]] 3 The new model
In the following AVM, a cyclical structure is needed In this section we present an extended descriptive to describe the “filler-gap” dependency in the rela- illustration of the model, using an example, rather tive clause [3, p. 19]: than a detailed formal model. There is only one basic structure in this model: def + the tuple, which is defined as a set of dimension- pred man value pairs. This is not a new data structure, and pred saw it has many different names in different formalisms: 1 dictionary in PostScript, hash-array in Perl, map comp 2 subj pred pro in C++, association list in Haskell, attribute-value obj h1 i list in XML, and tuple in Standard ML and Linda. We begin by proposing a possible encoding for Our model naturally encodes both such parse the sentence “LATEX 2ε is neat.” trees and these AVMs. [ type: sentence 2.3 Tables numberWord: 3 endPunctuation: [ type: unichar, code: 002E ] Xinxin Wang and Derick Wood [7] developed a gen- 0: [ type: TeXlogo eral model for tables, in which a table is a map- TeXcode: "\LaTeX\thinspace2\lower1pt ping from a multidimensional coordinate set to sets \hbox{\small$\varepsilon$}}" of contiguous cells. For them, the two-dimensional SimpleForm: [type: digiletters format commonly used to present a table is not the unicharstring: "LaTeX2e" ]] internal format. Here is an example of the use of a 1: [ type: word numberChar: 2 3-dimensional coordinate set from their paper: 0: [ type: unichar, code: 0069 ]
476 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Multidimensional text
1: [ type: unichar, code: 0073 ] ] numbers, whose structure is the unique count- 2: [ type: word able well-ordering. numberChar: 4 2: [ type: unichar, code: 0061 ] More precisely, the property dimensions form 1: [ type: unichar, code: 0065 ] an unstructured set, whilst the indexing dimensions 0: [ type: unichar, code: 006E ] form a subset of a structured set. We can, for exam- 3: [ type: unichar, code: 0074 ] ] ] ple, write a tuple of type typespec that defines the In this example, there are 12 tuples: sets needed for the dimensions in sentence tuples: • 1 sentence, [ type: typespec tupletype: sentence • 1 TEX logo, typedim: {type} • 1 character string consisting of a “word and dig- propdim: {numberWord, endPunctuation} its”, indexdim: N • 2 words, and ] • 7 Unicode characters. Note that, although the set of possible index dimen- The different dimensions in the example play sions N is the infinite set of all natural numbers, any different rˆoles. In fact, there are three categories of given tuple will use only a finite number of numbers dimensions: as indices. Also, it is important that, for example, the order of the characters in a word is defined by • The typing dimensions allow one to distinguish the ordering of their indexing dimensions as natural the different kinds of tuple. Thus all tuples numbers, not the order in which they appear in the must have a typing dimension. In the exam- written form of the “word tuple”. In this example ple, a special dimension, called type, provides we write the characters in the order of the numbers the information. This is the standard solution, of their Unicode slots (such an ordering may be very although we do not exclude the use of further efficient for certain applications). special dimensions, say subtype or version, for In all of these sets, of dimensions and possible further clarification. values, both equality and membership must be com- • The property dimensions are used to store any putable. In all usable examples, of course, equality- type of information about the tuple. In the ex- testing and other necessary set operations should be ample, these dimensions are: at worst of polynomial time complexity for reason- – numberWord and endPunctuation, for ably sized sets. tuples of type sentence; The tuples used as values may be of any type, – TeXcode and SimpleForm, for the thus, for example, they could include strings, files, TeXlogo tuples; programs, and so on. – unicharstring for digiletters tuples;1 There can be other interesting interplays be- – numberChar for word tuples; tween dimensions and values. Consider, for exam- – code for unichar tuples. ple, the Unicode character tuple: • The indexing dimensions are used to access the [ type: unichar, code: 002E ] substructures of a tuple (the content of the tu- Here the value 002E is a value used to index the Uni- ple) via an indexing mechanism or structure. code character database and, as Bella [2] has shown, The set of all the indexing dimensions available that database can be understood as a single tuple to a given type of tuple can be very large, con- indexed by the natural numbers whose entries are ceptually infinite, and will often carry a com- themselves tuples containing various kinds of infor- plex structure. mation about each Unicode character. In the following sections we will extensively The tuples used in this model are conceptual: develop examples of how the structure of these they can be implemented—both as data structures indexing dimensions can be used to encode com- in running programs and as sequentialised files on plex systems. In the example, both the sen- disk — in many different ways. Depending on the ex- tence and word tuples use the natural num- act applications, algorithms and programming lan- bers (N) to enumerate their content; thus the guages and environments, some solutions are more indexing dimensions available are the natural appropriate than others. 1 This could be replaced by a more formal tuple of type 4 Examples digiword that is like word: being an indexed collection of unichar tuples that can also contain digits. We hope it is now clear that our tuple structure
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 477 John Plaice, Blanca Mancilla and Chris Rowley can encode any sort of simple record or entry. In • Hierarchical document structure: The traditio- this section, we indicate with a few more examples nal book with frontmatter, chapters, sections how the tuple structure can encode different kinds and subsections is a typical example of a doc- of data structure. ument tree. This hierarchy can be extended As was described in the previous section, there downwards to paragraphs, sentences, phrases, are typing, property and indexing dimensions in a words and characters. tuple. If we restrict ourselves to the indexing di- mensions, then the tuple can be considered to be a • Attribute-value matrices: As explained in §2, (partial) function from an index set (the structured these contain shared structures. Below is the set of indexing dimensions) to a set of values, which AVM for the last example sentence. are in general themselves tuples, interpreting a sin- [ type: AVMentries gleton value as a type of tuple singleton and in- numberEntries: 3 dexing set of size 1 {0}. If, in the sense of datatypes, 1: [ type: AVM these singleton values are strongly typed, then these subj: 2 singleton values will need a datatype property di- comp: 3 mension. pred: seem tense: pres ] • Declined word-stem sequences: To enhance doc- 2: [ type: AVM ument search or to effect grammatical analy- agr: [ type: AVM sis, it is common to stem words, separating the pers: 3rd stem and the prefixes or suffixes of the words. num: sg ] We would then end up with entries such as: pred: mary ] [ type: verb 3: [ type: AVM language: English subj: 2 stem: carry pred: sleep mood: indicative tense: none ] ] tense: present perfect voice: active • Tables: The encoding of tables by Wang and person: 3rd Wood is our final example of the model. Here number: singular is a possible encoding, where the dimensions are unicharstring: "has carried" ] allowed to range over a set of possible values in order to encode the boxed values. or [ type: noun [ type: WWtable language: French [ type: WWindex stem: pomme year: 1991 gender: feminine term: Winter number: plural mark: Assignments.Ass1 unicharstring: "pommes" ] ] : 85 ... Should the system not be able to parse such [ type: WWindex a word/phrase, then it will store only its uni- year: 1991 charstring until it is appropriately updated. term: Winter..Fall • Parse trees: The result of the natural language mark: Assignments.Ass3 parsing of a sentence is a richer text structure ] : 75 that is often encoded in a tree structure. ... [ type: WWindex Below is a possible parse tree for the sentence year: 1992 “Mary seems to sleep.”, first presented in the term: Winter..Spring section on AVMs (§2): mark: Assignments.Ass3..Midterm [ type: sentence ] : 70 NP: Mary ... VP: [ type: verbPhrase [ type: WWindex V: seems year: 1992 VP: [ type: verbPhrase term: Fall Aux: to mark: Grade VP: [ type: verb ] : 70 V: sleep ]]]] ]
478 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Multidimensional text
5 Conclusions References The model introduced in this paper, based on gen- [1] R. A. Bailey, Cheryl E. Praeger, C. A. Rowley, eral tuples, is as simple as possible whilst being flex- and T. P. Speed. Generalized wreath products ible enough to encode a large range of different ap- of permutation groups. Proc. Lond. Math. Soc., proaches to the study and manipulation of text in s3–47(1):69–82, July 1983. all of its forms, as well as to support the encoding [2] G´abor Bella. Mod´elisation du texte num´erique and linguistic tools such as language dictionaries. multilingue: vers des mod`eles g´en´eraux et exten- However, the real power of the model is the idea sibles fond´essur le concept de text`eme. PhD the- of the index, built right into the model, which pro- sis, T´el´ecom Bretagne, Brest, France, 2008. vides access to any piece of data, thereby support- [3] Mark Johnson. Attribute-Value Logic and the ing the specification of algorithms and hypertext- Theory of Grammar. Center for the Study of like links between document components. Language and Information, Stanford University, The infinite index sets correspond to iterators 1988. into containers, as used, for example, in the C++ [4] Blanca Mancilla and John Plaice. Possible STL (Standard Template Library) for generic pro- worlds versioning. Mathematics in Computer gramming. Furthermore, using these sets it is even Science, 2008. In press. possible that certain tuples are, conceptually, count- able infinite (like lists in a functional programming [5] John Plaice and Chris Rowley. Characters are language), with the components being evaluated in not simply names, nor documents trees. In Glyph a lazy manner, on-demand. and Typesetting Workshop, East Asian Center In future papers we shall show how this unifying for Informatics in Humanities, Kyoto University, data model makes it easy to combine in a single sys- 2003. http://coe21.zinbun.kyoto-u.ac.jp/ tem myriad ways of editing, storing, manipulating papers/ws-type-2003/009-plaice.pdf. and presenting text and to manipulate all of these [6] Chris Rowley and John Plaice. New directions in together. document formatting: What is text? In Glyph and Typesetting Workshop, East Asian Center for Informatics in Humanities, Kyoto University, 2003. http://coe21.zinbun.kyoto-u.ac.jp/ papers/ws-type-2003/001-rowley.pdf. [7] Xinxin Wang and Derick Wood. A concep- tual model for tables. In Ethan V. Munson, Charles K. Nicholas, and Derick Wood, editors, PODDP, volume 1481 of Lecture Notes in Com- puter Science, pages 10–23. Springer, 1998.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 479 Data mining: Role of TEX files
Manjusha Susheel Joshi Bhaskaracharya Institute of Mathematics Pune 411004, India manjusha dot joshi (at) gmail dot com http://www.bprim.org
1 Background When users access the particular year of the In a recent data mining project, I was trying to un- journal, they can also see the overall topics easily, derstand how to extract important words from a doc- and a large set of keywords to help navigate through ument. I realised that while writing a document, an the articles. Even though authors provide keywords author usually emphasizes important words by us- now, they usually merely highlight the topic or main ing italics, bold face, underlining, or quotes. This is theme of the article. Here we are considering more a general observation for electronic documents. feature words from the document. While working on the project to try to find out If a user asks for some particular keyword, since appropriate information from the document set, I all the articles are already grouped according to their was looking for a better file format. In this regard, topics, a search program can show the user corre- sponding articles, by looking at the feature word the TEX file format attracted my attention. data. Thus, searches can be faster and better when Nowadays most research journals accept TEX files from authors. Journals then put pdf files of the TEX documents are available. articles or abstracts on the web site, or submit them Query User =⇒ Journal website =⇒ Specific paper for printing. TEX PDF 3 Why TEX files and not PDF in general? Authors =⇒ research journals =⇒ User 1. PDF files are rather heavy in size, while TEX At present, the research journals typically clas- files are light. sify papers by the year of publication or volume of 2. One can collect feature words when the file is in the issue in which a given paper is published. TEX format. TEX files are plain text, so rules Almost always, users have access only to PDF to process them are fairly easy to design. For files, and the journals do not publish the T X sources E instance, we can find boldface words in the TEX of the articles in any way. file with the rule ‘Search for the pattern ‘{\bf ’ 2 Situation and save words until the matching }. Another example: ‘Ignore words starting with \’. Ulti- Journals mostly have their own style files that take mately we can collect the actual content of the care of their abstract formatting, section heading document. Once collection is over, we do not style, headers, footers, and so on. They generally need to process the TEX file again. support keywords, citation, index, content, etc. All 3. Text extracted from PDF files often doesn’t un- these features make the T X file a very special doc- E derstand ‘fi’ or ‘ff’ ligatures properly; moreover, ument — special in the sense that one can extract Greek letters α,β,... are not understood by ‘feature words’ from the document relatively easily. present text extractors. Specifically, words in the index, abstract, sec- tion headings, and emphasized words in the docu- 4. For figures, text extractors typically find ASCII ment body are words which we can call feature words codes instead of text; many times we have ob- served garbage when a figure is present in the of the document. So for such TEX files, these feature words can be extracted, and submitted along with text file. the TEX file to the journal’s website. 5. Submission of the feature word file along with Now, suppose we have available a collection of the PDF file is possible. In fact, each PDF such TEX files for a year. Then from all the fea- file can be represented by the collection of fea- ture words associated with the TEX files, a program ture words for that file. When a user makes can collect feature words, understand which word a query, instead of searching in the entire PDF is from which group and make groups of these doc- file, searching can be done in only these key- uments based on clustering techniques (http://en. word collections, which would be considerably wikipedia.org/wiki/Data_clustering). faster and produce more relevant results.
480 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Data mining: Role of TEX files
4 Why TEX files and not other markup? There are many commands with which the TEX Consider HTML files: for line breaks, paragraphs, compiler collects important information from the even for extra space, explicit commands are required, TEX file to highlight in the output file. Some well- which makes the source full of commands which do known examples: not hold any information with respect to the content • For emphasis, LATEX uses commands such as of the article. {\bf ...}, \textbf{...}. So we can identify On the other hand, if we look at a typical TEX emphasized words fairly easily. file, it uses fewer formatting commands in compari- • To add a word to an index: \index{word}. son to an HTML file. For example, paragraphs are • For section headings: \section{...}. indicated simply by blank lines. Thus, there is less So we are assuming that the (LA)T X compiler disturbance when extracting text from the source. E understands that some words are to be highlighted 5 User wishes for some reason. We are interested in these words given some special importance by the author. Thus, Suppose a user wants to search for a general concept the general picture looks like this: in a repository. This would usually require a full text search, which is time-consuming. To speed up the TEX file search, what if the documents were already clustered by subject or concepts? If we can classify documents beforehand, the word extractor TEX compiler search process could be more like this: • Do we have the query results already saved? If . so, return them. file of feature words . • If not, the query is made against the ‘cluster Suppose there is a file of 2000 words, out of representatives’, described below, to return the which we found say 300 words that are special words. appropriate documents. Clearly we can increase speed of searches. 6 Challenges 7 Looking ahead This leads to the question of how to form such con- We can cluster documents based on their feature cepts or clusters beforehand, without knowing the words. For example, we might try this rule: query. How should such clusters be represented, and If documents have more than 70% of the fea- how do we find the representatives? ture words in common, group them together. In a document source file, the author highlights words with additional, perhaps invisible, markup. Now clusters will be defined according to their These words presumably help to represent the doc- common feature words, 70% in this case. The re- ument. How do we capture these representatives? maining 30% of words for each document in the clus- From a PDF file, text extraction is not simple, ter can be a secondary representative of the cluster. as we have seen. Another common format, RTF (rich To handle a query, we can search in the pri- text format), is even less straightforward. mary representation of the clusters and if not found, HTML format is somewhat better, although the search in secondary representations. This will make additional commands mentioned above complicate our search even faster. the job. Journals can provide this facility to their users, and it would be useful for other areas where TEX files are in use.
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 481 Abstracts
Editor’s note: Many of the conference presenta- open-source standard, and Google is on track to replace tions are available at http://www.river-valley. Microsoft. tv/conferences/tug2008 in video form, thanks to In 1997 the LATEX3 project said that LATEX3 would Kaveh Bazargan and River Valley Technologies. provide: • A new input document syntax, that aligns with −−∗−− SGML/XML. MathTran and TEX as a web service • A new class file interface, that aligns with Jonathan Fine SGML/XML. • A new style-designer interface that can work with a In 2006/7 I developed and set up the public MathTran visually-oriented, menu-driven specification system. web service www.mathtran.org. This was done with fund- • An effective interactive help system for document ing from JISC and the Open University. It provides authors. translation of T X-notation formulas into high-quality E • Thoroughly documented and modular source code. bitmaps. In April 2008 it served 48,000 images a day for a growing range of sites. After tuning, the server could These goals are still worth achieving. This talk will focus provide about 2 million images a day. It takes about 10 on some recent progress, and in particular: milliseconds to process a small formula. • Use of key-value syntax within tags. MathTran images contain rich metadata, including • Separation of parsing from processing. the TEX source for the image and the dvi and log outputs • An improved development environment. due to that source. This makes it straightforward to • On-line interactive help systems. edit and resize such images, or convert them to another format, such as SVG or PostScript. LuaTEX, MPlib, and random punk MathTran, used with JavaScript, makes it consid- Hans Hagen erably easier to put mathematics on a web page. In We use new LuaTEX and MPlib features to generate particular, the author of the page does not need to in- random characters from Donald Knuth’s punk font. This stall any special software, and does not have to store was the ‘surprise’ talk on the TUG’08 program. The full thousands of image files. paper will appear in a future issue of MAPS. The MathTran project is now focussed on the au- thoring of mathematical content. It has produced a Image handling in LuaTEX prototype instant preview document editor. Funded by Hartmut Henkel the 2008 Google Summer of Code, Christoph Hafemeister is developing JavaScript to provide autocompletion for The Lua language allows for defining new variable types, and LuaT X uses this concept for types like ‘node’ and commands and analysis of TEX errors, all integrated with E an online help system embedded in the web page. Sepa- ‘font’. In this talk an image library as part of the LuaTEX rate work is focussed on developing MathTran plugins engine is presented, built around a new ‘image’ type, giv- ing extended image handling and embedding capabilities. for WYSIWYG editor web-page components. This talk will present progress and prospects. It The image primitives inherited from pdfTEX are still fully will also discuss some of the broader implications for the functional for compatibility. First the process of image embedding and its limi- TEX community and software, such as tations using the pdfTEX primitives is described. Then, • People using TEX without installing TEX on their after a short introduction about Lua libraries, the ‘image’ machine. type of LuaTEX is presented together with the set of • Help components for web pages. new Lua functions for image handling, and their use is • Integration with third-party products. illustrated by examples. As work is still ongoing, possible • Standards for TEX-notation mathematics. future extensions are discussed as well. • Learning and teaching TEX. LuaTEX: what has been done, and A Why we need LTEX3 what will be done Jonathan Fine Taco Hoekwater
The LATEX3 project started in 1992. Since then, much At TUG 2007 in San Diego, the first beta version of has changed. XML has replaced SGML and along with LuaTEX was presented. This year the team presents a X/HTML has become the dominant markup language. version where significant parts of the TEX-Lua API are CSS has replaced explicit style attributes in HTML pages, stable. This talk will give an overview of the components and is now a widely understood and used language for that make up LuaTEX: what libraries do we have and specifying design. Internet access is considerably more what callbacks are available. The team has some ideas widespread, the web has gone from 1.0 to 2.0, Microsoft about the next stages of development and these will be has replaced IBM, Linux went from nothing in 1991 to an presented as well.
482 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting Abstracts
The galley Module or: How I Learned to Stop The data structures required for this model are Worrying and Love the Whatsit remarkably simple, and will allow the rapid development Morten Høgholm of many different kinds of algorithms.
TEX has a well-deserved good reputation for its line break- Windows of opportunity: A (biased) personal A ing algorithm, which has found its way into other software history of two decades of LTEX development — over the years. When it comes to inter-paragraph ma- Are there lessons to be learned? terial such as penalties, skips and whatsits, things start Frank Mittelbach getting murky as TEX provides little help in this area, especially on the main vertical list where most of the Looking back at twenty-odd years involvement in LATEX action is. development and maintenance the author highlights the This article describes the galley module which seeks (in his opinion) most important milestones and pitfalls. to control line breaking as well as taking care of inter- • What are significant events that came at the right paragraph material being added at the right time. In moment? other words, galley can assist packages such as breqn, • Which important events came at the wrong mo- whose goal is to construct paragraph shapes on the fly ment? while taking current ones into account as well as ensuring • What were the biggest failures and why? the output routine doesn’t get tricked by penalties, skips From this data the article attempts to draw conclusions and whatsits appearing in places where they could allow as to how the future of LATEX could be shaped in a breakpoints where none are intended. way beneficial to everybody involved and what needs to Minion Math: The design of a new happen to make this possible. math font family A pragmatic toolchain: TEX and friends and Johannes K¨uster friends of friends “Minion Math” is a set of mathematical fonts I have Steve Peter developed over the past 6 years. Designed as an add-on In this talk, we present the toolchain used to produce package to Adobe’s Minion Pro font family, it consists of the award-winning Pragmatic Bookshelf titles (http:// 20 OpenType fonts (4 weights, times 5 optical sizes). In www.pragprog.com) and examine some of the pleasures future releases it will cover the complete Unicode math and pitfalls encountered using T X, XML, XSLT, Ruby symbols, and more. E and other open technologies. In the design I tried to remove constraints and to avoid flaws and shortcomings of other math fonts, with Parallel typesetting the aim of creating the most comprehensive and versatile Toby Rahilly, John Plaice, Blanca Mancilla set of math fonts to date. Here I present the main design principles of Minion We present the general mechanism by which logical con- Math, and the most important design decisions I took. I tent, arranged in multiple interacting containers, can will show samples of the fonts and will compare the fonts be typeset into a set of visual substrates. The overall to other math fonts as well. algorithm is iterative, with the successive iterations re- Trademark attribution: Minion is either a trade- fining a multidimensional context that parameterises the mark or registered trademark of Adobe Systems Incor- behavior of the algorithm. porated in the United States and/or other countries and Each iteration consists of three parts. First, each used under license. visual substrate is informed which parts of which logical containers are to be placed thereon. Second, in parallel, Multiple simultaneous galleys: the content placed in the substrates is typeset. Third, the A simpler model for electronic documents resulting layout in each substrate is assessed for goodness, Blanca Mancilla, John Plaice, Toby Rahilly thereby resulting in the refinement to the overall context. We present a general model for electronic documents In the talk, we will present the theory and the supporting parallel containers of content, tied together practice behind this algorithm. through link components. This model is usable for a wide Three typefaces for mathematics range of documents, including simple textual documents Daniel Rhatigan with footnotes and floats, complex critical editions with multiple levels of footnotes and critical apparatus, maps This paper examines the issues involved in the design of with multiple layers of visual presentation, and music typefaces for mathematics. After a brief discussion of scores. some of the typographic and technical requirements of This model is inspired from the C++ Standard Tem- maths composition, three case studies in the development plate Library, whose basis is that Containers + Iterators of maths types are presented: Times 4-line Mathematics + Algorithms = Programs. In our approach, the ‘iter- Series 569, a complement to the Times New Roman text ators’ are pointers into the parallel containers, keeping types as set with Monotype equipment; American Math- track of callouts for notes, floats, and parallel links. ematical Society Euler, an experimental design intended
TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting 483 Abstracts to contrast against non-mathematical typefaces set with extended the algorithm for a number of complex cases, TEX; and Cambria Math, designed in concert with a new including correct drawing of consanguinic relationships, text face to take advantage of new Microsoft solutions twins and many others. for screen display and maths composition. In this talk we review the facilities of the current ver- In all three cases, the typefaces were created to show sion of the program and the new challenges in computer- the capabilities of new technological solutions for setting aided drawing of medical pedigrees. maths. The technical advances inherent in each font We try to make the talk interesting to TEXnicians are shown to be as central to its function as its visual by discussing the experience of design a TEX-based ap- characteristics. plication working in a “real world”. By looking at each typeface and technology in turn, and then comparing and contrasting the issues that are Observations of a TEXnician for hire addressed in each case, it becomes apparent that even Boris Veytsman though certain challenges are overcome with technical Several years ago the author was tempted by extremely advances, the need to consider the specific behaviours of cheap rates for TUGboat advertisements, and declared type in a maths setting remains constant. urbi et orbi he was a TEX consultant. This audacious See http://www.ultrasparky.org/school for the step led to many interesting experiences. Some results of complete paper and other typographical items. this work were published on CTAN and listed at http:// borisv.lk.net/latex.html (the list includes both com- Medical pedigrees with TEX and PSTricks: New advances and challenges missioned packages and the ones I wrote for my own purposes). Boris Veytsman, Leila Akhmadeeva In this talk I report on my past projects, big and A medical pedigree is an important tool for researchers, small, and discuss the lessons learned from my journeys clinicians, students and patients. It helps to diagnose in the fascinating world of publishers, editors and authors. many hereditary diseases, estimate risks for family mem- I describe writing book and journal styles, communica- bers, etc. Recently we reported a comprehensive package tion with customers and other issues relevant for TEX for automatic pedigree drawing. Since then we have consulting.
IBM Corporation, Springer-Verlag Heidelberg, TUG T J Watson Research Center, Heidelberg, Germany Yorktown, New York Institutional Stanford Linear Accelerator Center Institute for Defense Analyses, (SLAC), Stanford, California Members Center for Communications Stanford University, Research, Princeton, New Jersey Computer Science Department, Aalborg University, Department MacKichan Software, Stanford, California Washington/New Mexico, USA of Mathematical Sciences, Stockholm University, Department Aalborg, Denmark Marquette University, of Mathematics, Stockholm, Sweden Department of Mathematics, American Mathematical Society, Universit´eLaval, Statistics and Computer Science, Providence, Rhode Island Ste-Foy, Qu´ebec, Canada Milwaukee, Wisconsin Aware Software, Inc., Universiti Tun Hussein Masaryk University, Midland Park, New Jersey Onn Malaysia, Faculty of Informatics, Pusat Teknologi Maklumat, Banca d’Italia, Brno, Czech Republic Roma, Italy Batu Pahat, Johor, Malaysia Moravian College, University College, Cork, Center for Computing Sciences, Department of Mathematics and Computer Centre, Bowie, Maryland Computer Science, Cork, Ireland Certicom Corp., Bethlehem, Pennsylvania University of Delaware, Mississauga, Ontario, Canada MOSEK ApS, Computing and Network Services, Copenhagen, Denmark CSTUG, Newark, Delaware Praha, Czech Republic New York University, University of Oslo, Academic Computing Facility, Florida State University, Institute of Informatics, New York, New York School of Computational Science Blindern, Oslo, Norway and Information Technology, Princeton University, Vanderbilt University, Tallahassee, Florida Department of Mathematics, Nashville, Tennessee Princeton, New Jersey
484 TUGboat, Volume 29 (2008), No. 3 — Proceedings of the 2008 Annual Meeting TUGboat, Volume 29 (2008), No. 3 485
Calendar
2008 2009
Sep 16 – 19 ACM Symposium on Document Jan 8 – 10 College Book Art Association Engineering, S˜ao Paolo, Brazil. Biennial Conference, “Art, Fact, and www.documentengineering.org Artifact: The Book in Time and Sep 17 – 21 Association Typographique Internationale Place”, University of Iowa Center (ATypI) annual conference, for the Book, Iowa City, Iowa. “The Old · The New”, St. Petersburg, uicb.grad.uiowa.edu/uicb-cbaa-conference th Russia. www.atypi.org Apr 29 – BachoTEX2009: 17 BachoTEX Oct 4 – 5 Oak Knoll Fest XV, “Celebrating a May 3 Conference, Bachotek, Poland. ‘Hot Metal Man’ ”, honoring For information, visit Henry Morris and his 50th anniversary www.gust.org.pl/BachoTeX/2009 in printing, New Castle, Delaware. May 15 – “Marking Time”: A traveling juried www.oakknoll.com/fest Aug 15 exhibition of books by members Oct 6 Journ´ee GUTenberg & Assembl´ee of the Guild of Book Workers. g´en´erale, “XML et TEX”, Minnesota Center for Book Arts, Centre FIAP, Paris, France. Minneapolis. Sites and dates are listed at www.gutenberg.eu.org/manifestations palimpsest.stanford.edu/byorg/gbw Oct 10 – 12 American Printing History Association Jun 22 – 25 Digital Humanities 2009, Association of 2008 annual conference, “Saving the Literary and Linguistic Computing / History of Printing”, Grolier Club Association for Computers and the and Columbia University, New York, Humanities, University of Maryland. New York. www.printinghistory.org www.mith2.umd.edu/dh09 Oct 16 – 18 Guild of Book Workers, Jun 23 – 27 SHARP 2009, “Tradition & Innovation: Standards of Excellence The State of Book History”, Society for Annual Seminar, Toronto, Ontario. the History of Authorship, Reading & palimpsest.stanford.edu/byorg/gbw Publishing, Toronto, Ontario. www.sharpweb.org Oct 18 GuIT meeting 2008 (Gruppo utilizzatori Italiani di TEX), Pisa, Italy. Jun 24 – 27 DANTE: Exhibitor at LinuxTag, Berlin, www.guit.sssup.it/guitmeeting/2008 Germany. www.dante.de Oct 18 NTG 42nd meeting, Dordrecht, Netherlands. www.ntg.nl/bijeen/bijeen42.html TUG 2009 Oct 21 “Glasgow 501: Out of Print”, an University of Notre Dame, Notre Dame, Indiana th illustrated talk at St Bride Library, Jul 28 – 31 The 30 annual meeting of the TEX London, England. stbride.org/events Users Group. www.tug.org/tug2009 Oct 25 – 27 The Sixth International Conference on the Book, Catholic University Aug 3 – 7 SIGGRAPH 2009, “Network Your of America, Washington, DC. Senses”, New Orleans, Louisiana. b08.cg-conference.com www.siggraph.org/s2009 rd Nov 7 “Letterpress: a celebration”, conference Aug 24 – 28 EuroTEX 2009 and 3 ConTEXt meeting, at St Bride Library, London, England. The Hague, The Netherlands. stbride.org/events www.ntg.nl/EuroTeX2009 Dec 8 – 10 XML-in-Practice 2008, Arlington, Virginia. www.idealliance.org/xml2008
Status as of 15 September 2008 For additional information on TUG-sponsored events listed here, contact the TUG office (+1 503 223-9994, fax: +1 206 203-3960, e-mail: [email protected]). For events sponsored by other organizations, please use the contact address provided. An updated version of this calendar is online at www.tug.org/calendar. University of Notre Dame IN me, Da tre Notre Dame, IN 46556 USA No TEX July 28–31, 2009 USERS
Sponsored by the T X Users Group, GROUP E Notre Dame Journal of Formal Logic, and the University of Notre Dame http://tug.org/tug2009 TUGboat, Volume 29 (2008), No. 3 487
TEX Consultants
The information here comes from the consultants writing documentation. I also tweak and design fonts. themselves. We do not include information we know I have an MA in Linguistics from Harvard University to be false, but we cannot check out any of the and live in the New York metro area. information; we are transmitting it to you as it was given to us and do not promise it is correct. Also, this Shanmugam, R. is not an official endorsement of the people listed here. No. 38/1 (New No. 65), Veerapandian Nagar, Ist St. We provide this list to enable you to contact service Choolaimedu, Chennai-600094, Tamilnadu, India providers and decide for yourself whether to hire one. +91 9841061058 Email: rshanmugam (at) yahoo.com TUG also provides an online list of consultants at As a Consultant I provide consultation, technical http://tug.org/consultants.html. training, and full service support to the individuals, authors, corporates, typesetters, publishers, Khalighi, Vafa organizations, institutions, etc. and I also 4/34 Sorrell Street support to leading BPO/KPO/ITES/Publishing Parramatta NSW 2150 / Australia companies in implementing latest technologies +006 1420969496 with high level of automation in the field Email: m.aicart (at) menta.net of Typesetting/Prepress/Composition, Web: http://www.vafakhalighi.com ePublishing, XML2PAGE, WEBTechnology, I am a professional and experienced LAT Xacker/ E DataConversion, Digitization, Cross-media LAT Xnician and have been using T X and mates for E E publishing, etc. I have sound knowledge in creating 10 years. I typeset English, Persian, Arabic and Macros/Styles/Templates/Scripts and Conversions mathematics typesetting in those languages as well as with automation using latest softwares in industry. drawing any diagrams (mathematics diagrams, physics diagrams, game diagrams, musical notes and chemistry Sievers, Martin diagrams). My favourite language for drawing is Max-Planck-Str. 6–8, 54296 Trier, Germany PSTricks, however I am also familar with xy-pic and +1 49 651 81009-780 MetaPost. Email: Martin (at) TeXBeratung.com Web: http://www.TeXBeratung.com Martinez, Merc`eAicart o a As a mathematician I offer T X and LAT X services Tarragona 102 4 2 E E and consulting for the whole academic sector and 08015 Barcelona, Spain everybody looking for a high-quality output. From +34 932267827 setting up entire book projects to last-minute help, Email: m.aicart (at) menta.net from creating citation styles to typesetting your math, Web: http://www.edilatex.com tables or graphics — just contact me with information We provide, at reasonable low cost, T X and LAT X E E on your project. typesetting services to authors or publishers world- wide. We have been in business since the beginning of Veytsman, Boris 1990. For more information visit our web site. 46871 Antioch Pl. Sterling, VA 20164 Peter, Steve +1 703 915-2406 310 Hana Road Email: borisv (at) lk.net Edison, NJ 08817 Web: http://borisv.lk.net +1 732 287-5392 T X and LAT X consulting, training and seminars. Email: speter (at) dandy.net E E Integration with databases, automated document Specializing in foreign language, linguistic, and preparation, custom LAT X packages, conversions and technical typesetting using T X, LAT X, and ConT Xt, E E E E much more. I have about twelve years of experience in I have typeset books for Oxford University Press, T X and twenty-five years of experience in teaching & Routledge, and Kluwer, and have helped numerous E training. I have authored several packages on CTAN authors turn rough manuscripts, some with dozens of and published papers in T X related journals. languages, into beautiful camera-ready copy. I have E extensive experience in editing, proofreading, and 488 TUGboat, Volume 29 (2008), No. 3
2009 TEX Users Group Election 2009 TUG Election — Nomination Form Barbara Beeton Only TUG members whose dues have been paid for 2009 for the Elections Committee will be eligible to participate in the election. The sig- natures of two (2) members in good standing at the The positions of TUG President and of eight mem- time they sign the nomination form are required in ad- bers of the Board of Directors will be open as of the dition to that of the nominee. Type or print names 2009 Annual Meeting, which will be held in July clearly, using the name by which you are known to TUG. Names that cannot be identified from the TUG member- 2009 at the University of Notre Dame, Notre Dame, ship records will not be accepted as valid. Indiana. The current President, Karl Berry, has stated his The undersigned TUG members propose the nomi- willingness to stand for re-election. The directors whose nation of: terms will expire in 2009 are: Steve Grathwohl, Jim Hefferon, Klaus H¨oppner, Dick Koch, Arthur Ogawa, Name of Nominee: Steve Peter, and Dave Walden. One additional direc- Signature: tor position is currently unoccupied. Continuing direc- tors, with terms ending in 2011, are: Barbara Beeton, Date: Jon Breitenbucher, Kaja Christiansen, Susan DeMeritt, Ross Moore, Cheryl Ponchin, and Philip Taylor. for the position of (check one): The election to choose the new President and ✷ TUG President Board members will be held in Spring of 2009. Nom- ✷ Member of the TUG Board of Directors inations for these openings are now invited. for a term beginning with the 2009 Annual Meeting, The Bylaws provide that “Any member may be July 2009. nominated for election to the office of TUG Presi- dent/to the Board by submitting a nomination pe- Members supporting this nomination: tition in accordance with the TUG Election Proce- 1. dures. Election . . . shall be by written mail ballot (please print) of the entire membership, carried out in accordance with those same Procedures.” The term of President (signature) (date) is two years. The name of any member may be placed in 2. nomination for election to one of the open offices by (please print) submission of a petition, signed by two other mem- bers in good standing, to the TUG office at least two weeks (14 days) prior to the mailing of ballots. (signature) (date) (A candidate’s membership dues for 2009 will be ex- pected to be paid by the nomination deadline.) The Return this nomination form to the TUG office (FAXed term of a member of the TUG Board is four years. forms will be accepted). Nomination forms and all re- A nomination form follows this announcement; quired supplementary material (photograph, biography and personal statement for inclusion on the ballot) must forms may also be obtained from the TUG office, or be received in the TUG office no later than 1 February via http://tug.org/election. 2009.1 It is the responsibility of the candidate to ensure Along with a nomination form, each candidate must that this deadline is met. Under no circumstances will supply a passport-size photograph, a short biography, incomplete applications be accepted. and a statement of intent to be included with the bal- lot; the biography and statement of intent together may ✷ nomination form not exceed 400 words. The deadline for receipt at the ✷ photograph TUG office of nomination forms and ballot information ✷ biography/personal statement is 1 February 2009. Ballots will be mailed to all members within 30 days after the close of nominations. Marked ballots must be TEX Users Group FAX: +1 206 203-3960 returned no more than six (6) weeks following the mail- Nominations for 2009 Election ing; the exact dates will be noted on the ballots. P. O. Box 2311 Ballots will be counted by a disinterested party not Portland, OR 97208-2311 part of the TUG organization. The results of the election U.S.A. should be available by early June, and will be announced 1 Supplementary material may be sent separately from in a future issue of TUGboat as well as through various the form, and supporting signatures need not all appear on TEX-related electronic lists. one form. olume 29, Number 3 — TUG 2008 Conference Proceedings 2008 TUGBOAT Volume 29 (2008), No. 3
Table of Contents (ordered by difficulty) Introductory 418 Dave Crossland / Why didn’t METAFONT catch on? • personality types, METAFONT’s interface, and font vs. typeface design 352 Peter Flynn / TUG 2008: TEX’s 30th birthday • general review of the TUG 2008 conference 444 Manjusha Joshi / Smart ways of drawing PSTricks figures • using GUI programs with PSTricks for geometric figures 362 Jonathan Kew / TEXworks: Lowering the barrier to entry • report on a new TEX front end focused on ease-of-use and simplicity Intermediate 380 Taco Hoekwater / MetaPost developments: MPlib project report • translation of MetaPost functionality to a re-usable library 480 Manjusha Joshi / Data mining: Role of TEX files • extracting feature words from TEX files to improve searching 356 Niall Mansfield / How to develop your own document class — our experience • practical techniques for creating custom classes and styles A 376 Joe McCool / A newbie’s experiences with Lilypond, Lilypond-book, L TEX and Perl • producing nicely typeset traditional Irish music 421 Karel P´ıˇska / Creating cuneiform fonts with MetaType1 and FontForge • development process and results for a cuneiform font collection 413 Chris Rowley / Vistas for TEX: liberate the typography! (Part I) • extracting core TEX typesetting functionality for reuse 372 Joachim Schrod / Xindy revisited: Multi-lingual index creation for the UTF-8 age • introduction to and rationale for xindy’s main features Intermediate Plus 383 Hans Hagen / The TEX–Lua mix • introduction to the combination of TEX and the embedded scripting language Lua 365 J´erˆome Laurens / Direct and reverse synchronization with SyncTEX • synchronizing between TEX input and output via modifying the base engine 454 Mojca Miklavec and Arthur Reutenauer / Putting the Cork back in the bottle — Improving Unicode support in TEX • recasting hyphenation patterns to support both UTF-8 and 8-bit encodings 464 Ross Moore / Advanced features for publishing mathematics, in PDF and on the Web • using PDF and JavaScript to improve mathematics presentation and navigation 392 Kriszti´an P´ocza, Mih´aly Bicz´o and Zolt´an Porkol´ab / docx2tex: Word 2007 to TEX • free XML-based conversion software from Word 2007 (OOXML) to TEX 435 Ameer Sherif and Hossam Fahmy / Meta-designing parameterized Arabic fonts for AlQalam • using METAFONT to render Arabic with calligrapher-level quality 426 Ulrik Vieth / Do we need a ‘Cork’ math font encoding? • review of OpenType and Unicode math features, subsuming 8-bit encodings Advanced 462 Hans Hagen / The LuaTEX way: \framed • generalizing paragraph manipulation in LuaTEX 446 Hans Hagen / The MetaPost library and LuaTEX • using the new standalone MetaPost library from LuaTEX and ConTEXt 401 Jean-Michel Hufflen / Languages for bibliography styles • language comparison of bst, nbst, Perl, DSSSL, XSLT, etc. 474 John Plaice, Blanca Mancilla and Chris Rowley / Multidimensional text • a theoretical underpinning of documents as generalized tuples A 458 Stanislav Jan Sarmanˇ / Writing Gregg Shorthand with METAFONT and L TEX • an online system converting English text to Gregg shorthand Reports and notices 350 TUG 2008 conference information 351 TUG 2008 conference program 482 Abstracts (Fine, Hagen, Henkel, Hoekwater, Høgholm, K¨uster, Mancilla et al., Mittelbach, Peter, Rahilly et al., Rhatigan, Veytsman & Akhmadeeva, Veytsman) 484 Institutional members 485 Calendar 486 TUG 2009 announcement 487 TEX consulting and production services 488 TUG 2009 election TUGBOAT
Volume 29, Number 3 / 2008 TUG 2008 Conference Proceedings
TUG 2008 350 Conference program, delegates, and sponsors th 352 Peter Flynn / TUG 2008: TEX’s 30 birthday LATEX 356 Niall Mansfield / How to develop your own document class — our experience Software & Tools 362 Jonathan Kew / TEXworks: Lowering the barrier to entry 365 J´erˆome Laurens / Direct and reverse synchronization with SyncTEX 372 Joachim Schrod / Xindy revisited: Multi-lingual index creation for the UTF-8 age 380 Taco Hoekwater / MetaPost developments: MPlib project report 383 Hans Hagen / The TEX–Lua mix 376 Joe McCool / A newbie’s experiences with Lilypond, Lilypond-book, LATEX and Perl 392 Kriszti´an P´ocza, Mih´aly Bicz´oand Zolt´an Porkol´ab / docx2tex: Word 2007 to TEX 401 Jean-Michel Hufflen / Languages for bibliography styles
Dreamboat 413 Chris Rowley / Vistas for TEX: liberate the typography! (Part I) Fonts 418 Dave Crossland / Why didn’t METAFONT catch on? 421 Karel P´ıˇska / Creating cuneiform fonts with MetaType1 and FontForge 426 Ulrik Vieth / Do we need a ‘Cork’ math font encoding? 435 Ameer Sherif and Hossam Fahmy / Meta-designing parameterized Arabic fonts for AlQalam
Graphics 444 Manjusha Joshi / Smart ways of drawing PSTricks figures 446 Hans Hagen / The MetaPost library and LuaTEX Philology 454 Mojca Miklavec and Arthur Reutenauer / Putting the Cork back in the bottle — Improving Unicode support in TEX 458 Stanislav Jan Sarmanˇ / Writing Gregg Shorthand with METAFONT and LATEX Macros 462 Hans Hagen / The LuaTEX way: \framed Electronic Documents 464 Ross Moore / Advanced features for publishing mathematics, in PDF and on the Web 474 John Plaice, Blanca Mancilla and Chris Rowley / Multidimensional text 480 Manjusha Joshi / Data mining: Role of TEX files Abstracts 482 Abstracts (Fine, Hagen, Henkel, Hoekwater, Høgholm, K¨uster, Mancilla et al., Mittelbach, Peter, Rahilly et al., Rhatigan, Veytsman & Akhmadeeva, Veytsman) News 485 Calendar 486 TUG 2009 announcement
Advertisements 487 TEX consulting and production services TUG Business 484 TUG institutional members 488 TUG 2009 election TUGBOAT Volume 29 (2008), No. 3
Table of Contents (ordered by difficulty) Introductory 418 Dave Crossland / Why didn’t METAFONT catch on? • personality types, METAFONT’s interface, and font vs. typeface design 352 Peter Flynn / TUG 2008: TEX’s 30th birthday • general review of the TUG 2008 conference 444 Manjusha Joshi / Smart ways of drawing PSTricks figures • using GUI programs with PSTricks for geometric figures 362 Jonathan Kew / TEXworks: Lowering the barrier to entry • report on a new TEX front end focused on ease-of-use and simplicity Intermediate 380 Taco Hoekwater / MetaPost developments: MPlib project report • translation of MetaPost functionality to a re-usable library 480 Manjusha Joshi / Data mining: Role of TEX files • extracting feature words from TEX files to improve searching 356 Niall Mansfield / How to develop your own document class — our experience • practical techniques for creating custom classes and styles A 376 Joe McCool / A newbie’s experiences with Lilypond, Lilypond-book, L TEX and Perl • producing nicely typeset traditional Irish music 421 Karel P´ıˇska / Creating cuneiform fonts with MetaType1 and FontForge • development process and results for a cuneiform font collection 413 Chris Rowley / Vistas for TEX: liberate the typography! (Part I) • extracting core TEX typesetting functionality for reuse 372 Joachim Schrod / Xindy revisited: Multi-lingual index creation for the UTF-8 age • introduction to and rationale for xindy’s main features Intermediate Plus 383 Hans Hagen / The TEX–Lua mix • introduction to the combination of TEX and the embedded scripting language Lua 365 J´erˆomeLaurens / Direct and reverse synchronization with SyncTEX • synchronizing between TEX input and output via modifying the base engine 454 Mojca Miklavec and Arthur Reutenauer / Putting the Cork back in the bottle — Improving Unicode support in TEX • recasting hyphenation patterns to support both UTF-8 and 8-bit encodings 464 Ross Moore / Advanced features for publishing mathematics, in PDF and on the Web • using PDF and JavaScript to improve mathematics presentation and navigation 392 Kriszti´anP´ocza,Mih´alyBicz´o and Zolt´anPorkol´ab / docx2tex: Word 2007 to TEX • free XML-based conversion software from Word 2007 (OOXML) to TEX 435 Ameer Sherif and Hossam Fahmy / Meta-designing parameterized Arabic fonts for AlQalam • using METAFONT to render Arabic with calligrapher-level quality 426 Ulrik Vieth / Do we need a ‘Cork’ math font encoding? • review of OpenType and Unicode math features, subsuming 8-bit encodings Advanced 462 Hans Hagen / The LuaTEX way: \framed • generalizing paragraph manipulation in LuaTEX 446 Hans Hagen / The MetaPost library and LuaTEX • using the new standalone MetaPost library from LuaTEX and ConTEXt 401 Jean-Michel Hufflen / Languages for bibliography styles • language comparison of bst, nbst, Perl, DSSSL, XSLT, etc. 474 John Plaice, Blanca Mancilla and Chris Rowley / Multidimensional text • a theoretical underpinning of documents as generalized tuples A 458 Stanislav Jan Sarmanˇ / Writing Gregg Shorthand with METAFONT and L TEX • an online system converting English text to Gregg shorthand Reports and notices 350 TUG 2008 conference information 351 TUG 2008 conference program 482 Abstracts (Fine, Hagen, Henkel, Hoekwater, Høgholm, K¨uster,Mancilla et al., Mittelbach, Peter, Rahilly et al., Rhatigan, Veytsman & Akhmadeeva, Veytsman) 484 Institutional members 485 Calendar 486 TUG 2009 announcement 487 TEX consulting and production services 488 TUG 2009 election