G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
EXTENDING GNU TROFF TOPRODUCE HTML THROUGH THE TECHNIQUE OF NEXT EVENT SIMULATION
GAIUS MULLEY and WERNER LEMBERG School of Computing,University of Glamorgan, CF37 1DL, UK E-Mail: [email protected] Kl. Beurhausstr.1,44137 Dortmund, Germany E-Mail: wl@gnu.org
Abstract: This paper reports on a technique used to generate accurate HTML output from GNU Troff. GNU Troff is a typesetting package which reads plain text mixed with formatting commands and produces formatted output. It supports a number of devices and nowsupports the production of HTML.The paper discusses the design of the HTML device driver grohtml and modifications made to GNU Troff. The front end program troff wasmodi- fied to maintain a reduced state machine which is examined each time a glyph is passed to the back end device driver(post-grohtml). Anychange in system state between the production of twoglyphs results in a sequence of events being passed to the device driver. There is a direct correspondence between this technique and creating ascript for a next event simulation queue. Furthermore the device driverreconstructs the system state and for- mats the HTML according to state changes caused when processing the event queue. This technique works well, as it minimises the state information passed from front end to back end device driverwhilst still preserving the high levellayout of the text. Using this technique GNU Troffeffectively translates input source into another mark-up language and thus this technique could be extended to translate GNU Troffdocuments into anyofthe OpenOffice supported formats. Troffhas been in use for three decades nowand is still actively used by authors. Troff’sbiggest use, however, istoformat manual pages for GNU/Linux and other UNIX likeoperating systems. Introducing this facility into GNU Troffprovides users with the ability to translate legacy documents into HTML and in the future to a format supported by OpenOffice.
Ke ywords: groff, troff, grohtml, HTML, simulation.
table [Lesk, 1976a], picture [Kernighan, 1991, Wyk, 1982] and equation handling [Kernighan, 1977, INTRODUCTION Kernighan, 1976] (refer, tbl, pic and eqn respec- tively). These programs were executed in a pipeline GNU Troffisareimplementation of the program and troff transformed the heavily preprocessed troff which is available on the UNIX operating sys- formatting commands and text into device dependent tem. The original troff waswritten in PDP-11 output [Kernighan, 1978]. Although the input to assembly language by Joe Ossanna in 1973. Tw o troff wasdevice independent it was also very low years later it was rewritten in C and afterwards it leveland therefore users were encouraged to use went through a series of revisions until Joe Ossanna macro sets when producing documents [Lesk, died in 1977. 1976b]. During the 1980s the internals were mod- estly revised and a number of newmacro sets were Brian Kernighan continued troff development for the next 15 years and it is testament to the original written [Allman, 1980, Smith, 1980]. The macro design that the input language remained the same. sets provided freedom in document styling (similar Much of the original documentation receivedminor to modern HTML style sheets) and theyincluded: arbitrary style headers and footers; arbitrary style changes as new troff releases were issued and therefore Joe’sname was retained on these manu- footnotes; automatic sequence numbering for para- als [Ossanna, 1992]. graphs, sections, etc; multiple column output; dynamic font and point-size control; arbitrary hori- Brian Kernighan modified troff so that it could zontal and vertical local motions at anypoint; math- produce output for a number of different typesetting ematical bracket construction, and line drawing devices, while at the same time retaining the same functions [Ossanna, 1992]. input language specification. The input language wassorobust that a number of preprocessors were During the 1980s troff wasextended to handle written to provide reference [Tuthill, 1986], manydifferent devices, this was accomplished by
I.J. of SIMULATION Vol. 6 No 7-8 37 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML splitting the task of troff into twocomponents. commands to format abstracts, titles, footnotes, para- The front end ditroff produced device indepen- graphs, lists, hanging indents etc. Consequently, dent code and the back end device driverwhich sim- translating troff input source into HTML cannot be ply translated the device independent code into the achievedbywriting a device driverwhich simply target device commands. reads ditroff input and produces HTML output. Consider the diagram in figure 1 which shows the James Clark begantowork on the GNU implemen- keycomponents of a simple groff command line tation of the troff family of tools in 1989. This was invocation together with a synopsis of the data pass- to be a completely newimplementation of all the ing though the pipeline. In figure 1 the troff input preprocessors, the ditroff program and the macro consists of requests or calls to macros (lines prefixed sets. The first release of groff (version 0.3.1) with a period) and text. Troffinput might also occurred in June 1990 and it included a replacement include text with escapes, for example the word for ditroff, eqn, tbl and pic.Italso included a program can be typeset by temporarily reducing the replacement for the me macros and the man macros. point size by 1 and altering the font to Courier.The The replacement programs were mostly written in word and escapes can be encoded as \s-1\fCpro- C++ and often supported extensions and removed gram\s+1\fP.Infigure 1 the final output is Post- various static data size limitations. Since 1999 the Script (the default device) and the invocation also groff package has had newmaintainers and has includes the ms macro package. Figure 2 shows how undergone active dev elopment. It supports the com- aPostScript printer interprets the output from Figure mon macro sets associated with troff (man, me, ms, 1. mdoc and mm). However groff also provides a modern macro set (mom)and also provides newpre- processors and support for colour. Abasic title
DESCRIPTION OF THE PROBLEM 1. Heading at level1 1.1. Heading at level2 Troffwas widely used in the 1970s and 1980s. All UNIX documentation, release notes for both the 1.1.1. Heading at level3 First paragraph body AT&T and BSD UNIX variants was written in troff.Manypapers in the CACM and Software Figure 2: displaying the PostScript output Practice and Experience journals were also produced Notice howthe ditroff output only knows which using troff.Eventoday all UNIX and GNU/Linux font is to be used, the font size and position that the manual pages are written in troff using the man glyph should be placed. Another example that an macro set and some authors advocate using troff HTML device drivermust translate accurately is above other WYSIWYG tools to typeset their shown in figure 3. books [Tanenbaum, 1997, Stevens, 1998, Schaffter, 2004]. The start of an indented paragraph example in which line 1, line 2 and line 3 are Groffprovides compatibility with troff as well as vertically aligned. manymodern enhancements, image handling, colour .LP and limited pdf mark capability.Clearly the addition .IP once of an HTML device driverwould be useful. line 1 .IP twice line 2 GNU Troffcopied the original design of UNIX Troff .IP threefold ,which maps the input source onto a physical device line 3 through the use of a device independent language. Figure 3: examples of indented paragraphs Through this language it effectively plots each glyph at Cartesian coordinate position on a page. The dif- Here we see from the output shown in figure 4 that ficulty in translating troff source into HTML is the ms macro set diverted the indented paragraph exacerbated by the fact that both HTML and troff label parameter once, twice and threefold into a source are mark-up languages. As the GNU Troff macro. It then tests to see whether the macro width and original UNIX Troffpackages rely heavily on the is greater than a default length and if so it breaks the pipeline principle it has naturally led to the pre- line before starting the indented paragraph. Clearly processors (pic, tbl and eqn)translating their high the HTML device driverneeds to cope with these levelcommands onto much lower level troff com- constructs, as theyare also heavily used in manual mands. In turn, this has meant that much of the high pages. levelinformation (for example where a table starts and ends) is lost. Furthermore, using anyofthe macro packages will result in manylow lev el
I.J. of SIMULATION Vol. 6 No 7-8 38 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
Troffsource ditroffintemediate language postscript output
.TL xTps %!PS-Adobe-3.0 Abasic title xres 72000 %%Creator: groff .NH 1 xinit %%DocumentNeed Heading at level1 troff-ms p1 grops %%+font Times-Ro .NH 2 xfont 6 TB %%DocumentSupp Heading at level2 f6 %%Pages: 1 .LP s12 H123k V 123k %%PageOrder: Asc First paragraph tA %%Orientation: body wh3000 %%EndComments tbasic
Figure 1: Simple title, heading and paragraph
Forexample HTML has a restricted set of mathemat- The start of an indented paragraph example in which line 1, line 2 and line ical glyphs and it is restricted in its ability to accu- 3are vertically aligned. once line 1 rately position these glyphs [Musciano, 1998]. Groff twice in contrast can be used to typeset mathematical for- line 2 mulae and users will at least expect the groff HTML threefold line 3 device drivertoreproduce the formulae as an image. Figure 7 shows Troffinput, which describes the defi- Figure 4: result of processing figure 3 with groff -ms nite integral.
Furthermore manydocuments will include encapsu- .EQ lated PostScript as shown in figure 5. The processed delim $$ output is shown in figure 6. The issue here is that .EN PostScript has become the default device driverand .LP groffallows encapsulated PostScript to be handled As $ delta x -> 0 $ then .EQ as a special device case. Nevertheless this feature is sum from x=a to b y . delta x ˜ approx ˜ \ so useful and it would be expected to be imple- int from a to b y ˜ dx mented in the HTML device driver. Groffalso allows .EN users to embed their own PostScript inside Troff Figure 7: eqn example source files (a technique similar to inlining assembly language within a high levellanguage). In both When processed with the command groff -e -ms cases users could reasonably expect the HTML it produces the output as shown in figure 8. device drivertotranslate the PostScript into a PNG image which is referenced by the HTML output.
.TL As δ x → 0then An example of encapsulated PostScript b b .LP Σ y. δ x ≈ ∫ ydx .PSPIC -L tiger.eps 1i 1i x=a a Figure 5: example of encapsulated PostScript usage Figure 8: output of figure 7 as rendered on a PostScript device
An example of encapsulated postscript Finally the troff language is very rich in com- mands and requests. The language has the ability to construct macros at run time and it does this by diverting text and commands into a diversion.Some authors of macro sets perform some highly impres- sive programming tricks using this technique. For Figure 6: result of processing figure 5 with groff -ms example this technique has been used in the ms macro set to handle the IP request and the TL request seen earlier.Itisalso used to manage Afurther problem which needs to be addressed is abstracts, authors names and references for papers. the limited number of glyphs available in HTML. The problem is that when troff generates the
I.J. of SIMULATION Vol. 6 No 7-8 39 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML ditroff output the meaning of the text is lost. The extensible troff translator which translates troff HTML device driverneeds to knowwhether a line files into HTML-2.0.Itsupports the me, ms,and man wascentred, if the line was indented, when the cen- macro sets as well as the front end preprocessors tring ends, when indentation ends, when text is a tbl and eqn.This is an ambitious project and it has section heading and when text is a title. It cannot been successful in translating manual pages and guess this reliably by examining the glyph position troff documents. The success is partly due to and font information. unroff understanding manyofthe underlying troff requests and having extra tag information Conveying the indentation information to the device inside its macro set descriptions. The main disadvan- driverisparticularly difficult as diversions can be tages to unroff are that it does not have a seamless created to manipulate this value when the diversion interface to the pic preprocessor and currently it is re-read. Consider the example giveninfigure 9. does not handle encapsulated PostScript or diver- sions. .LP .di foo .br The doclifter project [Raymond, 2005a] provides .in +1i ameans whereby manydocuments written in mm, ms, hello me and mdoc can be translated into XML-Doc- .in -1i book [Walsh, 1999]. Eric Raymond designed .br doclifter to operate at the semantic leveland he .di .in +2i acknowledges that to do a really good job requires .br some human polishing. Nevertheless doclifter .foo has successfully translated 95% of all manual pages .in +1i in Fedora Core 3. .br .foo Doclifter translates tbl tables into DocBook table Figure 9: diversion and indentation example markup and pic input into SVG,howev er eqn is not translated. When processed with the command groff -ms it produces the output as shown in figure 9. OVERVIEW OF GROHTML
hello The approach taken by the authors in the design of hello grohtml wastomodify the front end groff binary Figure 10: output of figure 9 as rendered on a so that when the HTML device is requested the PostScript device troff input is run through the PostScript device driverand later through the HTML device driver. All Clearly an HTML device drivermust also understand images are created by converting the relevant Post- the diversions sufficiently so that the important for- Script page into a bitmap image after which the matting requests are accurately translated. required area is cut and converted into the PNG [Roelofs, 1999] format. The pipeline structure PREVIOUS WORK is outlined in figure 11. The preprocessors tbl, eqn and pic were modified to indicate where the start There have been a number of scripting solutions and end of a table, equation and picture occur.This implemented which translate troff documents into information is then transparently read by the front HTML.These scripts operate at the macro set level end and determines the area of the bitmap file which and theyindependently translate the functionality of is required for an image. The advantage of this the various macros into HTML.For example scripts approach is that grohtml can fully support encapsu- exists to translate the ms macros IP, TL, LP, PP, DS lated PostScript inclusion and it also allows users to into indented paragraph; title; left aligned paragraph; use this feature together with anycombination of paragraph and block text. The advantage of this glyphs to generate web images. approach is its simplicity but the disadvantages are that the scripts need to be maintained independently The macro sets ms and man were also slightly modi- to the corresponding macro set and more importantly fied to issue a tag when troff encounters a section the scripts do not understand user defined macros. heading, title or indented paragraph. All tags are Often these scripts do not honour all the macro emitted together with the ditroff intermediate optional arguments or allowinclusion of encapsu- code to the device driver post-grohtml. lated PostScript. The troff program was also modified to handle Another solution is unroff [Laumann, 1996], another escape \O which has fivevariants and they which is a Scheme-based, programmable and are described in table 1. These escapes are
I.J. of SIMULATION Vol. 6 No 7-8 40 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
groff-ms -e -Tps -rps4html=1 intemediate users troffdocument p PostScript r output e generate Definite integral g images image coordinates .EQ r from sum from x=a to b... o PostScript .EN h file png images 1 2 3 t m l groff-Thtml -e -ms -Z post HTML grohtml output
Figure 11: the groff -Thtml -ms -e pipe line
Escape Meaning \O0 Disable anyglyphs from being emitted at the outer level. † \O1 Enable output of glyphs, provided that the escape occurs at the outer level. † \O2 Providing the escape occurs at the outer level, enable output of glyphs and also write out to stderr the page number and four registers encompassing the glyphs previously writ- ten since the last call to \O. \O3 Begin a nesting level. \O4 End a nesting level. \O[5Pfilename] HTML device specific. Provided that this escape occurs at the outer nesting levelwrite the filename to stderr.The position of the image, P,must be specified and must be one of l, r, c or i (left, right, centre, inline). Table 1: new troff escapes and their meaning important as theyallowpreprocessors to tell troff Wheneveraglyph is about to be written in the to skip a block of text or issue a box of coordinates ditroff output, troff needs to check whether this which define the area for a set of glyphs. This func- state has changed since the last glyph was written. If tionality is used during the PostScript pass to deter- the state has changed then troff emits a series of mine the region used by a table, picture or equation. tags which define the change in state. There is clear Conversely it is used by the HTML pass to stop pro- correspondence between tags appearing in the ducing glyphs when a table, equation or picture is ditroff intermediate output and a next event queue encountered; but replace these items with an HTML albeit without anytime component. The HTML reference to an image. driver post-grohtml reconstructs the system state according to the events it reads and in the process STATEMACHINE SOLUTION translates the glyphs into HTML text surrounded by the appropriate HTML tags [Musciano, 1998]. Fur- If grohtml is implemented as described in the last thermore troff can handle requests read from section it will be able to translate all: encapsulated diversions by pushing the current minimal HTML PostScript, tbl, pic, eqn,section headings, num- state before a diversion is read and popping it later bered headings and titles accurately.Howev eritwill once the diversion has finished. This allows diver- still fail to accurately translate: centred lines, any sions to centre lines, alter the indentation, and emit indentation, preformatted blocks, tabulated text, line preformatted text, whilst restoring the state once the breaks and vertical spaces. Asolution to this prob- diversion has ended. lem is to extend the repertoire of tags which are passed from troff to post-grohtml.This The complete set of tags which post-grohtml under- requires further modification to the troff so that it stands is shown in table 2. Figure 16 shows the will issue tags wheneverthe above formatting extended ditroffoutput when GNU Troffprocesses requests are encountered. In essence the back end of Figure 9. The command line to generate this troff maintains a simple state machine which rep- resents the minimum information required to cor- †Both \O0 and \O1 reset four troff registers which mark the top left and bottom right hand corners of an rectly reproduce the formatting requests listed above. area containing all written glyphs.
I.J. of SIMULATION Vol. 6 No 7-8 41 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML information is shown below: into HTML and images and figure 15 shows that indentation within diversions behave correctly. groff -Z -ms -Thtml ex9.n At present the GNU Troffmacro sets: ms, man,and and the output in Figure 16 contains the minimum mwww have the extra HTML tags included. Figure 17 tags necessary in order to convey the meaning of the shows the output of a component from the GNU pic glyph formatting. The example shown in figure 9 guide [Raymond, 2005b] when processed through produces twolines with a single word hello on the following pipeline. each line. The first line hello is indented at three inches and on the second line hello is indented at groff -pet -ms -Thtml pic.ms | html2ps -s2 four inches. The example starts by declaring a left adjusted paragraph .LP and then it creates the diver- It can be seen that grohtml correctly determines sion .di foo.All subsequent text and requests are that the equation is inside a picture and generates diverted into foo until the second .di is encoun- one image. Figure 18 shows the grohtml manual tered. The next request .in +2i informs the type- page after it was processed by the pipeline: setter that an indentation of twoinches is required. groff -pet -man -Thtml grohtml.man When .foo is encountered it rereads the diversion which performs another relative indent of one inch and viewed through a browser.Inthis case we can before emitting the glyph hello.The diversion clearly see the vertical alignment correctly repre- then reduces the indentation by one inch before fin- sented between lines which contain a command line ishing. Textual input is resumed after the diversion option and those lines which are a component of a .foo at the request .in +1i which performs a rela- hanging paragraph. Figure 19 shows howthe GNU tive indent of one inch (at this point the indentation Troffhomepage appears when printed from a is set at three inches). Lastly the diversion is run browser.The homepage was created by the follow- again which emits the final hello four inches from ing command line. the left hand margin. groff -pet -ms -Thtml webpage.ms Figure 16 shows the ditroffoutput together with an annotated description. It also shows extensions The file webpage.ms makeextensive use of the made to ditroffwhich include the extra tag informa- mwww macro set which provides drop capital, tion (lines starting with xXdevtag: convey the unordered line list, URL and image related macros. It minimum events necessary such that post-gro- also makes use of colour which was recently added html can reconstruct the troff state machine. As to GNU Troff. post-grohtml reconstructs the state machine it generates the appropriate HTML.Critically for this CONCLUSIONS AND FURTHER WORK example it can be seen that here are only two xX devtag:.in ev ents, one at 3 inches and the second In conclusion the grohtml device driverhas been at 4 inches, despite the example in figure 9 having constructed and integrated into groff.This work four .in requests. has taken six years of part time effort and amounts to approximately 9600 lines of C++ code (not includ- EXAMPLES OF GROHTML OUTPUT ing macro set modifications). In due course the groff-1.19.2 will be formally released and this The examples shown in this section were produced will be the first beta release of grohtml.During the with groff version 1.19.2 and the test cases are development of grohtml anumber of additional those from figures 3, 5, 7 and 9. All the examples features were needed to produce useful HTML. were constructed using the following pipeline: These features were applied generically as possible and include the introduction of colour.Colour is now groff -Thtml -e -ms filename.ms | supported by the ASCII,PostScript, HTML,and DVI html2ps devices. GNU pic has also been extended to sup- port colour. and these are shown in figures 12, 13, 14 and 15 respectively.The test results appear satisfactory and The principles behind the HTML device driver showthat the indentation using the IP macro works appear sound, howeverthere are still minor bugs in as expected. This macro uses a diversion to detect grohtml.Future work will include bug fixing and the width of its first argument before emitting the also modifications to tbl so that troff tables can paragraph. The example shows howeach line 1, line be translated into HTML