G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
EXTENDING GNU TROFF TOPRODUCE HTML THROUGH THE TECHNIQUE OF NEXT EVENT SIMULATION
GAIUS MULLEY,WERNER LEMBERG School of Computing,University of Glamorgan, CF37 1DL, UK E-Mail: [email protected] Kl. Beurhausstr.1,44137 Dortmund, Germany E-Mail: [email protected]
Abstract: This paper reports on a technique used to generate accurate HTML output from GNU Troff. GNU Troff is a typesetting package which reads plain text mixed with formatting commands and produces formatted output. It supports a number of devices and nowsupports the production of HTML.The paper discusses the design of the HTML device driver grohtml and modifications made to GNU Troff. The front end program troff wasmodi- fied to maintain a reduced state machine which is examined each time a glyph is passed to the back end device driver(post-grohtml). Anychange in system state between the production of twoglyphs results in a sequence of events being passed to the device driver. There is a direct correspondence between this technique and creating ascript for a next event simulation queue. Furthermore the device driverreconstructs the system state and for- mats the HTML according to state changes caused when processing the event queue. This technique works well, as it minimises the state information passed from front end to back end device driverwhilst still preserving the high levellayout of the text. Using this technique GNU Troffeffectively translates input source into another mark-up language and thus this technique could be extended to translate GNU Troffdocuments into anyofthe OpenOffice supported formats. Troffhas been in use for three decades nowand is still actively used by authors. Troff’sbiggest use, however, istoformat manual pages for GNU/Linux and other UNIX likeoperating systems. Introducing this facility into GNU Troffprovides users with the ability to translate legacy documents into HTML and in the future to a format supported by OpenOffice.
Ke ywords: groff, troff, grohtml, HTML, simulation.
Brian Kernighan modified troff so that it could INTRODUCTION produce output for a number of different typesetting devices, while at the same time retaining the same GNU Troffisareimplementation of the program input language specification. The input language troff which is available on the UNIX operating sys- wassorobust that a number of preprocessors were tem. The original troff waswritten in PDP-11 written to provide reference [Tuthill, 1986], ta- assembly language by Joe Ossanna in 1973. Tw o ble [Lesk, 1976a], picture [Kernighan, 1991, Wyk, years later it was rewritten in C and afterwards it 1982] and equation handling [Kernighan, 1977, went through a series of revisions until Joe Ossanna Kernighan, 1976] (refer, tbl, pic and eqn respec- died in 1977. tively). These programs were executed in a pipeline and troff transformed the heavily preprocessed Brian Kernighan continued troff development for the next 15 years and it is testament to the original formatting commands and text into device dependent design that the input language remained the same. output [Kernighan, 1978]. Although the input to Much of the original documentation receivedminor troff wasdevice independent it was also very low leveland therefore users were encouraged to use changes as new troff releases were issued and therefore Joe’sname was retained on these manu- macro sets when producing documents [Lesk, als [Ossanna, 1992]. 1976b]. During the 1980s the internals were mod- estly revised and a number of newmacro sets were written [Allman, 1980, Smith, 1980]. The macro sets provided freedom in document styling (similar Published in the International Journal of to modern HTML style sheets) and theyincluded: SIMULATION Vol. 6 No 7-8, ISSN arbitrary style headers and footers; arbitrary style 1473-804x online, 1473-8031 print, 2005 footnotes; automatic sequence numbering for
I.J. of SIMULATION Vol. 6 No 7-8 37 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML paragraphs, sections, etc; multiple column output; and original UNIX Troffpackages rely heavily on the dynamic font and point-size control; arbitrary hori- pipeline principle it has naturally led to the pre- zontal and vertical local motions at anypoint; math- processors (pic, tbl and eqn)translating their high ematical bracket construction, and line drawing levelcommands onto much lower level troff com- functions [Ossanna, 1992]. mands. In turn, this has meant that much of the high levelinformation (for example where a table starts During the 1980s troff wasextended to handle and ends) is lost. Furthermore, using anyofthe manydifferent devices, this was accomplished by macro packages will result in manylow lev elcom- splitting the task of troff into twocomponents. mands to format abstracts, titles, footnotes, para- The front end ditroff produced device indepen- graphs, lists, hanging indents etc. Consequently, dent code and the back end device driverwhich sim- translating troff input source into HTML cannot be ply translated the device independent code into the achievedbywriting a device driverwhich simply target device commands. reads ditroff input and produces HTML output. Consider the diagram in Figure 1 which shows the James Clark begantowork on the GNU implemen- keycomponents of a simple groff command line tation of the troff family of tools in 1989. This was invocation together with a synopsis of the data pass- to be a completely newimplementation of all the ing though the pipeline. In Figure 1 the troff input preprocessors, the ditroff program and the macro consists of requests or calls to macros (lines prefixed sets. The first release of groff (version 0.3.1) with a period) and text. Troffinput might also occurred in June 1990 and it included a replacement include text with escapes, for example the word for ditroff, eqn, tbl and pic.Italso included a program can be typeset by temporarily reducing the replacement for the me macros and the man macros. point size by 1 and altering the font to Courier.The The replacement programs were mostly written in word and escapes can be encoded as C++ and often supported extensions and removed \s-1\fCprogram\s+1\fP. various static data size limitations. Since 1999 the In Figure 1 the final output is PostScript (the default groff package has had newmaintainers and has device) and the invocation also includes the ms undergone active dev elopment. It supports the com- macro package. Figure 2 shows howaPostScript mon macro sets associated with troff (man, me, ms, printer interprets the output from Figure 1. mdoc and mm). However groff also provides a modern macro set (mom)and also provides newpre- processors and support for colour. Abasic title
DESCRIPTION OF THE PROBLEM 1. Heading at level1
1.1. Heading at level2 Troffwas widely used in the 1970s and 1980s. All UNIX documentation, release notes for both the 1.1.1. Heading at level3 AT&T and BSD UNIX variants was written in First paragraph body troff.Manypapers in the CACM and Software Figure 2: displaying the PostScript output Practice and Experience journals were also produced Notice howthe output only knows which using troff.Eventoday all UNIX and GNU/Linux ditroff font is to be used, the font size and the position that manual pages are written in troff using the man the glyph should be placed. Another example that macro set and some authors advocate using troff above other WYSIWYG tools to typeset their an HTML device drivermust translate accurately is books [Tanenbaum, 1997, Stevens, 1998, Schaffter, shown in Figure 3. 2004]. The start of an indented paragraph example in which line 1, line 2 and line 3 are Groffprovides compatibility with troff as well as vertically aligned. manymodern enhancements, image handling, colour .LP and limited pdf mark capability.Clearly the addition .IP once of an HTML device driverwould be useful. line 1 .IP twice GNU Troffcopied the original design of UNIX Troff, line 2 which maps the input source onto a physical device .IP threefold line 3 through the use of a device independent language. Figure 3: examples of indented paragraphs Through this language it effectively plots each glyph at Cartesian coordinate position on a page. The dif- Here we see from the output shown in Figure 4 that ficulty in translating troff source into HTML is the ms macro set diverted the indented paragraph exacerbated by the fact that both HTML and troff label parameter once, twice and threefold into a source are mark-up languages. As the GNU Troff macro. It then tests to see whether the macro width
I.J. of SIMULATION Vol. 6 No 7-8 38 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
Troffsource ditroffintemediate language postscript output
.TL xTps %!PS-Adobe-3.0 Abasic title xres 72000 %%Creator: groff .NH 1 xinit %%DocumentNeed Heading at level1 troff-ms p1 grops %%+font Times-Ro .NH 2 xfont 6 TB %%DocumentSupp Heading at level2 f6 %%Pages: 1 .LP s12 H123k V 123k %%PageOrder: Asc First paragraph tA %%Orientation: body wh3000 %%EndComments tbasic
Figure 1: Simple title, heading and paragraph is greater than a default length and if so it breaks the line before starting the indented paragraph. Clearly An example of encapsulated postscript the HTML device driverneeds to cope with these constructs, as theyare also heavily used in manual pages.
The start of an indented paragraph example in which line 1, line 2 and line 3are vertically aligned. Figure 6: result of processing Figure 5 with groff -ms once line 1 twice line 2 threefold Afurther problem which needs to be addressed is line 3 the limited number of glyphs available in HTML. Figure 4: result of processing Figure 3 with groff -ms Forexample HTML has a restricted set of mathemat- ical glyphs and it is restricted in its ability to accu- Furthermore manydocuments will include encapsu- rately position these glyphs [Musciano, 1998]. Groff lated PostScript as shown in Figure 5. The pro- in contrast can be used to typeset mathematical for- cessed output is shown in Figure 6. The issue here is mulae and users will at least expect the groff HTML that PostScript has become the default device driver device drivertoreproduce the formulae as an image. and groffallows encapsulated PostScript to be han- Figure 7 shows Troffinput, which describes the defi- dled as a special device case. Nevertheless this fea- nite integral. ture is so useful and it would be expected to be implemented in the HTML device driver. Groffalso .EQ allows users to embed their own PostScript inside delim $$ Troffsource files (a technique similar to inlining .EN .LP assembly language within a high levellanguage). In As $ delta x -> 0 $ then both cases users could reasonably expect the HTML .EQ device drivertotranslate the PostScript into a PNG sum from x=a to b y . delta x ˜ approx ˜ \ image which is referenced by the HTML output. int from a to b y ˜ dx .EN .TL Figure 7: eqn example An example of encapsulated PostScript .LP When processed with the command groff -e -ms .PSPIC -L tiger.eps 1i 1i it produces the output as shown in figure 8. Figure 5: example of encapsulated PostScript usage
As δ x → 0then
b b Σ y. δ x ≈ ∫ ydx = x a a Figure 8: output of Figure 7 as rendered on a
I.J. of SIMULATION Vol. 6 No 7-8 39 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
PostScript device exist to translate the ms macros IP, TL, LP, PP, DS into indented paragraph; title; left aligned paragraph; Finally the troff language is very rich in com- paragraph and block text. The advantage of this mands and requests. The language has the ability to approach is its simplicity but the disadvantages are construct macros at run time and it does this by that the scripts need to be maintained independently diverting text and commands into a diversion.Some of the corresponding macro set and more impor- authors of macro sets perform some highly impres- tantly the scripts do not understand user defined sive programming tricks using this technique. For macros. Often these scripts do not honour all the example this technique has been used in the ms macro optional arguments or allowinclusion of macro set to handle the IP request and the TL encapsulated PostScript. request seen earlier.Itisalso used to manage abstracts, authors names and references for papers. Another solution is unroff [Laumann, 1996], The problem is that when troff generates the which is a Scheme-based, programmable and exten- ditroff output the meaning of the text is lost. The sible troff translator which translates troff files HTML device driverneeds to knowwhether a line into HTML-2.0.Itsupports the me, ms,and man wascentred, if the line was indented, when the cen- macro sets as well as the front end preprocessors tring ends, when indentation ends, when text is a tbl and eqn.This is an ambitious project and it has section heading and when text is a title. It cannot been successful in translating manual pages and guess this reliably by examining the glyph position troff documents. The success is partly due to and font information. unroff understanding manyofthe underlying troff requests and having extra tag information Conveying the indentation information to the device inside its macro set descriptions. The main disadvan- driverisparticularly difficult as diversions can be tages to unroff are that it does not have a seamless created to manipulate this value when the diversion interface to the pic preprocessor and currently it is re-read. Consider the example giveninFigure 9. does not handle encapsulated PostScript or diver- sions. .LP .di foo .br The doclifter project [Raymond, 2005a] provides .in +1i ameans whereby manydocuments written in mm, ms, hello me and mdoc can be translated into XML-Doc- .in -1i book [Walsh, 1999]. Eric Raymond designed .br doclifter to operate at the semantic leveland he .di .in +2i acknowledges that to do a really good job requires .br some human polishing. Nevertheless doclifter .foo has successfully translated 95% of all manual pages .in +1i in Fedora Core 3. .br .foo Doclifter translates tbl tables into DocBook table Figure 9: diversion and indentation example markup and pic input into SVG,howev er eqn is not translated. When processed with the command groff -ms it produces the output as shown in Figure 9. OVERVIEW OF GROHTML
hello The approach taken by the authors in the design of hello grohtml wastomodify the front end groff binary Figure 10: output of Figure 9 as rendered on a so that when the HTML device is requested the PostScript device troff input is run through the PostScript device driverand later through the HTML device driver. All Clearly an HTML device drivermust also understand images are created by converting the relevant Post- the diversions sufficiently so that the important for- Script page into a bitmap image after which the matting requests are accurately translated. required area is cut and converted into the PNG [Roelofs, 1999] format. The pipeline structure PREVIOUS WORK is outlined in Figure 11. The preprocessors tbl, eqn and pic were modified to indicate where the start There have been a number of scripting solutions and end of a table, equation and picture occur.This implemented which translate troff documents into information is then transparently read by the front HTML.These scripts operate at the macro set level end and determines the area of the bitmap file which and theyindependently translate the functionality of is required for an image. The advantage of this the various macros into HTML.For example scripts approach is that grohtml can fully support
I.J. of SIMULATION Vol. 6 No 7-8 40 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML encapsulated PostScript inclusion and it also allows diversion has ended. users to use this feature together with anycombina- tion of glyphs to generate web images. The complete set of tags which post-grohtml under- stands is shown in table 2. Figure 16 shows the The macro sets ms and man were also slightly modi- extended ditroffoutput when GNU Troffprocesses fied to issue a tag when troff encounters a section Figure 9. The command line to generate this infor- heading, title or indented paragraph. All tags are mation is shown below: emitted together with the ditroff intermediate code to the device driver post-grohtml. groff -Z -ms -Thtml ex9.n
The troff program was also modified to handle and the output in Figure 16 contains the minimum another escape \O which has fivevariants and they tags necessary in order to convey the meaning of the are described in table 1. These escapes are impor- glyph formatting. The example shown in Figure 9 tant as theyallowpreprocessors to tell troff to skip produces twolines with a single word hello on ablock of text or issue a box of coordinates which each line. The first line hello is indented at three define the area for a set of glyphs. This functionality inches and on the second line hello is indented at is used during the PostScript pass to determine the four inches. The example starts by declaring a left region used by a table, picture or equation. Con- adjusted paragraph .LP and then it creates the diver- versely it is used by the HTML pass to stop produc- sion .di foo.All subsequent text and requests are ing glyphs when a table, equation or picture is diverted into foo until the second .di is encoun- encountered; but replace these items with an HTML tered. The next request .in +2i informs the type- reference to an image. setter that an indentation of twoinches is required. When .foo is encountered it rereads the diversion STATEMACHINE SOLUTION which performs another relative indent of one inch before emitting the glyph hello.The diversion If grohtml is implemented as described in the last then reduces the indentation by one inch before fin- section it will be able to translate all: encapsulated ishing. Textual input is resumed after the diversion PostScript, tbl, pic, eqn,section headings, num- .foo at the request .in +1i which performs a rela- bered headings and titles accurately.Howev eritwill tive indent of one inch (at this point the indentation still fail to accurately translate: centred lines, any is set at three inches). Lastly the diversion is run indentation, preformatted blocks, tabulated text, line again which emits the final hello four inches from breaks and vertical spaces. Asolution to this prob- the left hand margin. lem is to extend the repertoire of tags which are passed from troff to post-grohtml.This Figure 16 shows the ditroffoutput together with an requires further modification to the troff so that it annotated description. It also shows extensions will issue tags wheneverthe above formatting made to ditroffwhich include the extra tag informa- requests are encountered. In essence the back end of tion (lines starting with xXdevtag: convey the troff maintains a simple state machine which rep- minimum events necessary such that post-gro- resents the minimum information required to cor- html can reconstruct the troff state machine. As rectly reproduce the formatting requests listed above. post-grohtml reconstructs the state machine it Wheneveraglyph is about to be written in the generates the appropriate HTML.Critically for this ditroff output, troff needs to check whether this example it can be seen that here are only two xX state has changed since the last glyph was written. If devtag:.in ev ents, one at 3 inches and the second the state has changed then troff emits a series of at 4 inches, despite the example in Figure 9 having tags which define the change in state. There is clear four .in requests. correspondence between tags appearing in the ditroff intermediate output and a next event queue EXAMPLES OF GROHTML OUTPUT albeit without anytime component. The HTML The examples shown in this section were produced driver post-grohtml reconstructs the system state with groff version 1.19.2 and the test cases are according to the events it reads and in the process those from Figures 3, 5, 7 and 9. All the examples translates the glyphs into HTML text surrounded by were constructed using the following pipeline: the appropriate HTML tags [Musciano, 1998]. Fur- thermore troff can handle requests read from groff -Thtml -e -ms filename.ms | html2ps diversions by pushing the current minimal HTML state before a diversion is read and popping it later and these are shown in Figures 12, 13, 14 and 15 once the diversion has finished. This allows diver- respectively.The test results appear satisfactory and sions to centre lines, alter the indentation, and emit showthat the indentation using the IP macro works preformatted text, whilst restoring the state once the as expected. This macro uses a diversion to detect
I.J. of SIMULATION Vol. 6 No 7-8 41 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML
groff-ms -e -Tps -rps4html=1 intemediate users troffdocument p PostScript r output e generate Definite integral g images image coordinates .EQ r from sum from x=a to b... o PostScript .EN h file png images 1 2 3 t m l groff-Thtml -e -ms -Z post HTML grohtml output
Figure 11: the groff -Thtml -ms -e pipe line
Escape Meaning \O0 Disable anyglyphs from being emitted at the outer level. † \O1 Enable output of glyphs, provided that the escape occurs at the outer level. † \O2 Providing the escape occurs at the outer level, enable output of glyphs and also write out to stderr the page number and four registers encompassing the glyphs previously writ- ten since the last call to \O. \O3 Begin a nesting level. \O4 End a nesting level. \O[5Pfilename] HTML device specific. Provided that this escape occurs at the outer nesting levelwrite the filename to stderr.The position of the image, P,must be specified and must be one of l, r, c or i (left, right, centre, inline). ______†Both \O0 and \O1 reset four troff registers which mark the top left and bottom right hand corners of an area containing all written glyphs.
Table 1: new troff escapes and their meaning the width of its first argument before emitting the represented between lines which contain a command paragraph. The example shows howeach line 1, line line option and those lines which are a component of 2and line 3 are vertically aligned. Notice that line 3 ahanging paragraph. Figure 19 shows howthe has a vertical space between itself and threefold. GNU Troffhomepage appears when printed from a Figure 13 shows that groff can translate eqn input browser.The homepage was created by the follow- into HTML and images and Figure 15 shows that ing command line. indentation within diversions behavescorrectly. groff -pet -ms -Thtml webpage.ms At present the GNU Troffmacro sets: ms, man,and mwww have the extra HTML tags included. Figure 17 The file webpage.ms makeextensive use of the shows the output of a component from the GNU pic mwww macro set which provides drop capital, guide [Raymond, 2005b] when processed through unordered line list, URL and image related macros. It the following pipeline. also makes use of colour which was recently added to GNU Troff. groff -pet -ms -Thtml pic.ms | html2ps -s2 CONCLUSIONS AND FURTHER WORK It can be seen that grohtml correctly determines that the equation is inside a picture and generates In conclusion the grohtml device driverhas been one image. Figure 18 shows the grohtml manual constructed and integrated into groff.This work page after it was processed by the pipeline: has taken six years of part time effort and amounts to approximately 9600 lines of C++ code (not includ- groff -pet -man -Thtml grohtml.man ing macro set modifications). In due course the groff-1.19.2 will be formally released and this and viewed through a browser.Inthis case we can will be the first beta release of grohtml.During the clearly see the vertical alignment correctly development of grohtml anumber of additional
I.J. of SIMULATION Vol. 6 No 7-8 42 ISSN 1473-804x online, 1473-8031 print G. MULLEY and W.LEMBERG: EXTENDING GNU TROFF TOPRODUCE HTML features were needed to produce useful HTML. These features were applied as generically as possi- hello ble and include the introduction of colour.Colour is hello nowsupported by the ASCII,PostScript, HTML,and Figure 15: the result of HTML output from Figure 9 DVI devices. GNU pic has also been extended to support colour. TagDescription The principles behind the HTML device driver appear sound, howeverthere are still minor bugs in .SH start of section heading grohtml.Future work will include bug fixing and .NH start of numbered heading also modifications to tbl so that troff tables can .col n start of column n be translated into HTML