Babel speaks Hindi
Zdeněk Wagner Vinohradská 114 13000 Prague 3 Czech Republic zdenek dot wagner (at) gmail dot com http://icebearsoft.euweb.cz
Abstract Babel provides a unified interface for creation of multilingual documents. Un- fortunately no Indic languages are currently supported, so typesetting in In- dic languages is based on specialised packages. The most advanced of these is Velthuis Devan¯agar¯ı for TEX, because it already provides Hindi values for language-dependent strings as well as a macro for a European-style date. A language definition file for plugging Hindi into Babel has therefore been recently developed. The second part of the paper explains differences between Unicode and Velthuis transliteration. This is important for understanding the tool that can convert Hindi and Sanskrit documents from MS Word and OpenOffice.org into TEX via an XSLT 2.0 processor and a Perl script, as well as a method of making the PDF files searchable. Finally the paper discusses some possibilities of further development: the advantages offered by X TE EX and the forthcoming integration of Lua into pdfTEX.
1 Introduction guage definition file which will integrate Hindi into the Babel system. Packages for typesetting in various Indic languages in both plain TEX and LATEX have been available 2 Birth of the Language Definition File from CTAN for a long time. The authors of these The aim of our work was to enable transparent use packages have made substantial efforts to support of Hindi in multilingual documents by means of the the Indic scripts, which present difficulties that can- standard Babel invocation: not be solved by TEX itself. For example, although generation of conjuncts and placing dependent vow- \usepackage[hindi]{babel} els (matras) to subscripts and superscripts could be Preparing a language definition file for Babel is handled as ligatures in TFM files, the form of a con- not a difficult task. It involves defining language- junct depends also on language. While conjuncts Ä dependent macros such as \chaptername, and the (kta) and à (nna) are used in Sanskrit and tradi- date macro \today. These definitions were already tional Hindi, they are nowadays often replaced with present in the Devan¯agar¯ı package and they even the half forms ?t and n, respectively. Such matters properly handled switching between the Bombay, can be solved only by using a preprocessor bound to Calcutta and Nepali variants of the Devan¯agar¯ıfonts. a (LA)TEX macro package. However, placing them into the language definition The existing packages are used by indologists file is not enough. Rendering these words in the De- from all over the world as well as by people from In- van¯agar¯ıfont requires a lot of special macros. More- dia. It is therefore unfortunate that the packages over, as mentioned previously, the Hindi text cannot support only the script and lack features to sup- be fed directly to TEX — preprocessing is manda- port the language. An exception is the Velthuis De- tory. van¯agar¯ıfor TEX package [1]. Since version 2.13, it It therefore seemed useless to copy the macros contains definitions for language-dependent strings from the Devan¯agar¯ıpackage to the language defini- as well as a European-style date, and macros for tion file and follow the same development in two switching between them and an English version. It different places. It was therefore decided just to therefore provides a kind of a mini-Babel. The fur- load devanagari.sty and redefine its options as lan- ther natural development step was to prepare a lan- guage attributes. As a matter of fact, the language
176 TUGboat, Volume 27 (2006), No. 2 — Proceedings of the 2006 Annual Meeting Babel speaks Hindi
definition file itself without the fonts and the prepro- In contrast, the transliteration scheme devel- cessor would not work at all. Thus the requirement oped by Frans Velthuis tries to be as close to schol- to have the Devan¯agar¯ıpackage installed does not arly practice as possible. Devan¯agar¯ıis traditionally pose a real limitation. transliterated into the Roman alphabet where long The devanagari.sty package already contains vowel, retroflex consonants, etc., are marked with captions and date macros for English. These defini- diacritics [10]. The practice varies slightly between tions must not be removed because they are already different textbooks and dictionaries. The Velthuis documented and their removal might damage legacy transliteration is a 7-bit encoding so the text can be documents. On the other hand these definitions col- input on a standard US keyboard. Transliteration lide with the Babel core. is based on pronunciation, although in Hindi a short The package was therefore modified so that the a in the middle of words, as well as at the end of macros are declared by means of \providecommand words, is very often not pronounced. The inherent and the definition delayed via \AtBeginDocument. a in the middle of the word must always be written The macros are thus guaranteed to exist and the but the final short a can be omitted. Thus krnA Babel definitions have higher preference no matter must be input as karanaa while Gr can be written in which order the files are loaded. The language just ghar. definition file checks the package version and will Important aspects of the differences between complain if an old version of devanagari.sty is in- these two approaches will be shown in the follow- stalled. ing subsections. As already mentioned, it is not sufficient to ac- tivate the Hindi language either as the main lan- guage or by any of the Babel language switching 3.1 Conversion into the Velthuis environments. The text must be enclosed within a transliteration {\dn ...} group, or the preprocessor will not find Book production is often a collaborative work of au- it. thor(s), editor(s), and a compositor. The authors rarely supply the texts in TEX; they commonly use other text editors, mostly MS Word. The first task 3 Unicode vs. Velthuis transliteration is therefore conversion of the supplied manuscript Devan¯agar¯ıoriginates in an ancient Br¯ahm¯ıscript, into TEX. Almost all markup must be removed and and is an abugida. Each consonant (vyanjana) also replaced in order to achieve a particular graphical represents an inherent following vowel — a in the design. case of Devan¯agar¯ı. Other vowels are added as di- We have found it is advantageous to open the acritics in the vicinity of the consonant. Groups of manuscript in OpenOffice.org, and then save in its consonants often form conjuncts, with only a minor- native format, which is XML. Although various tools ity of them written using a virama sign. Initial form are freely available, they still retain too much of of vowels have a different shape than vowel diacritics the author’s markup. Use of TEXML will also re- (matras). quire much work which can hardly be reused in other Unicode is based on characters. The inherent books. Since the OpenOffice.org file format is XML, a is not encoded. Thus U+0915 represents the k conversion can be performed by XSLT. A simple (ka) syllable (akshara). The syllable Ek (ki) is rep- stylesheet can remove nearly all markup, keeping resented by the two Unicode characters U+0915 and just boldface, italics and footnotes. U+093F. Reversing the order of glyphs in displayed Another difficulty is that some Unicode charac- output is left to the rendering engine [2]. ters are not directly available in TEX. Fortunately, Independent vowels (initial forms) have distinct Saxon 8.x [5], which implements XSLT 2.0, offers codes, e. g. the code of i (i) is U+0907. The three character maps, which allow replacement of such characters U+0915 U+094D U+0924 denote a San- characters with TEX control sequences. However, skrit conjunct Ä (kta). However, such a glyph may this is not enough for the texts in the Devan¯agar¯ı not be present in modern Hindi fonts. The render- script. If we just convert Unicode characters to cor- ing engine will then display ?t. In order to force responding Roman letters, all inherent a’s would be the rendering engine to display the latter form even lost. though the Sanskrit ligature is present in the font, Although the result of the XSLT transformation zero-width-joiner must be inserted. The Unicode can be fed into TEX, some postprocessing is recom- encoding will then be U+0915 U+904D U+200D mended. Sometimes bold text is entered such that U+0924. each character is emboldened separately. Merging
TUGboat, Volume 27 (2006), No. 2 — Proceedings of the 2006 Annual Meeting 177 Zdeněk Wagner
these into a single \textbf command is more eas- be found by hyperlinks, it is also desirable to be ily done later in a Perl script than directly in the able to search for words and sentences. Here, In- stylesheet. Also, the whole OpenOffice.org docu- dic scripts present a big problem. PDF, similar to ment is output on a single line. We need to edit PostScript, deals with glyphs, but the field in the the resulting file in order to wrap it to a reasonable search dialogue accepts Unicode characters. Files width. Without any programming, this can be done generated by the Devan¯agar¯ıpackage are therefore by the Perl Text::Wrap module. unsearchable. X TE EX [13] can use OpenType fonts, In addition to such formatting issues, we also but it turns out that a PDF file created by X TE EX split the main job of conversion between XSLT and is not searchable either, although the situation is Perl. First of all, long vowels can be encoded in not simple. X TE EX can use several engines for PDF the Velthuis transliteration either by doubling or creation and the results differ. OpenOffice.org is by uppercasing. The stylesheet always uses upper- slightly more successful because all simple words case for long vowels in order to prevent ambiguities. which do not contain conjuncts are searchable. For instance, kI will be converted into kaI because The key problem stems from the distinction be- kaii will be rendered as k{i, which is wrong. Empty tween glyphs and characters. Mappings between braces would also solve the problem, but using up- glyphs and characters can be inserted into so-called percase is easier. The independent vowels are trans- ‘ToUnicode’ maps. This feature is already imple- formed directly into the corresponding letter(s) in mented in the cmap.sty package. An experimen- the Velthuis transliteration. The dependent vowels tal ToUnicode map was therefore developed for the (matras) are preceded with an equal sign. The con- Velthuis Devan¯agar¯ıpackage. Since pdfTEX inserts sonants are followed by equal signs and the virama these maps according to the font encoding, each In- is transformed into an underscore. dic script will require its own encoding name for Afterwards the Perl script takes its turn. In LATEX. Currently Devan¯agar¯ı, Bengali, and Gur- the input, each paragraph appears on a single line mukhi use encoding U, and all of them rely on pre- to be processed. The first task is to form conjuncts. processors with analogous functionality. In the pres- This will work for Sanskrit words as well as mod- ent experimental project, encoding X0900 was used ern Hindi, where some ligatures are not used, e. g. in order to refer to the corresponding Unicode block. in aÚ^ ÚA. The virama will be added by the prepro- The dvng fonts contain single characters, liga- cessor. Conjuncts are created by tures and pieces which are glued together by TEX while (s/(\{\\dn [^}]*)=_/$1/) {} macros. Single characters and ligatures are directly mapped to Unicode characters. Half forms of the In the next step matras are handled. Double consonants are mapped to the corresponding char- equal signs are simply removed and lone equal signs acters followed by a virama. Vattu is added to a denote missing inherent a’s to be added. This is full consonant, therefore it is mapped to the r (ra) achieved by the following lines. consonant preceded by virama. This makes words while with conjuncts searchable. (s/(\{\\dn [^}]*)==([aAiIuU.eo])/$1$2/) {} Unfortunately it is not possible to solve all prob- while (s/(\{\\dn [^}]*)=/$1a/) {} lems. Since PDF works with glyphs, i-matras always Next, we remove the final inherent a unless we precede the consonants. When searchng such words, are converting a Sanskrit document. it is necessary to type them into the search field as they appear, i. e. to write the i-matra in front of while (!$opt_sanskrit && the consonant. Acrobat Reader is also confused by s/(\{\\dn [^}]*)a([^.a-zA-Z])/$1$2/) {} placing vowel diacritics below or above consonants Finally the line is wrapped. as well as by vattus. An extra word boundary is eval { created. Thus when searching for k Sl , two words print wrap(’’, ’’, $_); 1; k Sl must be typed into the search field. The } or do { word Ú~ Aivr must be entered in a way impossible in warn "Warning: $@"; print; the Velthuis transliteration, namely as two words Ú~ }; Aivr. Unicode allows starting a word with a depen- dent vowel which is, of course, incorrect. This mis- 3.2 Searchable PDF files feature enables making such words searchable. Ex- PDF files play an important role as online docu- tra word boundaries are not formed if the akshara is ments. Although the chapter and section names drawn as a glyph in the dvng font. The words zknA may be placed into outlines and related parts may and mA/A can be searched for as such. More detailed
178 TUGboat, Volume 27 (2006), No. 2 — Proceedings of the 2006 Annual Meeting Babel speaks Hindi
information is present in the documentation of the as well as output in Lua, it will be easier to imple- experimental ToUnicode map [12]. ment indexing software in Indic languages. The ToUnicode map is also used when copy- ing and pasting the text from a PDF file to another 5 Requirements for multilingual application. It is no surprise that we get into the environment same problems. The pasted text will contain extra Multilingual support is required not only in TEX, spaces after vattu and both short and long u-matras. but also in the whole operating system. First, it Words with i-matras will look visually correct but is necessary to display the Unicode characters cor- their Unicode representation will be wrong. If a rectly. Groups of consonants with viramas must be word such as EÚSlF is copied and pasted from PDF properly combined into conjuncts and matras moved and then sent to a sorting algorithm, it will appear to the correct place. It is achieved in Linux by one of in a nonsensical place, because the program will see the following libraries: ICU [4] and Pango [8]. How- a word starting with i-matra which is not allowed ever, these libraries are not yet used by all programs. by Hindi orthography. Even Firefox does not use Pango by default, it must first be activated by setting MOZ_PANGO_ENABLE=1. MS Internet Explorer displays Hindi texts correctly. 4 Future development The text must also be prepared using Unicode. The greatest disadvantage of contemporary Velthuis There remain problems with editors under Linux. Devan¯agar¯ıfor TEX is the necessity of using the pre- OpenOffice.org, gEdit, and
TUGboat, Volume 27 (2006), No. 2 — Proceedings of the 2006 Annual Meeting 179 Zdeněk Wagner
Figure 1: Comparison of search for Ú~ Aivr (input as Ú~ Aivr) in Adobe Acrobat Reader 7 in Linux (left) and Windows XP (right).
6 Conclusion [2] Joan Aliprand et al. The Unicode Standard, The paper describes how the Babel module for Hindi chapter South Asian Scripts. The Unicode was made. It further presents thoughts concerning Consortium, 2003. http://www.unicode. the use of Indic texts as online documents. Tools for org/faq/indic.html#5. conversion from MS Word and OpenOffice.org into [3] Hans Hagen. LuaTEX: Howling to the TEX and for making PDF files with Devan¯agar¯ıtexts moon. TUGboat, 26(2):152–157, 2005. searchable are presented. These tools are available http://www.tug.org/TUGboat/Contents/ from the author’s web page [11]. contents26-2.html. [4] International components for Unicode. 7 Acknowledgement http://icu.sourceforge.net/. The author would like to acknowledge the work [5] Michael Kay. Saxon, the XSLT and XQuery of other developers of the Velthuis Devan¯agar¯ıfor processor. http://saxon.sourceforge.net/. T X, as the package is the base for this project, as E [6] Petr Olšák. encTEX. well as Karel Píška for converting the Indic fonts http://www.olsak.net/enctex.html. to the Type 1 format [9]. Special thanks belong [7]
180 TUGboat, Volume 27 (2006), No. 2 — Proceedings of the 2006 Annual Meeting