The Asian Journal of TEX, Volume 2, No. 1, April 2008 Article revision 2008/3/15 KTS THE KOREAN TEXSOCIETY SINCE 2007 Unicode Support in the CJK Package Werner Lemberg Municipal theatre of Koblenz, Germany Kleine Beurhausstraße 1 44 137 Dortmund, Germany [email protected] KEYWORDS Unicode, UTF-8, CJK,LATEX, CJKutf8, font encoding, character set, X TE EX, PDF, bookmarks. ABSTRACT This article describes how the CJK package for LATEX handles Unicode encoded characters. Using the CJKutf8.sty file, LATEX’s native Unicode handling is com- bined with CJK’s capabilities for CJKV scripts. Another recent extension is sup- port for X TE EX which is here described also. 1 Basics I want to start this article with the introduction of some important concepts which are needed to correctly understand the sometimes tricky details of this topic. Many people use the terms described in this section quite sloppily; this often leads to heavy confusion, as can be easily seen, for example, in the archives of the CJK mailing list [1]. 1.1 Characters and glyphs Characters are entities which have a semantic meaning. Visual presentation forms of characters are called glyphs. A character can be represented by more than a single glyph — just think of an italic A and a sans-serif A. However, this mapping is not a one-to-one relationship; for example, the glyph A could be, among other characters, either a Latin, a Cyrillic, or a Greek uppercase letter. TEX, as do basically all text formatting programs, expects characters on the input side and emits glyphs on the output side. 1.2 Character sets and input encodings A character set is a collection of characters. An input encoding assigns code points to each element of a character set. A corollary of those separate definitions is that it is possible to have more than a single encoding for a given character set. For example, the Korean character set KS X 1001:1992 (formerly called KS C 5601-1992) can be used as the main encoding in an EUC (Extended Unix Code) encoding with code points in the range 0xA1A1–0xFDFE (with the second byte covering only the range 0xA1–0xFE), or it can be mapped onto Unicode (with many gaps for all Unicode characters not in KS X 1001). Copyright © 2008 by the Korean TEX Society 12 THE ASIAN JOURNAL OF TEX An excellent reference for Asian character sets and encodings is Ken Lunde’s book CJKV Information Processing [2]. 1.3 Font encodings Internally, each font, regardless of its format, uses either indices or names to access its glyphs. For some font formats, this is directly visible; below I discuss the most important ones of this kind, namely PostScript Type 1 and CID-keyed PostScript fonts. 1.3.1 Glyph names and indices Type 1 fonts [3] can contain more than 256 glyphs, but the standard way to access them is using an encoding vector which maps at most 256 glyph names to codes 0x00–0xFF; the rest of them are unencoded. In case you want to access unencoded glyphs you have to change the encoding vector to make them encoded (and others unencoded). Glyphs of CID-keyed PostScript fonts [4] do not have names. Instead, the devel- oper of this font technique, Adobe, has defined unique character collections (which are ordered glyph collections actually, using the terminology introduced in subsec- tion 1.2), giving each glyph a fixed index (the CID, character ID, which should rather be called a glyph ID). An example for such a collection is ‘Adobe-GB1-0’, which de- fines an ordering of glyphs for the Chinese GB 2312-80 character set. Note that col- lections for Asian scripts contain, among other things, some extra glyphs for vertical typesetting [5]. TEX fonts belong in this category too but their format is the most primitive of all of them: At most 256 glyphs are contained in a TEX font, to be accessed with indices only. 1.3.2 Character maps Other font formats like OpenType [6] use a different route to access glyphs by provid- ing character maps (also called ‘cmaps’), depending on the used platform and encoding. A typical OpenType font to be used with MS Windows in Unicode encoding has a (3,1) cmap: Value 3 is the PID, the platform ID (Windows), and value 1 is the EID, the en- coding ID (Unicode). Character maps exist for CID-keyed fonts too (called ‘CMaps’ in Adobe’s parlance). For example, the CMap ‘KSC-V’ maps the ISO-2022-KR input encoding to the ‘Adobe- Korea1-0’ collection, using vertical glyph representation forms [5]. While CMaps for CID-keyed fonts can select glyphs for either horizontal or vertical writing, this is not possible with OpenType cmaps. Instead, a different mechanism is used: the selection of features, based on scripts and languages. Assuming that an OpenType font supports vertical typesetting, it is possible, for example, to select the feature ‘vrt2’ (vertical representation forms1) for the script ‘hani’ (CJK ideographic script) and language ‘dflt’ (default, that is, to be applied for all languages belonging to the ‘hani’ tag). The sets of tags for feature, script, and language names are predefined.2 X TE EX supports selection of OpenType features, scripts, and languages. 1. An older version of this feature is called ‘vert’, which contains a subset of the glyphs in the ‘vrt2’ feature. 2. It is possible to extend these sets if necessary. VOLUME 2, NO. 1, APRIL 2008 13 Unicode binary representation UTF-8 binary representation U+0000–U+007F 00000000 00000000 0xxxxxxx 0xxxxxxx U+0080–U+07FF 00000000 00000yyy yyxxxxxx 110yyyyy 10xxxxxx U+0800–U+FFFF 00000000 zzzzyyyy yyxxxxxx 1110zzzz 10yyyyyy 10xxxxxx U+10000–U+10FFFF 000uuuzz zzzzyyyy yyxxxxxx 11110uuu 10zzzzzz 10yyyyyy 10xxxxxx TABLE 1. The relationship between Unicode values and UTF-8 encoding. 1.4 Unicode and UTF-8 Unicode [7] is a very large character set; the current major version 5.0 contains 99 024 characters; 70 229 of them are CJKV ideographic characters. A Unicode value is a number between 0x0 and 0x10FFFF (comprising 1 114 112 code points); this means that about 9% of the available code space is already allocated, with plenty of space for future additions. Note that about 12% of the code space is reserved for private use.3 Using Unicode values directly within a data stream is not possible since they are, as mentioned earlier, numbers and nothing else. Various formats have been defined for data exchange; the most common encoding forms are UTF-16 (each Unicode char- acter has a length of either two or four bytes), UTF-32 (all Unicode characters have a length of four bytes), and UTF-8 (Unicode characters have variable length). For TEX, only UTF-8 is of interest; the other two formats cannot be handled.4 Table 1 shows how Unicode is represented as UTF-8. The layout of UTF-8 is very clever, for a number of reasons: 1. The ASCII characters stay unmodified. 2. Looking at the leading byte of a UTF-8 byte sequence it is immediately known how many bytes follow; 3. All non-leading bytes have the same format, and can never be mistaken for lead- ing bytes. Property 1 guarantees that UTF-8 works within single-byte environments. Proper- ties 2 and 3 allow easy resynchronization in case of a data stream error — most other multibyte encodings, in particular all EUC encodings, SJIS, and Big 5, are lacking this very useful feature: A single missing byte probably makes the rest of a document unreadable. 1.5 CJKV scripts and Unicode As Unicode encodes characters, not glyphs, CJKV characters (this is, the ideographic script developed in China thousands of years ago which is or was in use for Chinese, 3. For the ordinary user, Unicode and the ISO/IEC 10 646 standard are the same. I will not go into more detail here. 4. However, X TE EX can digest UTF-16 too. 14 THE ASIAN JOURNAL OF TEX ¨ ¨ TABLE 2. The Unicode character U+9AA8 in a font for traditional Chi- nese (left) and simplified Chinese (right). 8 8 N TABLE 3. The Unicode character U+9038 in a font for traditional Chi- nese (left), Korean (middle), and Japanese (right). Japanese, Korean, and Vietnamese) are ‘unified’ if possible.5 The example given in the Unicode standard book to demonstrate this unification process is character U+9AA8 (see table 2). A more extreme example is character U+9038 (table 3), where even the writing order and stroke numbers differ. Given these examples it is obvious that unification on the input side needs local- ization on the output side. In other words, you get ugly results if you use a Chinese font for Japanese and vice versa. It is thus a bad idea to use a single font like the noto- rious cyberbit.ttf to ‘print CJK characters’, as naïve users may believe — it can be used, say, as a fall-back font for Web browsers if no other font can handle a particular character, but for good typography it should be avoided. 2 LATEX’s Unicode handling Within LATEX, the standard inputenc mechanism is used to load support for UTF-8 en- coding; the macros can be found in file utf8.def. The mechanism is quite simple: Assuming that we have input character Œ (which is U+0152, LATIN CAPITAL LIGA- TURE OE), utf8.def maps it to the macro \u8:Œ. Here, ‘u8:Œ’ is a single macro name defined with \csname; the ‘Œ’ in this name is encoded in UTF-8.