Edinburgh Research Explorer Omega Becomes a Sign Processor Citation for published version: Haralambous, Y & Bella, G 2005, 'Omega Becomes a Sign Processor', TUGboat, vol. 27, no. 0, pp. 99-110. <https://www.tug.org/TUGboat/tb27-0/haralambous.pdf> Link: Link to publication record in Edinburgh Research Explorer Document Version: Publisher's PDF, also known as Version of record Published In: TUGboat General rights Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s) and / or other copyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associated with these rights. Take down policy The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer content complies with UK legislation. If you believe that the public display of this file breaches copyright please contact [email protected] providing details, and we will remove access to the work immediately and investigate your claim. Download date: 26. Sep. 2021 Proceedings EuroTEX2005 – Pont-à-Mousson, France MOT02 Omega Becomes a Sign Processor Yannis Haralambous ENST Bretagne [email protected] http://omega.enstb.org/yannis G´abor Bella ENST Bretagne [email protected] Characters and Glyphs not one but four equivalence classes of shapes: ara- bic initial letter jeem, arabic medial letter The distinction between “characters” and “glyphs” jeem, and so on. But are these “characters”? is a rather new issue in computing, although the Answering to this question requires a pragmatic problem is as old as humanity: our species turns out approach. What difference will it make if we have to be a writing one because, amongst other things, one or rather four characters for letter jeem? There our brain is able to interpret images as symbols be- will indeed be a difference in operations such as longing to a given writing system. Computers deal searching, indexing, etc. A good question to ask with text in a more abstract way. When we agree is: “when I’m searching in an Arabic document, am that, in computing, all possible “capital A” letters I looking for specific forms of letters?” Most of the are represented by the number 65, then we cut short time, the answer is negative.1 Form-independent all information on how a given instance of capital let- searching will, most of the times, produce better re- ter A is drawn. Modern computing jargon describes sults and this implies that having a single character this process as “going from glyphs to characters.” 2 for all forms is probably a better choice. If a glyph is the image of a writing system’s atomic Unicode is a character encoding. In other unit, a character is an interpretation of that image, words, it contains descriptions of characters and an interpretation shared by many glyphs drawn by tries hard to define characters properly by avoiding different people in different places at different times. dependence on glyphs.3 If all these drawings are equivalent in terms of in- terpretation, we can consider character as an equiv- alence class of glyphs. To be operational such an 1 Arabic words are not always visually segmented as En- equivalence class must be described in a clear and glish ones—there is, for example, no guarantee that the first letter of a word will always be in initial form: if a word start- unambiguous way. This is why we define charac- ing with jeem is preceded by the definite article al, then the ter as being a description of an equivalence class of jeem will end up being in medial form. glyphs [7, pp. 53–58], [6]. 2 Greek is different: sigma doesn’t “become” final because Arabic text provides a typical illustration it “happens” to be at the end of a word. While medial sigma can appear anywhere, final sigma is used mainly for the end- ground for the concepts of character and glyph. In ings of particular grammatical forms and in onomatopeias or Arabic alphabet, letters are contextual, in the sense foreign words. One would hardly ever search for both the fi- that a given letter will change form according to nal and medial form of sigma since their rˆoles are distinct. the presence or absence of other surrounding ones. To illustrate this, when we abreviate a word by a period at a sigma then the latter does remain medial despite being the When we refer to an Arabic letter and represent it final letter: φιλοσοφ¬α → φιλοσ. Hence it is quite logical to graphically, we use the isolated form. We can also use distinct characters for medial and final sigma. 3 refer to it by its description (for example: arabic This is not always the case because of Unicode’s letter jeem) and this can be considered as de- tenth founding principle, namely convertibility of legacy encodings—and legacy encodings contain all kinds of things. scription of a “character”: the equivalence class of For example, again in the case of Arabic, the main Unicode shapes this letter can take in millions of Arabic doc- Arabic table indeed contains only form-independent “char- uments. While there may be millions of instances of acters.” But, hidden towards the end of the first Unicode this letter, according to Arabic grammar they all be- plane, one finds several hundreds of codepoints containing Arabic letters and ligatures in fixed forms, for legacy reasons. long to one of only four forms: isolated , initial , Like human history (or Stephen King’s movies) Unicode has medial , or final . Hence, we could choose to have shadowy places which people try to avoid and even to forget that they exist. Omega Becomes a Sign Processor 99 Yannis Haralambous, Gábor Bella MOT02 Proceedings EuroTEX2005 – Pont-à-Mousson, France The character vs. glyph issue is far from being to have any relation whatsoever with its character solved. In this paper we give an attempt to tran- code. Obviously, we can hardly talk about charac- scend it by introducing a new concept: the sign. ters at this point: we have crossed the bridge to Glyphland. A sign is a set {c, p1 = v1, . , pn = vn, g} where c Another very interesting node type is the “lig- is a character, g a glyph, and pi an arbitrary num- ature node” [10, §143]. This one contains a font ID, ber of named properties taking values vi. Charac- a glyph position in the font table, and a pointer to ter, glyph, number of properties, their names and a linked list of charnodes. This list is in fact the their values can be changed at any time, by spe- “decomposition” of the ligature. T X needs it in cial syntax in the input file, or by OTPs, or by E case it has to “break” the ligature during paragraph interaction with fonts. building, for example when a word needs to be hy- Using the term “sign” we clearly refer to a Saus- phenated inside a ligature. surian linguistics tradition whose relevance for Talking about hyphenation, there is a node nowadays semiotics needs not to be proven. For called “discretionary node” [10, §145]. This node Saussure [11, p. 99], sign is the basic discrete unit contains two pointers to horizontal lists, as well as of meaning. It is a dual entity made up of a signifier an integer. These horizontal lists are what is type- and a signified. For instance, when we write or pro- set when we break a word, before and after the line nounce the word “tree,” the visual or auditory image break (in standard cases the first one contains only a of this word is the signifier and the concept of tree is hyphen, and the second one is empty). The integer the signified. Inspired of this analysis one could at- is the number of nodes of the main horizontal list tempt to apply the notion of sign to characters and to delete if the word is hyphenated (in other words: glyphs, by asserting that glyph is signifier, and char- how many nodes to replace by the two horizontal acter is signified. Nevertheless, in semiotics things lists we mentioned). are not that simple because linguists generally deal As we see, in the three node types de- with units of meaning rather than with words per scribed above only glyphs are used—never charac- se, and even less with letters. A letter inside a word ters. There seems to be a contradiction with the is not considered to be a Saussurian sign. very nature of hyphenation: after all, words are hy- This is why we are bound to warn the reader phenated according to rules given by natural lan- that our concept of sign is inspired from but not guage grammars, and these grammars apply to char- identical to Saussurian sign. acters, not to glyphs. Indeed, would you hyphenate a word differently if some letters had calligraphic In Principio Creavit Knuth T Xum E shapes? Certainly not, but for TEX, these letters How does TEX deal with characters and/or glyphs? are glyphs in font tables, and if variant forms exist, First of all, .tex files contain characters. When then their positions in the font tables are necessar- TEX reads a file, it converts the data stream into to- ily different from the standard ones. How does TEX kens. A token ([10, §289] or [8], which is an exegesis deal with this? of Knuth’s B) is either a “character token” (that is, There is in TEX a primitive called \lccode (and two numbers: a character code and the character’s a corresponding WEB macro called lc code).