– *TALO’s LANGUAGE TECHNOLOGY HYPHENATORS SPELL CHECKERS DICTIONARIES Dr. J.C. Woestenburg, – *TALO b.v., Lijsterlaan 379, 1403 AZ Bussum, The Netherlands. tel: +31 35 69 32 801 fax: +31 35 69 75 993 e-mail: [email protected] http://www.talo.nl/ Spelling of lexical and grammatical collocations Enlarged edition Completely revised License the best language technology! Revision October 2021 Please, view PDF versions of this book with Adobe Reader Windows, Linux (x86) or Mac on- ly. For other platforms or viewers, fonts might miss some glyphs (character images) or PDF engines might incorrectly position some combining (diacritical) glyphs. 20th Edition, October 2021 – Copyright © *TALO b.v., 2001, 2002, 2003, 2004, 2005, ..., 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021. All rights reserved. Without limiting the rights under copyright reserved above, no part of this production may be reproduced, stored in or introduced into a retrieval system or transmitted, in any form or by any means (electronic, mechanical, photocopying, recording or otherwise), without the prior written permission of both the copyright owner and the above publisher of this book The greatest care has been taken in compiling this book. However, no responsibility can be accepted by the publisher or author for the accuracy of the information presented. Content Preface vii – Who is *TALO? viii – Insert *TALO in your software x – 1. TALO’s Language Technology 2 – 1.1. TALO’s Speller for the European, American, Austra- lian, African and Asian languages 2 1.1.1. Speller: languages and sizes of dictionaries 6 – 1.2. TALO’s Hyphenator for the European, American, Aus- tralian, African and Asian languages 17 1.2.1. Hyphenator: languages and varieties 21 1.3. Hyphenation and spelling worries: how to de-Babel- ize? 29 1.4. The complexity of languages 31 The linear model 33 Why you should use lingual Quarks! 34 1.5. References 35 2. The hyphenation rules for the European langua- ges 36 2.1. Overview Hyphenation Rules 36 2.1.1. The Afrikaans hyphenation rules (Skeiding van Woord- dele) 36 2.1.2. The Basque hyphenation rules (hitzen ebaketa silabik- oa) 37 2.1.3. The Catalan hyphenation rules (trencar un mot al final de ratlla, nova ortografia) 38 2.1.4. The Danish Hyphenation rules (orddeling men ofta stave.nøglen) 39 Dividing attached articles 40 2.1.5. The Dutch hyphenation rules (Woordafbreking) 40 Notes! 41 Dividing compounds 41 Special hyphenations 42 Aesthetics 42 2.1.6. The English hyphenation rules 42 Confusing examples 44 Aesthetics 44 2.1.7. The Estonian hyphenation rules (silbitusreeglid) 44 2.1.8. The Finnish hyphenation rules (tavujako, tavutussään- nöt) 45 2.1.9. The French hyphenation rules (syllabisation) 46 Note! 46 Etymological and phonological rules 47 Hyphenation in publishing matters 47 2.1.10. The Frisian hyphenation rules (staveringshifker/-ôf- brekking) 48 2.1.11. The Galician hyphenation rules (separazón silábica). 48 2.1.12. The German hyphenation rules (Silbentrennung) 48 Pre-1996 Special hyphenations 49 Differences between the Swiss German and German language 49 German reformed 50 2.1.13. The Icelandic hyphenation rules (orðskipihlutinn) 51 2.1.14. The Italian hyphenation rules (divisione in sillabe) 52 Dividing prefixes 53 2.1.15. The Norwegian hyphenation rules (deling av ord) 53 Special hyphenations 54 2.1.16. The Portuguese hyphenation rules (regras de hifeniza- çao, separaçao sílabas) 54 2.1.17. The Spanish hyphenation rules (separación silábica). 55 Conjugations 56 2.1.18. The Swedish hyphenation rules (avstavning) 56 Dividing compound words 57 2.1.19. The hyphenation rules of the Greek language 58 2.1.20. The languages of Eastern Europe and the Balkan 59 2.1.20.1. Hyphenation of the Slavic languages 59 2.1.20.2. Hyphenation of the Hungarian language 62 2.1.20.3. Hyphenation of the Baltic languages 63 2.1.20.4. The language of the Dacians 63 2.1.20.5. On the border of Europe: Turkish, Azerbaijanian and Kazakh 64 2.1.21. Hyphenation of the Near Eastern languages 65 2.1.21.1. Hebrew 65 2.1.21.2. Arabic and the Arabic script 65 2.1.22. Hyphenation of the Austronesian languages 65 2.2. Epilogue 66 2.3. References 67 3. A heritage of 5000 years 70 3.1. Today’s Languages and Indo-European Ancestry 73 3.1.1. The Germanic languages 74 3.1.1.1. The Scandinavian languages 74 Swedish 75 Norwegian 76 Danish 77 3.1.1.2. The West Germanic languages 78 3.1.2. The Italic languages 87 3.1.3. The Balto-Slavic languages 96 (c) TALO Bsm Preface – The word *TALO is a reconstruction of an Old Germanic word meaning a notch on a tally. The original marks on a tally can be thought of as the very first digital carvings in wood, being similar to bits in a present-day computer language word. The emergence of computers enabled the development of mathematical methods for pro- cessing large volumes of text. The mathematical engineering approach of the 80s, howev- er, does not fit into the psychology of language and doesn’t solve problems that are inher- ent in a language. In other words: the nature and complexity of language were not includ- ed in the work done by the mathematicians and engineers. Consequently, the mechanical approach didn’t solve the basic problems encountered in hyphenation and spell checking. The diversity and nature of languages force us to reconsider the way technology is set up. All languages have their roots and linguistic family relations in the distant past. Scientists have proposed hypotheses about the earliest emergence of speech, but the re- al origins of speech can not be established. Factual knowledge about languages dates back to about 5000 or 6000 years ago. Early hypotheses assumed an onomatopoetic root as (wo)men tried to imitate animal sounds. Other hypotheses assume that speech has its origins in the expression of emo- tions or in the communication needed during cooperative efforts (labour). Some theories assume an underlying application of rules of generative innate grammar. More recently, the semantic factor of language has been emphasized. The structure of words doesn’t seem to be just the result of a chaotic process, but it also doesn’t seem to be just the re- sult of the application of certain rules. This dominant semantic factor, a trace of meaning, has guided us in the construction of tools for processing written language. The basic elements of meaning have been used to find the smallest building blocks of meaning that help to decide where to hyphenate. These building blocks vastly improve hyphenation as well as spelling. They are applied in a set of tools described in the first chapter of this book. This book is intended to provide a better view of language. It reviews the different hyphen- ations in different languages as well as the relations between families of languages and the origins of languages. When our ancestors, about 1500 generations ago, came up with names for man (*manu- in Proto-Indo-European) and for family members, and later on started riding horses, in- vented the wheel, and built fortified elevated places (referred to in word combinations con- taining bhergh-), all of these activities and subjects were given names. These ancient roots are still present in current languages despite the fact that through the ages langua- ges have continuously been subject to changes. Development of relevant, practical and successful language technology requires the – knowledge and understanding that a company like *TALO has of the evolution and peculi- arities of many individual languages. J.C.Woestenburg, PhD, Bussum, 28 September 2006 *TALO–’s Language Technology (c) TALO Bsm – Who is *TALO? – *TALO is a Netherlands-based software company, specializing in the develop- ment of language modules. We conceived, designed, and developed the *TA- – LO hyphenation system from a psychological, cognitive, and language perspec- tive. The system is based on a unified structure for syllabification, that works in every alphabetical language. Our first product was called the “Dutch hyphena- tor”, and was integrated into the Dutch version of WordPerfect in 1986. Our next step was building a software generator to create hyphenators for other lan- guages. The creation of hyphenator modules was followed by the development of spelling modules, based on the same language principles. The development of a hyphenator begins with the thorough research into the language involved, followed by the action of the powerful generator. The reliabili- ty of the hyphenator depends on the quality of the research. In a short period of time, determined by the complexity of a language, the generator completes the creation of the hyphenator. The fundamental language model based on morpho- logical principles produces reliable hyphenations. The mechanism of the hy- phenator is a process of pattern recognition, based on the psychology of the working brain, and reacts instantaneously. Information is available in a very short time. The combination of this mechanism and the language model results in the hyphenator’s high speed. The hyphenators are currently comparable to the end state of a neural network. The hyphenators follow the rules and exceptions of hyphenation governed by leading academies. They internally take care of every peculiarity of a language. They correctly hyphenate new words and even complex compounds. – The *TALO speller/correctors are conceptually different from other ‘spellers’. The morphological principles applied to their mechanism stem from our vast ex- perience with hyphenators. Reliability is ensured by the principle of fuzzy dis- tance: an alternative is presented that really resembles the misspelled word, in- cluding reversals (e.g. the instead of the) and missing (initial) letters.