DOCSLIB.ORG

Language Technology Meets Documentary Linguistics: What We Have to Tell Each Other

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Language Technology Meets Documentary Linguistics: What We Have to Tell Each Other Language technology meets documentary linguistics: What we have to tell each other Language technology meets documentary linguistics: What we have to tell each other Trond Trosterud Giellatekno, Centre for Saami Language Technology http://giellatekno.uit.no/ . February 15, 2018 . Language technology meets documentary linguistics: What we have to tell each other Contents Introduction Language technology for the documentary linguist Language technology for the language society Conclusion . Language technology meets documentary linguistics: What we have to tell each other Introduction Introduction I Giellatekno: started in 2001 (UiT). Research group for language technology on Saami and other northern languages Gramm. modelling, dictionaries, ICALL, corpus analysis, MT, ... I Trond Trosterud, Lene Antonsen, Ciprian Gerstenberger, Chiara Argese I Divvun: Started in 2005 (UiT < Min. of Local Government). Infrastructure, proofing tools, synthetic speech, terminology I Sjur Moshagen, Thomas Omma, Maja Kappfjell, Børre Gaup, Tomi Pieski, Elena Paulsen, Linda Wiechetek . Language technology meets documentary linguistics: What we have to tell each other Introduction The most important languages we work on . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Language technology for documentary linguistics ... Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist ... what’s in it for the language community I work for? I Let’s pretend there are two types of language communities: 1. Language communities without plans for revitalisation or use in domains other than oral use 2. Language communities with such plans . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Language communities without such plans I Gather empirical material and do your linguistic analysis I (The triplet: Grammar, text collection and dictionary) I ... thereby doing much of the same groundwork as e.g. the neogrammarians: I A dictionary containing all words + (parallel) text collections I A phonemic and grammatical analysis I A reconstruction of protolanguages at different levels, based upon the analysis of related languages I The Fenno-Ugricists actually did this . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Computational methods as tools for the field linguist? I Simple letter string analysis: I Reverse-sort your lexica I ... | rev | sort | rev | ... I «Give me all stems with the vocalism -a-e-» I ... | egrep 'a[ˆaeiou]+e' | ... I Do wordform statistics (also taking variation into account) I [å|á|à|ä] (->) a , ... I «Unix for poets» and beyond . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Test your grammar I Let’s say you are a grammarian, I you write a morphophonological analysis of a language I Vowel harmony seems to be conditioned by these stem patterns I or want to make a generalisation over Sandhi phenomena I Combining affixes with the following segments will result in these and these changes I then you could write a computational model of your grammar I ... and test whether you generalisations actually hold: I on a corpus of (all) available text I or by generating paradigms with your alleged rules and inspect the result I (or you could do it manually, as before) . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Is it really worth it (for the linguist)? I The reverse-sorting? Definitely, yes. I But for the automatic grammar testing it is not that clear: I Pro: I Making a computational model of your grammar will force you to be explicit and comprehensive I And yes, it will tell you when your generalisations do (not) hold, and to what extent there is variation in your material I Con: I It is a machinery to learn I ... and perhaps you are explicit and comprehensive anyway . Language technology meets documentary linguistics: What we have to tell each other Language technology for the documentary linguist Note: There is an extra bonus! I Our imagined language society had no intention whatsoever to use the language outside traditional oral settings, or revitalise it but... I Suddenly the next generation changes its mind (many poorly documented languages have lots of speakers) . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society Language communities with plans for revitalisation / language use in the modern society I You now need to write a model for your language, and standardise it to some extent I Note the conflicting interests: I The linguist wants an orthography with all distinctions (you know what I mean) I The L1 speaker needs to produce and recognize the words I The L2 speaker needs cues to pronounce the words, and a stable norm to remember them I (We linguists have a sad tradition of confusing transliteration and orthography, and our needs with the needs of the language community) . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society For the computer, variation is not a problem I Machines are far better than humans in handling complexity and variation I Our way of modelling grammars in Tromsø: The finite state transducer . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society The solution that fell out of fashion Table: The Chomsky hierarchy Grammar Languages Automaton Production rules (constraints) Type-0 Recursively Turing machine α ! β (no restr.) enumerable Type-1 Context-sensitive Linear-bounded αAβ ! αγβ nondeterministic Turing machine Type-2 Context-free Non-deterministic A ! γ pushdown automa- ton Type-3 Regular Finite state A ! a and A ! aB automaton (FST) . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society The limitation of finite state transducers I To Chomsky: There are aspects of human language that FSTs do not cover (e.g. Swiss German) I But this is ok, we may ignore those aspects I ... and there is (in my opinion) a difference between syntax and morphology anyway I To me (my dissertation): FSTs tell me what, but not why I But this is ok as well, this is the limitation of all formal systems I The answer to the question why must be sought from outside the formal system . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society So why FSTs, and what do they give us? I A way of turning a formal model of the phonology and morphology into a bidirectional machine I The technology is not new (Koskenniemi 1983), but its virtues do not lie in its fashionability I This bidirectional machine may I ... immediately tell us whether our generalisations are empirically adequate or not I ... and provide the foundation for practical programs . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society What is an FST anyway? . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society How we write it Morphology: LEXICON NOUNS koira CASELEX ; katu CASELEX ; ... LEXICON CASELEX +N+Pl+Nom:ˆWGt # ; ... Morphophonology: t:d <=> _ Vow ˆWG: ; . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society The task for the grammarian: I Distinguish segmental from suprasegmental I make one model of each, and join them together . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society FSTs as spell checkers . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society The effect of introducing proofing tools I ... is unfortunately an under-studied topic I A personal note I My Saami correspondence is dependent upon the speller, and others tell the same I Ávvir, the North Saami daily, is dependent upon it I ... whereas NRK Sápmi do not use proofing tools, and have a home page with 7.2% Saami I The reactions in Greenland to getting a spellchecker were: I «Of course there is a spellchecker for Greenlandic! The Danes have it, so why not we?» . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society Proofing tools may be controversial I Fluent but analphabetic elderly speakers may be positive I The people who invented / govern the orthography may protest against proofing tools I They see their knowledge monopoly crumble: I «The young will not need to learn the language when there are proofing tools» I Revitalisers and language learners love the proofing tools . Language technology meets documentary linguistics: What we have to tell each other Language technology for the language society Our number 1 success story in language communities: e-dictionaries Table: Amount of base forms
Load more

Recommended publications
  • Enhanced Thesaurus Terms Extraction for Document Indexing
    Enhanced Thesaurus Terms Extraction for Document Indexing Frane ari¢, Jan najder, Bojana Dalbelo Ba²i¢, Hrvoje Ekli¢ Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, 10000 Zagreb, Croatia E-mail:{Frane.Saric, Jan.Snajder, Bojana.Dalbelo, Hrvoje.Eklic}@fer.hr Abstract. In this paper we present an mogeneous due to diverse background knowl- enhanced method for the thesaurus term edge and expertise of human indexers. The extraction regarded as the main support to task of building semi-automatic and auto- a semi-automatic indexing system. The matic systems, which aim to decrease the enhancement is achieved by neutralising burden of work borne by indexers, has re- the eect of language morphology applying cently attracted interest in the research com- lemmatisation on both the text and the munity [4], [13], [14]. Automatic indexing thesaurus, and by implementing an ecient systems still do not achieve the performance recursive algorithm for term extraction. of human indexers, so semi-automatic sys- Formal denition and statistical evaluation tems are widely used (CINDEX, MACREX, of the experimental results of the proposed MAI [10]). method for thesaurus term extraction are In this paper we present a method for the- given. The need for disambiguation methods saurus term extraction regarded as the main and the eect of lemmatisation in the realm support to semi-automatic indexing system. of thesaurus term extraction are discussed. Term extraction is a process of nding all ver- batim occurrences of all terms in the text. Keywords. Information retrieval, term Our method of term extraction is a part of extraction, NLP, lemmatisation, Eurovoc.
    [Show full text]
  • Arxiv:1908.07448V1
    Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing Milan Straka and Jana Strakova´ and Jan Hajicˇ Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {strakova,straka,hajic}@ufal.mff.cuni.cz Abstract Shared Task (Zeman et al., 2018). • We report our best results on UD 2.3. The We present an extensive evaluation of three addition of contextualized embeddings im- recently proposed methods for contextualized 25% embeddings on 89 corpora in 54 languages provements range from relative error re- of the Universal Dependencies 2.3 in three duction for English treebanks, through 20% tasks: POS tagging, lemmatization, and de- relative error reduction for high resource lan- pendency parsing. Employing the BERT, guages, to 10% relative error reduction for all Flair and ELMo as pretrained embedding in- UD 2.3 languages which have a training set. puts in a strong baseline of UDPipe 2.0, one of the best-performing systems of the 2 Related Work CoNLL 2018 Shared Task and an overall win- ner of the EPE 2018, we present a one-to- A new type of deep contextualized word repre- one comparison of the three contextualized sentation was introduced by Peters et al. (2018). word embedding methods, as well as a com- The proposed embeddings, called ELMo, were ob- parison with word2vec-like pretrained em- tained from internal states of deep bidirectional beddings and with end-to-end character-level word embeddings. We report state-of-the-art language model, pretrained on a large text corpus. results in all three tasks as compared to results Akbik et al.
    [Show full text]
  • Extended and Enhanced Polish Dependency Bank in Universal Dependencies Format
    Extended and Enhanced Polish Dependency Bank in Universal Dependencies Format Alina Wróblewska Institute of Computer Science Polish Academy of Sciences ul. Jana Kazimierza 5 01-248 Warsaw, Poland [email protected] Abstract even for languages with rich morphology and rel- atively free word order, such as Polish. The paper presents the largest Polish Depen- The supervised learning methods require gold- dency Bank in Universal Dependencies for- mat – PDBUD – with 22K trees and 352K standard training data, whose creation is a time- tokens. PDBUD builds on its previous ver- consuming and expensive process. Nevertheless, sion, i.e. the Polish UD treebank (PL-SZ), dependency treebanks have been created for many and contains all 8K PL-SZ trees. The PL- languages, in particular within the Universal De- SZ trees are checked and possibly corrected pendencies initiative (UD, Nivre et al., 2016). in the current edition of PDBUD. Further The UD leaders aim at developing a cross- 14K trees are automatically converted from linguistically consistent tree annotation schema a new version of Polish Dependency Bank. and at building a large multilingual collection of The PDBUD trees are expanded with the en- hanced edges encoding the shared dependents dependency treebanks annotated according to this and the shared governors of the coordinated schema. conjuncts and with the semantic roles of some Polish is also represented in the Universal dependents. The conducted evaluation exper- Dependencies collection. There are two Polish iments show that PDBUD is large enough treebanks in UD: the Polish UD treebank (PL- for training a high-quality graph-based depen- SZ) converted from Składnica zalezno˙ sciowa´ 1 and dency parser for Polish.
    [Show full text]
  • Universal Dependencies According to BERT: Both More Specific and More General
    Universal Dependencies According to BERT: Both More Specific and More General Tomasz Limisiewicz and David Marecekˇ and Rudolf Rosa Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics Charles University, Prague, Czech Republic {limisiewicz,rosa,marecek}@ufal.mff.cuni.cz Abstract former based systems particular heads tend to cap- ture specific dependency relation types (e.g. in one This work focuses on analyzing the form and extent of syntactic abstraction captured by head the attention at the predicate is usually focused BERT by extracting labeled dependency trees on the nominal subject). from self-attentions. We extend understanding of syntax in BERT by Previous work showed that individual BERT examining the ways in which it systematically di- heads tend to encode particular dependency verges from standard annotation (UD). We attempt relation types. We extend these findings by to bridge the gap between them in three ways: explicitly comparing BERT relations to Uni- • We modify the UD annotation of three lin- versal Dependencies (UD) annotations, show- ing that they often do not match one-to-one. guistic phenomena to better match the BERT We suggest a method for relation identification syntax (x3) and syntactic tree construction. Our approach • We introduce a head ensemble method, com- produces significantly more consistent depen- dency trees than previous work, showing that bining multiple heads which capture the same it better explains the syntactic abstractions in dependency relation label (x4) BERT. • We observe and analyze multipurpose heads, At the same time, it can be successfully ap- containing multiple syntactic functions (x7) plied with only a minimal amount of supervi- sion and generalizes well across languages.
    [Show full text]
  • An Evaluation of Machine Learning Approaches to Natural Language Processing for Legal Text Classification
    Imperial College London Department of Computing An Evaluation of Machine Learning Approaches to Natural Language Processing for Legal Text Classification Supervisors: Author: Prof Alessandra Russo Clavance Lim Nuri Cingillioglu Submitted in partial fulfillment of the requirements for the MSc degree in Computing Science of Imperial College London September 2019 Contents Abstract 1 Acknowledgements 2 1 Introduction 3 1.1 Motivation .................................. 3 1.2 Aims and objectives ............................ 4 1.3 Outline .................................... 5 2 Background 6 2.1 Overview ................................... 6 2.1.1 Text classification .......................... 6 2.1.2 Training, validation and test sets ................. 6 2.1.3 Cross validation ........................... 7 2.1.4 Hyperparameter optimization ................... 8 2.1.5 Evaluation metrics ......................... 9 2.2 Text classification pipeline ......................... 14 2.3 Feature extraction ............................. 15 2.3.1 Count vectorizer .......................... 15 2.3.2 TF-IDF vectorizer ......................... 16 2.3.3 Word embeddings .......................... 17 2.4 Classifiers .................................. 18 2.4.1 Naive Bayes classifier ........................ 18 2.4.2 Decision tree ............................ 20 2.4.3 Random forest ........................... 21 2.4.4 Logistic regression ......................... 21 2.4.5 Support vector machines ...................... 22 2.4.6 k-Nearest Neighbours .......................
    [Show full text]
  • Using Lexico-Syntactic Ontology Design Patterns for Ontology Creation and Population
    Using Lexico-Syntactic Ontology Design Patterns for ontology creation and population Diana Maynard and Adam Funk and Wim Peters Department of Computer Science University of Sheffield Regent Court, 211 Portobello S1 4DP, Sheffield, UK Abstract. In this paper we discuss the use of information extraction techniques involving lexico-syntactic patterns to generate ontological in- formation from unstructured text and either create a new ontology from scratch or augment an existing ontology with new entities. We refine the patterns using a term extraction tool and some semantic restrictions derived from WordNet and VerbNet, in order to prevent the overgener- ation that occurs with the use of the Ontology Design Patterns for this purpose. We present two applications developed in GATE and available as plugins for the NeOn Toolkit: one for general use on all kinds of text, and one for specific use in the fisheries domain. Key words: natural language processing, relation extraction, ontology generation, information extraction, Ontology Design Patterns 1 Introduction Ontology population is a crucial part of knowledge base construction and main- tenance that enables us to relate text to ontologies, providing on the one hand a customised ontology related to the data and domain with which we are con- cerned, and on the other hand a richer ontology which can be used for a variety of semantic web-related tasks such as knowledge management, information re- trieval, question answering, semantic desktop applications, and so on. Automatic ontology population is generally performed by means of some kind of ontology-based information extraction (OBIE) [1, 2]. This consists of identi- fying the key terms in the text (such as named entities and technical terms) and then relating them to concepts in the ontology.
    [Show full text]
  • LASLA and Collatinus
    L.A.S.L.A. and Collatinus: a convergence in lexica Philippe Verkerk, Yves Ouvrard, Margherita Fantoli, Dominique Longrée To cite this version: Philippe Verkerk, Yves Ouvrard, Margherita Fantoli, Dominique Longrée. L.A.S.L.A. and Collatinus: a convergence in lexica. Studi e saggi linguistici, ETS, In press. hal-02399878v1 HAL Id: hal-02399878 https://hal.archives-ouvertes.fr/hal-02399878v1 Submitted on 9 Dec 2019 (v1), last revised 14 May 2020 (v2) HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. L.A.S.L.A. and Collatinus: a convergence in lexica Philippe Verkerk, Yves Ouvrard, Margherita Fantoli and Dominique Longrée L.A.S.L.A. (Laboratoire d'Analyse Statistique des Langues Anciennes, University of Liège, Belgium) has begun in 1961 a project of lemmatisation and morphosyntactic tagging of Latin texts. This project is still running with new texts lemmatised each year. The resulting files have been recently opened to the interested scholars and they now count approximatively 2.500.000 words, the lemmatisation of which has been checked by a philologist. In the early 2.000's, Collatinus has been developed by Yves Ouvrard for teaching.
    [Show full text]
  • Experiments in Clustering Urban-Legend Texts
    How to Distinguish a Kidney Theft from a Death Car? Experiments in Clustering Urban-Legend Texts Roman Grundkiewicz, Filip Gralinski´ Adam Mickiewicz University Faculty of Mathematics and Computer Science Poznan,´ Poland [email protected], [email protected] Abstract is much easier to tap than the oral one, it becomes feasible to envisage a system for the machine iden- This paper discusses a system for auto- tification and collection of urban-legend texts. In matic clustering of urban-legend texts. Ur- this paper, we discuss the first steps into the cre- ban legend (UL) is a short story set in ation of such a system. We concentrate on the task the present day, believed by its tellers to of clustering of urban legends, trying to reproduce be true and spreading spontaneously from automatically the results of manual categorisation person to person. A corpus of Polish UL of urban-legend texts done by folklorists. texts was collected from message boards In Sec. 2 a corpus of 697 Polish urban-legend and blogs. Each text was manually as- texts is presented. The techniques used in pre- signed to one story type. The aim of processing the corpus texts are discussed in Sec. 3, the presented system is to reconstruct the whereas the clustering process – in Sec. 4. We manual grouping of texts. It turned out that present the clustering experiment in Sec. 5 and the automatic clustering of UL texts is feasible results – in Sec. 6. but it requires techniques different from the ones used for clustering e.g. news ar- 2 Corpus ticles.
    [Show full text]
  • Removing Boilerplate and Duplicate Content from Web Corpora
    Masaryk University Faculty}w¡¢£¤¥¦§¨ of Informatics !"#$%&'()+,-./012345<yA| Removing Boilerplate and Duplicate Content from Web Corpora Ph.D. thesis Jan Pomik´alek Brno, 2011 Acknowledgments I want to thank my supervisor Karel Pala for all his support and encouragement in the past years. I thank LudˇekMatyska for a useful pre-review and also for being so kind and cor- recting typos in the text while reading it. I thank the KrdWrd team for providing their Canola corpus for my research and namely to Egon Stemle for his help and technical support. I thank Christian Kohlsch¨utterfor making the L3S-GN1 dataset publicly available and for an interesting e-mail discussion. Special thanks to my NLPlab colleagues for creating a great research environment. In particular, I want to thank Pavel Rychl´yfor inspiring discussions and for an interesting joint research. Special thanks to Adam Kilgarriff and Diana McCarthy for reading various parts of my thesis and for their valuable comments. My very special thanks go to Lexical Computing Ltd and namely again to the director Adam Kilgarriff for fully supporting my research and making it applicable in practice. Last but not least I thank my family for all their love and support throughout my studies. Abstract In the recent years, the Web has become a popular source of textual data for linguistic research. The Web provides an extremely large volume of texts in many languages. However, a number of problems have to be resolved in order to create collections (text corpora) which are appropriate for application in natural language processing. In this work, two related problems are addressed: cleaning a boilerplate and removing duplicate and near-duplicate content from Web data.
    [Show full text]
  • The State of (Full) Text Search in Postgresql 12
    Event / Conference name Location, Date The State of (Full) Text Search in PostgreSQL 12 FOSDEM 2020 Jimmy Angelakos Senior PostgreSQL Architect Twitter: @vyruss https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 Contents ● (Full) Text Search ● Indexing ● Operators ● Non-natural text ● Functions ● Collation ● Dictionaries ● Other “text” types ● Examples ● Maintenance https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 Your attention please Allergy advice ● This presentation contains linguistics, NLP, Markov chains, Levenshtein distances, and various other confounding terms. ● These have been known to induce drowsiness and inappropriate sleep onset in lecture theatres. https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 What is Text? (Baby don’t hurt me) ● PostgreSQL character types – CHAR(n) – VARCHAR(n) – VARCHAR, TEXT ● Trailing spaces: significant (e.g. for LIKE / regex) ● Storage – Character Set (e.g. UTF-8) – 1+126 bytes → 4+n bytes – Compression, TOAST https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 What is Text Search? ● Information retrieval → Text retrieval ● Search on metadata – Descriptive, bibliographic, tags, etc. – Discovery & identification ● Search on parts of the text – Matching – Substring search – Data extraction, cleaning, mining https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 Text search operators in PostgreSQL ● LIKE, ILIKE (~~, ~~*) ● ~, ~* (POSIX regex) ● regexp_match(string text, pattern text) ● But are SQL/regular expressions enough? – No ranking of results – No concept of language – Cannot be indexed ● Okay okay, can be somewhat indexed* ● SIMILAR TO → best forget about this one https://www.2ndQuadrant.com FOSDEM Brussels, 2020-02-02 What is Full Text Search (FTS)? ● Information retrieval → Text retrieval → Document retrieval ● Search on words (on tokens) in a database (all documents) ● No index → Serial search (e.g.
    [Show full text]
  • A Diachronic Treebank of Russian Spanning More Than a Thousand Years
    Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5251–5256 Marseille, 11–16 May 2020 c European Language Resources Association (ELRA), licensed under CC-BY-NC A Diachronic Treebank of Russian Spanning More Than a Thousand Years Aleksandrs Berdicevskis1, Hanne Eckhoff2 1Språkbanken (The Swedish Language Bank), Universify of Gothenburg 2Faculty of Medieval and Modern Languages, University of Oxford [email protected], [email protected] Abstract We describe the Tromsø Old Russian and Old Church Slavonic Treebank (TOROT) that spans from the earliest Old Church Slavonic to modern Russian texts, covering more than a thousand years of continuous language history. We focus on the latest additions to the treebank, first of all, the modern subcorpus that was created by a high-quality conversion of the existing treebank of contemporary standard Russian (SynTagRus). Keywords: Russian, Slavonic, Old Church Slavonic, treebank, diachrony, conversion and a richer inventory of syntactic relation labels. 1. Introduction Secondary dependencies are used to indicate external The Tromsø Old Russian and OCS Treebank (TOROT, subjects, for example in control structures, and to indicate Eckhoff and Berdicevskis 2015) has been available in shared dependents, for example in structures with various releases since its beginnings in 2013, and its East coordinated verbs. Empty nodes are allowed to give more Slavonic part now contains approximately 230K words.1 information on elliptic structures and asyndetic This paper describes the TOROT 20200116 release,2 which coordination. The empty nodes are limited to verbs and adds a conversion of the SynTagRus treebank. Former conjunctions, the scheme does not allow empty nominal TOROT releases consisted of Old East Slavonic and nodes.
    [Show full text]
  • Universal Dependencies for Japanese
    Universal Dependencies for Japanese Takaaki Tanaka∗, Yusuke Miyaoy, Masayuki Asahara}, Sumire Uematsuy, Hiroshi Kanayama♠, Shinsuke Mori|, Yuji Matsumotoz ∗NTT Communication Science Labolatories, yNational Institute of Informatics, }National Institute for Japanese Language and Linguistics, ♠IBM Research - Tokyo, |Kyoto University, zNara Institute of Science and Technology [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] Abstract We present an attempt to port the international syntactic annotation scheme, Universal Dependencies, to the Japanese language in this paper. Since the Japanese syntactic structure is usually annotated on the basis of unique chunk-based dependencies, we first introduce word-based dependencies by using a word unit called the Short Unit Word, which usually corresponds to an entry in the lexicon UniDic. Porting is done by mapping the part-of-speech tagset in UniDic to the universal part-of-speech tagset, and converting a constituent-based treebank to a typed dependency tree. The conversion is not straightforward, and we discuss the problems that arose in the conversion and the current solutions. A treebank consisting of 10,000 sentences was built by converting the existent resources and currently released to the public. Keywords: typed dependencies, Short Unit Word, multiword expression, UniDic 1. Introduction 2. Word unit The definition of a word unit is indispensable in UD an- notation, which is not a trivial question for Japanese, since The Universal Dependencies (UD) project has been de- a sentence is not segmented into words or morphemes by veloping cross-linguistically consistent treebank annota- white space in its orthography.
    [Show full text]