DOCSLIB.ORG

The VISL System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The VISL System TheVISLSystem: ResearchandapplicativeaspectsofIT-basedlearing EckhardBick e-mail:[email protected] web:http://visl.sdu.dk 1.Abstract Thepaperpresentsanintegratedinter activeuserinterfaceforteachinggrammaticalanalysis throughtheInternetmedium(VisualInteractiveSyntaxLearning),developedatSouthernDenmark University, covering14differentlanguages ,halfofwhicharesupportedbylivegrammatical analysisofrunningtext.Forreasonsofrobustness,efficiencyandcorrectness,the system'sinternal toolsarebasedontheConstraintGrammarformalism(Karlsson,1990,1995),butusersarefreeto choosefromavarietyofnotationalfilters,supportingdifferentdescriptionalparadigms , witha currentteachingfocuson syntactictreestructuresand theform -functiondichotomy.Theoriginal kernelofprogramswasbuiltaroundamulti -levelparserforPortuguese(Bick, 1996, 2000) developedinadissertation frameworkatÅrhusUniversity andusedasapointofdeparturefor similarsystemsinother languages .Over thepast 5years,VISLhasgrownfromateaching initiativeintoafullblownresearch anddevelopment project withawiderangeof secondary projects,activitiesandlanguagetechnologyproducts. Examplesofapplicationorientedresearch areNLP -basedteachinggames, machinetranslationandgrammaticalspellchecking. TheVISL group has repeatedly attractedoutsidefundingfor thedevelopment of grammar teachingtools, semanticsbasedConstraintGrammarsandtheconstructionofannotatedcorpora. Online Proceedings of NODALIDA 2001 1.Background WhentheVISLprojectstartedin1996,itsprimarygoalwastofurthertheintegration ofITtoolsandITbasedcommunicationroutinesintotheuniversitylanguage teachingmilieuatOdenseUniversity(Denmark),andmorespecifically,todevelop toolsforVisualInteractiveSyntaxLearning.Theinitiativewasfundedjointlyby CTU(Centerfor Teknologi-StøttetUddannelse)andOdenseUniversityfor3years, andthelanguagesinvolvedwereEnglish,German,FrenchandPortuguese. Alreadyintheearlystagesoftheprojectitbecameclearthatadistinction wouldhavetobemadeastowhetherthe languagedatatobeusedintheteaching interfacewouldbelimitedtextbookexamplesorunlimitednaturallanguagetext.We decidedtodevelopbotha"closed"andan"open"system,andtodesigntheteaching applicationsformaximalsynergy,suchthatt heywouldbeabletotakeinputfrom boththeclosedandopenlanguagedatasources, -anddosoinalargelylanguage independentway. Fortheclosedsystemanotationalformalismwasdevelopedthatallowedthe textualexpressionofgraphicalsyntactict reestructures,anddatabasesofmanually analysedsentenceswerebuiltforallparticipatinglanguages,theoriginaltargetbeing 500textbooksentencesand500runningtextsentences.Withthehelpofenthusiastic studentsandteachers,these"closedcorpus"databasesareconstantlybeingenlarged, andtodayVISLcovers14languages,amongthemthebasicRomanceandGermanic languagesaswellasanumberofmoreexoticspecimen,likeArabic,Japaneseand Esperanto. Theopensystemisresearchbasedan dcenteredaroundtheConstraint Grammarparadigm,introducedbyFredKarlssonatHelsinkiUniversityintheearly 1990ies(Karlsson,1991,1995).The1996rolemodelforthesyntacticVISLsystem wasmyPortugueseCGparsingsystem(Bick1996,2000),whic hfeaturedafull dependencyanalysisofsubclausestructureandaprototypeCG -to-treesyntax transformationgrammar.IhavesincedevelopedsimilarCGbasedsystemsfor Danish,SpanishandEsperanto.ForEnglishandGerman,VISLhas correcteda nd amplifiedlicensedcommercialCGsystemsfromtheFinnishsoftwarefirmLingsoft. 2.Aunifiedapproachtogrammar ThecentralprincipleofVISL'slanguageanalysisis itsfocusonsurfacestructure (expressedaseitherdependencyrelationsorsyntactictrees tructures)andtheform - functiondichotomy.FollowingBacheet.al.(1993,1999),functionsymbolsstartwith uppercaseletters,formsymbolswithlowercaseletters,andboth arecombinedina combinedcolon -separatedsymbol(text)orfunction -over-formsy mbol(graphics). Forthedependencynotation,internationalCGconventionsarefollowed,withupper caselettersforallprimarytags,usingthe@ -symboltointroducefunctiontags,and arrowheads(>,<)forheadorienteddependencymarkers. Online Proceedings of NODALIDA 2001 VISLlightverticaltree VISLverticaltree (non-graphicalnotation) (non-graphicalnotation) UTT:cl(fcl) STA:fcl S:propVISL S:propVISL P:v(v-pr) er P:v-fin(v-pr) er Cs:g(np) Cs:np =D:art et =DN:art et =H:n forskningsprojekt =H:n forskningsprojekt =D:cl(fcl) =DN:fcl ==S:pron(pron-rel) der ==S:pron-rel der ==P:v(v-pr) involverer ==P:v-fin(v-pr) involverer ==Od:g(np) ==Od:np ===D:pron(pron-indef) mange ===DN:pron-indef mange ===D:adj forskellige ===DN:adj forskellige ===H:n sprog ===H:n sprog VISL [VISL] <heur><*> PROPNOM @SUBJ> er [være] <vk> VPRAKT @FMV et [en] ARTNEUSIDF @>N forskningsprojekt [forskningsprojekt] NNEUSIDFNOM @<SC , der [der] <rel> INDPnGnNNOM @SUBJ> involverer [involvere] <vt> VPRAKT&MV @FS-N< mange [mange] <quant> DETnGPNOM @>N forskellige [forskellig] ADJnGPnDNOM @>N sprog [sprog] NNEUPIDFNOM @<ACC . Meetingregularlyover4years,theVISLgroupofuniversityteachershasinvested considerableeffortindiscussingthecompatibilities,incompatibilitiesandblindspots ofdifferentnationalandlinguisticgrammartraditions,andagreedonacommon supersetofsymbols.Recently,areducedsymbolsetforpropedeuticuseandschools, "VISLlight",wasagreedupon,andtheDanishX-and-O-systemadaptedtomatchthe functioncategoriesusedinVISLlight. Atthelowe stlevel,11wordclassesand14 primaryfunctionsareused. Predicator(P),Verbal(V) ¡ Auxiliary(Vaux),alsoas<>(D) £ ¢ Mainverb(V*,Vm),alsoas (H,K) ¥ ¤ Verbchainparticle(Vp),alsoas<>(D),simplifiedas (A) ¦ Infinitivemarker(Vi,INFM),alsoas<>(D) § Subject(S) ( § ) Formalorprovisionalsubject(Sf),possiblywiththesubclassofsituativesubject(Ss) ¨ Direct(accusative)object(Od) ( ¨ ) Formalorprovisionalobject(Of) © Indirect(dative)object(Oi) Prepositionalobject(Op),emneled,evt.forenkletsom ¥ (A) ⊗ Subjectcomplement(Cs),Subjectpredicative(Ps) [⊗] Freesubjectpredicative(fPs,fCs),simplifiedas ¥ (A) Online Proceedings of NODALIDA 2001 ⊕ Objectcomplement(Co),Objectpredicative(Po) ⊕ [ ] Freeobjectpredicative(fPo,fCo),simplifiedas ¥ (A) ¥ ¥ Adverbial(A),withpossiblesubdivisionoffree( fA)orbound( bA,bAs,bAo) £ Head(H),Kernel(K) <> Dependents(D) Subordinator(SUB) ↔ Co-ordinator(CO) # Conjunct(CJT) «» Underspecifiedconstituentatclauselevel(e.g.clausebody) 3.Internetbasedteachingtools OnelessontobelearnedfromtheVISLproject,isthatitisnotatalleasyto introduceIT-basedtoolsintoanexistingteachingenvironment.Ap artfromhardware problems(thereneverbeingenough-compatibleandupdated-machinesintheright roomattherighttime),thereistheverycentralproblemofpsychologicalresistance againstthenewmedium,simplybecauseitmayfeeltoo"technical". Allthings technicalhaveaverylowacceptancerateintheHumanities,andteachersoftenresent thepersonalinvestmentintimeandeffortnecessarytoacquirethenecessaryskills - nottomentionchangesinteachingmaterialandexams.Thereis,ofco urse,a fundamentaldifference intermsof"technicality"betweenahumanteacheranda computerterminal,-thelatterlackstheteacher's naturalness,interactivity,flexibility and tutoringcapacities.Ontheotherhand,computersdohaveevidentteachin g advantages-theycanintegratethesenses,makinguseofcolours,picturesandsounds inamoreflexibleandimpressivemannerthanpapercan.Also,acomputerprogram can"know"more-intermsoffactsandexamples,andwithinawell -definedsubject matter-thanahumanteacher.Andlast,butnotleast,acomputersystem,especially ifaccessiblethroughtheinternet,canteachanunlimitednumberofstudentsatthe sametimeinwhatoptimallystillamountstoanindividualmanner. Giventheseadvantages,itmakessensetoinvestsomeeffortinaddressingthe fourmaindisadvantages,aslistedabove.TheVISL grammarteachinginterfacetries tomakeadvanceswithregardtothefollowingfourprinciples: (i)Flexibility TheVISLinterfaceisnotationallyf lexible,i.e.th eusercanchoosebetween severalnotationalconventions(e.g.flatdependencygrammar,enrichedtext,meta textnotation,treestructures),andmovebackandforthbetweendifferentlevelsof complexity.Forinstance,dependingontheexer cisechosen,thetypeandnumberof grammaticalcategoriesused(e.g.wordclasses)maybechanged.Inordertomake workmorecolourful,itisalsopossibletomovebetweentextbookmaterial,copied "live"texts,randomizedtestsentencesandone'sowncreativeidiolect. Inthetreestructureexamplebelow,theusercanswitchbackandforthbetween lettersymbolsandgraphicalsymbols,morethandoublethenumberofcategories,or reducethetreetoapurefunctiontree(greenonly). Online Proceedings of NODALIDA 2001 Lettersymbols Graphicalsymbols VISL'suniqueintegrationofteachingandresearchtoolswouldevenallowtheuserto experimentwithdifferentkindsofsubjectsor addacoupleofplaceandtime adverbialsandrerunthesentenceinfree-textmode–withexactlythesamegraphical setupandpaedagogicalfunctionality. (ii)Interactivity VISL'sjava -treeinterfaceforgrammaticalanalysisallowsthestep -by-step interactiveinspection,constructionandlabellingofsyntactictreesusingmenus, mouseclicksanddrag -and-dropmovements,allknownfromb asictextprocessor functionality. Inthe firstexamplebelow, astudenthasrecognizedthenp "minhest",buthas yettoassemble "lyst"ontothepredicator ("har")oftheadverbialsubclausetothe left. Online Proceedings of NODALIDA 2001 Whenasentenceprovesproblematicorincomprehensible,theusercanmodifyit,or askforthecomputer 'sopinion(show -meoption).Ingrammargameslike Paintbox, PostOffice or Shoot-the-Verb, interactivityinegratesac ertainelementof competition,andisfurtherenhancedbysoundeffects,timersandhigh-scores. Online Proceedings of NODALIDA 2001 (iii)Naturalness Amajordrawbackofmostlanguageteachingsoftware(or,forthatmatter, languageanalysissoftware)isthattheydonotrunonfree,na turallanguage,butonly onasmallsetofpredefinedsentencesorstructures("toylexica"or"toygrammars"),
Load more

Recommended publications
  • Using Constraint Grammar for Treebank Retokenization
    Using Constraint Grammar for Treebank Retokenization Eckhard Bick University of Southern Denmark [email protected] Abstract multiple tokens, in this case allowing the second part (the article) to become part of a separate np. This paper presents a Constraint Grammar-based method for changing the Tokenization is often regarded as a necessary tokenization of existing annotated data, evil best treated by a preprocessor with an establishing standard space-based abbreviation list, but has also been subject to ("atomic") tokenization for corpora methodological research, e.g. related to finite- otherwise using MWE fusion and state transducers (Kaplan 2005). However, there contraction splitting for the sake of is little research into changing the tokenization of syntactic transparency or for semantic a corpus once it has been annotated, limiting the reasons. Our method preserves ingoing comparability and alignment of corpora, or the and outgoing dependency arcs and allows evaluation of parsers. The simplest solution to the addition of internal tags and structure this problem is making conflicting systems for MWEs. We discuss rule examples and compatible by changing them into "atomic evaluate the method against both a tokenization", where all spaces are treated as Portuguese treebank and live news text token boundaries, independently of syntactic or annotation. semantic concerns. This approach is widely used in the machine-learning (ML) community, e.g. 1 Introduction for the Universal Dependencies initiative (McDonald et al. 2013). The method described in In an NLP framework, tokenization can be this paper can achieve such atomic tokenization defined as the identification of the smallest of annotated treebank data without information meaningful lexical units in running text.
    [Show full text]
  • Using Danish As a CG Interlingua: a Wide-Coverage Norwegian-English Machine Translation System
    Using Danish as a CG Interlingua: A Wide-Coverage Norwegian-English Machine Translation System Eckhard Bick Lars Nygaard Institute of Language and The Text Laboratory Communication University of Southern Denmark University of Oslo Odense, Denmark Oslo, Norway [email protected] [email protected] Abstract running, mixed domain text. Also, some languages, like English, German and This paper presents a rule-based Japanese, are more equal than others, Norwegian-English MT system. not least in a funding-heavy Exploiting the closeness of environment like MT. Norwegian and Danish, and the The focus of this paper will be existence of a well-performing threefold: Firstly, the system presented Danish-English system, Danish is here is targeting one of the small, used as an «interlingua». «unequal» languages, Norwegian. Structural analysis and polysemy Secondly, the method used to create a resolution are based on Constraint Norwegian-English translator, is Grammar (CG) function tags and ressource-economical in that it uses dependency structures. We another, very similar language, Danish, describe the semiautomatic as an «interlingua» in the sense of construction of the necessary translation knowledge recycling (Paul Norwegian-Danish dictionary and 2001), but with the recycling step at the evaluate the method used as well SL side rather than the TL side. Thirdly, as the coverage of the lexicon. we will discuss an unusual analysis and transfer methodology based on Constraint Grammar dependency 1 Introduction parsing. In short, we set out to construct a Norwegian-English MT Machine translation (MT) is no longer system by building a smaller, an unpractical science. Especially the Norwegian-Danish one and piping its advent of corpora with hundreds of output into an existing Danish deep millions of words and advanced parser (DanGram, Bick 2003) and an machine learning techniques, bilingual existing, robust Danish-English MT electronic data and advanced machine system (Dan2Eng, Bick 2006 and 2007).
    [Show full text]
  • Floresta Sinti(C)Tica : a Treebank for Portuguese
    )ORUHVWD6LQWi F WLFD$WUHHEDQNIRU3RUWXJXHVH 6XVDQD$IRQVR (FNKDUG%LFN 5HQDWR+DEHU 'LDQD6DQWRV *VISL project, University of Southern Denmark Institute of Language and Communication, Campusvej, 55, 5230 Odense M, Denmark [email protected], [email protected] ¡ SINTEF Telecom & Informatics, Pb 124, Blindern, NO-0314 Oslo, Norway [email protected],[email protected] $EVWUDFW This paper reviews the first year of the creation of a publicly available treebank for Portuguese, Floresta Sintá(c)tica, a collaboration project between the VISL and the Computational Processing of Portuguese projects. After briefly describing the main goals and the organization of the project, the creation of the annotated objects is presented in detail: preparing the text to be annotated, applying the Constraint Grammar based PALAVRAS parser, revising its output manually in a two-stage process, and carefully documenting the linguistic options. Some examples of the kind of interesting problems dealt with are presented, and the paper ends with a brief description of the tools developed, the project results so far, and a mention to a preliminary inter-annotator test and what was learned from it. supporting 16 different languages. VISL's Portuguese ,QWURGXFWLRQ0RWLYDWLRQDQGREMHFWLYHV system is based on the PALAVRAS parser (Bick, 2000), There are various good motives for creating a and has been functioning as a role model for other Portuguese treebank, one of them simply being the desire languages. More recently, VISL has moved to incorporate to make a new research tool available to the Portuguese semantic research, machine translation, and corpus language community, another the wish to establish some annotation proper.
    [Show full text]
  • Instructions for Preparing LREC 2006 Proceedings
    Translating the Swedish Wikipedia into Danish Eckhard Bick University of Southern Denmark Rugbjergvej 98, DK 8260 Viby J [email protected] Abstract Abstract. This paper presents a Swedish-Danish automatic translation system for Wikipedia articles (WikiTrans). Translated articles are indexed for both title and content, and integrated with original Danish articles where they exist. Changed or added articles in the Swedish Wikipedia are monitored and added on a daily basis. The translation approach uses a grammar-based machine translation system with a deep source-language structural analysis. Disambiguation and lexical transfer rules exploit Constraint Grammar tags and dependency links to access contextual information, such as syntactic argument function, semantic type and quantifiers. Out-of-vocabulary words are handled by derivational and compound analysis with a combined coverage of 99.3%, as well as systematic morpho-phonemic transliterations for the remaining cases. The system achieved BLEU scores of 0.65-0.8 depending on references and outperformed both STMT and RBMT competitors by a large margin. 1. Introduction syntactic function tags, dependency trees and a The amount of information available in Wikipedia semantic classification of both nouns and named differs greatly between languages, and many topics are entities. badly covered in small languages, with short, missing or stub-style articles. This asymmetry can be found 2. The Translation System (Swe2Dan) between Scandinavian languages, too. Thus, the In spite of the relatedness of Swedish and Danish, a Swedish Wikipedia has 6 times more text than its one-on-one translation is possible in less than 50% of Danish equivalent. Robot-created articles have helped all tokens.
    [Show full text]
  • 17Th Nordic Conference of Computational Linguistics (NODALIDA
    17th Nordic Conference of Computational Linguistics (NODALIDA 2009) NEALT Proceedings Series Volume 4 Odense, Denmark 14 – 16 May 2009 Editors: Kristiina Jokinen Eckhard Bick ISBN: 978-1-5108-3465-1 Printed from e-media with permission by: Curran Associates, Inc. 57 Morehouse Lane Red Hook, NY 12571 Some format issues inherent in the e-media version may also appear in this print version. Copyright© (2009) by the Association for Computational Linguistics All rights reserved. Printed by Curran Associates, Inc. (2017) For permission requests, please contact the Association for Computational Linguistics at the address below. Association for Computational Linguistics 209 N. Eighth Street Stroudsburg, Pennsylvania 18360 Phone: 1-570-476-8006 Fax: 1-570-476-0860 [email protected] Additional copies of this publication are available from: Curran Associates, Inc. 57 Morehouse Lane Red Hook, NY 12571 USA Phone: 845-758-0400 Fax: 845-758-2633 Email: [email protected] Web: www.proceedings.com Contents Contents iii Preface vii Commitees ix Conference Program xi I Invited Papers 1 JEAN CARLETTA Developing Meeting Support Technologies: From Data to Demonstration (and Beyond) 2 RALF STEINBERGER Linking News Content Across Languages 4 II Tutorial 6 GRAHAM WILCOCK Text Annotation with OpenNLP and UIMA 7 III Regular papers 9 LENE ANTONSEN,SAARA HUHMARNIEMI AND TROND TROSTERUD Interactive pedagogical programs based on constraint grammar 10 JARI BJÖRNE,FILIP GINTER,JUHO HEIMONEN,SAMPO PYYSALO AND TAPIO SALAKOSKI Learning to Extract Biological Event and
    [Show full text]
  • Frag, a Hybrid Constraint Grammar Parser for French
    FrAG, a Hybrid Constraint Grammar Parser for French Eckhard Bick University of Southern Denmark Odense, Denmark eckhard.bick @ mail.dk Abstract This paper describes a hybrid system (FrAG) for tagging / parsing French text, and presents results from ongoing development work, corpus annotation and evaluation. The core of the system is a sentence scope Constraint Grammar (CG), with linguist-written rules. However, unlike traditional CG, the system uses hybrid techniques on both its morphological input side and its syntactic output side. Thus, FrAG draws on a pre-existing probabilistic Decision Tree Tagger (DTT) before and in parallel with its own lexical stage, and feeds its output into a Phrase Structure Grammar (PSG) that uses CG syntactic function tags rather than ordinary terminals in its rewriting rules. As an alternative architecture, dependency tree structures are also supported. In the newest version, dependencies are assigned within the CG-framework itself, and can interact with other rules. To provide semantic context, a semantic prototype ontology for nouns is used, covering a large part of the lexicon. In a recent test run on Parliamentary debate transcripts, FrAG achieved F-scores of 98.7 % for part of speech (PoS) and between 93.1 % and 96.2 % for syntactic function tags. Dependency links were correct in 95.9 %. 1 CG with probabilistic input lexemes with information on This paper describes a hybrid tagger/parser for French, the PoS, paradigmatical 65.470 French Annotation Grammar (FrAG), and presents verbal valency 6.218 preliminary results from ongoing development work, nominal valency 230 corpus annotation and evaluation. The core of the system semantic class (nouns) 17.860 is a sentence scope Constraint Grammar (CG), with linguist-written rules modelled on similar systems for Table 1: Lexical information types Portuguese and Danish (Bick 2000).
    [Show full text]
  • Prescriptive Infinitives in the Modern North Germanic Languages: An
    Nor Jnl Ling 39.3, 231–276 C Nordic Association of Linguists 2016 doi:10.1017/S0332586516000196 Johannessen, Janne Bondi. 2016. Prescriptive infinitives in the modern North Germanic languages: An ancient phenomenon in child-directed speech. Nordic Journal of Linguistics 39(3), 231–276. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. Prescriptive infinitives in the modern North Germanic languages: An ancient phenomenon in child-directed speech Janne Bondi Johannessen The prescriptive infinitive can be found in the North Germanic languages, is very old, and yet is largely unnoticed and undescribed. It is used in a very limited pragmatic context of a pleasant atmosphere by adults towards very young children, or towards pets or (more rarely) adults. It has a set of syntactic properties that distinguishes it from the imperative: Negation is pre-verbal, subjects are pre-verbal, subjects are third person and are only expressed by lexical DPs, not personal pronouns. It can be found in modern child language corpora, but probably originated before AD 500. The paper is largely descriptive, but some theoretical solutions to the puzzles of this construction are proposed. Keywords: child-directed speech, context roles, finiteness, imperatives, negation, North Germanic languages, prescriptive infinitives, subjects, word order Janne Bondi Johannessen, University of Oslo, MultiLing & Text Lab, Department of Linguistics and Scandinavian Studies, P.O. Box 1102 Blindern, N–0317 Oslo, Norway.
    [Show full text]
  • A Morphological Lexicon of Esperanto with Morpheme Frequencies
    A Morphological Lexicon of Esperanto with Morpheme Frequencies Eckhard Bick University of Southern Denmark Campusvej 55, DK-5230 Odense M Email: [email protected] Abstract This paper discusses the internal structure of complex Esperanto words (CWs). Using a morphological analyzer, possible affixation and compounding is checked for over 50,000 Esperanto lexemes against a list of 17,000 root words. Morpheme boundaries in the resulting analyses were then checked manually, creating a CW dictionary of 28,000 words, representing 56.4% of the lexicon, or 19.4% of corpus tokens. The error percentage of the EspGram morphological analyzer for new corpus CWs was 4.3% for types and 6.4% for tokens, with a recall of almost 100%, and wrong/spurious boundaries being more common than missing ones. For pedagogical purposes a morpheme frequency dictionary was constructed for a 16 million word corpus, confirming the importance of agglutinative derivational morphemes in the Esperanto lexicon. Finally, as a means to reduce the morphological ambiguity of CWs, we provide POS likelihoods for Esperanto suffixes. Keywords: Morphological Analysis, Esperanto, Affixation, Compounding, Morpheme Frequencies 1. Introduction 2. Morphological analysis As an artificial language with a focus on regularity and From a language technology perspective, inflexional facilitation of language acquisition, Esperanto was regularity, morphological transparency and surface-based designed with a morphology that allows (almost) free, access to semantic features turn POS tagging of Esperanto productive combination of roots, affixes and inflexion into a non-task, and facilitate the parsing of syntactic and endings. Thus, the root 'san' (healthy) not only accepts its semantic structures (Bick 2007).
    [Show full text]
  • Degrees of Orality in Speechlike Corpora
    Degrees of Orality in Speech-like Corpora: Comparative Annotation of Chat and E-mail Eckhard Bick University of Southern Denmark [email protected] Background Spoken language data are difficult to obtain in large quantities (very time & labour consuming) Hypothesis: Certain written data may approximate some of the linguistic features of spoken language ● Candidates: chat, e-mail, broadcasts, speech and discussion transcripts, film subtitle files This paper discusses data, tools, pitfalls and results of such an approach: ● suitable corpora (from the CorpusEye initiative at SDU) ● suitable tokenization and annotation methodology (CG) ● linguistic insights and cross-corpus comparison The corpora http://www.cs.cmu.edu/~enron/ Enron E-mail Dataset: corporate e-mail (CALO Project) Chat Corpus 2002-2004 (Project JJ) ● (a) Harry Potter, (b) Goth Chat, (c) X Underground, (d) Amarantus: War in New York Europarl - English section (Philipp Koehn) ● transcribed parliamentary debates BNC (British National Corpus) ● split in (a) written and (b) spoken sections Grammatical Annotation Constraint Grammar (Karlsson et al. 1995, Bick 2000) ● reductionist rules, tag-based information ● rules remove, select, add or substitute information REMOVE VFIN IF (*-1C PRP BARRIER NON-PRE-N) ((0 N) OR (*1C N BARRIER NON-PRE-N)) EngGram (CG3 style grammar) ● modular architecture: morphological analysis --> disambiguation --> syntactic function --> dependency ● CG3: integrates local statistical information, uses unification ● robust and accurate (F-pos 99%,
    [Show full text]
  • Arborest – a Growing Treebank of Estonian
    Arborest – a Growing Treebank of Estonian by Eckhard Bick Institute of Language and Communication University of Southern Denmark [email protected] Heli Uibo and Kaili Müürisep Institute of Computer Science University of Tartu, Estonia {heli.uibo, kaili.muurisep}@ut.ee 1. Introduction Treebank creation is a very labor-consuming task, especially if the applications intended include machine learning, gold standard parser evaluation or teaching, since only a manually checked syntactically annotated corpus can provide optimal support for these purposes. There are, however, possibilities to make the annotation process (partly) automatic, saving (manual) annotation time and/or allowing the creation of larger corpora. Whenever possible, existing resources – both corpora and grammars – should be reused. In the case of the Estonian treebank project Arborest, we have therefore opted to make use of existing technology and experiences from the VISL project1, where two-stage systems including both Constraint Grammar (CG)- and Phrase Structure Grammar (PSG)-parsers have been used to build treebanks for several languages (Bick, 2003 [1]). Moreover, the VISL annotation scheme has been 1 URL: http://visl.sdu.dk adopted as a standard for tagging the parallel corpus in Nordic Treebank Network2. For Estonian, there already exists a shallow syntactically annotated – and proof-read – corpus, allowing us to bypass the first step in treebank construction (CG-parsing). This paper describes how a VISL-style hybrid treebank of Estonian has been semi-automatically derived from this corpus with a special Phrase Structure Grammar, using as terminals not words, but CG function tags. We will analyze the results of the experiment and look more thoroughly at adverbials, non-finite verb constructions and complex noun phrases.
    [Show full text]
  • The English Wikipedia in Esperanto
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DSpace at Tartu University Library WikiTrans: The English Wikipedia in Esperanto Eckhard Bick GrammarSoft ApS & University of Southern Denmark [email protected] Abstract: and as large as e.g. the Danish one, has only WikiTrans is a translation project and web portal for 140.000 articles, while the English Wikipedia with translated Wikipedias. Using the GrammarSoft's rule- its 3.4 million articles (or 2.345.000.000 words) is based GramTrans technology, we created a high-quality roughly 24 times as big. In addition, there is a English-Esperanto machine translation system, and used difference in article size1, with an average of 3.600 it to translate the entire English Wikipedia (ca. 3.000.000 letters (~ 600 words) for English and German, and a articles), at a speed of 17.000 articles a day. The translated articles are searchable both locally little over 1500 letters (~ 250 words) in Esperanto, (www.wikitrans.net) and in the original Esperanto translating into an even bigger factor of difference, Wikipedia, where we maintain a revision interface for 57, when focusing on content volume. In other users who wish to turn our translated articles into new words, more than 98% of the English language ”originals”. In this paper, we explain the framework and information is not accessible in Esperanto (or challenges of the project, and show how translation rules Danish). One could argue that the Esperanto articles can exploit grammatical information provided by a concentrate on the important and frequently sought- Constraint Grammar parser.
    [Show full text]
  • The Nordic Dialect Corpus – an Advanced Research Tool
    The Nordic Dialect Corpus – an advanced research tool Janne Bondi Johannessen Joel Priestley University of Oslo University of Oslo Oslo, Norway Oslo, Norway [email protected] [email protected] Kristin Hagen Tor Anders Åfarli University of Oslo Norwegian Univ. of Science & Tech. Oslo, Norway Trondheim, Norway [email protected] [email protected] Øystein Alexander Vangsnes University of Tromsø Tromsø, Norway [email protected] 2 Introduction Abstract In this paper, we describe the first, completed part of the Nordic Dialect Corpus. The corpus The paper describes the first part of the Nordic has a variety of features that combined makes it a Dialect Corpus. This is a tool that combines a very advanced tool for language researchers. number of useful features that together makes These features include: Linguistic contents (dia- it a unique and very advanced resource for re- lects from five closely related languages), anno- searchers of many fields of language search. tation (tagging and two types of transcription), The corpus is web-based and features full search interface (advanced possibilities for com- audio-visual representation linked to tran- bining a large array of search criteria and results scripts. presentation in an intuitive and simple interface), many search variables (linguistics-based, infor- 1 Credits mant-based, time-based), multimedia display (linking of sound and video to transcriptions), The Nordic Dialect Corpus is the result of close display of informant details (number of words collaboration between the partners in the re- and other information on informants), advanced search networks Scandinavian Dialect Syntax results handling (concordances, collocations, and Nordic Centre of Excellence in Microcom- counts and statistics shown in a variety of parative Syntax.
    [Show full text]