DOCSLIB.ORG

PANACEA Project Grant Agreement No.: 248064

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
PANACEA Project Grant Agreement No.: 248064 SEVENTH FRAMEWORK PROGRAMME THEME 3 Information and communication Technologies PANACEA Project Grant Agreement no.: 248064 Platform for Automatic, Normalized Annotation and Cost-Effective Acquisition of Language Resources for Human Language Technologies D5.1 Parallel technology tools and resources Dissemination Level: Public Delivery Date: July 16th 2010 Status – Version: Final v1.0 Author(s) and Affiliation: Pavel Pecina (DCU), Antonio Toral (DCU), Gregor Thurmair (LinguaTec), Andy Way (DCU) Parallel technology tools and resources Table of contents Table of contents ........................................................................................................................... 1 1 Introduction ........................................................................................................................... 4 2 Terminology .......................................................................................................................... 4 2.1 Definitions ..................................................................................................................... 4 2.2 Acronyms ....................................................................................................................... 5 2.3 Related documents ......................................................................................................... 6 3 Task description .................................................................................................................... 6 3.1 Alignment ...................................................................................................................... 6 3.1.1 Sentential alignment ................................................................................................. 6 3.1.2 Sub-sentential alignment .......................................................................................... 6 3.2 Bilingual dictionary induction ....................................................................................... 7 3.3 Transfer grammar induction .......................................................................................... 8 3.3.1 Problem description .................................................................................................. 8 3.3.2 Scope and limitations ................................................................................................ 8 4 Current state of the art ........................................................................................................... 9 4.1 Alignment .................................................................................................................... 10 4.1.1 Sentential alignment ............................................................................................... 10 4.1.2 Sub-sentential alignment ........................................................................................ 11 4.2 Bilingual dictionary induction ..................................................................................... 14 4.3 Transfer grammar induction ........................................................................................ 15 4.3.1 Learning structural transfer rules ............................................................................ 15 4.3.2 Learning simple lexical transfer: Extracting transfer entries from corpora ............ 15 4.3.3 Selecting the best transfer for a given context ........................................................ 16 5 Existing tools ....................................................................................................................... 26 5.1 Alignment .................................................................................................................... 26 5.1.1 Sentential alignment ............................................................................................... 26 5.1.2 Sub-sentential alignment ........................................................................................ 26 5.2 Bilingual dictionary induction ..................................................................................... 27 5.3 Transfer grammar induction ........................................................................................ 27 5.3.1 Apertium ................................................................................................................. 28 5.3.2 Required tools ......................................................................................................... 28 1 Parallel technology tools and resources 6 Resource description ........................................................................................................... 29 6.1 Alignment .................................................................................................................... 29 6.1.1 Types of parallel corpora ........................................................................................ 29 6.1.2 Domains and languages .......................................................................................... 29 6.2 Bilingual dictionary induction ..................................................................................... 30 6.3 Transfer grammar induction ........................................................................................ 30 7 Solution path and work plan ................................................................................................ 33 7.1 Alignment .................................................................................................................... 33 7.1.1 Strategy ................................................................................................................... 33 7.1.2 Setup ....................................................................................................................... 34 7.1.3 Preprocessing .......................................................................................................... 34 7.1.4 Parallel sentence alignment and extraction from comparable corpora ................... 34 7.1.5 Sub-sentential alignment ........................................................................................ 34 7.1.6 Additional experiments ........................................................................................... 35 7.1.7 Testing .................................................................................................................... 35 7.2 Bilingual dictionary induction ..................................................................................... 35 7.2.1 Strategy ................................................................................................................... 35 7.2.2 Setup ....................................................................................................................... 35 7.2.3 Methodology ........................................................................................................... 36 7.2.4 Testing .................................................................................................................... 36 7.3 Transfer grammar induction ........................................................................................ 36 7.3.1 Strategy ................................................................................................................... 37 7.3.2 Setup ....................................................................................................................... 37 7.3.3 Preprocessing .......................................................................................................... 38 7.3.4 Topic test processing .............................................................................................. 39 7.3.5 Grammatical testing ................................................................................................ 39 7.3.6 Conceptual context determination .......................................................................... 40 7.3.7 Order of tests, entry packages ................................................................................. 40 7.3.8 Testing .................................................................................................................... 41 8 Bibliography ........................................................................................................................ 42 A. Tool Documentation Forms.................................................................................................... 53 A. 1 Hunalign .......................................................................................................................... 53 A.2 Geometric Mapping and Alignment ................................................................................. 55 2 Parallel technology tools and resources A.3 Bilingual Sentence Aligner .............................................................................................. 57 A.4 Giza++ .............................................................................................................................. 59 A.5 Berkeley Aligner .............................................................................................................. 62 A.6 OpenMaTrEx .................................................................................................................... 64 A.7 Subtree Aligner ................................................................................................................ 66 3 Parallel technology tools and resources 1 Introduction The main
Load more

Recommended publications
  • Standardisation Action Plan for Clarin
    Standardisation Action Plan for Clarin State: Proposal to CLARIN community Nuria Bel, Jonas Beskow, Lou Boves, Gerhard Budin, Nicoletta Calzolari, Khalid Choukri, Erhard Hinrichs, Steven Krauwer, Lothar Lemnitzer, Stelios Piperidis, Adam Przepiorkowski, Laurent Romary, Florian Schiel, Helmut Schmidt, Hans Uszkoreit, Peter Wittenburg August 2009 Summary This document describes a proposal for a Standardisation Action Plan (SAP) for the Clarin initiative in close synchronization with other relevant initiatives such as Flarenet, ELRA, ISO and TEI. While Flarenet is oriented towards a broader scope since it is also addressing standards that are typically used in industry, CLARIN wants to be more focussed in its statements to the research domain. Due to the overlap it is agreed that the Flarenet and CLARIN documents on standards need to be closely synchronized. This note covers standards that are generic (XML, UNICODE) as well as standards that are domain specific where naturally the LRT community has much more influence. This Standardization Action Plan wants to give an orientation for all practical work in CLARIN to achieve a harmonized domain of language resources and technology stepwise and therefore its core message is to overcome fragmentation. To meet these goals it wants to keep its message as simple as possible. A web-site will be established that will contain more information about examples, guidelines, explanations, tools, converters and training events such as summer schools. The organization of the document is as follows: • Chapter 1: Introduction to the topic. • Chapter 2: Recommended standards that CLARIN should endorse page 4 • Chapter 3: Standards that are emerging and relevant for CLARIN page 8 • Chapter 4: General guidelines that need to be followed page 12 • Chapter 5: Reference to community practices page 14 • Chapter 6: References This document tries to be short and will give comments, recommendations and discuss open issues for each of the standards.
    [Show full text]
  • Proceedings of the Workshop on Multilingual Language Resources and Interoperability, Pages 1–8, Sydney, July 2006
    COLING •ACL 2006 MLRI’06 Multilingual Language Resources and Interoperability Proceedings of the Workshop Chairs: Andreas Witt, Gilles Sérasset, Susan Armstrong, Jim Breen, Ulrich Heid and Felix Sasaki 23 July 2006 Sydney, Australia Production and Manufacturing by BPA Digital 11 Evans St Burwood VIC 3125 AUSTRALIA c 2006 The Association for Computational Linguistics Order copies of this and other ACL proceedings from: Association for Computational Linguistics (ACL) 209 N. Eighth Street Stroudsburg, PA 18360 USA Tel: +1-570-476-8006 Fax: +1-570-476-0860 [email protected] ISBN 1-932432-82-5 ii Table of Contents Preface .....................................................................................v Organizers . vii Workshop Program . ix Lexical Markup Framework (LMF) for NLP Multilingual Resources Gil Francopoulo, Nuria Bel, Monte George, Nicoletta Calzolari, Monica Monachini, Mandy Pet and Claudia Soria . 1 The Role of Lexical Resources in CJK Natural Language Processing Jack Halpern . 9 Towards Agent-based Cross-Lingual Interoperability of Distributed Lexical Resources Claudia Soria, Maurizio Tesconi, Andrea Marchetti, Francesca Bertagna, Monica Monachini, Chu-Ren Huang and Nicoletta Calzolari. .17 The LexALP Information System: Term Bank and Corpus for Multilingual Legal Terminology Consolidated Verena Lyding, Elena Chiocchetti, Gilles Sérasset and Francis Brunet-Manquat . 25 The Development of a Multilingual Collocation Dictionary Sylviane Cardey, Rosita Chan and Peter Greenfield. .32 Multilingual Collocation Extraction: Issues and Solutions Violeta Seretan and Eric Wehrli . 40 Structural Properties of Lexical Systems: Monolingual and Multilingual Perspectives Alain Polguère . 50 A Fast and Accurate Method for Detecting English-Japanese Parallel Texts Ken’ichi Fukushima, Kenjiro Taura and Takashi Chikayama . 60 Evaluation of the Bible as a Resource for Cross-Language Information Retrieval Peter A.
    [Show full text]
  • Standardization Work Isotiger Transcription of Spoken Language CQLF: Corpus Query Lingua Franca
    Recent Initiatives towards New Standards for Language Resources Gottfried Herzog1, Ulrich Heid2, Thorsten Trippel3, Piotr Ba´nski4, Laurent Romary5, Thomas Schmidt4, Andreas Witt4, Kerstin Eckart6 1DIN Deutsches Institut fur¨ Normung e. V., Berlin, 2Universit¨at Hildesheim, 3Universit¨at Tubingen,¨ 4Institut fur¨ Deutsche Sprache, Mannheim, 5Inria, 6Universit¨at Stuttgart E-mail: [email protected] Standardization Work Transcription of spoken language • Who? A representation format to compare, interchange and combine { Experts from industry, academia and administrations orthography-based transcriptions of spoken language. { Experts are nominated { based on expertise and interest The standard is developed in cooperation with TEI proposals in the field. [Schmidt 2011] { ISO committee TC 37/SC 4 and national mirror committees, Based on e.g. for Germany: Normenausschuss NA-105-00-06 AA • State of the art tools and formats for creating, editing, Sprachressourcen publishing and querying transcribed data DIN { Deutsches Institut f¨ur Normung e.V. • Widely used transcription systems • How? { Stepwise procedures: Encoded components Example on handout Proposals { working drafts { (draft) international standards • Metadata: based on TEI header { Consensus-based: drafting { commenting { ballot • Macrostructure: timeline, single and grouped utterances, { National standards organizations provide infrastructure elements outside utterances (e.g. <pause> and <incident>) • Microstructure: Excerpt of the list of standards and standard proposals by ISO
    [Show full text]
  • Mise En Page 1
    Catalogue général Janvier 2019 27-37 St George’s Road – London SW19 4EU — United Kingdom www.iste-editions.fr – www.openscience.fr – www.iste.co.uk – www.istegroup.com Table des matières Table of contents Organisation éditoriale, SCIENCES .................................................................................. 4 Biologie, médecine et santé • Biology, Medecine and Health ........................................ 7 Bioingénierie médicale • Biomedical Engineering ................................................................ 8 Biologie • Biology ................................................................................................................. 11 Ingénierie de la santé et société • Health Engineering and Society ..................................... 12 Chimie • Chemistry ............................................................................................................ 17 Agriculture, science des aliments et nutrition ....................................................................... 18 Agriculture, Food Science and Nutrition Chimie • Chemistry ............................................................................................................... 20 Écologie et environnement • Ecology and Environment ................................................ 23 Écologie • Ecological Sciences ............................................................................................ 24 Environnement • Environmental Sciences ........................................................................... 28
    [Show full text]
  • A Method for Reusing and Re-Engineering Non-Ontological Resources for Building Ontologies
    Departamento de Inteligencia Artificial Facultad de Informatica´ PhD Thesis A Method for Reusing and Re-engineering Non-ontological Resources for Building Ontologies Author : Msc. Boris Marcelo Villazon´ Terrazas Advisor : Prof. Dr. Asuncion´ Gomez´ Perez´ 2011 ii Tribunal nombrado por el Sr. Rector Magfco. de la Universidad Politecnica´ de Madrid, el d´ıa...............de.............................de 20.... Presidente : Vocal : Vocal : Vocal : Secretario : Suplente : Suplente : Realizado el acto de defensa y lectura de la Tesis el d´ıa..........de......................de 20...... en la E.T.S.I. /Facultad...................................................... Calificacion´ .................................................................................. EL PRESIDENTE LOS VOCALES EL SECRETARIO iii iv Abstract Current well-known methodologies for building ontologies do not consider the reuse and possible subsequent re-engineering of existing knowledge resources. The ontologization of non-ontological resources has led to the design of several specific methods, techniques and tools. These are mainly specific to a particular resource type, or to a particular resource implementation. Thus, everytime ontol- ogy engineers are confronted with the task of re-engineering a new resource into an ontology, they develop ad-hoc solutions for transforming such resource into a single ontology. Within the context of the NeOn project, we propose a novel methodology for building ontology networks: the NeOn Methodology, a methodology based on sce- narios. One
    [Show full text]
  • LMF for a Selection of African Languages Chantal Enguehard, Mathieu Mangeot
    LMF for a selection of African Languages Chantal Enguehard, Mathieu Mangeot To cite this version: Chantal Enguehard, Mathieu Mangeot. LMF for a selection of African Languages. Francopoulo, Gil. LMF: Lexical Markup Framework, theory and practice, Hermès science, pp.8, 2013. hal-00959228 HAL Id: hal-00959228 https://hal.archives-ouvertes.fr/hal-00959228 Submitted on 1 Apr 2014 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Chapter 71 LMF for a selection of African Languages Chantal Enguehard LINA, 2 rue de la Houssiniere, BP 92208, 44322 Nantes Cedex 03, France Mathieu Mangeot GETALP-LIG, 41 rue des Mathématiques, BP 53 F-38041 GRENOBLE CEDEX 9 7.1. Introduction Electronic resources are scarce regarding less-resourced languages, so it is wise to take published dictionaries and convert them into a standard format usable by automated tools for natural language processing. We introduce the notion of less- resourced languages and then discuss the methodology of conversion that we have defned and implemented. The fourth part presents examples of conversion from the initial published format to the LMF format. The last part describes some diffculties encountered when representing certain information into LMF format.
    [Show full text]
  • An Experiment in Lexical Information Extraction
    LEXIE - an Experiment in Lexical Information Extraction John J. Camilleri Michael Rosner Dept. Intelligent Computer Systems Dept. Intelligent Computer Systems University of Malta University of Malta MSD2080 Msida, Malta MSD2080 Msida, Malta john @johnjcamilleri.com mike.rosner @um.edu.mt Abstract to the general approach adopted there. However, the This document investigates the possibility of ex- two main differences we see between their approach tracting lexical information automatically from and ours are that (a) Boguraev’s work was oriented the pages of a printed dictionary of Maltese. An rather heavily towards the GPSG grammatical frame- experiment was carried out on a small sample work and (b) they were able to access the original of dictionary entries using hand-crafted rules to source tapes of the dictionary. The input they used parse the entries. Although the results obtained was therefore faithful to the orginal. were quite promising, a major problem turned out to errors introduced by OCR and the incon- In our case we had to rely upon OCR input as de- sistent style adopted for writing dictionary en- scribed further in section 4. The inherent errors caused tries. certain problems. One of the aims of this work is to establish a viable method for extracting lexical entries for lesser-studied languages lacking the usual battery of language re- Keywords sources in machine readable form. Most languages of the world fall into this category. The availability of a lexicon extraction, lexical information, lexicon, semitic paper dictionary, is, however, fairly widespread, even for exotic languages so that the methods being pro- posed could provide a robust alternative to dictionary 1 Introduction extraction for a relatively large number of languages.
    [Show full text]
  • Toward an Architecture for the Global Wordnet Initiative
    1 Toward an Architecture for the Global Wordnet Initiative Andrea Marchetti 1, Maurizio Tesconi 1, Francesco Ronzano 1, Marco Rosella 1 Francesca Bertagna 2, Monica Monachini 2, Claudia Soria 2, Nicoletta Calzolari 2, Chu-Ren Huang 3, Shu-Kai Hsieh 3 1Istituto di Informatica e Telematica-CNR, Via Moruzzi 1, Pisa, Italy {andrea.marchetti, maurizio.tesconi, francesco.ronzano, marco.rosella}@iit.cnr.it 2Istituto di Linguistica Computazionale-CNR, Via Moruzzi 1, Pisa, Italy {francesca.bertagna, monica.monachini, claudia.soria, nicoletta.calzolari}@ilc.cnr.it, 3Academia Sinica, Nankang, Taipei, Taiwan, [email protected], [email protected] handle and define the semantics of data. As a consequence, we Abstract — Enhancing the development of multilingual lexicons have assisted to the first attempts of integration of lexical is of foremost importance for intercultural collaboration to take resource in the Semantic Web infrastructure and content place, as multilingual lexicons are the cornerstone of several organization model. Nevertheless, large-scale multilingual multilingual applications. However, the development and maintenance of large-scale, robust multilingual dictionaries is a lexical resources are not as widely available and are very tantalizing task. Moreover, Semantic Web’s growing interest costly to construct. towards the availability of high-quality lexical resources and their The previous trend in lexical resource was oriented to multilingual interoperability, is focusing more and more attention maximization of effort by building large-scale, general- on this topic. In this paper we present a tool, based on a web purpose lexicons. However, these lexical resources are not service architecture, enabling semi-automatic generation of always satisfactory despite the tremendous amount of work bilingual lexicons through linking of distributed monolingual lexical resources.
    [Show full text]
  • Multilingual Language Resources and Interoperability
    Published in: Language Resources & Evaluation vol. 43 (2009) no. 1, pp. 1-14. Multilingual language resources and interoperability Andreas Witt Æ Ulrich Heid Æ Felix Sasaki Æ Gilles Se´rasset Abstract This article introduces the topic of ‘‘Multilingual language resources and interoperability’’. We start with a taxonomy and parameters for classifying language resources. Later we provide examples and issues of interoperatability, and resource architectures to solve such issues. Finally we discuss aspects of linguistic formalisms and interoperability. Keywords Language Á Resources Á Interoperability 1 Introduction This special issue of Language Resources and Evaluation, entitled ‘‘Multilingual language resources and interoperability’’, is composed of extended versions of selected papers from the COLING/ACL Workshop on Multilingual language resources and interoperability, held in 2006, in Sydney (cf. Witt et al. 2006 ). This introduction does not attempt to provide a complete overview of this vast topic, but A. Witt (&) IDS, Mannheim, Germany e-mail: [email protected] U. Heid University of Stuttgart, Stuttgart, Germany e-mail: [email protected] F. Sasaki Keio University, Fujisawa, Japan e-mail: [email protected] G. Se´rasset Universite´ Joseph Fourier, Grenoble Cedex 9, France e-mail: [email protected] 2 rather sketches the background against which the articles assembled in this issue are to be read. In particular, we examine the notions of (multilingual) language resources (Sect. 2) and interoperability of resources (Sect. 3), and assess resource architectures (Sect. 4) and linguistic representation formalisms (Sect. 5) with respect to their potential to support resource interoperability. This background provides a framework in which each paper in this issue of the journal is then situated.
    [Show full text]
  • Interoperability and Standards
    Common Language Resources and Technology Infrastructure Interoperability and Standards 2010-12-23 Version: 1 Editors: Erhard Hinrichs, Iris Vogel www.clarin.eu Common Language Resources and Technology Infrastructure The ultimate objective of CLARIN is to create a European federation of existing digital repositories that include language-based data, to provide uniform access to the data, wherever it is, and to provide existing language and speech technology tools as web services to retrieve, manipulate, enhance, explore and exploit the data. The primary target audience is researchers in the humanities and social sciences and the aim is to cover all languages relevant for the user community. The objective of the current CLARIN Preparatory Phase Project (2008-2010) is to lay the technical, linguistic and organizational foundations, to provide and validate specifications for all aspects of the infrastructure (including standards, usage, IPR) and to secure sustainable support from the funding bodies in the (now 23) participating countries for the subsequent construction and exploitation phases beyond 2010. CLARIN-D5C-3 2 Common Language Resources and Technology Infrastructure Interoperability and Standards CLARIN-D5C-3 EC FP7 project no. 212230 Deliverable: D5.C-3 - Deadline: 31.12.2010 Responsible: Erhard Hinrichs CLARIN-D5C-3 3 Common Language Resources and Technology Infrastructure Contributing Partners: UTU, UHEL, IPPBAS, KTH, IMCS, IPIPAN, ILC-CNR, ILSP, MPI Contributing Members: BBAW, U Hamburg, U Stuttgart, Vienna With contributions
    [Show full text]
  • LMF Reloaded
    LMF Reloaded 1,3,4 1,2,3 8 1,6,7 Laurent Romary ,​ Mohamed Khemakhem ,​ Fahad Khan ,​ Jack Bowers ,​ Nicoletta ​ 8 5 ​ 5 ​ 9 ​ Calzolari ,​ Monte George ,​ Mandy Pet ,​ Piotr Bański ​ ​ ​ ​ 1. Inria-ALMAnaCH - Automatic Language Modelling and ANAlysis & Computational Humanities 2. UPD7 - Université Paris Diderot - Paris 7 3. CMB - Centre Marc Bloch 4. BBAW - Berlin-Brandenburg Academy of Sciences and Humanities 5. ANSI- American National Standards Institute 6. EPHE - École Pratique des Hautes Études ​ 7. ÖAW - Austrian Academy of Sciences 8. CNR-ILC - Istituto di Linguistica Computazionale "Antonio Zampolli" 9. IDS - Institut für Deutsche Sprache Abstract The Lexical Markup Framework (LMF) or ISO 24613 [1] is a de jure standard ​ which constitutes a framework for modelling and encoding lexical information both in retrodigitised print dictionaries as well as in NLP lexical databases. An in-depth review is currently underway within the standardisation sub-committee, ISO-TC37/SC4/WG4 with the goal of creating a more modular, flexible and durable follow up to the original LMF standard published by ISO in 2008. In this paper we will showcase some of the major improvements which have so far been implemented in the new version of LMF. Key words: ISO 24613, LMF, Lexical resources ​ 1. Introduction The previous version of LMF, published by ISO in 2008 [1] offered a framework for modelling, publishing and sharing lexical resources with a special focus on requirements arising from the domain of Natural Language Processing (NLP). Due to the potential richness and the multi-layered nature of linguistic descriptions in lexical resources the LMF meta-model ended up taking on a great deal of complexity in its attempt to reflect these various different linguistic facets.
    [Show full text]
  • Language Description and Hypertext: Nunggubuyu As a Case Study
    Language Documentation & Conservation Special Publication No. 4 (October 2012) Electronic Grammaticography ed. by Sebastian Nordhoff pages 63-77 http://nf lrc.hawaii.edu/ldc http://hdl.handle.net/10125/4530 3 http://nf lrc.hawaii.edu/ldc/sp04 Language description and hypertext: Nunggubuyu as a case study Simon Musgrave♠ , Nick Thieberger~ ♠Monash University, ~University of Melbourne Any reasonably complete description of a language is a complex object, typically com- posed of a grammar, a dictionary, and a text collection with internal relationships that can be represented as hyperlinks. The information would be fully searchable, links be- tween text and media could be implemented, and the presentation would be based on a well-defined data structure with advantages for archiving and reusability. We present a small fragment from Heath’s Nunggubuyu text collection with links to parts of the other elements of the description to demonstrate the benefit which this approach can bring. This initial step involves a certain amount of hand-coding but establishes a basis for the necessary data structure which will then be used in a second phase where we develop techniques for the automatic processing of scanned versions of Heath’s work. Grammatical descriptions written with the kinds of structure we are developing, or ca- pable of being converted to that structure (while being ‘born digital’) are likely to be in short supply. Presentations of old materials in new formats will inform new electronic grammars, and help gain the acceptance of the linguistic community for preferred for- mats. 1 Introduction Any reasonably complete description of a language is a complex object.
    [Show full text]