A Standardized Project Gutenberg Corpus for Statistical Analysis of Natural Language and Quantitative Linguistics
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Construction of English-Bodo Parallel Text
International Journal on Natural Language Computing (IJNLC) Vol.7, No.5, October 2018 CONSTRUCTION OF ENGLISH -BODO PARALLEL TEXT CORPUS FOR STATISTICAL MACHINE TRANSLATION Saiful Islam 1, Abhijit Paul 2, Bipul Syam Purkayastha 3 and Ismail Hussain 4 1,2,3 Department of Computer Science, Assam University, Silchar, India 4Department of Bodo, Gauhati University, Guwahati, India ABSTRACT Corpus is a large collection of homogeneous and authentic written texts (or speech) of a particular natural language which exists in machine readable form. The scope of the corpus is endless in Computational Linguistics and Natural Language Processing (NLP). Parallel corpus is a very useful resource for most of the applications of NLP, especially for Statistical Machine Translation (SMT). The SMT is the most popular approach of Machine Translation (MT) nowadays and it can produce high quality translation result based on huge amount of aligned parallel text corpora in both the source and target languages. Although Bodo is a recognized natural language of India and co-official languages of Assam, still the machine readable information of Bodo language is very low. Therefore, to expand the computerized information of the language, English to Bodo SMT system has been developed. But this paper mainly focuses on building English-Bodo parallel text corpora to implement the English to Bodo SMT system using Phrase-Based SMT approach. We have designed an E-BPTC (English-Bodo Parallel Text Corpus) creator tool and have been constructed General and Newspaper domains English-Bodo parallel text corpora. Finally, the quality of the constructed parallel text corpora has been tested using two evaluation techniques in the SMT system. -
Using Digital Libraries: Search Strategies for Family Historians
Using Digital Libraries: Search Strategies for Family Historians Elizabeth M. O’Neal PO Box 1259, Lompoc, CA 93436 [email protected] https://www.swangenealogy.net https://mydescendantsancestors.com Looking for books about your family history? You may be able to find them without leaving the comfort of your home! Millions of books have already been digitized and are free to use, as well as download to your personal library. Learn where to find the best digital book collections, how to strategically search them, and how to save your finds to your computer or cloud storage for later reference. Digital Libraries – Family History FamilySearch Digital Library - https://www.familysearch.org/library/books The newly-updated FamilySearch Digital Library contains more than 440,000 digitized genealogy and family history books and publications from the archives of family history libraries such as the Allen County Public Library, the Family History Library in Salt Lake City, and others. Included in the collection are family histories, county and local histories, genealogy magazines and how-to books, gazetteers, medieval histories and pedigrees. While some books are only viewable in a Family History Center, many can be viewed from – as well as downloaded to – your home computer. To access the FamilySearch Digital Library, visit FamilySearch.org, and click “Search” in the top menu. In the drop-down menu, select “Books.” On the home page of the digital library, you will see a simple search bar. Here, you can type in a surname, historical events, groups of people, or names of places. A search will cover every word of text. -
Corpora: Google Ngram Viewer and the Corpus of Historical American English
EuroAmerican Journal of Applied Linguistics and Languages E JournALL Volume 1, Issue 1, November 2014, pages 48 68 ISSN 2376 905X DOI - - www.e journall.org- http://dx.doi.org/10.21283/2376905X.1.4 - Exploring mega-corpora: Google Ngram Viewer and the Corpus of Historical American English ERIC FRIGINALa1, MARSHA WALKERb, JANET BETH RANDALLc aDepartment of Applied Linguistics and ESL, Georgia State University bLanguage Institute, Georgia Institute of Technology cAmerican Language Institute, New York University, Tokyo Received 10 December 2013; received in revised form 17 May 2014; accepted 8 August 2014 ABSTRACT EN The creation of internet-based mega-corpora such as the Corpus of Contemporary American English (COCA), the Corpus of Historical American English (COHA) (Davies, 2011a) and the Google Ngram Viewer (Cohen, 2010) signals a new phase in corpus-based research that provides both novice and expert researchers immediate access to a variety of online texts and time-coded data. This paper explores the applications of these corpora in the analysis of academic word lists, in particular, Coxhead’s (2000) Academic Word List (AWL). Coxhead (2011) has called for further research on the AWL with larger corpora, noting that learners’ use of academic vocabulary needs to address for the AWL to be useful in various contexts. Results show that words on the AWL are declining in overall frequency from 1990 to the present. Implications about the AWL and future directions in corpus-based research utilizing mega-corpora are discussed. Keywords: GOOGLE N-GRAM VIEWER, CORPUS OF HISTORICAL AMERICAN ENGLISH, MEGA-CORPORA, TREND STUDIES. ES La creación de megacorpus basados en Internet, tales como el Corpus of Contemporary American English (COCA), el Corpus of Historical American English (COHA) (Davies, 2011a) y el Visor de Ngramas de Google (Cohen, 2010), anuncian una nueva fase en la investigación basada en corpus, pues proporcionan, tanto a investigadores noveles como a expertos, un acceso inmediato a una gran diversidad de textos online y datos codificados con time-code. -
Usage of IT Terminology in Corpus Linguistics Mavlonova Mavluda Davurovna
International Journal of Innovative Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075, Volume-8, Issue-7S, May 2019 Usage of IT Terminology in Corpus Linguistics Mavlonova Mavluda Davurovna At this period corpora were used in semantics and related Abstract: Corpus linguistic will be the one of latest and fast field. At this period corpus linguistics was not commonly method for teaching and learning of different languages. used, no computers were used. Corpus linguistic history Proposed article can provide the corpus linguistic introduction were opposed by Noam Chomsky. Now a day’s modern and give the general overview about the linguistic team of corpus. corpus linguistic was formed from English work context and Article described about the corpus, novelties and corpus are associated in the following field. Proposed paper can focus on the now it is used in many other languages. Alongside was using the corpus linguistic in IT field. As from research it is corpus linguistic history and this is considered as obvious that corpora will be used as internet and primary data in methodology [Abdumanapovna, 2018]. A landmark is the field of IT. The method of text corpus is the digestive modern corpus linguistic which was publish by Henry approach which can drive the different set of rules to govern the Kucera. natural language from the text in the particular language, it can Corpus linguistic introduce new methods and techniques explore about languages that how it can inter-related with other. for more complete description of modern languages and Hence the corpus linguistic will be automatically drive from the these also provide opportunities to obtain new result with text sources. -
Enriching an Explanatory Dictionary with Framenet and Propbank Corpus Examples
Proceedings of eLex 2019 Enriching an Explanatory Dictionary with FrameNet and PropBank Corpus Examples Pēteris Paikens 1, Normunds Grūzītis 2, Laura Rituma 2, Gunta Nešpore 2, Viktors Lipskis 2, Lauma Pretkalniņa2, Andrejs Spektors 2 1 Latvian Information Agency LETA, Marijas street 2, Riga, Latvia 2 Institute of Mathematics and Computer Science, University of Latvia, Raina blvd. 29, Riga, Latvia E-mail: [email protected], [email protected] Abstract This paper describes ongoing work to extend an online dictionary of Latvian – Tezaurs.lv – with representative semantically annotated corpus examples according to the FrameNet and PropBank methodologies and word sense inventories. Tezaurs.lv is one of the largest open lexical resources for Latvian, combining information from more than 300 legacy dictionaries and other sources. The corpus examples are extracted from Latvian FrameNet and PropBank corpora, which are manually annotated parallel subsets of a balanced text corpus of contemporary Latvian. The proposed approach augments traditional lexicographic information with modern cross-lingually interpretable information and enables analysis of word senses from the perspective of frame semantics, which is substantially different from (complementary to) the traditional approach applied in Latvian lexicography. In cases where FrameNet and PropBank corpus evidence aligns well with the word sense split in legacy dictionaries, the frame-semantically annotated corpus examples supplement the word sense information with clarifying usage examples and commonly used semantic valence patterns. However, the annotated corpus examples often provide evidence of a different sense split, which is often more coarse-grained and, thus, may help dictionary users to cluster and comprehend a fine-grained sense split suggested by the legacy sources. -
20 Toward a Sustainable Handling of Interlinear-Glossed Text in Language Documentation
Toward a Sustainable Handling of Interlinear-Glossed Text in Language Documentation JOHANN-MATTIS LIST, Max Planck Institute for the Science of Human History, Germany 20 NATHANIEL A. SIMS, The University of California, Santa Barbara, America ROBERT FORKEL, Max Planck Institute for the Science of Human History, Germany While the amount of digitally available data on the worlds’ languages is steadily increasing, with more and more languages being documented, only a small proportion of the language resources produced are sustain- able. Data reuse is often difficult due to idiosyncratic formats and a negligence of standards that couldhelp to increase the comparability of linguistic data. The sustainability problem is nicely reflected in the current practice of handling interlinear-glossed text, one of the crucial resources produced in language documen- tation. Although large collections of glossed texts have been produced so far, the current practice of data handling makes data reuse difficult. In order to address this problem, we propose a first framework forthe computer-assisted, sustainable handling of interlinear-glossed text resources. Building on recent standardiza- tion proposals for word lists and structural datasets, combined with state-of-the-art methods for automated sequence comparison in historical linguistics, we show how our workflow can be used to lift a collection of interlinear-glossed Qiang texts (an endangered language spoken in Sichuan, China), and how the lifted data can assist linguists in their research. CCS Concepts: • Applied computing → Language translation; Additional Key Words and Phrases: Sino-tibetan, interlinear-glossed text, computer-assisted language com- parison, standardization, qiang ACM Reference format: Johann-Mattis List, Nathaniel A. -
Linguistic Annotation of the Digital Papyrological Corpus: Sematia
Marja Vierros Linguistic Annotation of the Digital Papyrological Corpus: Sematia 1 Introduction: Why to annotate papyri linguistically? Linguists who study historical languages usually find the methods of corpus linguis- tics exceptionally helpful. When the intuitions of native speakers are lacking, as is the case for historical languages, the corpora provide researchers with materials that replaces the intuitions on which the researchers of modern languages can rely. Using large corpora and computers to count and retrieve information also provides empiri- cal back-up from actual language usage. In the case of ancient Greek, the corpus of literary texts (e.g. Thesaurus Linguae Graecae or the Greek and Roman Collection in the Perseus Digital Library) gives information on the Greek language as it was used in lyric poetry, epic, drama, and prose writing; all these literary genres had some artistic aims and therefore do not always describe language as it was used in normal commu- nication. Ancient written texts rarely reflect the everyday language use, let alone speech. However, the corpus of documentary papyri gets close. The writers of the pa- pyri vary between professionally trained scribes and some individuals who had only rudimentary writing skills. The text types also vary from official decrees and orders to small notes and receipts. What they have in common, though, is that they have been written for a specific, current need instead of trying to impress a specific audience. Documentary papyri represent everyday texts, utilitarian prose,1 and in that respect, they provide us a very valuable source of language actually used by common people in everyday circumstances. -
The Project Gutenberg Ebook of Things to Make, by Archibald Williams This Ebook Is for the Use of Anyone Anywhere at No Cost
The Project Gutenberg EBook of Things To Make, by Archibald Williams This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.net Title: Things To Make Author: Archibald Williams Release Date: January 11, 2005 [EBook #14664] Language: English Character set encoding: ASCII *** START OF THIS PROJECT GUTENBERG EBOOK THINGS TO MAKE *** Produced by Don Kostuch Transcriber's Note: If the pdf version of the book is viewed using facing pages with even numbered pages on the left, you will see a close approximation of the original book. Notations of the form "(1,650) 2" appear at the bottom of some pages; they are probably printer's references for assembling to book. The text only version is of limited use because of the many figures used. I recommend the pdf or rtf versions. Some of the projects should be approached with care since they involve corrosive or explosive chemicals, electricity and steam boilers. Do not use lead solder, particularly on cooking utensils. Whether you simply want to travel back into the mind of a young boy at the beginning of the twentieth century, or want to try your hand at some interesting projects in carpentry, machinery, kites and many other areas, have fun. The following four pages have definitions of unusual (to me) terms used frequently in the text. Terms Batten - Narrow strip of wood. -
Text and Corpus Analysis: Computer-Assisted Studies of Language and Culture
© Copyright Michael Stubbs 1996. This is chapter 1 of Text and Corpus Analysis which was published by Blackwell in 1996. TEXT AND CORPUS ANALYSIS: COMPUTER-ASSISTED STUDIES OF LANGUAGE AND CULTURE MICHAEL STUBBS CHAPTER 1. TEXTS AND TEXT TYPES Attested language text duly recorded is in the focus of attention for the linguist. (Firth 1957b: 29.) The main question discussed in this book is as follows: How can an analysis of the patterns of words and grammar in a text contribute to an understanding of the meaning of the text? I discuss traditions of text analysis in (mainly) British linguistics, and computer-assisted methods of text and corpus analysis. And I provide analyses of several shorter and longer texts, and also of patterns of language across millions of words of text corpora. The main emphasis is on the analysis of attested, naturally occurring textual data. An exclusive concentration on the text alone is, however, not an adequate basis for text interpretation, and this introductory chapter discusses the appropriate balance for text analysis, between the text itself and its production and reception. Such analyses of attested texts are important for both practical and theoretical reasons, and the main reasons can be easily and briefly stated. First, there are many areas of the everyday world in which the systematic and critical interpretation of texts is an important social skill. One striking example occurs in courtrooms, where very important decisions, by juries as well as lawyers, can depend on such interpretations. Second, much of theoretical linguistics over the past fifty years has been based on the study of isolated sentences. -
Downloading Ebooks from Project Gutenberg to Your Computer, Kindle, Nook, Or Ipad
Downloading eBooks from Project Gutenberg to Your Computer, Kindle, Nook, or iPad Downloading to Your Computer Downloading to Your iPad Downloading to Your Kindle Downloading to Your Nook Downloading to Your Computer Click the Download this ebook to your ereader, tablet PC, or computer link. This will take you to the download page for the current title. Select “EPUB” format. A window will pop up with – “Save file” – click “OK,” choose an appropriate place to save, and click “Save.” This ebook format must be viewed using Adobe Digital Editions. Click here to download. Enjoy! Downloading to Your iPad Click the Download this ebook to your ereader, tablet PC, or computer link. This will take you to the download page for the current title. Click on “EPUB” format. Click “Open in iBooks.” The ebook will automatically download and open in iBooks. Enjoy! Downloading to Your Kindle Using USB Cord Using Email Click the Download this ebook to your ereader, Click the Download this ebook to your ereader, tablet PC, or computer link. This will take you to the tablet PC, or computer link. This will take you to the download page for the current title. download page for the current title. Use the “Kindle” format. Attach the USB cord to your computer and Kindle. A window will pop up with – “Save file” – click “OK,” Use the “Kindle” format. choose an appropriate place to save, and click A window will pop up with – “Save file” – click “OK,” “Save.” This will save your selection to your choose an appropriate place to save, and click computer, and it can then be emailed to your “Save.” This will save your selection to your Kindle. -
Arxiv:2007.16007V2 [Cs.CL] 17 Apr 2021 Beddings in Deep Networks Like the Transformer Can Improve Performance
Exploring Swedish & English fastText Embeddings for NER with the Transformer Tosin P. Adewumi, Foteini Liwicki & Marcus Liwicki [email protected] EISLAB SRT Department Lule˚aUniversity of Technology Sweden Abstract In this paper, our main contributions are that embeddings from relatively smaller cor- pora can outperform ones from larger corpora and we make the new Swedish analogy test set publicly available. To achieve a good network performance in natural language pro- cessing (NLP) downstream tasks, several factors play important roles: dataset size, the right hyper-parameters, and well-trained embeddings. We show that, with the right set of hyper-parameters, good network performance can be reached even on smaller datasets. We evaluate the embeddings at both the intrinsic and extrinsic levels. The embeddings are deployed with the Transformer in named entity recognition (NER) task and significance tests conducted. This is done for both Swedish and English. We obtain better performance in both languages on the downstream task with smaller training data, compared to re- cently released, Common Crawl versions; and character n-grams appear useful for Swedish, a morphologically rich language. Keywords: Embeddings, Transformer, Analogy, Dataset, NER, Swedish 1. Introduction The embedding layer of neural networks may be initialized randomly or replaced with pre- trained vectors, which act as lookup tables. One of such pre-trained vector tools include fastText, introduced by Joulin et al. Joulin et al. (2016). The main advantages of fastText are speed and competitive performance to state-of-the-art (SotA). Using pre-trained em- arXiv:2007.16007v2 [cs.CL] 17 Apr 2021 beddings in deep networks like the Transformer can improve performance. -
A Collection of Swedish Diachronic Word Embedding Models Trained on Historical
A Collection of Swedish Diachronic Word Embedding Models Trained on Historical Newspaper Data DATA PAPER SIMON HENGCHEN NINA TAHMASEBI *Author affiliations can be found in the back matter of this article ABSTRACT CORRESPONDING AUTHOR: Simon Hengchen This paper describes the creation of several word embedding models based on a large Språkbanken Text, Department collection of diachronic Swedish newspaper material available through Språkbanken of Swedish, University of Text, the Swedish language bank. This data was produced in the context of Språkbanken Gothenburg, SE Text’s continued mission to collaborate with humanities and natural language [email protected] processing (NLP) researchers and to provide freely available language resources, for the development of state-of-the-art NLP methods and tools. KEYWORDS: word embeddings; semantic change; newspapers; diachronic word embeddings TO CITE THIS ARTICLE: Hengchen, S., & Tahmasebi, N. (2021). A Collection of Swedish Diachronic Word Embedding Models Trained on Historical Newspaper Data. Journal of Open Humanities Data, 7: 2, pp. 1–7. DOI: https://doi. org/10.5334/johd.22 1 OVERVIEW Hengchen and Tahmasebi 2 Journal of Open We release diachronic word2vec (Mikolov et al., 2013) and fastText (Bojanowski et al., 2017) models Humanities Data DOI: 10.5334/johd.22 in their skip-gram with negative sampling (SGNS) architecture. The models are trained on 20-year time bins, with two temporal alignment strategies: independently-trained models for post-hoc alignment (as introduced by Kulkarni et al. 2015), and incremental training (Kim et al., 2014). In the incremental scenario, a model for t1 is trained and saved, then updated with the data from t2.