DOCSLIB.ORG

Better Web Corpora for Corpus Linguistics and NLP

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Masaryk University Faculty of Informatics Better Web Corpora For Corpus Linguistics And NLP Doctoral Thesis Vít Suchomel Brno, Spring 2020 Masaryk University Faculty of Informatics Better Web Corpora For Corpus Linguistics And NLP Doctoral Thesis Vít Suchomel Brno, Spring 2020 Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Vít Suchomel Advisor: Pavel Rychlý i Acknowledgements I would like to thank my advisors, prof. Karel Pala and prof. Pavel Rychlý for their problem insight, help with software design and con- stant encouragement. I am also grateful to my colleagues from Natural Language Process- ing Centre at Masaryk University and Lexical Computing, especially Miloš Jakubíček, Pavel Rychlý and Aleš Horák, for their support of my work and invaluable advice. Furthermore, I would like to thank Adam Kilgarriff, who gave me a wonderful opportunity to work for a leading company in the field of lexicography and corpus driven NLP and Jan Pomikálek who helped me to start. I thank to my wife Kateřina who supported me a lot during writing this thesis. Of those who have always accepted me and loved me in spite of my failures, God is the greatest. ii Abstract The internet is used by computational linguists, lexicographers and social scientists as an immensely large source of text data for various NLP tasks and language studies. Web corpora can be built in sizes which would be virtually impossible to achieve using traditional corpus creation methods. This thesis presents a web crawler designed to obtain texts from the internet allowing to build large text corpora for NLP and linguistic applications. An asynchronous communication design (rather than usual synchronous multi-threaded design) was implemented for the crawler to provide an easy to maintain alternative to other web spider software. Cleaning techniques were devised to transform the messy nature of data coming from the uncontrolled environment of the internet. However, it can be observed that usability of recently built web cor- pora is hindered by several factors: The results derived from statistical processing of corpus data are significantly affected by the presence of non-text (web spam, computer generated text and machine trans- lation) in text corpora. It is important to study the issue to be able to avoid non-text at all or at least decrease its size in web corpora. Another observed factor is the case of web pages or their parts written in multiple languages. Multilingual pages should be recognised, lan- guages identified and text parts separated to respective monolingual corpora. This thesis proposes additional cleaning stages in the process of building text corpora which help to deal with these issues. Unlike traditional corpora made from printed media in the past decades, sources of web corpora are not categorised and described well, thus making it difficult to control the content of the corpus. Rich annotation of corpus content is dealt with in the last part of the thesis. An inter-annotator agreement driven English genre annotation and two experiments with supervised classification of text types in English and Estonian web corpora are presented. iii Keywords Web corpora, Web crawling, Text processing, Language identification, Discerning similar languages, Spam removal, Corpus annotation, Inter- annotator agreement, Text types, Text topic, Text genre iv Contents Introduction 1 0.1 Large, Clean and Rich Web Corpora .............1 0.2 Contents of the Thesis & Relation to Publications ......5 1 Efficient Web Crawling For Large Text Corpora 8 1.1 Building Corpora From the Web ...............8 1.2 SpiderLing, an Asynchronous Text Focused Web Crawler .. 14 1.2.1 General Web Crawler Architecture . 14 1.2.2 SpiderLing Architecture . 17 1.2.3 Yield Rate Aware Efficient Crawling . 28 1.2.4 Deployment of SpiderLing in Corpus Projects . 38 1.3 Brno Corpus Processing Pipeline ............... 41 2 Cleaner Web Corpora 47 2.1 Discerning Similar Languages ................ 48 2.1.1 Method Description . 48 2.1.2 Evaluation on VarDial Datasets . 51 2.1.3 Comparison to Other Language Detection Tools 56 2.1.4 Application to Web Corpora . 57 2.2 Non-Text Removal ...................... 63 2.2.1 Web Spam in Text Corpora . 63 2.2.2 Removing Spam from an English Web Corpus through Supervised Learning . 73 2.2.3 Semi-manual Efficient Classification of Non-text in an Estonian Web Corpus . 85 2.2.4 Web Spam Conclusion . 91 3 Richer Web Corpora 92 3.1 Genre Annotation of Web Corpora: Scheme and Issues ... 94 3.1.1 Genre Selection and Reliability of Classification 94 3.1.2 Experiment Setup . 97 3.1.3 Inter-annotator Agreement . 103 3.1.4 Dealing with a Low Agreement . 107 3.2 Text Type Annotation of Web Corpora ............ 110 3.2.1 Topic Annotation of an English Web Corpus through Learning from a Web Directory . 110 v 3.2.2 Semi-manual Efficient Annotation of Text Types in Estonian National Corpus . 117 4 Summary 120 4.1 Author’s Contribution .................... 120 4.2 Future Challenges of Building Web Corpora ......... 122 A Appendices 124 A.1 Genre Definition for Annotators ............... 125 A.2 Text Size after Processing Steps ................ 127 A.3 FastText Hyperparameters for English Topic Classification . 129 A.4 Selected Papers ........................ 130 Bibliography 140 vi List of Tables 1.1 Sums of downloaded and final data size for all domains above the given yield rate threshold. 31 1.2 The yield rate threshold as a function of the number of downloaded documents. 33 1.3 Yield rate of crawling of the web of selected target languages in 2019: The ratio of the size of the plaintext output of the crawler to the size of all data downloaded is calculated in the fourth column ‘YR’. The ratio of the size of the plaintext after discerning similar languages and near paragraph de-duplication to the size of all data downloaded is calculated in the last, cumulative yield rate column ‘CYR’. Cs & sk denotes Czech and Slovak languages that were crawled together. 39 2.1 Sizes of wordlists used in the evaluation. Large web sources – TenTen, Aranea and WaC corpora – were limited to respective national TLDs. Other wordlists were built from the training and evaluation data of DSL Corpus Collection and parts of GloWbE corpus. Columns Web, DSL and GloWbE contain the count of words in the respective wordlist. 53 2.2 Overall accuracy using large web corpus wordlists and DSL CC v. 1 training data wordlists on DSL CC v. 1 gold data. The best result achieved by participants in VarDial 2014 can be found in the last column. 54 2.3 Performance of our method on VarDial DSL test data compared to the best score achieved by participants of the competition at that time. 55 2.4 Comparison of language identification tools on 952 random paragraphs from Czech and Slovak web. The tools were set to discern Czech, Slovak and English. 57 2.5 Discriminating similar languages in Indonesian web corpus from 2010 (Indonesian WaC corpus v. 3 by Siva Reddy): Document count and token count of corpus parts in languages discerned. 58 vii 2.6 Discriminating similar languages in the Norwegian web corpus from 2015 (noTenTen15): Document count and token count of corpus parts in languages discerned. 59 2.7 Overview of removal of unwanted languages in recently built web corpora (gaTenTen20, enTenTen19, etTenTen19, frTenTen19, huTenTen12, itTenTen19, roTenTen16). Document count and token count of corpus data before and after language filtering. ‘Removed’ stands for the percent of data removed. 60 2.8 Languages recognised in the Estonian web corpus from 2019 (etTenTen19). Document count and token count of corpus parts in languages discerned. 61 2.9 Languages recognised in the output of SpiderLing crawling Czech and Slovak web in 2019. Document count and token count of corpus parts in languages discerned. 62 2.10 Comparison of the 2015 English web corpus before and after spam removal using the classifier. Corpus sizes and relative frequencies (number of occurrences per million words) of selected words are shown. By reducing the corpus to 55 % of the former token count, phrases strongly indicating spam documents such as cialis 20 mg, payday loan, essay writing or slot machine were almost removed while innocent phrases not attracting spammers from the same domains such as oral administration, interest rate, pass the exam or play games were reduced proportionally to the whole corpus. 75 2.11 Top collocate objects of verb ‘buy’ before and after spam removal in English web corpus (enTenTen15, Word Sketches). Corpus frequency of the verb: 14,267,996 in the original corpus and 2,699,951 in the cleaned corpus – 81 % reduction by cleaning (i.e. more than the average reduction of a word in the corpus). 80 2.12 Top collocate subjects of verb ‘buy’ before and after spam removal in English web corpus (enTenTen15, Word Sketches). 81 viii 2.13 Top collocate modifiers of noun ‘house’ before and after spam removal in English web corpus (enTenTen15, Word Sketches). Corpus frequency of the noun: 10,873,053 in the original corpus and 3,675,144 in the cleaned corpus – 66 % reduction by cleaning. 82 2.14 Top collocate nouns modified by adjective ‘online’ before and after spam removal in English web corpus (enTenTen15, Word Sketches). Corpus frequency of the adjective: 20,903,329 in the original corpus and 4,118,261 in the cleaned corpus – 80 % reduction by cleaning. 83 2.15 Top collocate nouns modified by adjective ‘green’ before and after spam removal in English web corpus (enTenTen15, Word Sketches).
Recommended publications
  • Talk Bank: a Multimodal Database of Communicative Interaction

    Talk Bank: a Multimodal Database of Communicative Interaction

    Talk Bank: A Multimodal Database of Communicative Interaction 1. Overview The ongoing growth in computer power and connectivity has led to dramatic changes in the methodology of science and engineering. By stimulating fundamental theoretical discoveries in the analysis of semistructured data, we can to extend these methodological advances to the social and behavioral sciences. Specifically, we propose the construction of a major new tool for the social sciences, called TalkBank. The goal of TalkBank is the creation of a distributed, web- based data archiving system for transcribed video and audio data on communicative interactions. We will develop an XML-based annotation framework called Codon to serve as the formal specification for data in TalkBank. Tools will be created for the entry of new and existing data into the Codon format; transcriptions will be linked to speech and video; and there will be extensive support for collaborative commentary from competing perspectives. The TalkBank project will establish a framework that will facilitate the development of a distributed system of allied databases based on a common set of computational tools. Instead of attempting to impose a single uniform standard for coding and annotation, we will promote annotational pluralism within the framework of the abstraction layer provided by Codon. This representation will use labeled acyclic digraphs to support translation between the various annotation systems required for specific sub-disciplines. There will be no attempt to promote any single annotation scheme over others. Instead, by promoting comparison and translation between schemes, we will allow individual users to select the custom annotation scheme most appropriate for their purposes.
  • International Computer Science Institute Activity Report 2007

    International Computer Science Institute Activity Report 2007

    International Computer Science Institute Activity Report 2007 International Computer Science Institute, 1947 Center Street, Suite 600, Berkeley, CA 94704-1198 USA phone: (510) 666 2900 (510) fax: 666 2956 [email protected] http://www.icsi.berkeley.edu PRINCIPAL 2007 SPONSORS Cisco Defense Advanced Research Projects Agency (DARPA) Disruptive Technology Offi ce (DTO, formerly ARDA) European Union (via University of Edinburgh) Finnish National Technology Agency (TEKES) German Academic Exchange Service (DAAD) Google IM2 National Centre of Competence in Research, Switzerland Microsoft National Science Foundation (NSF) Qualcomm Spanish Ministry of Education and Science (MEC) AFFILIATED 2007 SPONSORS Appscio Ask AT&T Intel National Institutes of Health (NIH) SAP Volkswagen CORPORATE OFFICERS Prof. Nelson Morgan (President and Institute Director) Dr. Marcia Bush (Vice President and Associate Director) Prof. Scott Shenker (Secretary and Treasurer) BOARD OF TRUSTEES, JANUARY 2008 Prof. Javier Aracil, MEC and Universidad Autónoma de Madrid Prof. Hervé Bourlard, IDIAP and EPFL Vice Chancellor Beth Burnside, UC Berkeley Dr. Adele Goldberg, Agile Mind, Inc. and Pharmaceutrix, Inc. Dr. Greg Heinzinger, Qualcomm Mr. Clifford Higgerson, Walden International Prof. Richard Karp, ICSI and UC Berkeley Prof. Nelson Morgan, ICSI (Director) and UC Berkeley Dr. David Nagel, Ascona Group Prof. Prabhakar Raghavan, Stanford and Yahoo! Prof. Stuart Russell, UC Berkeley Computer Science Division Chair Mr. Jouko Salo, TEKES Prof. Shankar Sastry, UC Berkeley, Dean
  • Arabic Corpus and Word Sketches

    Arabic Corpus and Word Sketches

    Journal of King Saud University – Computer and Information Sciences (2014) 26, 357–371 King Saud University Journal of King Saud University – Computer and Information Sciences www.ksu.edu.sa www.sciencedirect.com arTenTen: Arabic Corpus and Word Sketches Tressy Arts a, Yonatan Belinkov b, Nizar Habash c,*, Adam Kilgarriff d, Vit Suchomel e,d a Chief Editor Oxford Arabic Dictionary, UK b MIT, USA c New York University Abu Dhabi, United Arab Emirates d Lexical Computing Ltd, UK e Masaryk Univ., Czech Republic Available online 7 October 2014 KEYWORDS Abstract We present arTenTen, a web-crawled corpus of Arabic, gathered in 2012. arTenTen con- Corpora; sists of 5.8-billion words. A chunk of it has been lemmatized and part-of-speech (POS) tagged with Lexicography; the MADA tool and subsequently loaded into Sketch Engine, a leading corpus query tool, where it Morphology; is open for all to use. We have also created ‘word sketches’: one-page, automatic, corpus-derived Concordance; summaries of a word’s grammatical and collocational behavior. We use examples to demonstrate Arabic what the corpus can show us regarding Arabic words and phrases and how this can support lexi- cography and inform linguistic research. The article also presents the ‘sketch grammar’ (the basis for the word sketches) in detail, describes the process of building and processing the corpus, and considers the role of the corpus in additional research on Arabic. Ó 2014 Production and hosting by Elsevier B.V. on behalf of King Saud University. 1. Introduction ber of the TenTen Corpus Family (Jakubı´cˇek et al., 2013).
  • Usage of IT Terminology in Corpus Linguistics Mavlonova Mavluda Davurovna

    Usage of IT Terminology in Corpus Linguistics Mavlonova Mavluda Davurovna

    International Journal of Innovative Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075, Volume-8, Issue-7S, May 2019 Usage of IT Terminology in Corpus Linguistics Mavlonova Mavluda Davurovna At this period corpora were used in semantics and related Abstract: Corpus linguistic will be the one of latest and fast field. At this period corpus linguistics was not commonly method for teaching and learning of different languages. used, no computers were used. Corpus linguistic history Proposed article can provide the corpus linguistic introduction were opposed by Noam Chomsky. Now a day’s modern and give the general overview about the linguistic team of corpus. corpus linguistic was formed from English work context and Article described about the corpus, novelties and corpus are associated in the following field. Proposed paper can focus on the now it is used in many other languages. Alongside was using the corpus linguistic in IT field. As from research it is corpus linguistic history and this is considered as obvious that corpora will be used as internet and primary data in methodology [Abdumanapovna, 2018]. A landmark is the field of IT. The method of text corpus is the digestive modern corpus linguistic which was publish by Henry approach which can drive the different set of rules to govern the Kucera. natural language from the text in the particular language, it can Corpus linguistic introduce new methods and techniques explore about languages that how it can inter-related with other. for more complete description of modern languages and Hence the corpus linguistic will be automatically drive from the these also provide opportunities to obtain new result with text sources.
  • Collection of Usage Information for Language Resources from Academic Articles

    Collection of Usage Information for Language Resources from Academic Articles

    Collection of Usage Information for Language Resources from Academic Articles Shunsuke Kozaway, Hitomi Tohyamayy, Kiyotaka Uchimotoyyy, Shigeki Matsubaray yGraduate School of Information Science, Nagoya University yyInformation Technology Center, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan yyyNational Institute of Information and Communications Technology 4-2-1 Nukui-Kitamachi, Koganei, Tokyo, 184-8795, Japan fkozawa,[email protected], [email protected], [email protected] Abstract Recently, language resources (LRs) are becoming indispensable for linguistic researches. However, existing LRs are often not fully utilized because their variety of usage is not well known, indicating that their intrinsic value is not recognized very well either. Regarding this issue, lists of usage information might improve LR searches and lead to their efficient use. In this research, therefore, we collect a list of usage information for each LR from academic articles to promote the efficient utilization of LRs. This paper proposes to construct a text corpus annotated with usage information (UI corpus). In particular, we automatically extract sentences containing LR names from academic articles. Then, the extracted sentences are annotated with usage information by two annotators in a cascaded manner. We show that the UI corpus contributes to efficient LR searches by combining the UI corpus with a metadata database of LRs and comparing the number of LRs retrieved with and without the UI corpus. 1. Introduction Thesaurus1. In recent years, such language resources (LRs) as corpora • He also employed Roget’s Thesaurus in 100 words of and dictionaries are being widely used for research in the window to implement WSD.
  • A Resource of Corpus-Derived Typed Predicate Argument Structures for Croatian

    A Resource of Corpus-Derived Typed Predicate Argument Structures for Croatian

    CROATPAS: A Resource of Corpus-derived Typed Predicate Argument Structures for Croatian Costanza Marini Elisabetta Ježek University of Pavia University of Pavia Department of Humanities Department of Humanities costanza.marini01@ [email protected] universitadipavia.it Abstract 2014). The potential uses and purposes of the resource range from multilingual The goal of this paper is to introduce pattern linking between compatible CROATPAS, the Croatian sister project resources to computer-assisted language of the Italian Typed-Predicate Argument learning (CALL). Structure resource (TPAS1, Ježek et al. 2014). CROATPAS is a corpus-based 1 Introduction digital collection of verb valency structures with the addition of semantic Nowadays, we live in a time when digital tools type specifications (SemTypes) to each and resources for language technology are argument slot, which is currently being constantly mushrooming all around the world. developed at the University of Pavia. However, we should remind ourselves that some Salient verbal patterns are discovered languages need our attention more than others if following a lexicographical methodology they are not to face – to put it in Rehm and called Corpus Pattern Analysis (CPA, Hegelesevere’s words – “a steadily increasing Hanks 2004 & 2012; Hanks & and rather severe threat of digital extinction” Pustejovsky 2005; Hanks et al. 2015), (2018: 3282). According to the findings of initiatives such as whereas SemTypes – such as [HUMAN], [ENTITY] or [ANIMAL] – are taken from a the META-NET White Paper Series (Tadić et al. shallow ontology shared by both TPAS 2012; Rehm et al. 2014), we can state that and the Pattern Dictionary of English Croatian is unfortunately among the 21 out of 24 Verbs (PDEV2, Hanks & Pustejovsky official languages of the European Union that are 2005; El Maarouf et al.
  • Student Research Workshop Associated with RANLP 2011, Pages 1–8, Hissar, Bulgaria, 13 September 2011

    Student Research Workshop Associated with RANLP 2011, Pages 1–8, Hissar, Bulgaria, 13 September 2011

    RANLPStud 2011 Proceedings of the Student Research Workshop associated with The 8th International Conference on Recent Advances in Natural Language Processing (RANLP 2011) 13 September, 2011 Hissar, Bulgaria STUDENT RESEARCH WORKSHOP ASSOCIATED WITH THE INTERNATIONAL CONFERENCE RECENT ADVANCES IN NATURAL LANGUAGE PROCESSING’2011 PROCEEDINGS Hissar, Bulgaria 13 September 2011 ISBN 978-954-452-016-8 Designed and Printed by INCOMA Ltd. Shoumen, BULGARIA ii Preface The Recent Advances in Natural Language Processing (RANLP) conference, already in its eight year and ranked among the most influential NLP conferences, has always been a meeting venue for scientists coming from all over the world. Since 2009, we decided to give arena to the younger and less experienced members of the NLP community to share their results with an international audience. For this reason, further to the first successful and highly competitive Student Research Workshop associated with the conference RANLP 2009, we are pleased to announce the second edition of the workshop which is held during the main RANLP 2011 conference days on 13 September 2011. The aim of the workshop is to provide an excellent opportunity for students at all levels (Bachelor, Master, and Ph.D.) to present their work in progress or completed projects to an international research audience and receive feedback from senior researchers. We have received 31 high quality submissions, among which 6 papers have been accepted as regular oral papers, and 18 as posters. Each submission has been reviewed by
  • Linguistic Annotation of the Digital Papyrological Corpus: Sematia

    Linguistic Annotation of the Digital Papyrological Corpus: Sematia

    Marja Vierros Linguistic Annotation of the Digital Papyrological Corpus: Sematia 1 Introduction: Why to annotate papyri linguistically? Linguists who study historical languages usually find the methods of corpus linguis- tics exceptionally helpful. When the intuitions of native speakers are lacking, as is the case for historical languages, the corpora provide researchers with materials that replaces the intuitions on which the researchers of modern languages can rely. Using large corpora and computers to count and retrieve information also provides empiri- cal back-up from actual language usage. In the case of ancient Greek, the corpus of literary texts (e.g. Thesaurus Linguae Graecae or the Greek and Roman Collection in the Perseus Digital Library) gives information on the Greek language as it was used in lyric poetry, epic, drama, and prose writing; all these literary genres had some artistic aims and therefore do not always describe language as it was used in normal commu- nication. Ancient written texts rarely reflect the everyday language use, let alone speech. However, the corpus of documentary papyri gets close. The writers of the pa- pyri vary between professionally trained scribes and some individuals who had only rudimentary writing skills. The text types also vary from official decrees and orders to small notes and receipts. What they have in common, though, is that they have been written for a specific, current need instead of trying to impress a specific audience. Documentary papyri represent everyday texts, utilitarian prose,1 and in that respect, they provide us a very valuable source of language actually used by common people in everyday circumstances.
  • Creating and Using Multilingual Corpora in Translation Studies Claudio Fantinuoli and Federico Zanettin

    Creating and Using Multilingual Corpora in Translation Studies Claudio Fantinuoli and Federico Zanettin

    Chapter 1 Creating and using multilingual corpora in translation studies Claudio Fantinuoli and Federico Zanettin 1 Introduction Corpus linguistics has become a major paradigm and research methodology in translation theory and practice, with practical applications ranging from pro- fessional human translation to machine (assisted) translation and terminology. Corpus-based theoretical and descriptive research has investigated written and interpreted language, and topics such as translation universals and norms, ideol- ogy and individual translator style (Laviosa 2002; Olohan 2004; Zanettin 2012), while corpus-based tools and methods have entered the curricula at translation training institutions (Zanettin, Bernardini & Stewart 2003; Beeby, Rodríguez Inés & Sánchez-Gijón 2009). At the same time, taking advantage of advancements in terms of computational power and increasing availability of electronic texts, enormous progress has been made in the last 20 years or so as regards the de- velopment of applications for professional translators and machine translation system users (Coehn 2009; Brunette 2013). The contributions to this volume, which are centred around seven European languages (Basque, Dutch, German, Greek, Italian, Spanish and English), add to the range of studies of corpus-based descriptive studies, and provide exam- ples of some less explored applications of corpus analysis methods to transla- tion research. The chapters, which are based on papers first presented atthe7th congress of the European Society of Translation Studies held in Germersheim in Claudio Fantinuoli & Federico Zanettin. 2015. Creating and using multilin- gual corpora in translation studies. In Claudio Fantinuoli & Federico Za- nettin (eds.), New directions in corpus-based translation studies, 1–11. Berlin: Language Science Press Claudio Fantinuoli and Federico Zanettin July/August 20131, encompass a variety of research aims and methodologies, and vary as concerns corpus design and compilation, and the techniques used to ana- lyze the data.
  • From CHILDES to Talkbank

    From CHILDES to Talkbank

    From CHILDES to TalkBank Brian MacWhinney Carnegie Mellon University MacWhinney, B. (2001). New developments in CHILDES. In A. Do, L. Domínguez & A. Johansen (Eds.), BUCLD 25: Proceedings of the 25th annual Boston University Conference on Language Development (pp. 458-468). Somerville, MA: Cascadilla. a similar article appeared as: MacWhinney, B. (2001). From CHILDES to TalkBank. In M. Almgren, A. Barreña, M. Ezeizaberrena, I. Idiazabal & B. MacWhinney (Eds.), Research on Child Language Acquisition (pp. 17-34). Somerville, MA: Cascadilla. Recent years have seen a phenomenal growth in computer power and connectivity. The computer on the desktop of the average academic researcher now has the power of room-size supercomputers of the 1980s. Using the Internet, we can connect in seconds to the other side of the world and transfer huge amounts of text, programs, audio and video. Our computers are equipped with programs that allow us to view, link, and modify this material without even having to think about programming. Nearly all of the major journals are now available in electronic form and the very nature of journals and publication is undergoing radical change. These new trends have led to dramatic advances in the methodology of science and engineering. However, the social and behavioral sciences have not shared fully in these advances. In large part, this is because the data used in the social sciences are not well- structured patterns of DNA sequences or atomic collisions in super colliders. Much of our data is based on the messy, ill-structured behaviors of humans as they participate in social interactions. Categorizing and coding these behaviors is an enormous task in itself.
  • Jazykovedný Ústav Ľudovíta Štúra

    Jazykovedný Ústav Ľudovíta Štúra

    JAZYKOVEDNÝ ÚSTAV ĽUDOVÍTA ŠTÚRA SLOVENSKEJ AKADÉMIE VIED 2 ROČNÍK 70, 2019 JAZYKOVEDNÝ ČASOPIS ________________________________________________________________VEDEcKÝ ČASOPIS PRE OTáZKY TEóRIE JAZYKA JOURNAL Of LINGUISTIcS ScIENTIfIc JOURNAL fOR ThE Theory Of LANGUAGE ________________________________________________________________ hlavná redaktorka/Editor-in-chief: doc. Mgr. Gabriela Múcsková, PhD. Výkonní redaktori/Managing Editors: PhDr. Ingrid Hrubaničová, PhD., Mgr. Miroslav Zumrík, PhD. Redakčná rada/Editorial Board: PhDr. Klára Buzássyová, CSc. (Bratislava), prof. PhDr. Juraj Dolník, DrSc. (Bratislava), PhDr. Ingrid Hrubaničová, PhD. (Bra tislava), doc. Mgr. Martina Ivanová, PhD. (Prešov), Mgr. Nicol Janočková, PhD. (Bratislava), Mgr. Alexandra Jarošová, CSc. (Bratislava), prof. PaedDr. Jana Kesselová, CSc. (Prešov), PhDr. Ľubor Králik, CSc. (Bratislava), PhDr. Viktor Krupa, DrSc. (Bratislava), doc. Mgr. Gabriela Múcsková, PhD. (Bratislava), Univ. Prof. Mag. Dr. Stefan Michael Newerkla (Viedeň – Ra- kúsko), Associate Prof. Mark Richard Lauersdorf, Ph.D. (Kentucky – USA), prof. Mgr. Martin Ološtiak, PhD. (Prešov), prof. PhDr. Slavomír Ondrejovič, DrSc. (Bratislava), prof. PaedDr. Vladimír Patráš, CSc. (Banská Bystrica), prof. PhDr. Ján Sabol, DrSc. (Košice), prof. PhDr. Juraj Vaňko, CSc. (Nitra), Mgr. Miroslav Zumrík, PhD. (Bratislava), prof. PhDr. Pavol Žigo, CSc. (Bratislava). Technický_______________________________________________________________ redaktor/Technical editor: Mgr. Vladimír Radik Vydáva/Published by: Jazykovedný
  • The British National Corpus Revisited: Developing Parameters for Written BNC2014 Abi Hawtin (Lancaster University, UK)

    The British National Corpus Revisited: Developing Parameters for Written BNC2014 Abi Hawtin (Lancaster University, UK)

    The British National Corpus Revisited: Developing parameters for Written BNC2014 Abi Hawtin (Lancaster University, UK) 1. The British National Corpus 2014 project The ESRC Centre for Corpus Approaches to Social Science (CASS) at Lancaster University and Cambridge University Press are working together to create a new, publicly accessible corpus of contemporary British English. The corpus will be a successor to the British National Corpus, created in the early 1990s (BNC19941). The British National Corpus 2014 (BNC2014) will be of the same order of magnitude as BNC1994 (100 million words). It is currently projected that the corpus will reach completion in mid-2018. Creation of the spoken sub-section of BNC2014 is in its final stages; this paper focuses on the justification and development of the written sub- section of BNC2014. 2. Justification for BNC2014 There are now many web-crawled corpora of English, widely available to researchers, which contain far more data than will be included in Written BNC2014. For example, the English TenTen corpus (enTenTen) is a web-crawled corpus of Written English which currently contains approximately 19 billion words (Jakubíček et al., 2013). Web-crawling processes mean that this huge amount of data can be collected very quickly; Jakubíček et al. (2013: 125) note that “For a language where there is plenty of material available, we can gather, clean and de-duplicate a billion words a day.” So, with extremely large and quickly-created web-crawled corpora now becoming commonplace in corpus linguistics, the question might well be asked why a new, 100 million word corpus of Written British English is needed.