Corpus Linguistics : A General Introduction Niladri Sekhar Dash 1. Introduction Corpus Linguistics is a multidimensional area. It is an area with a wide spectrum for encompassing all diversities of language use in all domains of linguistic interaction, communication, and comprehension. The introduction of corpus in language study and application has incorporated a new dimension to linguistics. In principle, Corpus Linguistics is an approach that aims at investigating language and all its properties by analysing large collections of text samples. This approach has been used in a number of research areas for ages: from descriptive study of a language, to language education, to lexicography, etc. It broadly refers to exhaustive analysis of any substantial amount of authentic, spoken and/or written text samples. In general, it covers large amount of machine-readable data of actual language use that includes the collections of literary and non-literary text samples to reflect on both the synchronic and diachronic aspects of a language. The uniqueness corpus linguistics lies in its way of using modern computer technology in collection of language data, methods used in processing language databases, techniques used in language data and information retrieval, and strategies used in application of these in all kinds language-related research and development activities. Electronic (digital) language corpus is a new thing. It has a history of nearly half a century. Therefore, we are yet to come to a common consensus as to what counts as corpus, and how it should be designed, developed, classified, processed and utilised. The basic philosophy behind corpus linguistics has two wings: (a) we have a cognitive drive to know how people use language in their daily communication activities, and (b) if it is possible to build up intelligent systems that can efficiently interact with human beings. With this motivation both computer scientists and linguists have come together to develop language corpus that can be used for designing intelligent systems (e.g., machine translation system, language processing system, speech understanding system, text analysis and understanding system, computer aided instruction system, etc.) for the benefit of the language community at large. All branches of linguistics and language technology can benefit from insights obtained from analysis of corpora. Thus, description and analysis of linguistic properties collected from a corpus becomes of paramount importance in all many areas of human knowledge and application. 2. What is Corpus? The term corpus is derived from Latin corpus "body". At present it means representative collection of texts of a given language, dialect or other subset of a language to be used for linguistic analysis. In finer definition, it refers to (a) (loosely) any body of text; (b) (most commonly) a body of machine- readable text; and (c) (more strictly) a finite collection of machine-readable texts sampled to be representative of a language or variety (McEnery and Wilson 1996: 218). Corpus contains a large collection of representative samples of texts covering different varieties of language used in various domains of linguistic interactions. Theoretically, corpus is (C)apable (O)f (R)epresenting (P)otentially (U)nlimited (S)elections of texts. It is compatible to computer, operational in research and application, representative of the source language, processable by man and machine, unlimited in data, and systematic in formation and representation (Dash 2005: 35). 2 3. Salient Features of Corpus • Quantity: It should be big in size containing large amount of data either in spoken or written form. Size is virtually the sum of its components, which constitute its body. • Quality (= authenticity). All texts should be obtained from actual examples of speech and writing. The role of a linguist is very important here. He has to verify if language data is collected from ordinary communication, and not from experimental conditions or artificial circumstances. • Representation: It should include samples from a wide range of texts. It should be balanced to all areas of language use to represent maximum linguistic diversities, as future analysis devised on it needs verification and authentication of information from the corpus representing a language. • Simplicity: It should contain plain texts in simple format. This means that we expect an unbroken string of characters (or words) without any additional linguistic information marked-up within texts. A simple plain text is opposed to any kind of annotation with various types of linguistic and non-linguistic information. • Equality: Samples used in corpus should be of even size. However, this is a controversial issue and will not be adopted everywhere. Sampling model may change considerably to make a corpus more representative and multi-dimensional. • Retreavability: Data, information, examples, and references should be easily retrievable from corpus by the end-users. This pays attention to preserving techniques of language data in electronic format in computer. The present technology makes it possible to generate corpus in PC and preserve it in such way that we can easily retrieve data as and when required. • Verifiability: Corpus should be open to any kind of empirical verification. We can use data form corpus for any kind of verification. This puts corpus linguistics steps ahead of intuitive approach to language study. • Augmentation: It should be increased regularly. This will put corpus 'at par' to register linguistic changes occurring in a language in course of time. Over time, by addition of new linguistic data, a corpus achieves historical dimension for diachronic studies, and for displaying linguistic cues to arrest changes in life and society. • Documentation: Full information of components should be kept separate from the text itself. It is always better to keep documentation information separate from the text, and include only a minimal header containing reference to documentation. In case of corpus management, this allows effective separation of plain texts from annotation with only a small amount of programming effort. 4. The TDIL Corpora of Indian Languages The generation of corpus for the Indian languages began in 1991, when the Department of Electronics, Govt. of India initiated a project ( Technology Development for Indian Languages ) to develop machine-readable corpus of texts in all the major Indian languages. Also emphasis was laid on development of software for language processing (e.g., POS tagging, text encoding, frequency counting, spelling checking, morphological processing, etc.) as well as for designing of systems for machine translation from English to Indian languages. N. S. Dash: Corpus Linguistics: A General Introduction. CIIL, Mysore, 25 th August 2010 3 Indian Institute of Technology, Delhi was given the task for developing corpus of Indian English, Hindi, and Punjabi; Central Institute of Indian Languages , Mysore was assigned for corpus of Tamil, Telugu, Kannada, and Malayalam; Deccan College, Pune developed corpus of Marathi and Gujarati; Indian Institute of Applied Language Sciences, Bhubaneswar developed corpus of Oriya, Bengali and Assamese; Sampurnananda Sanskrit University, Varanasi was entrusted with the development of corpus of Sanskrit; and Aligarh Muslim University , Aligarh was assigned the task for developing corpus of Urdu, Sindhi, and Kashmiri; Indian Institute of Technology, Kanpur took responsibility for designing systems and software for language processing and machine translation, while CIIL, Mysore took responsibility to archive the entire corpus database of all Indian languages for future utilization. After the completion of the TDIL project in 1995, works on corpus development and processing stopped for some reasons beyond the knowledge of the present author. However, the MICT, Govt. of India has revived the whole enterprise (http:\\www.mit.gov.in) with new enthusiasm and vision. The realisation of this enterprise is clearly manifested in formation of LDC-IL, although my personal view is that it should be NAIL (National Archive for the Indian Languages), rather than the LDC-IL (Dash 2003). 5. Conceptual Classification of Corpora Since electronic corpus is a new thing, we are yet to reach to a common consensus to what counts as a corpus, and how it should be classified. The classification scheme I propose here goes as far as it is prudent at the present moment. It offers a reasonable way to classify corpora, with clearly delimited categories wherever possible. Different criteria for classification are applied to corpora, sub-corpora, and their related components. Linguistic criteria may be external and internal. External criteria are largely mapped onto corpora from text typology concerned with participants, occasion, social setting, communicative function of language, etc. Internal criteria are concerned with recurrence of language patterns within the pieces of language. Taking all these issues under consideration I classify corpora in a broad scheme in the following manner: Genre of text, Nature of data, Type of text, Purpose of design, and Nature of application. 5.1 Genre of Text • Written Corpus: A written corpus (e.g., TDIL Corpus) by virtue of its genre contains only language data collected from various written, printed, published and electronic sources. • Speech Corpus: A speech corpus (e.g., Wellington Corpus of Spoken New Zealand English ) contains all formal and informal discussions,