DOCSLIB.ORG

Analyzing the Punjabi Language Stemmers: a Critical Approach

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Analyzing the Punjabi Language Stemmers: a Critical Approach 250 Analyzing the Punjabi Language Stemmers: A Critical Approach Harjit Singha APS Neighbourhood Campus, Punjabi University, Patiala, India Abstract Stemming is a procedure used to reduce the inflected words by removing affixes from them. It is widely used in Natural Language Processing systems developed for various languages like English, Hindi etc. Punjabi is a low resource availability language spoken in northern regions of India and Pakistan. In Pakistan, Shahmukhi script is used to write Punjabi. While in India, Gurmukhi script is used. Various approaches are used for stemming natural language words such as Brute Force approach, Rule based approach, Statistical approach etc. In Gurmukhi scripted Punjabi language stemming, either the Brute Force approach or Rule based approach or a combination of both of these approaches have been used till now. This paper presents a comparative analysis of various stemmers developed so far for stemming Gurmukhi scripted Punjabi language words based on their methodology and accuracy. It will motivate further research in developing more efficient stemmers for Punjabi language. It will benefit the researchers to compare and understand the used approaches and the problems faced in adopting each approach. Keywords Stemming, Natural Language Processing, Information Retrieval, Suffix Striping, Punjabi Language Processing 1. Introduction Various approaches are used for stemming natural language words such as Brute Force approach, Rule based approach, Statistical A procedure adopted to reduce the inflected approach etc. In brute force approach, a table words by removing any affixes attached to of inflected words along with their stem words them is called Stemming. Stemming is a very is maintained. To stem a word, that word is common and crucial task performed in Natural searched in the table for a match. If match Language Processing (NLP) applications. occurs with some word, the related stem word Many morphologically similar words can be is picked from the table. The approach is easy stemmed to the same stem word after to implement and the accuracy is dependent on removing any suffixes or prefixes from them. the table of words available in the database. In It improves the efficiency of further processes rule based approach, the affixes are removed adopted for information retrieval. It improves by using some fixed rules. The input words are the effectiveness of information retrieval by checked to have a group of characters at the helping to identify redundant information, to end or beginning of each word. That set of remove irrelevant information from retrieval characters (affix) is removed to get the stem and to optimize the retrieval by placing high word. In some words, an alternative set of ranked information on the top [1]. International Semantic Intelligence Conference, February 25–27, 2021, New Delhi, India EMAIL: [email protected](H. Singh) ORCID: 0000-0002-3215-4798 (H. Singh) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 251 characters is attached to the word to complete 2. Literature Review it. In statistical approach, the distance Jain et al. [3] used NLP technique to find measure is calculated between the two words. root word in Hindi language. The Hindi If the words are morphologically similar, they inflected word was stemmed to remove affix will have low distance measure. For from the end of the word. The stemmer was morphologically unrelated words, distance tested using 800 inflected words. They used measure will be high. Using this measure, the prefix, suffix and root word lists to remove the words of a document are grouped to make affixes and validate the root word. The clusters. In this way, morphologically similar stemmer provided the hit ratio of 0.93. Yadav words will be grouped in each cluster. Then et al. [4] proposed a method for automatic the central word of each cluster is taken as a summarization of text documents. They stem word [2]. generated a network of related sentences based In Gurmukhi scripted Punjabi language on semantic and lexical relation. Two stemming, either the Brute Force approach or sentences are related to each other using a Rule based approach or a combination of both certain strength that represents the number of of these approaches have been used till now. relationships between the two. Gros et al. [5] This paper analyses all those approaches and performed an analysis to test the effect on the compares them with each other based on the performance of information retrieval with methodology and accuracy. It will motivate anaphora resolution. They claimed the positive further research in developing more efficient effect on the results. In this way, the quality of stemmers for Punjabi language. It will benefit information retrieval results can be the researchers to compare and understand the significantly improved. used approaches and the problems faced in Agrawal et al. [6] proposed a translation adopting each approach. tool to translate Sanskrit language text to Hindi. They used stemming in preprocessing 1.1. Punjabi Language the Sanskrit words before converting them to Hindi. It was used as a major phase in Punjabi language belongs to Indo-Aryan translation tool development. Any affixes from group of languages. Its word order in sentence input Sanskrit words were removed before is Subject-Object-Verb and is spoken by using them for generating the equivalent Hindi almost 100 million people residing in India, text. A Sanskrit corpus was used to Pakistan and many other countries. It may be successfully stem the words by searching them called Punjabi community. Table 1 shows the in the corpus and validate. number of Punjabi speakers in some countries. Bhadwal et al. [7] developed Hindi- Sanskrit bilingual translation system. The Table 1 system was able to translate Hindi text to Punjabi Speakers in Some Countries Sanskrit and vice-versa using a number of modules. For Sanskrit to Hindi translation Country Punjabi Source flow, they used stemming as a major step Speakers before syntax analysis. The system was Pakistan 99,774,008 www.cia.gov developed in Java. Bhadwal et al. [8] proposed India 33,124,726 censusindia.gov.in a Hindi to Sanskrit machine translation Canada 543,495 www12.statcan.gc.ca system. The system includes a number of NLP UK 273,000 www.ons.gov.uk modules such as transliteration, POS-tagging, US 253,740 www.census.gov root verb extraction etc. In root verb UAE 191,000 ethnologue.com extraction, the algorithm finds the root verb before mapping to equivalent Sanskrit word. Punjabi is basically written in two different Although there are positive effects of scripts; Perso-Arabic and Gurmukhi. Perso- stemming in various NLP applications, but Arabic is related to Persian script which is also Wahbeh et al. [9] claimed negative effects on called Shahmukhi, and Gurmukhi is derived text classification in Arabic. They used from Landa, an old script of Indian sub- stemming as a preprocessing module in Arabic continent. 252 text classification and found that results were the knowledge of Punjabi and Gurmukhi negatively affected when stemming was used. script, because function of a stemmer is just to Review of literature shows that stemming remove any affixes attached to a word. is used for a variety of NLP applications. It Therefore only brief introduction to Punjabi motivated the analysis of stemmers developed and Gurmukhi script is enough. This paper for Punjabi language. Only a handful of discusses the stemmers developed for Punjabi research papers found for this low resource language Gurmukhi script. The paper availability Asian language. discusses the methodologies used by researchers to stem the inflected words. In this 3. Stemming Linguistics in paper, Gurmukhi scripted Punjabi language henceforth will be called Punjabi. Gurmukhi Punjabi The significance of this analysis is that it discloses the gap in the NLP research for In linguistics, an affix refers to either suffix Punjabi language. It will motivate further or prefix. In Punjabi language, both types of research in developing more efficient affixes are used to make proper words for stemmers for Punjabi using new ideas or by sentences. But in most cases, prefix changes extending existing approaches. the meaning of the word. So, in NLP applications, only suffixes are considered for 4.1. Earlier Stemmer for Punjabi removal. Table 2 shows some inflected words with their stem words. The table shows how the Punjabi words are dealt with linguistically Kumar et al. [10] proposed a Punjabi to remove some characters and if required, stemmer based on combination of brute force substitute some alternative set of characters to approach and suffix stripping technique. The complete the word. researchers divided the system in three units and named these units as Input unit, Output Table 2 unit and Processing unit. The Input unit takes a Punjabi word and Some Punjabi Inflected and Root words that word is sent to the Processing unit which Inflected Stem Suffix searches for the word in a database of Punjabi Word word Removal and inflected words and their stem words. If a Substitution word is found in the database, the corresponding stem word is considered as ਵਿਵਿਆਰਥੀਆਂ ਵਿਵਿਆਰਥੀ ਆਂ output stem word. If no match is found in ਬੱਸ拓 ਬੱਸ ਾ ਾਂ database, then suffix removal technique is and used to stem the word. The word ending is ਕੱਤੇ ਕੱਤ ਾੇ ਾ searched in a list of possible Punjabi suffixes. ਹਿ ਿ拓 ਹਿ ਿ拓 If the word ending matches with any one of the suffix present in the suffix list then the ਘਰ⸂ ਘਰ ਾ ਾਂ ending (suffix) is removed to get the stem ਮ ੜ ਮ ੜ ਾ word. The steps are shown in Algorithm 1 and graphically represented in Figure 1.
Load more

Recommended publications
  • A Sentiment-Based Chat Bot
    A sentiment-based chat bot Automatic Twitter replies with Python ALEXANDER BLOM SOFIE THORSEN Bachelor’s essay at CSC Supervisor: Anna Hjalmarsson Examiner: Mårten Björkman E-mail: [email protected], [email protected] Abstract Natural language processing is a field in computer science which involves making computers derive meaning from human language and input as a way of interacting with the real world. Broadly speaking, sentiment analysis is the act of determining the attitude of an author or speaker, with respect to a certain topic or the overall context and is an application of the natural language processing field. This essay discusses the implementation of a Twitter chat bot that uses natural language processing and sentiment analysis to construct a be- lievable reply. This is done in the Python programming language, using a statistical method called Naive Bayes classifying supplied by the NLTK Python package. The essay concludes that applying natural language processing and sentiment analysis in this isolated fashion was simple, but achieving more complex tasks greatly increases the difficulty. Referat Natural language processing är ett fält inom datavetenskap som innefat- tar att få datorer att förstå mänskligt språk och indata för att på så sätt kunna interagera med den riktiga världen. Sentiment analysis är, generellt sagt, akten av att försöka bestämma känslan hos en författare eller talare med avseende på ett specifikt ämne eller sammanhang och är en applicering av fältet natural language processing. Den här rapporten diskuterar implementeringen av en Twitter-chatbot som använder just natural language processing och sentiment analysis för att kunna svara på tweets genom att använda känslan i tweetet.
    [Show full text]
  • Named Entity Recognition System for Kashmiri Language Iamir Bashir Malik, Iikhushboo Bansal Istudent, M.Tech, Iiassistant Professor I,Iidept
    ISSN : 2347 - 8446 (Online) International Journal of Advanced Research in ISSN : 2347 - 9817 (Print) Vol. 3, Issue 2 (Apr. - Jun. 2015) Computer Science & Technology (IJARCST 2015) Named Entity Recognition System for Kashmiri Language IAmir Bashir Malik, IIKhushboo Bansal IStudent, M.Tech, IIAssistant Professor I,IIDept. of CSE, Desh Bhagat University, Mandi Gobindgarh, Punjab, India Abstract Named Entity Recognition (NER) is a task which helps in finding out Persons name, Location names, Organization names, Place, Date, Time etc. and classifies them into predefined different categories. Named Entity Recognition plays a major role in various Natural Language Processing (NLP) fields like Information Extraction, Machine Translations and Question Answering. Unfortunately Kashmiri language which is a scarce resourced language has not been taken into account. This paper describes the problems of NER in the context of Kashmiri Language and provides relevant solutions. Keywords Named Entity, Named Entity Recognition, Natural language process, Kashmiri language text. I. Introduction is as follows. The term Named Entity (NE) was evolved during the sixth (1) “Micromax”represent anorganization and “ Dec19, 2014” Message Understanding Conference (MUC -6, 1995).Named represent dateand “smartphone” represent entity and “had Entity Recognition (NER) is also knows as entity identification is a launched its on” represent others. subtask of information extraction (IE). NER extracts and classifies The named entities may be of any type such as given below
    [Show full text]
  • (And Potential) Language and Linguistic Resources on South Asian Languages
    CoRSAL Symposium, University of North Texas, November 17, 2017 Existing (and Potential) Language and Linguistic Resources on South Asian Languages Elena Bashir, The University of Chicago Resources or published lists outside of South Asia Digital Dictionaries of South Asia in Digital South Asia Library (dsal), at the University of Chicago. http://dsal.uchicago.edu/dictionaries/ . Some, mostly older, not under copyright dictionaries. No corpora. Digital Media Archive at University of Chicago https://dma.uchicago.edu/about/about-digital-media-archive Hock & Bashir (eds.) 2016 appendix. Lists 9 electronic corpora, 6 of which are on Sanskrit. The 3 non-Sanskrit entries are: (1) the EMILLE corpus, (2) the Nepali national corpus, and (3) the LDC-IL — Linguistic Data Consortium for Indian Languages Focus on Pakistan Urdu Most work has been done on Urdu, prioritized at government institutions like the Center for Language Engineering at the University of Engineering and Technology in Lahore (CLE). Text corpora: http://cle.org.pk/clestore/index.htm (largest is a 1 million word Urdu corpus from the Urdu Digest. Work on Essential Urdu Linguistic Resources: http://www.cle.org.pk/eulr/ Tagset for Urdu corpus: http://cle.org.pk/Publication/papers/2014/The%20CLE%20Urdu%20POS%20Tagset.pdf Urdu OCR: http://cle.org.pk/clestore/urduocr.htm Sindhi Sindhi is the medium of education in some schools in Sindh Has more institutional backing and consequent research than other languages, especially Panjabi. Sindhi-English dictionary developed jointly by Jennifer Cole at the University of Illinois Urbana- Champaign and Sarmad Hussain at CLE (http://182.180.102.251:8081/sed1/homepage.aspx).
    [Show full text]
  • Punjabi Language Characteristics and Role of Thesaurus in Natural
    Dharam Veer Sharma et al, / (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 2 (4) , 2011, 1434-1437 Punjabi Language Characteristics and Role of Thesaurus in Natural Language processing Dharam Veer Sharma1 Aarti2 Department of Computer Science, Punjabi University, Patiala, INDIA Abstract---This paper describes an attempt to explain various 2.2 Characteristics of the Punjabi Language characteristics of Punjabi language. The origin and symbols of Modern Punjabi is a very tonal language, making use of Punjabi language are presents in this paper. Various relations various tones to differentiate words that would otherwise be exist in thesaurus and role of thesaurus in natural language identical. Three primary tones can be identified: high-rising- processing also has been elaborated in this paper. falling, mid-rising-falling, and low rising. Following are characteristics of Punjabi language [3] [4]. Keywords---Thesaurus, Punjabi, characteristics, relations 2.2.1 Morphological characteristics Morphologically, Punjabi is an agglutinative language. That 1. INTRODUCTION is to say, grammatical information is encoded by way of A thesaurus links semantically related words and helps in the affixation (largely suffixation), rather than via independent selection of most appropriate words for given contexts [1]. A freestanding morphemes. Punjabi nouns inflect for number thesaurus contains synonyms (words which have basically the (singular, plural), gender (masculine, feminine), and same meaning) and as such is an important tool for many declension class (absolute, oblique). The absolute form of a applications in NLP too. The purpose is twofold: For writers, noun is its default or uninflected form. This form is used as it is a tool - one with words grouped and classified to help the object of the verb, typically when inanimate, as well as in select the best word to convey a specific nuance of meaning, measure or temporal (point of time) constructions.
    [Show full text]
  • Applying Ensembling Methods to BERT to Boost Model Performance
    Applying Ensembling Methods to BERT to Boost Model Performance Charlie Xu, Solomon Barth, Zoe Solis Department of Computer Science Stanford University cxu2, sbarth, zoesolis @stanford.edu Abstract Due to its wide range of applications and difficulty to solve, Machine Comprehen- sion has become a significant point of focus in AI research. Using the SQuAD 2.0 dataset as a metric, BERT models have been able to achieve superior performance to most other models. Ensembling BERT with other models has yielded even greater results, since these ensemble models are able to utilize the relative strengths and adjust for the relative weaknesses of each of the individual submodels. In this project, we present various ensemble models that leveraged BiDAF and multiple different BERT models to achieve superior performance. We also studied the effects how dataset manipulations and different ensembling methods can affect the performance of our models. Lastly, we attempted to develop the Synthetic Self- Training Question Answering model. In the end, our best performing multi-BERT ensemble obtained an EM score of 73.576 and an F1 score of 76.721. 1 Introduction Machine Comprehension (MC) is a complex task in NLP that aims to understand written language. Question Answering (QA) is one of the major tasks within MC, requiring a model to provide an answer, given a contextual text passage and question. It has a wide variety of applications, including search engines and voice assistants, making it a popular problem for NLP researchers. The Stanford Question Answering Dataset (SQuAD) is a very popular means for training, tuning, testing, and comparing question answering systems.
    [Show full text]
  • Online Guides to Indian Languages with Particular Reference to Hindi, Punjabi, and Sanskrit
    University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Library Philosophy and Practice (e-journal) Libraries at University of Nebraska-Lincoln 5-2012 Online Guides to Indian Languages with Particular Reference to Hindi, Punjabi, and Sanskrit Preeti Mahajan Panjab University, [email protected] Neeraj Kumar Singh Panjab University, [email protected] Follow this and additional works at: https://digitalcommons.unl.edu/libphilprac Part of the Library and Information Science Commons Mahajan, Preeti and Singh, Neeraj Kumar, "Online Guides to Indian Languages with Particular Reference to Hindi, Punjabi, and Sanskrit" (2012). Library Philosophy and Practice (e-journal). 749. https://digitalcommons.unl.edu/libphilprac/749 http://unllib.unl.edu/LPP/ Library Philosophy and Practice 2012 ISSN 1522-0222 Online Guides to Indian Languages with Particular Reference to Hindi, Punjabi, and Sanskrit Prof. Preeti Mahajan Department of Library and Information Science Panjab University Chandigarh, India Neeraj Kumar Singh Assistant Librarian A C Joshi Library Panjab University Chandigarh, India Introduction India is a multilingual country and the second most populated country on earth There are a quite a number of languages spoken in India. Some of these languages are accepted nationally while others are accepted as dialects of that particular region. The Indian languages belong to four language families namely Indo-European, Dravidian, Austroasiatic (Austric) and Sino-Tibetan. Majority of India's population are using Indo-European and Dravidian languages. The former are spoken mainly in northern and central regions and the latter in southern India. India has 22 officially recognised languages. But around 33 different languages and 2000 dialects have been identified in India.
    [Show full text]
  • Volume 33 (1) November 2018
    A Peer-reviewed Journal of Linguistic Society of Nepal Nepalese Linguistics Volume 33 (1) November 2018 Editor-in-Chief Kamal Poudel Editors Ram Raj Lohani Dr. Tikaram Poudel Office bearers for 2018-2020 President Bhim Narayan Regmi Vice President Krishna Prasad Chalise General Secretary Dr. Karnakhar Khatiwada Secretary (Office) Dr. Ambika Regmi Secretary (General) Dr. Tara Mani Rai Treasurer Ekku Maya Pun Member Dr. Narayan Prasad Sharma Member Dr. Ramesh Kumar Limbu Member Dr. Laxmi Raj Pandit Member Pratigya Regmi Member Shankar Subedi Editorial Board Editor-in-Chief Kamal Poudel Editors Ram Raj Lohani Dr. Tikaram Poudel Nepalese Linguistics is a peer-reviewed journal published by Linguistic Society of Nepal (LSN). LSN publishes articles related to the scientific study of languages, especially from Nepal. The authors are solely responsible for the views expressed in their articles. Published by: Linguistic Society of Nepal Kirtipur, Kathmandu Nepal Copies: 300 © Linguistic Society of Nepal ISSN 0259-1006 Price: NC 400/- (Nepal) IC 350/- (India) US$ 10/- The publication of this volume was supported by Nepal Academy. Editorial Linguistic Society of Nepal, since its inception in 1979, has been involved in preserving and promoting the languages of the Himalayan region through different activities such as organizing conferences, workshops and publications. As all our esteemed readers know that the journal Nepalese Linguistics is one of the major initiatives of the Society. The Board of Editors feels immense pleasure to bring out Volume 33.1 of Nepalese Linguistics in the eve of the 39th International Annual Conference of Linguistic Society of Nepal. The Society decided to peer-review the articles since this issue in order to ensure the quality of the journal.
    [Show full text]
  • Aconcorde: Towards a Proper Concordance for Arabic
    aConCorde: towards a proper concordance for Arabic Andrew Roberts, Dr Latifa Al-Sulaiti and Eric Atwell School of Computing University of Leeds LS2 9JT United Kingdom {andyr,latifa,eric}@comp.leeds.ac.uk July 12, 2005 Abstract Arabic corpus linguistics is currently enjoying a surge in activity. As the growth in the number of available Arabic corpora continues, there is an increased need for robust tools that can process this data, whether it be for research or teaching. One such tool that is useful for both groups is the concordancer — a simple tool for displaying a specified target word in its context. However, obtaining one that can reliably cope with the Arabic lan- guage had proved extremely difficult. Therefore, aConCorde was created to provide such a tool to the community. 1 Introduction A concordancer is a simple tool for summarising the contents of corpora based on words of interest to the user. Otherwise, manually navigating through (of- ten very large) corpora would be a long and tedious task. Concordancers are therefore extremely useful as a time-saving tool, but also much more. By isolat- ing keywords in their contexts, linguists can use concordance output to under- stand the behaviour of interesting words. Lexicographers can use the evidence to find if a word has multiple senses, and also towards defining their meaning. There is also much research and discussion about how concordance tools can be beneficial for data-driven language learning (Johns, 1990). A number of studies have demonstrated that providing access to corpora and concordancers benefited students learning a second language.
    [Show full text]
  • Hunspell – the Free Spelling Checker
    Hunspell – The free spelling checker About Hunspell Hunspell is a spell checker and morphological analyzer library and program designed for languages with rich morphology and complex word compounding or character encoding. Hunspell interfaces: Ispell-like terminal interface using Curses library, Ispell pipe interface, OpenOffice.org UNO module. Main features of Hunspell spell checker and morphological analyzer: - Unicode support (affix rules work only with the first 65535 Unicode characters) - Morphological analysis (in custom item and arrangement style) and stemming - Max. 65535 affix classes and twofold affix stripping (for agglutinative languages, like Azeri, Basque, Estonian, Finnish, Hungarian, Turkish, etc.) - Support complex compoundings (for example, Hungarian and German) - Support language specific features (for example, special casing of Azeri and Turkish dotted i, or German sharp s) - Handle conditional affixes, circumfixes, fogemorphemes, forbidden words, pseudoroots and homonyms. - Free software (LGPL, GPL, MPL tri-license) Usage The src/tools dictionary contains ten executables after compiling (or some of them are in the src/win_api): affixcompress: dictionary generation from large (millions of words) vocabularies analyze: example of spell checking, stemming and morphological analysis chmorph: example of automatic morphological generation and conversion example: example of spell checking and suggestion hunspell: main program for spell checking and others (see manual) hunzip: decompressor of hzip format hzip: compressor of
    [Show full text]
  • Package 'Ptstem'
    Package ‘ptstem’ January 2, 2019 Type Package Title Stemming Algorithms for the Portuguese Language Version 0.0.4 Description Wraps a collection of stemming algorithms for the Portuguese Language. URL https://github.com/dfalbel/ptstem License MIT + file LICENSE LazyData TRUE RoxygenNote 5.0.1 Encoding UTF-8 Imports dplyr, hunspell, magrittr, rslp, SnowballC, stringr, tidyr, tokenizers Suggests testthat, covr, plyr, knitr, rmarkdown VignetteBuilder knitr NeedsCompilation no Author Daniel Falbel [aut, cre] Maintainer Daniel Falbel <[email protected]> Repository CRAN Date/Publication 2019-01-02 14:40:02 UTC R topics documented: complete_stems . .2 extract_words . .2 overstemming_index . .3 performance . .3 ptstem_words . .4 stem_hunspell . .5 stem_modified_hunspell . .5 stem_porter . .6 1 2 extract_words stem_rslp . .7 understemming_index . .7 unify_stems . .8 Index 9 complete_stems Complete Stems Description Complete Stems Usage complete_stems(words, stems) Arguments words character vector of words stems character vector of stems extract_words Extract words Extracts all words from a character string of texts. Description Extract words Extracts all words from a character string of texts. Usage extract_words(texts) Arguments texts character vector of texts Note it uses the regex \b[:alpha:]+\b to extract words. overstemming_index 3 overstemming_index Overstemming Index (OI) Description It calculates the proportion of unrelated words that were combined. Usage overstemming_index(words, stems) Arguments words is a data.frame containing a column word a a column group so the function can identify groups of words. stems is a character vector with the stemming result for each word performance Performance Description Performance Usage performance(stemmers = c("rslp", "hunspell", "porter", "modified-hunspell")) Arguments stemmers a character vector with names of stemming algorithms.
    [Show full text]
  • Cross-Lingual and Supervised Learning Approach for Indonesian Word Sense Disambiguation Task
    Cross-Lingual and Supervised Learning Approach for Indonesian Word Sense Disambiguation Task Rahmad Mahendra, Heninggar Septiantri, Haryo Akbarianto Wibowo, Ruli Manurung, and Mirna Adriani Faculty of Computer Science, Universitas Indonesia Depok 16424, West Java, Indonesia [email protected], fheninggar, [email protected] Abstract after the target word), the nearest verb from the target word, the transitive verb around the target Ambiguity is a problem we frequently face word, and the document context. Unfortunately, in Natural Language Processing. Word neither the model nor the corpus from this research Sense Disambiguation (WSD) is a task to is made publicly available. determine the correct sense of an ambigu- This paper reports our study on WSD task for ous word. However, research in WSD for Indonesian using the combination of the cross- Indonesian is still rare to find. The avail- lingual and supervised learning approach. Train- ability of English-Indonesian parallel cor- ing data is automatically acquired using Cross- pora and WordNet for both languages can Lingual WSD (CLWSD) approach by utilizing be used as training data for WSD by ap- WordNet and parallel corpus. Then, the monolin- plying Cross-Lingual WSD method. This gual WSD model is built from training data and training data is used as an input to build it is used to assign the correct sense to any previ- a model using supervised machine learn- ously unseen word in a new context. ing algorithms. Our research also exam- ines the use of Word Embedding features 2 Related Work to build the WSD model. WSD task is undertaken in two main steps, namely 1 Introduction listing all possible senses for a word and deter- mining the right sense given its context (Ide and One of the biggest challenges in Natural Language Veronis,´ 1998).
    [Show full text]
  • A Comparison of Word Embeddings and N-Gram Models for Dbpedia Type and Invalid Entity Detection †
    information Article A Comparison of Word Embeddings and N-gram Models for DBpedia Type and Invalid Entity Detection † Hanqing Zhou *, Amal Zouaq and Diana Inkpen School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa ON K1N 6N5, Canada; [email protected] (A.Z.); [email protected] (D.I.) * Correspondence: [email protected]; Tel.: +1-613-562-5800 † This paper is an extended version of our conference paper: Hanqing Zhou, Amal Zouaq, and Diana Inkpen. DBpedia Entity Type Detection using Entity Embeddings and N-Gram Models. In Proceedings of the International Conference on Knowledge Engineering and Semantic Web (KESW 2017), Szczecin, Poland, 8–10 November 2017, pp. 309–322. Received: 6 November 2018; Accepted: 20 December 2018; Published: 25 December 2018 Abstract: This article presents and evaluates a method for the detection of DBpedia types and entities that can be used for knowledge base completion and maintenance. This method compares entity embeddings with traditional N-gram models coupled with clustering and classification. We tackle two challenges: (a) the detection of entity types, which can be used to detect invalid DBpedia types and assign DBpedia types for type-less entities; and (b) the detection of invalid entities in the resource description of a DBpedia entity. Our results show that entity embeddings outperform n-gram models for type and entity detection and can contribute to the improvement of DBpedia’s quality, maintenance, and evolution. Keywords: semantic web; DBpedia; entity embedding; n-grams; type identification; entity identification; data mining; machine learning 1. Introduction The Semantic Web is defined by Berners-Lee et al.
    [Show full text]