DOCSLIB.ORG

Hindi Named Entities Recognition (Ner) Using Natural Language Processing and Machine Learning

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Hindi Named Entities Recognition (Ner) Using Natural Language Processing and Machine Learning International Journal of Advances in Electronics and Computer Science, ISSN(p): 2394-2835 Volume-6, Issue-9, Sep.-2019 http://iraj.in HINDI NAMED ENTITIES RECOGNITION (NER) USING NATURAL LANGUAGE PROCESSING AND MACHINE LEARNING 1RIA MEHTA, 2DWEEP PANDYA, 3PRATIK CHAUDHARI, 4DEVIKA VERMA, 5KRISHNANJAN BHATTACHARJEE, 6SHIVA KARTHIK S, 7SWATI MEHTA, 8AJAI KUMAR 1,2,3,4Vishwakarma Institute of Information Technology, Pune, India 5,6,7,8Centre for Development of Advanced Computing, Pune, India E-mail: 1ria.mehta [email protected], 2dweep.pandya [email protected], 3pratik.chaudhari [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] Abstract - Named Entity Recognition (NER) is an important task in Natural Language Processing (NLP) that aims to auto identify and annotate Named Entities in the text, such as Person, Location, Organization etc. NER has been an essential component in various applications such as Information Extraction and Retrieval, Machine Translation, Question Answering (Q-A), Text Summarization etc. For NER in Hindi, while there have been a number of studies carried out, no high accuracy tool has yet been developed as per the Literature Survey. In this research, a methodology for Hindi Named Entities Recognition using NLP algorithms with RDF and Conditional Random Fields has been proposed. The results derived shows that the hybrid approach for NER achieves the recognition accuracy to 90.7% on Hindi texts. Keywords - Named Entity Recognition, Machine Learning, Natural Language Processing, CRF I. INTRODUCTION The accuracy of NER systems vary as per texts. The process of identifying Named Entities (NEs) Rule based systems have higher accuracy from a textual document and classifying them into but are not customizable different conceptual categories (Name, Place, Party, Limited tagset is considered in existing Designation) is an important step in the task of systems. Natural Language Processing (NLP). This process is Context of word is not taken into account called Named Entity Recognition (NER). In this age especially when it comes to ambiguous of World Wide Web, information is available in named entities. ( Location and Person Name abundance but in Indian Languages, there is very less being same) work done on NLP and Analytics per se. NER is Proper nouns are used as common nouns in especially a crucial proviso in applications of certain cases Information Retrieval, Machine Translation, Question No standard Gazetteer list available for Answering, Text Summarization, Efficient Search Names of people, organizations, cities, Engines etc. NER development for Indian languages states, companies, ruling parties etc. remain a challenge owning to syntactic and semantic Spelling variations complexities of Indian Languages and lack of Lack of high accuracy Part of Speech processing tools. Since Hindi is the official language Tagger of India, it has been selected for this project to In the current system, these problems are addressed develop NER tool with Hybrid NLP Ontology and by combining with indigenously built NLP algorithm rule based and Machine Learning based approach to using Linguistic rules and ontology depicted through gain better accuracy than available tools. Since Hindi RDF in tandem with the Machine Learning (ML) is part of Indo-Aryan family, the current approach model of Conditional Random Fields (CRF). RDF can be used for other Indo-Aryan languages such as has been used for creating a list of standard named Gujarati, Bengali, Marathi etc. as optimal NER tools entities compiled from various sources as cue to NLP are not present in the aforesaid languages as well. algorithm and ML. 10 tags for NER have been There are three main ways to perform NER. Using considered whereas most of the existing systems only Linguistic Rule-sets along with Ontology, Machine consider the 3 main tags- person, location, Learning or a Hybrid approach comprising both. organization. The named entities are also phrase Machine Learning has been the most successful in marked to better elicit information from the text. predicting unknown entities and Rule based systems give highest accuracy. Thus Hybrid NER systems are II. PREVIOUS WORK AND GAP ANALYSIS most efficient for Indian languages. Most common approaches to NER are Machine This system was developed to improve upon some of Learning (ML) algorithms and Hybrid of Rule-Based the issues in the current systems. Some of the issues and ML algorithms. realized are as follows: Stanford NER tagger is an existing available open- source NER tool. Stanford uses CRF as its classifier. Hindi Named Entities Recognition (NER) using Natural Language Processing and Machine Learning 59 International Journal of Advances in Electronics and Computer Science, ISSN(p): 2394-2835 Volume-6, Issue-9, Sep.-2019 http://iraj.in In the analysis and testing of this tool on 5400 Hindi They have identified 3 types of rules- Type 1: words. The precision and recall obtained were 0.45 Dictionary rule, Type 2: Bi-gram rule and Type 3: and 0.5 respectively. The F-score was 0.47. While the feature rule. From the result, it is observed that rules accuracy for English NER was 0.9223. Type 2 and Type 1 combined gives the highest Chopra, Joshi and Mathur [1] have used Hidden precision of 91.1% and recall of 67.89%. Markov Model (HMM) for NER in Hindi. As they have explained, HMM helps develop language- HMM has also been used for Gujarati language by Independent NER systems. It is also easy to scale and Vora, Vasant and Adhvaryu [6]. The paper mentions analyze these systems. An F-Measure of 97.14% was no details on the accuracy of their system. Named obtained on training data of 2343 tokens and testing Entity Recognition in Gujarati is however, rare and on 105 tokens. However, HMM has the label bias very less explored. HMM is easy to implement on problem. The size of training and testing corpus used any language thus making it desirable for Vora et al. in [1] is very limited. to use it for their NER in Gujarati language. As input Sharma and Goyal [2] considered a total of 29 they are taking printed versions of Gujarati Text. The features such as Context word, Word prefix, Word drawback they pointed out is that their system Suffix, POS information, Gazetteer Lists- list of wouldn’t recognise handwritten documents as writing person names, location names, organization names differs from person to person. etc. Overall the precision, recall and F-Score they get are 72.78%, 65.82% and 70.45% respectively. Furthermore, Das and Dhar [7] not only classify Sinha [3] focuses on disambiguating Ambiguous Named Entities like Organization, Person, Location Proper Names (APN) in Hindi using CRF. An APN is but entities have Before, Internal and End of a name which can also be used as a common noun. multiword tags as well that are represented by B-tag, In his paper, Sinha [3] also mentions the need for I-tag, E-tag. The aim is to improve the performance derivation of a relevant corpus. Subsequently, a of Question Answering, Auto Summarization, relevant corpus has been created tagging it into 3 Information Retrieval etc with a combination of POS categories namely: Names (NEP), Words in the Tagging and Entity Recognition. The Entity dictionary not used as names (NNE) and other words Recognition task was done using Conditional (OTH). A combined output of CRF and Rule-Based Random field (CRF). For training of CRF, they use where the output is essentially OR-ed gives an overall the ICON 2013 training datasets in open source using F-Score of 71.16%. the following web resource: CRF3 While surveying for NER on the Marathi Language, (https://crfsharp.codeplex.com/sourceControlilatest) we found that A. S. Patil, B. V. Pawar and N. V. Patil classifier which is written in C#. The entity set [4] address the problem of assigning correct named proposed is Name, Location, Organization, Symbol, entity class tag to each word using the Hidden Number, Date and Abbreviation. They have a Markov Model (HMM), a probabilistic one, trained maximum of 0.9156 and a minimum of 0.5714 on a manually tagged corpus for the language. precision values for symbol and abbreviation entity Proposed system in [4] reports an overall F1-Score of class respectively and maximum 0.8172 and 62.70% when no preprocessing was applied whereas minimum 0.4444 recall values for these two entity it reports an overall F1-Score of 77.79% when class on the 300 sentences randomly chosen from the preprocessing was applied on the same data. The ICON 2013 training dataset. system described [4] has recognized- the person, locations, numbers and measures well but other By combining the outputs of classifiers like named entities are not recognized satisfactorily. Maximum Entropy (ME), Conditional Random Field Reference [4] uses pre-processing techniques like (CRF) and Support Vector Machine (SVM) using a lemmatization to improve the efficiency which can be majority voting approach, Named Entity Recognition a bit costly. (NER) for the Bengali Language is done by Ekbal and Bandopadhyay [8]. They use four major NE tags: Association rule mining aims to find rules, given a set Person name, Location name, Organization name and of transactions. It helps discover relations between Miscellaneous name tags. The training set consists of variables of a large database. Jain Yadav and Tayal about 150K wordforms. Evaluation results of the [5] have a proposed a system for Hindi where for voted system for the gold standard test set of 30K every pair of itemsets A and B they find a set of rules wordforms have demonstrated the overall recall, where Support >= min (Support) and Confidence >= precision, and F-Score values of 87.11%, 83.61%, min (Confidence).
Load more

Recommended publications
  • Cs224n-2021-Lecture11-Qa.Pdf
    Lecture 11: Question Answering Danqi Chen Princeton University Lecture plan 1. What is question answering? (10 mins) Your default final project! 2. Reading comprehension (50 mins) ✓ How to answer questions over a single passage of text 3. Open-domain (textual) question answering (20 mins) ✓ How to answer questions over a large collection of documents 2 1. What is question answering? Question (Q) Answer (A) The goal of question answering is to build systems that automatically answer questions posed by humans in a natural language The earliest QA systems dated back to 1960s! (Simmons et al., 1964) 3 Question answering: a taxonomy Question (Q) Answer (A) • What information source does a system build on? • A text passage, all Web documents, knowledge bases, tables, images.. • Question type • Factoid vs non-factoid, open-domain vs closed-domain, simple vs compositional, .. • Answer type • A short segment of text, a paragraph, a list, yes/no, … 4 Lots of practical applications 5 Lots of practical applications 6 Lots of practical applications 7 IBM Watson beated Jeopardy champions 8 IBM Watson beated Jeopardy champions Image credit: J & M, edition 3 (1) Question processing, (2) Candidate answer generation, (3) Candidate answer scoring, and (4) Confidence merging and ranking. 9 Question answering in deep learning era Image credit: (Lee et al., 2019) Almost all the state-of-the-art question answering systems are built on top of end- to-end training and pre-trained language models (e.g., BERT)! 10 Beyond textual QA problems Today, we will mostly focus on how to answer questions based on unstructured text. Knowledge based QA Image credit: Percy Liang 11 Beyond textual QA problems Today, we will mostly focus on how to answer questions based on unstructured text.
    [Show full text]
  • UNIVERSITY of CALIFORNIA Los Angeles
    UNIVERSITY OF CALIFORNIA Los Angeles Constructing Diasporic Identity Through Kathak Dance: Flexibility, Fixity, and Nationality in London and Los Angeles A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Culture and Performance by Shweta Saraswat 2019 © Copyright by Shweta Saraswat 2019 ABSTRACT OF THE DISSERTATION Constructing Diasporic Identity Through Kathak Dance: Flexibility, Fixity, and Nationality in London and Los Angeles by Shweta Saraswat Doctor of Philosophy in Culture and Performance University of California, Los Angeles, 2019 Professor Anurima Banerji, Chair This dissertation focuses on the role of the classical Indian dance form Kathak in negotiating questions of cultural identity and national affiliation among members of the Indian diaspora residing in London, UK, and Los Angeles, USA. This study considers how institutional actions and discourses related to the practice of Kathak dance in these two cities and the personal experiences of dancers themselves reflect certain political, aesthetic, and social values that impact the formation of diasporic identity. The dissertation argues that Indian diasporic subjects negotiate a fundamental tension through their practice of Kathak dance: the tension between Kathak’s inherent flexibility and contextual conditions of fixity. ii As described in Chapter 1, Kathak’s inherent flexibility refers to certain foundational elements of the dance that center around creative interpretation, improvisation, and immersive practice (riyaaz), as well as the expression of multiple identities that these foundational elements enable. A discourse of Kathak’s flexibility frames the dancer’s transcendence and/or transformation of socially assigned identifications as an act of aesthetic virtuosity with cross- cultural significance.
    [Show full text]
  • Abstractive Text Summarization Using Rich Semantic Graph for Marathi Sentence
    JASC: Journal of Applied Science and Computations ISSN NO: 1076-5131 Abstractive Text Summarization using Rich Semantic Graph for Marathi Sentence Sheetal Shimpikar, Sharvari Govilkar Computer Technology Department, Mumbai University [email protected] [email protected] Abstract — Text summarization helps to find, take out important sentences from the original document and links together to construct a short and clear summary. Large text documents are difficult to summarize manually. Using the computer program, a text can be reduced to get important points; the summary obtained from it is termed as Text summarization. The objective of the work is the representation and summarization of Indian language for “Marathi” text documents using abstractive text summarization techniques. The proposed approach takes Marathi documents as input text. The first step is pre-processing of the input text. Rich semantic graph method. The challenge in doing abstractive text summarization in Marathi documents is due to the complexity because it cannot be formulated mathematically or logically. And no work has been done on Marathi language. The Rich Semantic Graph based method gives the correct, bug free result. Keywords — Text Summarization, Rich Semantic Graph, Part of Speech Tagging, Name Entity Recognition, Ontology I. INTRODUCTION In current era of digital cultures and technologies, to understand the huge amount of information, text summarization is very important method for all. The purpose of text summarization is to minimize the text from the documents into meaningful form and save important contents. Abstractive text summarization methods use language understanding tools to generate a summary. They extract phrases and lexical chains from the documents.
    [Show full text]
  • Oil and Gas News 05 Feb 2021
    ONGC News, 05.02.2021 Print Page 1 of 61 Sanmarg Satat vikas ke liya gahan sahyog ki zarurat ONGC CMD Feb 5, 2021 | Kolkata | Pg No.: 13 | | Sq Cm:385 | AVE: 1107459 | PR Value: 5537294 Page 2 of 61 The Hindu Business Line HPCL Q3 net zooms Rs215% to Rs2,354.6 cr on inventory gains Feb 5, 2021 | Delhi | Pg No.: 2 | | Sq Cm:156 | AVE: 288260 | PR Value: 1441302 Page 3 of 61 The Asian Age HPCL REPORTS RECORD PROFIT Feb 5, 2021 | Delhi | Pg No.: 7 | | Sq Cm:42 | AVE: 147965 | PR Value: 739823 Page 4 of 61 The Times of India HPCL net trebles to Rs 2,354cr in Q3 Feb 5, 2021 | Delhi | Pg No.: 19 | | Sq Cm:15 | AVE: 314541 | PR Value: 1572707 Image: 2 / 2 Page 5 of 61 Mint HPCL's Q3 net profit sees three-fold rise Feb 5, 2021 | Delhi | Pg No.: 6 | | Sq Cm:118 | AVE: 473690 | PR Value: 2368450 Page 6 of 61 Business Standard HPCL REPORTS RECORD PROFIT ON INVENTORY... Feb 5, 2021 | Delhi | Pg No.: 1,4 | | Sq Cm:141 | AVE: 349778 | PR Value: 1748890 Page 7 of 61 Business Standard HPCL REPORTS RECORD PROFIT ON INVENTORY... Feb 5, 2021 | Delhi | Pg No.: 1,4 | | Sq Cm:141 | AVE: 349778 | PR Value: 1748890 Page 8 of 61 The Economic Times HPCL Q3 Profit Triples to Rs 2,355 Crore Feb 5, 2021 | Delhi | Pg No.: 1,10 | | Sq Cm:103 | AVE: 1055098 | PR Value: 5275488 Page 9 of 61 The Economic Times HPCL Q3 Profit Triples to Rs 2,355 Crore Feb 5, 2021 | Delhi | Pg No.: 1,10 | | Sq Cm:103 | AVE: 1055098 | PR Value: 5275488 Page 10 of 61 The Financial Express HPCL profit increases 215% to Rs2,355 crore Feb 5, 2021 | Delhi | Pg No.: 4 | | Sq Cm:157 | AVE: 514707
    [Show full text]
  • Word Sense Disambiguation Using Semantic Web for Tamil to English Statistical Machine Translation Santosh Kumar T.S
    IRA-International Journal of Technology & Engineering ISSN 2455-4480; Vol.05, Issue 02 (2016) Pg. no. 22-31 Institute of Research Advances http://research-advances.org/index.php/IRAJTE Word Sense Disambiguation Using Semantic Web for Tamil to English Statistical Machine Translation Santosh Kumar T.S. Bharathiar University, India. Type of Reviewed: Peer Reviewed. DOI: http://dx.doi.org/10.21013/jte.v5.n2.p1 How to cite this paper: T.S., Santosh Kumar (2016). Word Sense Disambiguation Using Semantic Web for Tamil to English Statistical Machine Translation. IRA-International Journal of Technology & Engineering (ISSN 2455-4480), 5(2), 22-31. doi:http://dx.doi.org/10.21013/jte.v5.n2.p1 © Institute of Research Advances This work is licensed under a Creative Commons Attribution-Non Commercial 4.0 International License subject to proper citation to the publication source of the work. Disclaimer: The scholarly papers as reviewed and published by the Institute of Research Advances (IRA) are the views and opinions of their respective authors and are not the views or opinions of the IRA. The IRA disclaims of any harm or loss caused due to the published content to any party. 22 IRA-International Journal of Technology & Engineering ABSTRACT Machine Translation has been an area of linguistic research for almost more than two decades now. But it still remains a very challenging task for devising an automated system which will deliver accurate translations of the natural languages. However, great strides have been made in this field with more success owing to the development of technologies of the web and off late there is a renewed interest in this area of research.
    [Show full text]
  • Hotstar Us App Download Hotstar Us App Download
    hotstar us app download Hotstar us app download. Completing the CAPTCHA proves you are a human and gives you temporary access to the web property. What can I do to prevent this in the future? If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware. If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices. Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the Chrome Web Store. Cloudflare Ray ID: 669b589c5b8ac433 • Your IP : 188.246.226.140 • Performance & security by Cloudflare. Hotstar Android. Hotstar for Android smartphones is the application that offers us the best movies, TV series and programs, and sports broadcasts on Indian television. 1 2 3 4 5 6 7 8 9 10. Watching TV on our phone is getting more and more usual thanks to the speed modern-day connections can offer us whether through 4G data plans or WiFi networks. That's why so many apps similar to You TV Player are popping up all over the place. Free online TV from India. One of those applications is Hotstar , the audiovisual contents of which are focused on India, offering its users series, movies, television programs, and live or pre-recorded sports broadcasts . Once you download its APK, you'll be able to appreciate that it comes along with an eye-catching and intuitive design that allows us to play certain contents via streaming and also download them to be able to watch them offline.
    [Show full text]
  • Question Answering System Using Ontology in Marathi Language
    International Journal of Artificial Intelligence and Applications (IJAIA), Vol.8, No.4, July 2017 QUESTION ANSWERING SYSTEM USING ONTOLOGY IN MARATHI LANGUAGE Sharvari S. Govilkar 1 and J. W. Bakal 2 1Department of Computer Engineering, PCE, Mumbai, India 2Department of Computer Engineering, SJCOE, Mumbai, India ABSTRACT Humans are always in a quest to extract information related to some topic or entity. Question answering system helps user to find the precise answer of the question articulated in natural language. Question answering system provides explicit, concise and accurate answer to user questions rather than providing set of relevant documents or web pages as answers as most of the information retrieval system does. The paper proposes question answering system for Marathi natural language by using concept of ontology as a formal representation of knowledge base for extracting answers. Ontology is used to express domain specific knowledge about semantic relations and restrictions in the given domains. The ontologies are developed with the help of domain experts and the query is analyzed both syntactically and semantically. The results obtained here are accurate enough to satisfy the query raised by the user. The level of accuracy is enhanced since the query is analyzed semantically. KEYWORDS Question answering system (QAS), Ontology, Marathi Natural language QA system (NLQA), Natural language processing (NLP) 1. INTRODUCTION With the rapid growth of the amount of online and electronic documents in Indian regional language, the keyword based approaches lack many important elements to enable QA driven process. So a system is required which can provide user with accurate answers for their queries .Question answering system provides user with functionality where they can ask questions in natural language and the system returns answer which is most accurate and precise of all the possible answers for the given input question.
    [Show full text]
  • Danqi Chen Princeton University Lecture Plan
    Lecture 11: Question Answering Danqi Chen Princeton University Lecture plan 1. What is question answering? (10 mins) Your default final project! 2. Reading comprehension (50 mins) ✓ How to answer questions over a single passage of text 3. Open-domain (textual) question answering (20 mins) ✓ How to answer questions over a large collection of documents 2 1. What is question answering? Question (Q) Answer (A) The goal of question answering is to build systems that automatically answer questions posed by humans in a natural language The earliest QA systems dated back to 1960s! (Simmons et al., 1964) 3 Question answering: a taxonomy Question (Q) Answer (A) • What information source does a system build on? • A text passage, all Web documents, knowledge bases, tables, images.. • Question type • Factoid vs non-factoid, open-domain vs closed-domain, simple vs compositional, .. • Answer type • A short segment of text, a paragraph, a list, yes/no, … 4 Lots of practical applications 5 Lots of practical applications 6 Lots of practical applications 7 IBM Watson beated Jeopardy champions 8 IBM Watson beated Jeopardy champions Image credit: J & M, edition 3 (1) Question processing, (2) Candidate answer generation, (3) Candidate answer scoring, and (4) Confidence merging and ranking. 9 Question answering in deep learning era Image credit: (Lee et al., 2019) Almost all the state-of-the-art question answering systems are built on top of end- to-end training and pre-trained language models (e.g., BERT)! 10 Beyond textual QA problems Today, we will mostly focus on how to answer questions based on unstructured text. Knowledge based QA Image credit: Percy Liang 11 Beyond textual QA problems Today, we will mostly focus on how to answer questions based on unstructured text.
    [Show full text]