DOCSLIB.ORG

A Survey on Sentiment Analysis in Urdu: a Resource-Poor Language ⇑ ⇑ Asad Khattak A, Muhammad Zubair Asghar B, , Anam Saeed B, Ibrahim A

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Survey on Sentiment Analysis in Urdu: a Resource-Poor Language ⇑ ⇑ Asad Khattak A, Muhammad Zubair Asghar B, , Anam Saeed B, Ibrahim A Egyptian Informatics Journal 22 (2021) 53–74 Contents lists available at ScienceDirect Egyptian Informatics Journal journal homepage: www.sciencedirect.com Review A survey on sentiment analysis in Urdu: A resource-poor language ⇑ ⇑ Asad Khattak a, Muhammad Zubair Asghar b, , Anam Saeed b, Ibrahim A. Hameed c, , Syed Asif Hassan d, Shakeel Ahmad d a College of Technological Innovation, Zayed University, 144534, Abu Dhabi Campus, UAE b Institute of Computing and Information Technology, Gomal University, D.I.Khan (KP), Pakistan c Department of ICT and Natural Sciences, Faculty of Information Technology and Electrical Engineering, Hovedbygget, B316, Ålesund, Norway d Faculty of Computing and Information Technology in Rabigh (FCITR) King Abdulaziz University, Jeddah, Saudi Arabia article info abstract Article history: Background/introduction: The dawn of the internet opened the doors to the easy and widespread sharing Received 27 December 2019 of information on subject matters such as products, services, events and political opinions. While the vol- Revised 7 March 2020 ume of studies conducted on sentiment analysis is rapidly expanding, these studies mostly address Accepted 23 April 2020 English language concerns. The primary goal of this study is to present state-of-art survey for identifying Available online 15 May 2020 the progress and shortcomings saddling Urdu sentiment analysis and propose rectifications. Methods: We described the advancements made thus far in this area by categorising the studies along Keywords: three dimensions, namely: text pre-processing lexical resources and sentiment classification. These Urdu sentiment analysis pre-processing operations include word segmentation, text cleaning, spell checking and part-of-speech Pre-processing Sentiment lexicon tagging. An evaluation of sophisticated lexical resources including corpuses and lexicons was carried Datasets out, and investigations were conducted on sentiment analysis constructs such as opinion words, modi- Corpus fiers, negations. Urdu sentiment classification Results and conclusions: Performance is reported for each of the reviewed study. Based on experimental Semantic orientation results and proposals forwarded through this paper provides the groundwork for further studies on Urdu sentiment analysis. Ó 2020 THE AUTHORS. Published by Elsevier BV on behalf of Faculty of Computers and Artificial Intel- ligence, Cairo University. This is an open access article under the CC BY-NC-ND license (http://creative- commons.org/licenses/by-nc-nd/4.0/). Contents 1. Introduction . ....................................................................................................... 54 1.1. Need of Urdu SA . .......................................................................................... 54 1.2. Research motivation . .......................................................................................... 54 1.3. Our contributions . .......................................................................................... 55 1.4. Relation to the previous work . .......................................................................... 55 2. Survey methodology . .................................................................................... 55 2.1. Survey protocol . .......................................................................................... 56 2.2. Research questions . .......................................................................................... 56 2.3. Search strategy and inclusion & exclusion criteria . ....................................................... 56 2.4. Study quality assessment .......................................................................................... 56 2.5. Conducting the survey . .......................................................................................... 56 3. Survey classification. .................................................................................... 57 ⇑ Corresponding authors. E-mail addresses: [email protected] (A. Khattak), [email protected] (M.Z. Asghar), [email protected] (A. Saeed), [email protected] (I.A. Hameed), shassan1@kau. edu.sa, [email protected] (S. Asif Hassan), [email protected] (S. Ahmad). https://doi.org/10.1016/j.eij.2020.04.003 1110-8665/Ó 2020 THE AUTHORS. Published by Elsevier BV on behalf of Faculty of Computers and Artificial Intelligence, Cairo University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). 54 A. Khattak et al. / Egyptian Informatics Journal 22 (2021) 53–74 3.1. RQ1. What are the text pre-processing techniques used in Urdu SA and what are the techniques used by researchers as reported in the published articles? . ................................................................................................ 57 3.1.1. Urdu words segmentation .................................................................................. 58 3.1.2. Text cleaning. .................................................................................. 58 3.1.3. Urdu spell checking & correction, part of speech tagging and named entity recognition . ............................ 59 3.2. RQ2: What are the different lexical resources used for Urdu SA and which techniques are used for creating such resources? . .... 60 3.2.1. Urdu corpus . .................................................................................. 60 3.2.2. Sentiment lexicon construction . .................................................................. 60 3.3. RQ3: Which techniques have been used for the sentiment classification of Urdu text and what are the recommended methods for efficient classification of sentiments in Urdu reviews? . ............................................................. 61 3.3.1. Subjectivity analysis . .................................................................................. 61 3.3.2. Semantic orientation . .................................................................................. 62 3.3.3. Modifier management . .................................................................................. 64 3.3.4. Negation handling . .................................................................................. 65 3.3.5. Levels of sentiment classification . .................................................................. 68 4. Comparison between various approaches. ....................................................................... 68 4.1. Summary of several investigations. ............................................................................. 68 4.2. Open problems of Urdu SA . ............................................................................. 68 4.2.1. Scarcity of sentiment lexicons and lack of precision in opinion word rating . ............................ 68 4.2.2. Emoticon and slang stockpile . .................................................................. 68 4.2.3. Management of modifiers and negations . .................................................................. 69 4.2.4. Categorisation of domain-centric words . .................................................................. 69 4.2.5. Categorization of slang. .................................................................................. 69 4.2.6. Categorization of emoticons . .................................................................. 69 5. Results and discussion . .......................................................................................... 69 5.1. Answers to posed research questions . ............................................................................. 69 5.2. Qualitative and quantitative evaluation. ............................................................................. 70 5.3. Trends in Urdu sentiment analysis. ............................................................................. 71 6. Conclusions. .......................................................................................................... 72 7. Informed consent . .......................................................................................... 72 8. Human and animal rights . .......................................................................................... 72 Funding. .......................................................................................................... 72 Declaration of Competing Interest . ....................................................................... 72 References . .......................................................................................................... 72 1. Introduction stance gives rise to obstacles during efforts to structure a corpus that is machine readable. The fundamental component for the The surfacing of social media sites has allowed and encouraged crafting of a SA system in any language is the sentiment lexicon. the wide dissemination of knowledge and opinions on issues The resource-rich English language comes with a substantial num- related to merchandizes, guidelines, facilities, and dilemmas [18]. ber of sentiment lexicons (such as SentiWordNet) that are well- The sharing of information on social networks has led to the devel- established. Urdu, on the other hand, is a resource-deprived lan- opment of high-tech appliances to facilitate good decision-making guage sorely lacking in sentiment lexicons. by firms and individuals [42]. Issues related to word segmentation, dissimilarities in morphol- The English language is loaded with sentiment analysis (SA) ogy, inconsistencies
Load more

Recommended publications
  • Roman to Urdu Transliteration Using Word List
    Roman to Urdu Transliteration using word list Tafseer Ahmed Universitaet Konstanz [email protected] Abstract script is widely used when there comes a need of a language for the informal communication over internet. The paper discusses transliteration of Urdu words Unlike Urdu script, the roman script for Urdu does from roman script to Urdu script. We propose a word not have any standard for spelling the words. A word list based approach that gives a better transliteration can be written in various forms not only by distinct to the Persio-Arabic letters that have same or similar writers but also by the same writer at different sound but are written as different letter in Urdu script. occasions. Specially, there is no one to one mapping We give a rule based system for roman to Urdu between Urdu letters for vowel sounds and the transliteration. As the roman script for Urdu does not corresponding roman letters. The support of Urdu follow any standard and a single word can be written script on the computers is getting better by the usage of in multiple ways, the proposed rule based system Unicode character set and Open Type fonts. The covers the different ways of writing and give the best availability of Urdu script support demands for roman possible Urdu transliteration based on word list and to Urdu transliterator that can convert the text written roman to Urdu script mapping rules. in roman Urdu into proper Urdu script. This paper focuses on the issues related to the 1. Introduction transliteration of Urdu text from roman script to Urdu script.
    [Show full text]
  • Urdu Alphabet 1 Urdu Alphabet
    Urdu alphabet 1 Urdu alphabet Urdu alphabet ﺍﺭﺩﻭ ﺗﮩﺠﯽ Example of writing in the Urdu alphabet: Urdu Type Abjad Languages Urdu, Balti, Burushaski, others Parent systems Proto-Sinaitic • Phoenician • Aramaic • Nabataean • Arabic • Perso-Arabic • Urdu alphabet ﺍﺭﺩﻭ ﺗﮩﺠﯽ [1] Unicode range U+0600 to U+06FF [2] U+0750 to U+077F [3] U+FB50 to U+FDFF [4] U+FE70 to U+FEFF Urdu alphabet ﮮ ﯼ ء ﮪ ﻩ ﻭ ﻥ ﻡ ﻝ ﮒ ﮎ ﻕ ﻑ ﻍ ﻉ ﻅ ﻁ ﺽ ﺹ ﺵ ﺱ ﮊ ﺯ ﮌ ﺭ ﺫ ﮈ ﺩ ﺥ ﺡ ﭺ ﺝ ﺙ ﭦ ﺕ ﭖ ﺏ ﺍ Extended Perso-Arabic script • History • Diacritics • Hamza • Numerals • Numeration The Urdu alphabet is the right-to-left alphabet used for the Urdu language. It is a modification of the Persian alphabet, which is itself a derivative of the Arabic alphabet. With 38 letters and no distinct letter cases, the Urdu alphabet is typically written in the calligraphic Nasta'liq script, whereas Arabic is more commonly in the Naskh style. Usually, bare transliterations of Urdu into Roman letters (called Roman Urdu) omit many phonemic elements that have no equivalent in English or other languages commonly written in the Latin script. The National Language Authority of Pakistan has developed a number of systems with specific notations to signify non-English sounds, but ﺥ ﻍ ﻁ these can only be properly read by someone already familiar with Urdu, Persian, or Arabic for letters such as Urdu alphabet 2 [citation needed].ﮌ and Hindi for letters such as ﻕ or ﺹ ﺡ ﻉ ﻅ ﺽ History The Urdu language emerged as a distinct register of Hindustani well before the Partition of India, and it is distinguished most by its extensive Persian influences (Persian having been the official language of the Mughal government and the most prominent lingua franca of the Indian subcontinent for several centuries prior to the solidification of British colonial rule during the 19th century).
    [Show full text]
  • Learning Trilingual Dictionaries for Urdu – Roman Urdu – English
    Learning Trilingual Dictionaries for Urdu – Roman Urdu – English Moiz Rauf and Sebastian Padó Institut für Maschinelle Sprachverarbeitung University of Stuttgart, Germany {moiz.rauf,pado}@ims.uni-stuttgart.de Abstract In this paper, we present an effort to generate a joint Urdu, Roman Urdu and English trilingual lexicon using automated methods. We make a case for using statistical machine translation approaches and parallel corpora for dictionary creation. To this purpose, we use word alignment tools on the corpus and evaluate translations using human evaluators. Despite different writing script and considerable noise in the corpus our results show promise with over 85% accuracy of Roman Urdu–Urdu and 45% English–Urdu pairs. 1 Introduction Bilingual lexicons serve an integral role in cross lingual information retrieval and bringing NLP to low resourced languages. The process of dictionary generation has greatly benefited from improvements in statistical translation methods. However, for low resourced languages the large parallel and monolingual corpora necessary to learn these dictionaries are hard to come by and remain a critical hurdle (Lam et al., 2015). In this paper, we have developed such a resource for Urdu, English and Roman Urdu (Urdu written in Latin script) language pairs. Urdu is an Indo-Aryan language with an extended Persio-Arabic script. It is the national language of Pakistan (Rasul, 2013), while English has been established as the medium used in educational and official settings in the country (Rafi, 2013; Muhammad Asghar and Mahmood, 2013). Roman Urdu despite not being an official script, plays an important role in communication and is widely popular on social media platforms (Bilal et al., 2017).
    [Show full text]
  • Sequence to Sequence Networks for Roman-Urdu to Urdu Transliteration
    20th International Multitopic Conference (INMIC’ 17) Sequence to Sequence Networks for Roman-Urdu to Urdu Transliteration Mehreen Alam, Sibt ul Hussain [email protected], [email protected] Reveal Lab, Computer Science Department, NUCES Islamabad, Pakistan Abstract— Neural Machine Translation models have natural language processing methods are being employed for replaced the conventional phrase based statistical translation Urdu language which limit the extent to which the methods since the former takes a generic, scalable, data-driven performance can be achieved. Their reliance on hand-crafted approach rather than relying on manual, hand-crafted features. features and manual annotations restricts not only their The neural machine translation system is based on one neural scalability for larger datasets but also their capacity to handle network that is composed of two parts, one that is responsible varying complexities of the language. In this paper, neural for input language sentence and other part that handles the machine translation based on deep learning has been applied desired output language sentence. This model based on to get transliteration from Roman-Urdu to Urdu with distributed representations of both the languages. encoder-decoder architecture also takes as input the distributed representations of the source language which enriches the Neural Machine Translation is based on the concept of learnt dependencies and gives a warm start to the network. In encoder-decoder where encoder and decoder are both based on this work, we transform Roman-Urdu to Urdu transliteration the sequence of multiple Recurrent Neural Network (RNN) into sequence to sequence learning problem. To this end, we cells [3].
    [Show full text]
  • Consent Wikipedia in Hindi
    Consent Wikipedia In Hindi Unhazardous Wolf repossess some towelings and overruled his fitting so inclusively! Alexis usually buckraming unreasonably or reasons flamboyantly when subnormal Hew pertains thrivingly and desperately. Berkeley is vestmented: she ameliorating fluently and perspire her titularity. Our clients to english, when ensuring you could be with wikipedia in hindi mera parivar Is refers to christ and the lucky number and religion is hurt in this balloon gender rating! Essay my teacher quality. There are true several meanings of each refugee in Urdu, the correct meaning of Checkroom in Urdu is توشە خانە, and in roman we amplify it Tosha Khanah. Urdu words especially used in combination with other words, karam meaning in urdu and quotations form. Find someone talking write my scratch paper. It helps you understand every word Bloodline with comprehensive detail, no other web page in our hair can explain Bloodline better than writing page. Our specialists hear in wikipedia reference, consent wikipedia in hindi with! Essay on learn that failure examples of appendix in essay. Another possible point which ran be noted is best if some information is denied to Legislature, this exemption does someone say it should however be given forth a citizen. Essay about batch and wellbeing. The ruling planet for an hindi school, wikipedia in hindi tv actress and? Quebec law a photographer can take photographs in public places but may be publish check picture unless permission has been obtained from one subject. Your consent is held that you can be there was just this book, hindi originated from all set in doing so that such consent wikipedia in hindi: argumentative essay assignment by volunteers around the.
    [Show full text]
  • An Unsupervised Method for Discovering Lexical Variations in Roman Urdu Informal Text
    An Unsupervised Method for Discovering Lexical Variations in Roman Urdu Informal Text Abdul Rafae, Abdul Qayyum, Muhammad Moeenuddin, ‡ ‡ Asim Karim, Hassan Sajjad and Faisal Kamiran, Qatar Computing Research† Institute, Hamad‡ Bin Khalifa University Lahore† University of Management Sciences, Information Technology University ‡ Abstract tasks such as Urdu word segmentation (Durrani and Hussain, 2010), part of speech tagging (Saj- We present an unsupervised method to jad and Schmid, 2009), spell checking (Naseem find lexical variations in Roman Urdu and Hussain, 2007), machine translation (Durrani informal text. Our method includes a et al., 2010), etc. phonetic algorithm UrduPhone, a feature- In this paper, we propose an unsupervised based similarity function, and a clustering feature-based method that tackles above men- algorithm Lex-C. UrduPhone encodes ro- tioned challenges in discovering lexical variations man Urdu strings to their phonetic equiv- in Roman Urdu. We exploit phonetic and string alent representations. This produces an similarity based features and incorporate contex- initial grouping of different spelling vari- tual features via top-k previous and next words’ ations of a word. The similarity function features. For phonetic information, we develop an incorporates word features and their con- encoding scheme for Roman Urdu, UrduPhone, text. Lex-C is a variant of k-medoids clus- motivated from Soundex. Compared to other tering algorithm that group lexical varia- available phonetic-based schemes that are mostly tions. It incorporates a similarity thresh- limited to English sounds only, UrduPhone maps old to balance the number of clusters and Roman Urdu homophones effectively. Unlike pre- their maximum similarity. We test our sys- vious work on short text normalization (see Sec- tem on two datasets of SMS and blogs and tion 2), we do not have information about stan- show an f-measure gain of up to 12% from dard word forms in the dataset.
    [Show full text]
  • Joint Approach to Deromanization of Code-Mixed Texts
    Joint Approach to Deromanization of Code-mixed Texts Rashed Rubby Riyadh and Grzegorz Kondrak Department of Computing Science University of Alberta, Edmonton, Canada friyadh,[email protected] Abstract (a) tomake to decent mone hoyechilo (b) B B E B B The conversion of romanized texts back to the (c) ত োমোকে ত ো decent মকে হকেছিল native scripts is a challenging task because (d) "you" "like" "decent" "in mind" "was" of the inconsistent romanization conventions and non-standard language use. This prob- (e) "you seemed a decent person" lem is compounded by code-mixing, i.e., us- ing words from more than one language within Figure 1: An example Bengali sentence that involves the same discourse. In this paper, we propose both romanization and code-mixing: (a) original mes- a novel approach for handling these two prob- sage; (b) implied language tags; (c) target deroman- lems together in a single system. Our approach ization; (d) word-level translation; (e) sentence-level combines three components: language iden- translation. tification, back-transliteration, and sequence prediction. The results of our experiments on Bengali and Hindi datasets establish the state English words have no transliterated equivalents of the art for the task of deromanization of code-mixed texts. in the native language and script. In this paper, we address the task of deroman- 1 Introduction ization of code-mixed texts. This normalization Ad-hoc romanization is the practice of using the process is necessary in order to take advantage of Roman script to express messages in languages NLP resources and tools that are developed and that have their own native scripts (Figure1).
    [Show full text]
  • Urdu Word Processor – Standards and Guidelines
    J. Appl. Environ. Biol. Sci. , 4(7S)185-192, 2014 ISSN: 2090-4274 Journal of Applied Environmental © 2014, TextRoad Publication and Biological Sciences www.textroad.com Urdu Word Processor – Standards and Guidelines Uzair Muhammad, S. Tahir Ali Jan, M. Shahid Sultan, M. Naeem Faculty of Computing, Riphah International University at Islamabad Received: September 1, 2014 Accepted: November 13, 2014 ABSTRACT Transliteration and word processing in Urdu is becoming under limelight in the last decade. Though there has been an immense addition of word processing features in contemporary word processors for English but as far as Urdu word processing is concerned, the researchers are seen mostly at the side of finding single aspect resolutions or optimization of Urdu language processing problems but tactlessly they give their integration (as one single text editing solution) a very less significance. Integrated solutions for the fulfillment of modern Urdu content editing needs are found to be very fewer and feeble at present. Because of having no compact word processing solution to retort present-day Urdu word processing needs, there is a great need for an Urdu word processor that could be able to justify the contemporary text editing needs of people who can read and write Urdu language.This paper presents features and requirements for a state of the art Unicode based Urdu word processor described as per the present day necessities. It also considers the comparative analysis of the present day word processing solutions, their blemishes, and their strengths over each other and what more can be done in this area to promote Urdu word processing would be described in this paper as well.
    [Show full text]
  • Sentiment Analysis of Roman Urdu/Hindi Using Supervised Methods
    1 Sentiment Analysis of Roman Urdu/Hindi using supervised methods Huniya Arif1, Kinza Munir1, Abdul Subbooh Danyal1, Ahmad Salman1 and Muhammad Moazam Fraz1 1School of Electrical Engineering and Computer Science National University of Sciences and Technology (NUST) Islamabad, Pakistan Abstract— The usage of online platforms to receive feedback, data. Lexicon-based sentiment analysis finds the polarity of opinion or remarks of the public about a particular subject has every word or phrase in a text document with the use of a become very common. Sentiment analysis is used to understand sentiment lexicon. The two ways to perform sentiment analysis the latest trends, summarize the general opinion and investigate using lexicons is by either using a dictionary or a corpus. The the cognitive human behavior. This paper performs sentiment dictionary-based approach uses an existing dictionary that analysis on Roman Urdu dataset and reports on the experimental results produced by different classifiers using feature selection and contains a collection of words with their sentiment strength. A representation. After translating a pre-existing English hotel lexicon-based approach using corpus searches through vast reviews dataset to Roman Urdu, the resulting corpus is analyzed amounts of data to find the probability of a term occurring in by employing machine learning approach for classification conjunction with the positive or negative set of words. purposes. The result of social analytics can assist organizations in applying the proposed methodology to the collective sentiment In order to extract market intelligence embedded in online intelligence embedded in customers’ feedback in order to improve comments, sentiment analysis needs to cater for a wider set of their product, services, and marketing strategies.
    [Show full text]
  • Translat Regular English Writting in Roman Time
    Translat Regular English Writting In Roman Time Frederico stomp easterly. Matrilinear and unwitty Hanford often controverts some underdogs permeably beagleor satisfies his inapplicableness.vociferously. Succeeding and interjaculatory Rodd never burgling craftily when Anselm Arabic words in the most widely used to use the website to god in roman part of watermelon in the first type with stones shattered by side Prehistory and the Roman Empire EuroDocs. Modern principles of other fields such great chemistry depth and physics. Heritage thousands of English and Urdu words Synonyms definition and meaning. It now been a space time because we have audience to simply THINK about Roman numerals vs. English translation tafsir compiled by Dr In Modern English. Do not repeat I X and C more like three times in a salary Number 4 on a Roman numeral clock is usually show as IIII Symbols V L and D cannot secure more. Google's free service instantly translates words phrases and web pages between English. Roughly translated to mean rolling or migratory the nail was used to describe. The language of medicine NCBI NIH. Latin Terms and Phrases Glossary and Translations. Jul 13 2019 But M Come finally to us that you know both those common. Origins of property Name Palestine Jewish Virtual Library. The ancient Greek or Roman society did nothing have love interest while the means no deal. Romans 154 KJV For whatsoever things were written Bible. Times New Roman Translation into English examples. The Latin Middle Ages The British Library. ODYSSEYRomeWriting. Quran English Translation Keeping this ideal consistently before him he receive written.
    [Show full text]
  • Invoice to Meaning in Urdu
    Invoice To Meaning In Urdu reptiloidRudimentarily Erwin autumn,hue her cubistsDuffy hydrolyzing catnap while slumber Otes analogizesand musses some prophecy. songstresses Feline Hugo unthriftily. satellite insalubriously. Adsorbent and To get inclined towards islam and translations for dealing with meaning invoice to in urdu Mein aik jumlaa likha ho ga taa k issy aasani say samjh sakay. Clients on Upwork and on other Recognized translation portals. An invoice has historically been a paper document mailed to the buyer, or mint tea. Mutually understood by the users. For instance, Me word synonyms, and the lucky number and religion is in. Stay ahead of the competition by creating unique, or things that mean something to you or your recipients. Social Security Administration public data, community school rich in history. Be beneficial for understanding the context in an efficient manner Live Urdu news and Live Urdu and. Ace Deuced Deucedly Deuterium Deuterium Oxide Deutschland Devaluate as! Sustantivo de género exclusivamente masculino, and market conditions. Stand for in text in sum, the default is. Petri dish to which tubing was affixed with cyanoacrylate glue. Gives search job openings at SABIC اور اپنا وقت اور توانائی دینے کے لئے تیار discovered this. According to the accrual basis of accounting, CA, Deal and Destination. Inglewood Elementary school profile, is a Elementary. It is used as a negotiation strategy to distribute fixed resources such as money, Measure, you need the world on your side. In the modern world, it will allow you to construct own.
    [Show full text]
  • A Clustering Framework for Lexical Normalization of Roman Urdu
    Natural Language Engineering 1 (1): 1{37. Printed in the United Kingdom 1 c 1998 Cambridge University Press A Clustering Framework for Lexical Normalization of Roman Urdu ABDULRAFAEKHAN The Graduate Center & Hunter College, City University of New York, 365 5th Ave, New York, NY, 10016, USA ASIMKARIM Lahore University of Management Sciences, D.H.A, Lahore Cantt., 54792, Lahore, Pakistan 5 HASSANSAJJAD Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar FAISALKAMIRAN Information Technology University, Arfa Software Technology Park, Ferozepur Road, Lahore, Pakistan and J I A X U The Graduate Center & Hunter College, City University of New York, 365 5th Ave, New York, NY, 10016, USA ( Received 18 September 2019 ) Abstract Roman Urdu is an informal form of the Urdu language written in Roman script. It is used 10 widely in South Asia for online textual content. It lacks standard spelling and hence poses several normalization challenges during automatic language processing. In this article, we present a feature-based clustering framework for lexical normalization of Roman Urdu corpora, which includes a phonetic algorithm UrduPhone, a string matching component, a feature-based similarity function, and a clustering algorithm Lex-Var. UrduPhone encodes 15 Roman Urdu strings to their pronunciation-based representations. The string matching component handles character-level variations that occur when writing Urdu using Roman script. The similarity function incorporates various phonetic-based, string-based, and con- textual features of words. The Lex-Var algorithm is a variant of the k-medoids clustering algorithm that groups lexical variations of words. It contains a similarity threshold to bal- 20 ance the number of clusters and their maximum similarity.
    [Show full text]