DOCSLIB.ORG

Metasoundex Phonetic Matching for English and Spanish

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Metasoundex Phonetic Matching for English and Spanish Global Journal of Enterprise Information System DOI: 10.18311/gjeis/2018/19822 MetaSoundex Phonetic Matching for English and Spanish K. Koneru* and C. Varol Department of Computer Science, Sam Houston State University, TX 77341, USA; [email protected], [email protected] Abstract Researchers confront major problems while searching for various kinds of data in large imprecise databases, as they are not spelled - gle, pronunciation of words was considered as one of the practices to solve the problem effectively. The technique used to acquire correctly or in the way they were expected to be spelled. As a result, they cannot find the word they sought. Over the years of strug Metaphone, Caverphone, DMetaphone, Phonex etc., are also used for information retrieval in different environments. The main con- words based on sounds is known as “Phonetic Matching”. Soundex was the first algorithm developed and other algorithms such as and Spanish languages. The newly developed MetaSoundex algorithm addresses the limitations of well-known phonetic matching tribution of this paper is to analyze and implement the newly proposed MetaSoundex algorithm for fixing ill-defined data in English Soundex and Metaphone algorithms and has higher precision compared to Soundex, thus reducing the noise in the considered arena. techniques, Metaphone and Soundex. Specifically, the new algorithm provided results that are more accurate compared to both Keywords: Information Retrieval, Metaphone, Metasoundex, Misspelled Words, Phonetic Matching, Soundex Paper Code (DOI): 19822; Originality Test Ratio: 19%; Submission Online: 23-Jan-2018; Manuscript Accepted: 30-Jan-2018; Originality Check: 03-Feb-2018; Peer Reviewers Comment: 13-Feb-2018; Double Blind Reviewers Comment: 20-Mar-2018; Author Revert: 24-Mar-2018; Camera-Ready-Copy: 27-Mar-2018; Editorial Board Excerpt: 30-Mar-2018. Editorial Board Excerpt: Initially at the Time of Submission (ToS) submitted paper had a 19% plagiarism which is an accepted percent- age for publication. The editorial board is of an observation that paper had a subsequent surveillance by the blind reviewer’s which at a later stages had been rectified and amended by an authors ( Koneru and Varol ) in various phases as and when required to do so. The reviewer’s had in an initial stages comment with minor revision with a following remark which at a short span restructured by an authors. The comments related to this manuscript is extremely noticeable both subject-wise and research wise by the reviewers during evaluation and further at blind review process too. All the comments had been shared at a variety of dates by the authors’ in due course of time and same had been integrated by the author in addition. By and large all the editorial and reviewer’s comments had been incorporated in this very paper at the end and further the paper had been earmarked and decided under “Empirical Research Paper” category as its highlights and emphasize the pure and first hand information in relation to MetaSoundex Phonetic Matching for English and Spanish. 1. Introduction Soundex was the naive algorithm proposed and other algo- rithms like Metaphone, Caverphone, DMetaphone, Phonex etc., Information deterioration is an intensive problem for organi- are also used for retrieving nearest matches for misspelled data. zations in the present era. With the increase in the amount of As per the research15, it was clearly observed that there is no con- information saved day by day, there is a desperate need for locat- crete technique for retrieving nearest matches. Soundex has high ing the mistyped data. Organizations are facing great challenge accuracy than other algorithms but has huge overhead due to its to maintain the quality of data in information systems with vari- high false positives. Metaphone has high efficiency in spite of its ous sources of data damage. Whenever the data is assimilated low accuracy, due to its low overhead. Hence, this paper mainly from multiple sources, it is a challenge to recognize the duplicate involves the proposal, implementation, and analysis of a hybrid information due to the existence of misspelled data for the same algorithm, MetaSoundex. It is observed that MetaSoundex has an record. As a result, the information of organization always ends accuracy of 84.5% for a real-life dataset, which is improved over up at risk. To address these challenges, techniques such as string Soundex (80%) and Metaphone (58%). matching, phonetic matching, and data linkage have been used. The rest of this paper is structured as follows. Differences Apart from other techniques that depend on variations in letters, between string matching and phonetic matching are discussed phonetic matching is mainly contingent on variations in sound in next section. Section three describes background of pho- to identify the misspelled data. As a result, the misspelled data netic matching algorithms. Section four explains in detail about from multilingual sources can also be identified using phonetic Soundex, Metaphone, and MetaSoundex, which is the initial matching. contribution of this paper. Section five defines the evaluation *Author for correspondence MetaSoundex Phonetic Matching for English and Spanish Table 1. String matching vs phonetic matching26,29 String Matching Phonetic Matching Matching Matches data based on patterns of substrings Matches data based on the similar pronunciations Involves Addition, Deletion or Substitution of Letters Conversion of data to phonetic patterns Applications Applied in Search Engines, Bio-Informatics, Used in name retrieval in enquiry lines, record linkage and fraud spell checkers, digital forensics etc. detection. Gaining its importance in spell checkers and; search engines. Prominence Mainly used for matching names and nouns Can be used in multi-lingual environment, where diversities in from English Language pronunciation or writing styles may be present. The study in this paper particularly focuses on phonetic matching because: a. It is not explored as much as string matching and still relies on old techniques. b. Of increase in the voice-to-text translation applications, where phonetic matching plays a crucial role. metrics and describes the experimental setup used in this study. 3.1 Evolution of Phonetic Matching The main contribution of this paper is presented in section six in Algorithms which the results of newly proposed MetaSoundex are analyzed and compared with the existing algorithms. Finally, this paper is The evolution of phonetic matching has come into frame when 5 concluded and future work is pointed out in section seven. there is a hardship in the retrieval of information . The main goal of phonetic matching algorithms is to encode homophones to the same representation so that they can be matched despite of minor 2. String Matching vs Phonetic differences in spelling1,10. The technique of obtaining words using Matching sounds was used in the US census since the late 1890’s, but a con- crete solution to this was first proposed and patented by Robert Phonetic matching is one of the important techniques that C. Russell in 1912 as Soundex algorithm27. The background of plays a major role in variety of fields such as digital investiga- various phonetic matching algorithms is discussed. tion involving voice memos and voice mails, transcriptions, and voice apps such as ‘Siri’, ‘Google’ voice etc.3 to provide suggestions 3.1.1 Soundex for misspelled words. Phonetic comparison meticulously identi- The earliest algorithm in the literature is Soundex developed by fies the words that are most likely to sound similar. It obtains Odell and Robert C. Russell in 1912, which generates a four-digit the quantitative analysis of pronunciations32 between speech code retaining its first letter. The authors patented the algorithm forms and spellings of words, whereas, string matching mainly in 191822. The process mainly encodes consonants while a vowel involves insertion, deletion, and substitution of letters to find the is not encoded unless it is the first letter. Arguably, Soundex is near matches26,29. Table 1 describes the most common differences one of the most widely known of all phonetic algorithms. It is between string matching and phonetic matching. used as a standard feature in applications like mySQL, oracle, etc. Because of the few disadvantages like dependency on the 3. Phonetic Matching Algorithms first letter, failure of detection of silent consonants, limit to the four characters of encoding, and high overhead in the retrieved Information retrieval is one of the major viewpoints of data mining matches, Soundex can only be used in applications where high 28 application areas . However, the information may not be consist- false positives and false negatives can be tolerated27. ent over the considered arena due to various causes. The different sources of variations can be spelling variations (typographical 3.1.2 Beider-Morse Phonetic Matching (BMPM) errors, substituted letters or by addition or omission of letters), phonetic variations (discrepancies in phonetic structure of words), Beider and Morse implemented an improvement to Soundex to double names or double first names (names having more than one reduce the number of false positives and false negatives, known as word), change of name27 (individual undergoes change of name). Beider-Morse Phonetic Matching (BMPM). Beider, et al, has also Of the different criteria mentioned above, the research in mentioned that the algorithm is extended to languages other than phonetic variations led to the development of phonetic matching English, with the application of some generic rules to obtain the 5 algorithms, which obtains worthwhile approximate matches to phonetic codes . Varol, et al, discussed BMPM as a hybrid tech- the misspelled words. nique with a 6-letter encoded code in which the percentage of 2 Vol 10 | Issue 1 | January-March 2018 | www.informaticsjournals.com/index.php/gjeis GJEIS | Print ISSN: 0975-153X | Online ISSN: 0975-1432 K. Koneru and C. Varol Empirical Research Paper irrelevant matches can be abated by 70%33. A set of tables repre- of the word.
Load more

Recommended publications
  • Phonetic Matching Toolkit with State-Of-The-Art Meta-Soundex
    PHONETIC MATCHING TOOLKIT WITH STATE-OF-THE-ART META-SOUNDEX ALGORITHM (ENGLISH AND SPANISH) _____________ A Thesis Presented to The Faculty of the Department of Computer Science Sam Houston State University _____________ In Partial Fulfillment of the Requirements for the Degree of Master of Science _____________ by Keerthi Koneru December, 2016 PHONETIC MATCHING TOOLKIT WITH STATE-OF-THE-ART META-SOUNDEX ALGORITHM (ENGLISH AND SPANISH) by Keerthi Koneru ______________ APPROVED: Cihan Varol, PhD Thesis Director Narasimha Shashidhar, PhD Committee Member Bing Zhou, PhD Committee Member John B. Pascarella, PhD Dean, College of Science and Engineering Technology DEDICATION I dedicate this dissertation to all my family members who had always motivated and supported me to successfully achieve this. A special dedication to my father, Koneru Paramesh Babu, whom I regard as the most influential person in my life, who always showed me the right direction and responsible for grooming me into a better individual. Last but not the least, I would like to dedicate to my brother Sridhar Reddy and my friend Subash Kumar, who constantly motivated me to work comprehensively and effectively, which helped me a lot in successfully achieving this milestone. iii ABSTRACT Koneru, Keerthi, Phonetic matching toolkit with state-of-the-art Meta-Soundex algorithm (English and Spanish). Master of Science (Computing and Information Science), December, 2016, Sam Houston State University, Huntsville, Texas. Researchers confront major problems while searching for various kinds of data in large imprecise databases, as they are not spelled correctly or in the way they were expected to be spelled. As a result, they cannot find the word they sought.
    [Show full text]
  • Using Phonetic Knowledge in Tools and Resources for Natural Language Processing and Pronunciation Evaluation
    Using phonetic knowledge in tools and resources for Natural Language Processing and Pronunciation Evaluation Gustavo Augusto de Mendonça Almeida Gustavo Augusto de Mendonça Almeida Using phonetic knowledge in tools and resources for Natural Language Processing and Pronunciation Evaluation Master dissertation submitted to the Instituto de Ciências Matemáticas e de Computação – ICMC- USP, in partial fulfillment of the requirements for the degree of the Master Program in Computer Science and Computational Mathematics. FINAL VERSION Concentration Area: Computer Science and Computational Mathematics Advisor: Profa. Dra. Sandra Maria Aluisio USP – São Carlos May 2016 Ficha catalográfica elaborada pela Biblioteca Prof. Achille Bassi e Seção Técnica de Informática, ICMC/USP, com os dados fornecidos pelo(a) autor(a) Almeida, Gustavo Augusto de Mendonça A539u Using phonetic knowledge in tools and resources for Natural Language Processing and Pronunciation Evaluation / Gustavo Augusto de Mendonça Almeida; orientadora Sandra Maria Aluisio. – São Carlos – SP, 2016. 87 p. Dissertação (Mestrado - Programa de Pós-Graduação em Ciências de Computação e Matemática Computacional) – Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, 2016. 1. Template. 2. Qualification monograph. 3. Dissertation. 4. Thesis. 5. Latex. I. Aluisio, Sandra Maria, orient. II. Título. SERVIÇO DE PÓS-GRADUAÇÃO DO ICMC-USP Data de Depósito: Assinatura: ______________________ Gustavo Augusto de Mendonça Almeida Utilizando conhecimento fonético em ferramentas e recursos de Processamento de Língua Natural e Treino de Pronúncia Dissertação apresentada ao Instituto de Ciências Matemáticas e de Computação – ICMC-USP, como parte dos requisitos para obtenção do título de Mestre em Ciências – Ciências de Computação e Matemática Computacional. VERSÃO REVISADA Área de Concentração: Ciências de Computação e Matemática Computacional Orientadora: Profa.
    [Show full text]
  • The Music in Plain Speech and Writing
    The Music in Plain Speech and Writing PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Sun, 19 May 2013 11:51:46 UTC Contents CMU Pronouncing Dictionary 1 Arpabet 2 International Phonetic Alphabet 5 Phonetic transcription 26 Phonemic orthography 30 Pronunciation 35 Syllable 36 Allophone 46 Homophone 49 Rhyme 52 Half rhyme 59 Internal rhyme 60 Assonance 62 Literary consonance 64 Alliteration 65 Mnemonic 68 Phonetic algorithm 73 Metaphone 74 Soundex 77 Rhyme Genie 79 References Article Sources and Contributors 81 Image Sources, Licenses and Contributors 84 Article Licenses License 85 CMU Pronouncing Dictionary 1 CMU Pronouncing Dictionary CMU Pronouncing Dictionary Developer(s) Carnegie Mellon University Stable release 0.7a / February 18, 2008 Development status Maintained Available in English License Public Domain [1] Website Homepage The CMU Pronouncing Dictionary (also known as cmudict) is a public domain pronouncing dictionary created by Carnegie Mellon University (CMU). It defines a mapping from English words to their North American pronunciations, and is commonly used in speech processing applications such as the Festival Speech Synthesis System and the CMU Sphinx speech recognition system. The latest release is 0.7a, which contains 133,746 entries (from 123,442 baseforms). Database Format The database is distributed as a text file of the format word <two spaces> pronunciation. If there are multiple pronunciations available for a word, all subsequent entries are followed by an index in parentheses. The pronunciation is encoded using a modified form of the Arpabet system. The difference is stress marks on vowels with levels 0, 1, 2; not all entries have stress however.
    [Show full text]
  • TUGBOAT Volume 24, Number 2 / 2003
    TUGBOAT Volume 24, Number 2 / 2003 General Delivery 159 From the president / Karl Berry 160 Editorial comments / Barbara Beeton 162 Daniel Taupin, 1936–2003 / John Plaice 163 What is TEX? / Douglas Waud Typography 165 Maths = Typography? / Richard Lawrence 169 On musical typesetting: Sonata for TEX and METAFONT, Op. 2 / Federico Garcia Font Forum 183 There is no end: Omega and Zapfino / William Adams 200 The METAFONT approach: Implicit, relative, and analytical font design / Timothy Hall Resources 205 CTAN plans / Robin Fairbairns, Jim Hefferon, Rainer Sch¨opf, Joachim Schrod, Graham Williams, Reinhard Zierke LATEX 208 Some notes on templates / Lars Hellstr¨om 211 Writing a big book — A first experience with LATEX / David Walden 216 Designing packages for Λ: An overview / Apostolos Syropoulos 221 LATEX news, issues 14–16, 2001–2003 / LATEX Project Team 224 ednotes — critical edition typesetting with LATEX / Uwe L¨uck Software & Tools 236 Hyphenation patterns for minority languages / Kevin Scannell 240 (LA)TEX, genealogy, and the LifeLines software / Andrew Caird 245 Generating LATEX documents through Matlab / S. E. Talole, S. B. Phadke 249 mlBibTEX’s Version 1.3 / Jean-Michel Hufflen Hints & Tricks 262 Glisterings / Peter Wilson 265 The treasure chest / Mark LaPlante Book Reviews 275 The LATEX Companion, Second Edition / Claudio Beccari 278 Guide to LATEX, 4th Edition / Douglas Waud and Mimi Burbank Abstracts 282 Les Cahiers GUTenberg: Contents of Issue 42 (July 2003) 283 MAPS: Contents of issues 27–28 (2002) 286 TEXemplares: Contents of issues 4–6 (2003–04) Reports 288 Report on the Pune workshop on LATEX and free mathematical software / S. A.
    [Show full text]
  • An Assessment of Name Matching Algorithms Abstract 1. Name
    An Assessment of Name Matching Algorithms A. J. Lait1 and B. Randell2 Department of Computing Science University of Newcastle upon Tyne Abstract In many computer applications involving the recording and processing of personal data there is a need to allow for variations in surname spelling, caused for example by transcription errors. A number of algorithms have been developed for name matching, i.e. which attempt to identify name spelling variations, one of the best known of which is the Soundex algorithm. This paper describes a comparative analysis of a number of these algorithms and, based on an analysis of their comparative strengths and weaknesses, proposes a new and improved name matching algorithm, which we call the Phonex algorithm. The analysis takes advantage of the recent creation of a large list of “equivalent surnames”, published in the book Family History Knowledge UK [Park1992]. This list is based on data supplied by some thousands of individual genealogists, and can be presumed to be representative of British surnames and their variations over the last two or three centuries. It thus made it possible to perform what we would argue were objective tests of name matching, the results of which provide a solid basis for the analysis that we have performed, and for our claims for the merits of the new algorithm, though these are unlikely to hold fully for surnames emanating largely from other countries. 1. Name Variations There are many applications of computer-based name-matching algorithms including record linkage and database searching where variations in spelling, caused for example by transcription errors, need to be allowed for.
    [Show full text]
  • Discovering Lexical Similarity Through Articulatory Feature-Based Phonetic Edit Distance
    Discovering Lexical Similarity Through Articulatory Feature-based Phonetic Edit Distance ∗ † ∗ Tafseer Ahmed ,​ Muhammad Suffian Nizami ,​ Muhammad Yaseen Khan ​ ∗ ​ † ​ Mohammad Ali Jinnah University, Karachi ,​ FAST NUCES, Faisalabad ​ ​ [email protected], [email protected], [email protected] Abstract Lexical Similarity (LS) between two languages uncovers many interesting linguistic insights such as genetic relationship, mutual intelligibility, and the usage of one’s vocabulary into other. There are various methods through which LS is evaluated. In the same regard, this paper presents a method of Phonetic Edit Distance (PED) that uses a soft comparison of letters using the articulatory features associated with them. The system converts the words into the corresponding International Phonetic Alphabet (IPA), followed by the conversion of IPA into its set of articulatory features. Later, the lists of the set of articulatory features are compared using the proposed method. As an example, PED gives edit distance of German word vater (/faːtər/) and Persian word ‘ (pidar, /pedær/) as ’ﭘﺪﺭ shalom, /ʃəlɒm/) and Arabic word ‘ ’ (salam, /səla:m/) as 0.93, whereas for a juxtapose) ’שלום‘ and similarly, Hebrew word ;0.82 ﺳﻼﻡ comparison, their IPA based edit distances are 4 and 2 respectively. Experiments are performed with six languages (Arabic, Hindi, Marathi, Persian, Sanskrit, and Urdu). In this regard, we extracted part of speech wise word-lists from the Universal Dependency corpora and evaluated the LS for every pair of language. Thus, with the proposed approach, we find the genetic affinity, similarity, and borrowing/loan-words despite having script differences and sound variation phenomena among these languages. Keywords: Lexical Similarity, Edit Distance, Phonetic Matching, Articulatory Features ​ Arabic, Hindi, Marathi, Persian, Sanskrit, and Urdu.
    [Show full text]