DOCSLIB.ORG

Faspell: a Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based on DAE-Decoder Paradigm

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Faspell: a Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based on DAE-Decoder Paradigm FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, Junhui Liu Intelligent Platform Division, iQIYI, Inc. fhongyuzhong, yuxianguo, heneng, liunan, [email protected] Abstract 1.1 Related work and bottlenecks Almost all previous Chinese spell checking mod- We propose a Chinese spell checker – FASPell based on a new paradigm which consists of els deploy a common paradigm where a fixed set a denoising autoencoder (DAE) and a de- of similar characters of each Chinese character coder. In comparison with previous state- (called confusion set) is used as candidates, and a of-the-art models, the new paradigm allows filter selects the best candidates as substitutions for our spell checker to be Faster in computa- a given sentence. This naive design is subjected to tion, readily Adaptable to both simplified and two major bottlenecks, whose negative impact has traditional Chinese texts produced by either been unsuccessfully mitigated: humans or machines, and to require much Simpler structure to be as much Powerful in • overfitting to under-resourced Chinese both error detection and correction. These four spell checking data. Since Chinese spell achievements are made possible because the new paradigm circumvents two bottlenecks. checking data require tedious professional First, the DAE curtails the amount of Chi- manual work, they have always been under- nese spell checking data needed for super- resourced. To prevent the filter from over- vised learning (to <10k sentences) by lever- fitting, Wang et al.(2018) propose an auto- aging the power of unsupervisedly pre-trained matic method to generate pseudo spell check- masked language model as in BERT, XLNet, ing data. However, the precision of their spell MASS etc. Second, the decoder helps to elim- checking model ceases to improve when the inate the use of confusion set that is deficient in flexibility and sufficiency of utilizing the generated data reaches 40k sentences. Zhao salient feature of Chinese character similarity. et al.(2017) use an extensive amount of ad hoc linguistic rules to filter candidates, only 1 Introduction to achieve worse performance than ours even though our model does not leverage any lin- There has been a long line of research on detect- guistic knowledge. ing and correcting spelling errors in Chinese texts since some trailblazing work in the early 1990s • inflexibility and insufficiency of confusion (Shih et al., 1992; Chang, 1995). However, de- set in utilizing character similarity. The spite the spelling errors being reduced to substitu- feature of Chinese character similarity is very tion errors in most researches1 and efforts of mul- salient as it is related to the main cause of tiple recent shared tasks (Wu et al., 2013; Yu et al., spelling errors (see subsection 2.2). How- 2014; Tseng et al., 2015; Fung et al., 2017), Chi- ever, the idea of confusion set is troublesome nese spell checking remains a difficult task. More- in utilizing it: over, the methods for languages like English can 1. inflexibility to address the issue that hardly be directly used for the Chinese language confusing characters in one scenario because there are no delimiters between words, may not be confusing in another. The whose lack of morphological variations makes the difference between simplified and tradi- syntactic and semantic interpretations of any Chi- tional Chinese shown in Table1 is an nese character highly dependent on its context. example. Wang et al.(2018) also sug- 1Likewise, this paper only covers substitution errors. gest that confusing characters for ma- 160 Proceedings of the 2019 EMNLP Workshop W-NUT: The 5th Workshop on Noisy User-generated Text, pages 160–169 Hong Kong, Nov 4, 2019. c 2019 Association for Computational Linguistics chines are different from those for hu- 2. The DAE-decoder paradigm is sequence- mans. Therefore, in practice, it is very to-sequence, which makes it resemble the likely that the correct candidates for sub- encoder-decoder paradigm in tasks like ma- stitution do not exist in a given confu- chine translation, grammar checking, etc. sion set, which harms recall. Also, con- However, in the encoder-decoder paradigm, sidering more similar characters to pre- the encoder extracts semantic information, serve recall will risk lowering precision. and the decoder generates texts that embody 2. insufficiency in utilizing character simi- the information. In contrast, in the DAE- larity. Since a cut-off threshold of quan- decoder paradigm, the DAE provides candi- tified character similarity (Liu et al., dates to reconstruct texts from the corrupted 2010; Wang et al., 2018) is used to pro- ones based on contextual feature, and the de- duce the confusion set, similar charac- coder3 selects the best candidates by incorpo- ters are actually treated indiscriminately rating other features. in terms of their similarity. This means Besides the new paradigm per se, there are two the information of character similarity additional contributions in our proposed Chinese is not sufficiently utilized. To compen- spell checking model: sate this, Zhang et al.(2015) propose a spell checker that has to consider many • we propose a more precise quantification less salient features such as word seg- method of character similarity than the ones mentation, which add more unnecessary proposed by Liu et al.(2010) and Wang et al. noises to their model. (2018) (see subsection 2.2); • we propose an empirically effective decoder 1.2 Motivation and contributions to filter candidates under the principle of get- The motivation of this paper is to circumvent the ting the highest possible precision with mini- two bottlenecks in subsection 1.1 by changing the mal harm to recall (see subsection 2.3). paradigm for Chinese spell checking. 1.3 Achievements As a major contribution and as exemplified by our proposed Chinese spell checking model in Fig- Thanks to our contributions mentioned in subsec- ure1, the most general form of the new paradigm tion 1.2, our model can be characterized by the fol- consists of a denoising autoencoder2 (DAE) and a lowing achievements relative to previous state-of- decoder. To prove that it is indeed a novel contri- the-art models, and thus is named FASPell. bution, we compare it with two similar paradigms • Our model is Fast. It is shown (subsection and show their differences as follows: 3.3) to be faster in filtering than previous state-of-the-art models either in terms of ab- 1. Similar to the old paradigm used in previous solute time consumption or time complexity. Chinese spell checking models, a model un- der the DAE-decoder paradigm also produces • Our model is Adaptable. To demonstrate this, candidates (by DAE) and then filters the can- we test it on texts from different scenarios didates (by the decoder). However, candi- – texts by humans, such as learners of Chi- dates are produced on the fly based on con- nese as a Foreign Language (CFL), and by texts. If the DAE is powerful enough, we machines, such as Optical Character Recog- should expect that all contextually suitable nition (OCR). It can also be applied to both candidates are recalled, which prevent the in- simplified Chinese and traditional Chinese, flexibility issue caused by using confusion despite the challenging issue that some er- set. The DAE will also prevent the overfit- roneous usages of characters in traditional ting issue because it can be trained unsuper- texts are considered valid usages in simpli- visedly using a large number of natural texts. fied texts (see Table1). To the best of our Moreover, character similarity can be used by knowledge, all previous state-of-the-art mod- the decoder without losing any information. els only focus on human errors in traditional Chinese texts. 2the term denoising autoencoder follows the same sense used by Yang et al.(2019), which is arguably more general 3The term decoder here is analogous as in Viterbi decoder than the one used by Vincent et al.(2008). in the sense of finding the best path along candidates. 161 Table 1: Examples on the left are considered valid ✓ ✓ usages in simplified Chinese (SC). Notes on the right 国 际 电 台 著 名 主 持 人 are about how they are erroneous in traditional Chi- nese (TC) and suggested corrections. This inconsis- tency is because multiple traditional characters were Confidence-Similarity Decoder merged into identical characters in the simplification process. Our model makes corrections for this type of errors only in traditional texts. In simplified texts, they 国 际 电 台 知 名 主 持 人 rank=1 are not detected as errors. 0.9994 0.9999 0.9999 0.9999 0.2878 0.9626 0.9994 0.9981 0.9999 國 際 听 话 著 音 广 目 者 rank=2 SC Examples Notes on TC usage 0.0002 0.0000 0.0000 0.0000 0.1999 0.0019 0.0002 0.0002 0.0000 世 家 节 视 报 台 演 主 手 rank=3 h+ (weekend) h ! 1 h only in h0, etc. 0.0000 0.0000 0.0000 0.0000 0.0429 0.0015 0.0000 0.0001 0.0000 Å8 8 J 8 8 (trip) ! only in 泳, etc. 台 界 讲 播 冠 闻 支 节 持 rank=4 6 (make) 6 ! 製 6 only in 6¦, etc. 0.0000 0.0000 0.0000 0.0000 0.0252 0.0014 0.0000 0.0001 0.0000 • Our model is Simple. As shown in Fig- Masked Language Model ure1, it has only a masked language model and a filter as opposed to multiple feature- producing models and filters being used in 国 际 电 台 苦 名 丰 持 人 ✖ ✖ previous state-of-the-art proposals. More- over, only a small training set and a set of Figure 1: A real example of how an erroneous sentence visual and phonological features of charac- which is supposed to have the meaning of "A famous international radio broadcaster" is successfully spell- ters are required in our model.
Load more

Recommended publications
  • Spell Checker
    International Journal of Scientific and Research Publications, Volume 5, Issue 4, April 2015 1 ISSN 2250-3153 SPELL CHECKER Vibhakti V. Bhaire, Ashiki A. Jadhav, Pradnya A. Pashte, Mr. Magdum P.G ComputerEngineering, Rajendra Mane College of Engineering and Technology Abstract- Spell Checker project adds spell checking and For handling morphology language-dependent algorithm is an correction functionality to the windows based application by additional step. The spell-checker will need to consider different using autosuggestion technique. It helps the user to reduce typing forms of the same word, such as verbal forms, contractions, work, by identifying any spelling errors and making it easier to possessives, and plurals, even for a lightly inflected language like repeat searches .The main goal of the spell checker is to provide English. For many other languages, such as those featuring unified treatment of various spell correction. Firstly the spell agglutination and more complex declension and conjugation, this checking and correcting problem will be formally describe in part of the process is more complicated. order to provide a better understanding of these task. Spell checker and corrector is either stand-alone application capable of A spell checker carries out the following processes: processing a string of words or a text or as an embedded tool which is part of a larger application such as a word processor. • It scans the text and selects the words contained in it. Various search and replace algorithms are adopted to fit into the • domain of spell checker. Spell checking identifies the words that It then compares each word with a known list of are valid in the language as well as misspelled words in the correctly spelled words (i.e.
    [Show full text]
  • Finite State Recognizer and String Similarity Based Spelling
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by BRAC University Institutional Repository FINITE STATE RECOGNIZER AND STRING SIMILARITY BASED SPELLING CHECKER FOR BANGLA Submitted to A Thesis The Department of Computer Science and Engineering of BRAC University by Munshi Asadullah In Partial Fulfillment of the Requirements for the Degree of Bachelor of Science in Computer Science and Engineering Fall 2007 BRAC University, Dhaka, Bangladesh 1 DECLARATION I hereby declare that this thesis is based on the results found by me. Materials of work found by other researcher are mentioned by reference. This thesis, neither in whole nor in part, has been previously submitted for any degree. Signature of Supervisor Signature of Author 2 ACKNOWLEDGEMENTS Special thanks to my supervisor Mumit Khan without whom this work would have been very difficult. Thanks to Zahurul Islam for providing all the support that was required for this work. Also special thanks to the members of CRBLP at BRAC University, who has managed to take out some time from their busy schedule to support, help and give feedback on the implementation of this work. 3 Abstract A crucial figure of merit for a spelling checker is not just whether it can detect misspelled words, but also in how it ranks the suggestions for the word. Spelling checker algorithms using edit distance methods tend to produce a large number of possibilities for misspelled words. We propose an alternative approach to checking the spelling of Bangla text that uses a finite state automaton (FSA) to probabilistically create the suggestion list for a misspelled word.
    [Show full text]
  • Spell Checking in Computer-Assisted Language Learning: a Study of Misspellings by Nonnative Writers of German
    SPELL CHECKING IN COMPUTER-ASSISTED LANGUAGE LEARNING: A STUDY OF MISSPELLINGS BY NONNATIVE WRITERS OF GERMAN Anne Rimrott Diplom, Justus Liebig Universitat, 2002 THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS In the Department of Linguistics O Anne Rimrott 2005 SIMON FRASER UNIVERSITY Spring 2005 All rights reserved. This work may not be reproduced in whole or in part, by photocopy or other means, without permission of the author. APPROVAL Name: Anne Rimrott Degree: Master of Arts Title of Thesis: Spell Checking in Computer-Assisted Language Learning: A Study of Misspellings by Nonnative Writers of German Examining Committee: Chair: Dr. Alexei Kochetov Assistant Professor, Department of Linguistics Dr. Trude Heift Senior Supervisor Associate Professor, Department of Linguistics Dr. Chung-hye Han Supervisor Assistant Professor, Department of Linguistics Dr. Maria Teresa Taboada Supervisor Assistant Professor, Department of Linguistics Dr. Mathias Schulze External Examiner Assistant Professor, Department of Germanic and Slavic Studies University of Waterloo Date DefendedIApproved: SIMON FRASER UNIVERSITY PARTIAL COPYRIGHT LICENCE The author, whose copyright is declared on the title page of this work, has granted to Simon Fraser University the right to lend this thesis, project or extended essay to users of the Simon Fraser University Library, and to make partial or single copies only for such users or in response to a request from the library of any other university, or other educational institution, on its own behalf or for one of its users. The author has further granted permission to Simon Fraser University to keep or make a digital copy for use in its circulating collection.
    [Show full text]
  • Exploiting Wikipedia Semantics for Computing Word Associations
    Exploiting Wikipedia Semantics for Computing Word Associations by Shahida Jabeen A thesis submitted to the Victoria University of Wellington in fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science. Victoria University of Wellington 2014 Abstract Semantic association computation is the process of automatically quan- tifying the strength of a semantic connection between two textual units based on various lexical and semantic relations such as hyponymy (car and vehicle) and functional associations (bank and manager). Humans have can infer implicit relationships between two textual units based on their knowledge about the world and their ability to reason about that knowledge. Automatically imitating this behavior is limited by restricted knowledge and poor ability to infer hidden relations. Various factors affect the performance of automated approaches to com- puting semantic association strength. One critical factor is the selection of a suitable knowledge source for extracting knowledge about the im- plicit semantic relations. In the past few years, semantic association com- putation approaches have started to exploit web-originated resources as substitutes for conventional lexical semantic resources such as thesauri, machine readable dictionaries and lexical databases. These conventional knowledge sources suffer from limitations such as coverage issues, high construction and maintenance costs and limited availability. To overcome these issues one solution is to use the wisdom of crowds in the form of collaboratively constructed knowledge sources. An excellent example of such knowledge sources is Wikipedia which stores detailed information not only about the concepts themselves but also about various aspects of the relations among concepts. The overall goal of this thesis is to demonstrate that using Wikipedia for computing word association strength yields better estimates of hu- mans’ associations than the approaches based on other structured and un- structured knowledge sources.
    [Show full text]
  • Words in a Text
    Cross-lingual geo-parsing for non-structured data Judith Gelernter Wei Zhang Language Technologies Institute Language Technologies Institute School of Computer Science School of Computer Science Carnegie Mellon University Carnegie Mellon University [email protected] [email protected] ABSTRACT to cross-language geo-information retrieval. We will examine A geo-parser automatically identifies location words in a text. We Strötgen’s view that normalized location information is language- have generated a geo-parser specifically to find locations in independent [16]. Difficulties in cross-language experiments are unstructured Spanish text. Our novel geo-parser architecture exacerbated when the languages use different alphabets, and when combines the results of four parsers: a lexico-semantic Named the focus is on proper names, as in this case, the names of Location Parser, a rules-based building parser, a rules-based street locations. parser, and a trained Named Entity Parser. Each parser has Geo-parsing structured versus unstructured text requires different different strengths: the Named Location Parser is strong in recall, language processing tools. Twitter messages are challenging to and the Named Entity Parser is strong in precision, and building geo-parse because of their non-grammaticality. To handle non- and street parser finds buildings and streets that the others are not grammatical forms, we use a Twitter tokenizer rather than a word designed to do. To test our Spanish geo-parser performance, we tokenizer. We use an English part of speech tagger created for compared the output of Spanish text through our Spanish geo- tweets, which was not available to us in Spanish.
    [Show full text]
  • Automatic Spelling Correction Based on N-Gram Model
    International Journal of Computer Applications (0975 – 8887) Volume 182 – No. 11, August 2018 Automatic Spelling Correction based on n-Gram Model S. M. El Atawy A. Abd ElGhany Dept. of Computer Science Dept. of Computer Faculty of Specific Education Damietta University Damietta University, Egypt Egypt ABSTRACT correction. In this paper, we are designed, implemented and A spell checker is a basic requirement for any language to be evaluated an end-to-end system that performs spellchecking digitized. It is a software that detects and corrects errors in a and auto correction. particular language. This paper proposes a model to spell error This paper is organized as follows: Section 2 illustrates types detection and auto-correction that is based on n-gram of spelling error and some of related works. Section 3 technique and it is applied in error detection and correction in explains the system description. Section 4 presents the results English as a global language. The proposed model provides and evaluation. Finally, Section 5 includes conclusion and correction suggestions by selecting the most suitable future work. suggestions from a list of corrective suggestions based on lexical resources and n-gram statistics. It depends on a 2. RELATED WORKS lexicon of Microsoft words. The evaluation of the proposed Spell checking techniques have been substantially, such as model uses English standard datasets of misspelled words. error detection & correction. Two commonly approaches for Error detection, automatic error correction, and replacement error detection are dictionary lookup and n-gram analysis. are the main features of the proposed model. The results of the Most spell checkers methods described in the literature, use experiment reached approximately 93% of accuracy and acted dictionaries as a list of correct spellings that help algorithms similarly to Microsoft Word as well as outperformed both of to find target words [16].
    [Show full text]
  • Development of a Persian Syntactic Dependency Treebank
    Development of a Persian Syntactic Dependency Treebank Mohammad Sadegh Rasooli Manouchehr Kouhestani Amirsaeid Moloodi Department of Computer Science Department of Linguistics Department of Linguistics Columbia University Tarbiat Modares University University of Tehran New York, NY Tehran, Iran Tehran, Iran [email protected] [email protected] [email protected] Abstract tions in tasks such as machine translation. Depen- dency treebanks are collections of sentences with This paper describes the annotation process their corresponding dependency trees. In the last and linguistic properties of the Persian syn- decade, many dependency treebanks have been de- tactic dependency treebank. The treebank veloped for a large number of languages. There are consists of approximately 30,000 sentences at least 29 languages for which at least one depen- annotated with syntactic roles in addition to morpho-syntactic features. One of the unique dency treebank is available (Zeman et al., 2012). features of this treebank is that there are al- Dependency trees are much more similar to the hu- most 4800 distinct verb lemmas in its sen- man understanding of language and can easily rep- tences making it a valuable resource for ed- resent the free word-order nature of syntactic roles ucational goals. The treebank is constructed in sentences (Kubler¨ et al., 2009). with a bootstrapping approach by means of available tagging and parsing tools and man- Persian is a language with about 110 million ually correcting the annotations. The data is speakers all over the world (Windfuhr, 2009), yet in splitted into standard train, development and terms of the availability of teaching materials and test set in the CoNLL dependency format and annotated data for text processing, it is undoubt- is freely available to researchers.
    [Show full text]
  • Study on Spell-Checking System Using Levenshtein Distance Algorithm
    International Journal of Recent Development in Engineering and Technology Website: www.ijrdet.com (ISSN 2347-6435(Online) Volume 8, Issue 9, September 2019) Study on Spell-Checking System using Levenshtein Distance Algorithm Thi Thi Soe1, Zarni Sann2 1Faculty of Computer Science 2Faculty of Computer Systems and Technologies 1,2FUniversity of Computer Studies (Mandalay), Myanmar Abstract— Natural Language Processing (NLP) is one of If the word is not found, it is considered to be an error, the most important research area carried out in the world of and an attempt may be made to suggest that word. When a Artificial Intelligence (AI). NLP supports the tasks of a word which is not within the dictionary is encountered, portion of AI such as spell checking, machine translation, most spell checkers provide an option to add that word to a automatic text summarization, and information extraction, so list of known exceptions that should not be flagged. An on. Spell checking application presents valid suggestions to the user based on each mistake they encounter in the user’s adaptive spelling checker tool based on Ternary Search document. The user then either makes a selection from a list Tree data structure is described in [1]. A comprehensive of suggestions or accepts the current word as valid. Spell- spelling checker application presented a significant checking program is often integrated with word processing challenge in producing suggestions for a misspelled word that checks for the correct spelling of words in a document. when employing the traditional methods in [2]. It learned Each word is compared against a dictionary of correctly spelt the complex orthographic rules of Bangla.
    [Show full text]
  • Spell Checker in CET Designer
    Linköping University | Department of Computer Science Bachelor thesis, 16 ECTS | Datateknik 2016 | LIU-IDA/LITH-EX-G--16/069--SE Spell checker in CET Designer Rasmus Hedin Supervisor : Amir Aminifar Examiner : Zebo Peng Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin- istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam- manhang som är kränkande för upphovsmannenslitterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circum- stances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose.
    [Show full text]
  • Discourse-Aware Statistical Machine Translation As a Context-Sensitive Spell Checker
    Discourse-aware Statistical Machine Translation as a Context-Sensitive Spell Checker Behzad Mirzababaei, Heshaam Faili and Nava Ehsan School of Electrical and Computer Engineering College of Engineering University of Tehran, Tehran, Iran {b.mirzababaei,hfaili,n.ehsan}@ut.ac.ir Phrase-based SMT is weak in handling long- Abstract distance dependencies between the sentence words. In order to capture this kind of Real-word errors or context sensitive spelling dependencies, which affects detecting the correct errors, are misspelled words that have been candidate word, mentioned SMT is augmented wrongly converted into another word of with a discourse-aware reranking method for vocabulary. One way to detect and correct reranking the N-best results of SMT. real-word errors is using Statistical Machine Our work can be regarded as an extension of Translation (SMT), which translates a text containing some real-word errors into a the method introduced by Ehsan and Faili correct text of the same language. In this (2013), in which they use SMT to detect and paper, we improve the results of mentioned correct the spelling errors of a document. But SMT system by employing some discourse- here, we use the N-best results of SMT as a aware features into a log-linear reranking candidate list for each erroneous word and rerank method. Our experiments on a real-world test the list by using a discourse-aware reranking data in Persian show an improvement of system which is just a log-linear ranker. about 9.5% and 8.5% in the recall of Shortly, the contributions of this paper can be detection and correction respectively.
    [Show full text]
  • Using the Hunspell Spell Checker
    integrated translation environment Using the Hunspell Spell Checker © 2004-2014 Kilgray Translation Technologies. All rights reserved. Using the Hunspell Spell Checker in memoQ Contents Contents ...................................................................................................................................... 2 Setting up the Hunspell spell checker for your language ................................................................ 3 Using the Hunspell spell checker during translation ....................................................................... 6 This guide covers memoQ 2014 and higher settings for Hunspell. It contains text items from the English user interface of the program. These items are under constant verification and are subject to change without prior notification. memoQ integrated translation environment Page 2 of 6 Using the Hunspell Spell Checker in memoQ Setting up the Hunspell spell checker for your language memoQ installs the Hunspell spell checker, an open-source application used in OpenOffice.org and Mozilla Firefox/Thunderbird. The most important benefits of using Hunspell are the following: . You no longer need to use Microsoft Word’s spell checker, which is not always available.1 . Spell checking becomes significantly faster in memoQ. You can check spelling in practically any language. You can spell check as you type. memoQ uses Hunspell for on-the-fly spelling when you have configured a Hunspell dictionary for your target language in Options > Spell settings. memoQ does not include any particular spelling dictionaries for Hunspell. Before you can use Hun- spell for a language of your choice, you need to set up the appropriate dictionaries. To set up dic- tionaries for a particular language, follow the steps below: 1. Start memoQ. 2. From the Tools menu or in the Quick Access Toolbar in memoQ 2014 R2, choose Options. In the category list to the left, click Spelling and grammar.
    [Show full text]
  • Analyzing the Impact of Spelling Errors on POS-Tagging and Chunking in Learner English
    Analyzing the Impact of Spelling Errors on POS-Tagging and Chunking in Learner English Tomoya Mizumoto1 and Ryo Nagata2 1 1 RIKEN Center for Advanced Intelligence Project 2 Konan University [email protected], nagata-ijcnlp @hyogo-u.ac.jp Abstract The heavy dependence on POS-tagging and chunking suggests that failures could degrade the Part-of-speech (POS) tagging and chunk- performance of NLP systems and linguistic analy- ing have been used in tasks targeting ses (Han et al., 2006; Sukkarieh and Blackmore, learner English; however, to the best 2009). For example, failure to recognize noun our knowledge, few studies have eval- phrases in a sentence could lead to failure in cor- uated their performance and no stud- recting related errors in article use and noun num- ies have revealed the causes of POS- ber. More importantly, such failures make it more tagging/chunking errors in detail. There- difficult to simply count the number of POSs and fore, we investigate performance and an- chunks, thereby causing inaccurate estimates of alyze the causes of failure. We focus their distributions. Note that such estimates are of- on spelling errors that occur frequently ten employed in linguistic analysis, including the in learner English. We demonstrate that above-mentioned studies. spelling errors reduced POS-tagging per- Despite its importance in related tasks, we also formance by 0.23% owing to spelling er- note that few studies have focused on performance rors, and that a spell checker is not neces- evaluations of POS-tagging and chunking. Only a sary for POS-tagging/chunking of learner few studies, including Nagata et al.(2011), Berzak English.
    [Show full text]