DOCSLIB.ORG

Approaches to Natural Language Processing in App- Development

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Approaches to Natural Language Processing in App- Development DEGREE PROJECT IN COMPUTER ENGINEERING, FIRST CYCLE, 15 CREDITS STOCKHOLM, SWEDEN 2018 Approaches to natural language processing in app- development CAMRAN DJOWEINI HENRIETTA HELLBERG KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF INFORMATION AND COMMUNICATION TECHNOLOGY 1 Acknowledgements Firstly, we would like to thank our advisors at KTH, the Royal Institute of Technology, Anders Sjögren and Fadil Galjic for their patience, continuous support and much appreciated guidance throughout the course of this project. Secondly, we would like to thank William Dalheim for his incredible enthusiasm and for the introduction to an exciting, and for us new, field. 2 Abstract Natural language processing is an on-going field that is not yet fully established. A high demand for natural language processing in applications creates a need for good development-tools and different implementation approaches developed to suit the engineers behind the applications. This project approaches the field from an engineering point of view to research approaches, tools, and techniques that are readily available today for development of natural language processing support. The sub-area of information retrieval of natural language ​ processing was examined through a case study, where prototypes were developed to get a deeper understanding of the tools and techniques used for such tasks from an engineering point of view. We found that there are two major approaches to developing natural language processing support for applications, high-level and low-level approaches. A categorization of tools and frameworks belonging to the two approaches as well as the source code, documentation and, evaluations, of two prototypes developed as part of the research are presented. The choice of approach, tools and techniques should be based on the specifications and requirements of the final product and both levels have their own pros and cons. The results of the report are, to a large extent, generalizable as many different natural language processing tasks can be solved using similar solutions even if their goals vary. Keywords Natural language processing, information retrieval, voice-control, implementation approaches, NLP. 3 4 Sammanfattning Datalingvistik (engelska natural language processing) är ett område inom datavetenskap som ännu inte är fullt etablerat. En hög efterfrågan av stöd för naturligt språk i applikationer skapar ett behov av tillvägagångssätt och verktyg anpassade för ingenjörer. Detta projekt närmar sig området från en ingenjörs synvinkel för att undersöka de tillvägagångssätt, verktyg och tekniker som finns tillgängliga att arbeta med för utveckling av stöd för naturligt språk i applikationer i dagsläget. Delområdet ‘information retrieval’ undersöktes genom en fallstudie, där prototyper utvecklades för att skapa en djupare förståelse av verktygen och teknikerna som används inom området. Vi kom fram till att det går att kategorisera verktyg och tekniker i två olika grupper, beroende på hur distanserad utvecklaren är från den underliggande bearbetningen av språket. Kategorisering av verktyg och tekniker samt källkod, dokumentering och utvärdering av prototyperna presenteras som resultat. Valet av tillvägagångssätt, tekniker och verktyg bör baseras på krav och specifikationer för den färdiga produkten. Resultaten av studien är till stor del generaliserbara eftersom lösningar till många problem inom området är likartade även om de slutgiltiga målen skiljer sig åt. Nyckelord Natural language processing, informationsinhämtning, röststyrning, implementerings tillvägagångssätt, NLP. 5 6 Table of contents 1 Introduction 12 1.1 An overview……………………………………………………………………………….. 12 ​ 1.2 Problem…………………………………………………………………………………….. 14 ​ 1.3 Aim…………………………………………………………………………………………... 15 ​ 1.4 Limitations………………………………………………………………………………... 15 ​ 2 Background theory 16 2.1 Natural language processing……………………………………………………….. 16 ​ 2.2 Estimating probability………………………………………………………………... 17 ​ 2.3 Artificial intelligence and natural language processing………………….. 18 ​ 2.4 String pre-processing, to prepare data for machine learning………….. 19 ​ 2.5 Parsing……………………………………………………………………………………... 19 ​ 2.6 Sequence tagging, labeling useful information……………………………... 20 ​ 2.7 Neural networks and natural language processing………………………... 20 ​ 2.8 Natural language processing algorithms………………………………………. 22 ​ 2.9 A model for development of natural language processing……………… 26 ​ 3 Method 28 3.1 Research methodology……………………………………………………………….. 28 ​ 3.2 Sub-questions of the research question……………………………………….. 30 ​ 3.3 Research method………………………………………………………………………. 32 ​ 3.4 Project method………………………………………………………………………….. 36 ​ 3.5 Documentation and modelling method……………………………………….. 38 ​ 4 Techniques and tools for natural language processing 40 4.1 Structuralizing development……………………………………………………….. 40 ​ 4.2 The underlying principles of natural language processing……………... 41 ​ 4.3 Tools for developing natural language processing support…………….. 42 ​ 4.4 Bridging the theory and the prototype development……………………... 48 ​ 5 Development of natural language processing support 50 5.1 Choice of techniques and tools…………………………………………………….. 50 ​ 5.2 The prototypes…………………………………………………………………………... 52 ​ 5.3 Prototype one……………………………………………………………………………. 54 ​ 5.4 Prototype two……………………………………………………………………………. 60 ​ 5.5 Alternative solutions………………………………………………………………….. 66 ​ 7 6 Discussion and conclusion 68 6.1 Implementing natural language processing support……………………... 68 ​ 6.2 Validity and reliability of the methodology…………………………………...70 ​ 6.3 The result………………………………………………………………………………….. 71 ​ 6.4 Sustainability, economic and ethical aspects………………………………... 72 ​ 6.5 Concluding remarks…………………………………………………………………... 73 ​ References 76 8 Author concepts and abbreviations NLP Natural Language Processing ​ NLU Natural Language Understanding ​ CL Computational Linguistics ​ ML Machine learning ​ PS Probability and Statistics ​ AI Artificial Intelligence ​ NER Named Entity Recognition ​ POS Part Of Speech ​ CRF Conditional random fields ​ HMM Hidden Markov Model ​ ANN Artificial Neural Network ​ RNN Recursive Neural Network ​ LSTM Long Short Term Memory implementation of a neural network ​ High-level techniques - High-level tools are an easily scalable package deal that distance ​ the programmer, to different extents, from the underlying principles of natural language processing. Low-level tools- Tools categorised as ‘low-level tools’ in this report are toolkits and ​ libraries used for natural language processing, where the developer is allowed more control of the implementation-specific details of the natural language processing techniques. They are, however, not termed low-low level tools based on the availability of working with adapting algorithms and mathematical formulas. 9 10 1 Introduction This chapter introduces the reader to the field of natural language processing, through an introductory section, and describes the context of the project. 1.1 An overview Interest in how computers understand and process language, natural language processing and machine learning, more generally, began in the late 1940’s, and around the time when pioneering computer scientist, Alan Turing, published an article on ‘Computing Machinery and Intelligence’, where he asked the question: “can machines think?”[1]. The question itself is deceptively simple, but it gets to the root of more complicated philosophical questions of the meaning of ‘thinking’ in the computer context and the social implications if computers were to become truly intelligent. Computer intelligence is more relevant today than it ever has been before, with the machine learning field having matured from a theoretical stage into something that already has started to improve the everyday life of people across the world, through intelligent software applications. Natural language processing, or NLP, is a field that is made up of several of the sub-areas of computer science, including machine learning and artificial intelligence. Natural language processing concerns itself with understanding and modelling of how computers can understand human language. The difference between the ease with which a human and a computer can learn and understand a language represents an interesting area of computer science that lies at the intersection of the human-computer interface. Interesting questions include: why is it so easy for a human to understand language and so hard for a computer to understand the context of a language whose “rules” it already knows? To us, this is a very interesting question that digs into the details of our understanding of how computers and machine learning works. The term “natural language” refers to human languages, and is the opposite of “artificial languages”, which have been created by humans for interacting with computers. “Natural language processing” lies at the intersection of natural and artificial language and is about how computers process, and understand, natural language input. As a summary, the field of natural language processing is concerned with the question: how can we make computers understand language in the same way we understand other people? 11 Since Alan Turing wrote the article in the 1950’s, the field of machine learning has shown significant progress, much of which can be applied to natural language processing. Yet, the field is still in its infancy, there is not one correct or best solution to a natural language problem [2]. In more recent years, natural language processing
Load more

Recommended publications
  • Aviation Speech Recognition System Using Artificial Intelligence
    SPEECH RECOGNITION SYSTEM OVERVIEW AVIATION SPEECH RECOGNITION SYSTEM USING ARTIFICIAL INTELLIGENCE Appareo’s embeddable AI model for air traffic control (ATC) transcription makes the company one of the world leaders in aviation artificial intelligence. TECHNOLOGY APPLICATION As featured in Appareo’s two flight apps for pilots, Stratus InsightTM and Stratus Horizon Pro, the custom speech recognition system ATC Transcription makes the company one of the world leaders in aviation artificial intelligence. ATC Transcription is a recurrent neural network that transcribes analog or digital aviation audio into text in near-real time. This in-house artificial intelligence, trained with Appareo’s proprietary dataset of flight-deck audio, takes the terabytes of training data and its accompanying transcriptions and reduces that content to a powerful 160 MB model that can be run inside the aircraft. BENEFITS OF ATC TRANSCRIPTION: • Provides improved situational awareness by identifying, capturing, and presenting ATC communications relevant to your aircraft’s operation for review or replay. • Provides the ability to stream transcribed speech to on-board (avionic) or off-board (iPad/ iPhone) display sources. • Provides a continuous representation of secondary audio channels (ATIS, AWOS, etc.) for review in textual form, allowing your attention to focus on a single channel. • ...and more. What makes ATC Transcription special is the context-specific training work that Appareo has done, using state-of-the-art artificial intelligence development techniques, to create an AI that understands the aviation industry vocabulary. Other natural language processing work is easily confused by the cadence, noise, and vocabulary of the aviation industry, yet Appareo’s groundbreaking work has overcome these barriers and brought functional artificial intelligence to the cockpit.
    [Show full text]
  • A Sentiment-Based Chat Bot
    A sentiment-based chat bot Automatic Twitter replies with Python ALEXANDER BLOM SOFIE THORSEN Bachelor’s essay at CSC Supervisor: Anna Hjalmarsson Examiner: Mårten Björkman E-mail: [email protected], [email protected] Abstract Natural language processing is a field in computer science which involves making computers derive meaning from human language and input as a way of interacting with the real world. Broadly speaking, sentiment analysis is the act of determining the attitude of an author or speaker, with respect to a certain topic or the overall context and is an application of the natural language processing field. This essay discusses the implementation of a Twitter chat bot that uses natural language processing and sentiment analysis to construct a be- lievable reply. This is done in the Python programming language, using a statistical method called Naive Bayes classifying supplied by the NLTK Python package. The essay concludes that applying natural language processing and sentiment analysis in this isolated fashion was simple, but achieving more complex tasks greatly increases the difficulty. Referat Natural language processing är ett fält inom datavetenskap som innefat- tar att få datorer att förstå mänskligt språk och indata för att på så sätt kunna interagera med den riktiga världen. Sentiment analysis är, generellt sagt, akten av att försöka bestämma känslan hos en författare eller talare med avseende på ett specifikt ämne eller sammanhang och är en applicering av fältet natural language processing. Den här rapporten diskuterar implementeringen av en Twitter-chatbot som använder just natural language processing och sentiment analysis för att kunna svara på tweets genom att använda känslan i tweetet.
    [Show full text]
  • Notice Report
    NetApp Notice Report Copyright 2020 About this document The following copyright statements and licenses apply to the software components that are distributed with the Active IQ Platform product released on 2020-05-13 05:40:47. This product does not necessarily use all the software components referred to below. Where required, source code is published at the following location. ftp://ftp.netapp.com/frm-ntap/opensource 1 Components: Component License Achilles Core 5.3.0 Apache License 2.0 An open source Java toolkit for Amazon S3 0.9.0 Apache License 2.0 Apache Avro 1.7.6-cdh5.4.2.1.1 Apache License 2.0 Apache Avro 1.7.6-cdh5.4.4 Apache License 2.0 Apache Commons BeanUtils 1.7.0 Apache License 2.0 Apache Commons BeanUtils 1.8.0 Apache License 2.0 Apache Commons CLI 1.2 Apache License 2.0 Apache Commons Codec 1.9 Apache License 2.0 Apache Commons Collections 3.2.1 Apache License 2.0 Apache Commons Compress 1.4.1 Apache License 2.0 Apache Commons Configuration 1.6 Apache License 2.0 Apache Commons Digester 1.8 Apache License 1.1 Apache Commons Lang 2.6 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 Apache Commons Math 3.1.1 Apache License 2.0 Apache Commons Net 3.1 Apache License 2.0 Apache Directory API ASN.1 API 1.0.0-M20 Apache License 2.0 Apache Directory LDAP API Utilities 1.0.0-M19 Apache License 2.0 Apache Directory LDAP API Utilities 1.0.0-M20 Apache License 2.0 Apache Directory Studio 2.0.0-M15 Apache License 2.0 Apache Directory Studio 2.0.0-M20 Apache License 2.0 2 Apache Hadoop Annotations 2.6.0-cdh5.4.4 Apache
    [Show full text]
  • Applying Ensembling Methods to BERT to Boost Model Performance
    Applying Ensembling Methods to BERT to Boost Model Performance Charlie Xu, Solomon Barth, Zoe Solis Department of Computer Science Stanford University cxu2, sbarth, zoesolis @stanford.edu Abstract Due to its wide range of applications and difficulty to solve, Machine Comprehen- sion has become a significant point of focus in AI research. Using the SQuAD 2.0 dataset as a metric, BERT models have been able to achieve superior performance to most other models. Ensembling BERT with other models has yielded even greater results, since these ensemble models are able to utilize the relative strengths and adjust for the relative weaknesses of each of the individual submodels. In this project, we present various ensemble models that leveraged BiDAF and multiple different BERT models to achieve superior performance. We also studied the effects how dataset manipulations and different ensembling methods can affect the performance of our models. Lastly, we attempted to develop the Synthetic Self- Training Question Answering model. In the end, our best performing multi-BERT ensemble obtained an EM score of 73.576 and an F1 score of 76.721. 1 Introduction Machine Comprehension (MC) is a complex task in NLP that aims to understand written language. Question Answering (QA) is one of the major tasks within MC, requiring a model to provide an answer, given a contextual text passage and question. It has a wide variety of applications, including search engines and voice assistants, making it a popular problem for NLP researchers. The Stanford Question Answering Dataset (SQuAD) is a very popular means for training, tuning, testing, and comparing question answering systems.
    [Show full text]
  • Automatic Correction of Real-Word Errors in Spanish Clinical Texts
    sensors Article Automatic Correction of Real-Word Errors in Spanish Clinical Texts Daniel Bravo-Candel 1,Jésica López-Hernández 1, José Antonio García-Díaz 1 , Fernando Molina-Molina 2 and Francisco García-Sánchez 1,* 1 Department of Informatics and Systems, Faculty of Computer Science, Campus de Espinardo, University of Murcia, 30100 Murcia, Spain; [email protected] (D.B.-C.); [email protected] (J.L.-H.); [email protected] (J.A.G.-D.) 2 VÓCALI Sistemas Inteligentes S.L., 30100 Murcia, Spain; [email protected] * Correspondence: [email protected]; Tel.: +34-86888-8107 Abstract: Real-word errors are characterized by being actual terms in the dictionary. By providing context, real-word errors are detected. Traditional methods to detect and correct such errors are mostly based on counting the frequency of short word sequences in a corpus. Then, the probability of a word being a real-word error is computed. On the other hand, state-of-the-art approaches make use of deep learning models to learn context by extracting semantic features from text. In this work, a deep learning model were implemented for correcting real-word errors in clinical text. Specifically, a Seq2seq Neural Machine Translation Model mapped erroneous sentences to correct them. For that, different types of error were generated in correct sentences by using rules. Different Seq2seq models were trained and evaluated on two corpora: the Wikicorpus and a collection of three clinical datasets. The medicine corpus was much smaller than the Wikicorpus due to privacy issues when dealing Citation: Bravo-Candel, D.; López-Hernández, J.; García-Díaz, with patient information.
    [Show full text]
  • A Comparison of Online Automatic Speech Recognition Systems and the Nonverbal Responses to Unintelligible Speech
    A Comparison of Online Automatic Speech Recognition Systems and the Nonverbal Responses to Unintelligible Speech Joshua Y. Kim1, Chunfeng Liu1, Rafael A. Calvo1*, Kathryn McCabe2, Silas C. R. Taylor3, Björn W. Schuller4, Kaihang Wu1 1 University of Sydney, Faculty of Engineering and Information Technologies 2 University of California, Davis, Psychiatry and Behavioral Sciences 3 University of New South Wales, Faculty of Medicine 4 Imperial College London, Department of Computing Abstract large-scale commercial products such as Google Home and Amazon Alexa. Mainstream ASR Automatic Speech Recognition (ASR) systems use only voice as inputs, but there is systems have proliferated over the recent potential benefit in using multi-modal data in order years to the point that free platforms such to improve accuracy [1]. Compared to machines, as YouTube now provide speech recognition services. Given the wide humans are highly skilled in utilizing such selection of ASR systems, we contribute to unstructured multi-modal information. For the field of automatic speech recognition example, a human speaker is attuned to nonverbal by comparing the relative performance of behavior signals and actively looks for these non- two sets of manual transcriptions and five verbal ‘hints’ that a listener understands the speech sets of automatic transcriptions (Google content, and if not, they adjust their speech Cloud, IBM Watson, Microsoft Azure, accordingly. Therefore, understanding nonverbal Trint, and YouTube) to help researchers to responses to unintelligible speech can both select accurate transcription services. In improve future ASR systems to mark uncertain addition, we identify nonverbal behaviors transcriptions, and also help to provide feedback so that are associated with unintelligible speech, as indicated by high word error that the speaker can improve his or her verbal rates.
    [Show full text]
  • Learned in Speech Recognition: Contextual Acoustic Word Embeddings
    LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS Shruti Palaskar∗, Vikas Raunak∗ and Florian Metze Carnegie Mellon University, Pittsburgh, PA, U.S.A. fspalaska j vraunak j fmetze [email protected] ABSTRACT model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13]. Using this model, we jointly learn to End-to-end acoustic-to-word speech recognition models have re- automatically segment and classify input speech into individual cently gained popularity because they are easy to train, scale well to words, hence getting rid of the problem of chunking or requiring large amounts of training data, and do not require a lexicon. In addi- pre-defined word boundaries. As our A2W model is trained at the tion, word models may also be easier to integrate with downstream utterance level, we show that we can not only learn acoustic word tasks such as spoken language understanding, because inference embeddings, but also learn them in the proper context of their con- (search) is much simplified compared to phoneme, character or any taining sentence. We also evaluate our contextual acoustic word other sort of sub-word units. In this paper, we describe methods embeddings on a spoken language understanding task, demonstrat- to construct contextual acoustic word embeddings directly from a ing that they can be useful in non-transcription downstream tasks. supervised sequence-to-sequence acoustic-to-word speech recog- Our main contributions in this paper are the following: nition model using the learned attention distribution. On a suite 1. We demonstrate the usability of attention not only for aligning of 16 standard sentence evaluation tasks, our embeddings show words to acoustic frames without any forced alignment but also for competitive performance against a word2vec model trained on the constructing Contextual Acoustic Word Embeddings (CAWE).
    [Show full text]
  • Aconcorde: Towards a Proper Concordance for Arabic
    aConCorde: towards a proper concordance for Arabic Andrew Roberts, Dr Latifa Al-Sulaiti and Eric Atwell School of Computing University of Leeds LS2 9JT United Kingdom {andyr,latifa,eric}@comp.leeds.ac.uk July 12, 2005 Abstract Arabic corpus linguistics is currently enjoying a surge in activity. As the growth in the number of available Arabic corpora continues, there is an increased need for robust tools that can process this data, whether it be for research or teaching. One such tool that is useful for both groups is the concordancer — a simple tool for displaying a specified target word in its context. However, obtaining one that can reliably cope with the Arabic lan- guage had proved extremely difficult. Therefore, aConCorde was created to provide such a tool to the community. 1 Introduction A concordancer is a simple tool for summarising the contents of corpora based on words of interest to the user. Otherwise, manually navigating through (of- ten very large) corpora would be a long and tedious task. Concordancers are therefore extremely useful as a time-saving tool, but also much more. By isolat- ing keywords in their contexts, linguists can use concordance output to under- stand the behaviour of interesting words. Lexicographers can use the evidence to find if a word has multiple senses, and also towards defining their meaning. There is also much research and discussion about how concordance tools can be beneficial for data-driven language learning (Johns, 1990). A number of studies have demonstrated that providing access to corpora and concordancers benefited students learning a second language.
    [Show full text]
  • Natural Language Toolkit Based Morphology Linguistics
    Published by : International Journal of Engineering Research & Technology (IJERT) http://www.ijert.org ISSN: 2278-0181 Vol. 9 Issue 01, January-2020 Natural Language Toolkit based Morphology Linguistics Alifya Khan Pratyusha Trivedi Information Technology Information Technology Vidyalankar Institute of Technology Vidyalankar Institute of Technology, Mumbai, India Mumbai, India Karthik Ashok Prof. Kanchan Dhuri Information Technology, Information Technology, Vidyalankar Institute of Technology, Vidyalankar Institute of Technology, Mumbai, India Mumbai, India Abstract— In the current scenario, there are different apps, of text to different languages, grammar check, etc. websites, etc. to carry out different functionalities with respect Generally, all these functionalities are used in tandem with to text such as grammar correction, translation of text, each other. For e.g., to read a sign board in a different extraction of text from image or videos, etc. There is no app or language, the first step is to extract text from that image and a website where a user can get all these functions/features at translate it to any respective language as required. Hence, to one place and hence the user is forced to install different apps do this one has to switch from application to application or visit different websites to carry out those functions. The which can be time consuming. To overcome this problem, proposed system identifies this problem and tries to overcome an integrated environment is built where all these it by providing various text-based features at one place, so that functionalities are available. the user will not have to hop from app to app or website to website to carry out various functions.
    [Show full text]
  • The Dzone Guide to Volume Ii
    THE D ZONE GUIDE TO MODERN JAVA VOLUME II BROUGHT TO YOU IN PARTNERSHIP WITH DZONE.COM/GUIDES DZONE’S 2016 GUIDE TO MODERN JAVA Dear Reader, TABLE OF CONTENTS 3 EXECUTIVE SUMMARY Why isn’t Java dead after more than two decades? A few guesses: Java is (still) uniquely portable, readable to 4 KEY RESEARCH FINDINGS fresh eyes, constantly improving its automatic memory management, provides good full-stack support for high- 10 THE JAVA 8 API DESIGN PRINCIPLES load web services, and enjoys a diverse and enthusiastic BY PER MINBORG community, mature toolchain, and vigorous dependency 13 PROJECT JIGSAW IS COMING ecosystem. BY NICOLAI PARLOG Java is growing with us, and we’re growing with Java. Java 18 REACTIVE MICROSERVICES: DRIVING APPLICATION 8 just expanded our programming paradigm horizons (add MODERNIZATION EFFORTS Church and Curry to Kay and Gosling) and we’re still learning BY MARKUS EISELE how to mix functional and object-oriented code. Early next 21 CHECKLIST: 7 HABITS OF SUPER PRODUCTIVE JAVA DEVELOPERS year Java 9 will add a wealth of bigger-picture upgrades. 22 THE ELEMENTS OF MODERN JAVA STYLE But Java remains vibrant for many more reasons than the BY MICHAEL TOFINETTI robustness of the language and the comprehensiveness of the platform. JVM languages keep multiplying (Kotlin went 28 12 FACTORS AND BEYOND IN JAVA GA this year!), Android keeps increasing market share, and BY PIETER HUMPHREY AND MARK HECKLER demand for Java developers (measuring by both new job 31 DIVING DEEPER INTO JAVA DEVELOPMENT posting frequency and average salary) remains high. The key to the modernization of Java is not a laundry-list of JSRs, but 34 INFOGRAPHIC: JAVA'S IMPACT ON THE MODERN WORLD rather the energy of the Java developer community at large.
    [Show full text]
  • Hunspell – the Free Spelling Checker
    Hunspell – The free spelling checker About Hunspell Hunspell is a spell checker and morphological analyzer library and program designed for languages with rich morphology and complex word compounding or character encoding. Hunspell interfaces: Ispell-like terminal interface using Curses library, Ispell pipe interface, OpenOffice.org UNO module. Main features of Hunspell spell checker and morphological analyzer: - Unicode support (affix rules work only with the first 65535 Unicode characters) - Morphological analysis (in custom item and arrangement style) and stemming - Max. 65535 affix classes and twofold affix stripping (for agglutinative languages, like Azeri, Basque, Estonian, Finnish, Hungarian, Turkish, etc.) - Support complex compoundings (for example, Hungarian and German) - Support language specific features (for example, special casing of Azeri and Turkish dotted i, or German sharp s) - Handle conditional affixes, circumfixes, fogemorphemes, forbidden words, pseudoroots and homonyms. - Free software (LGPL, GPL, MPL tri-license) Usage The src/tools dictionary contains ten executables after compiling (or some of them are in the src/win_api): affixcompress: dictionary generation from large (millions of words) vocabularies analyze: example of spell checking, stemming and morphological analysis chmorph: example of automatic morphological generation and conversion example: example of spell checking and suggestion hunspell: main program for spell checking and others (see manual) hunzip: decompressor of hzip format hzip: compressor of
    [Show full text]
  • Practice with Python
    CSI4108-01 ARTIFICIAL INTELLIGENCE 1 Word Embedding / Text Processing Practice with Python 2018. 5. 11. Lee, Gyeongbok Practice with Python 2 Contents • Word Embedding – Libraries: gensim, fastText – Embedding alignment (with two languages) • Text/Language Processing – POS Tagging with NLTK/koNLPy – Text similarity (jellyfish) Practice with Python 3 Gensim • Open-source vector space modeling and topic modeling toolkit implemented in Python – designed to handle large text collections, using data streaming and efficient incremental algorithms – Usually used to make word vector from corpus • Tutorial is available here: – https://github.com/RaRe-Technologies/gensim/blob/develop/tutorials.md#tutorials – https://rare-technologies.com/word2vec-tutorial/ • Install – pip install gensim Practice with Python 4 Gensim for Word Embedding • Logging • Input Data: list of word’s list – Example: I have a car , I like the cat → – For list of the sentences, you can make this by: Practice with Python 5 Gensim for Word Embedding • If your data is already preprocessed… – One sentence per line, separated by whitespace → LineSentence (just load the file) – Try with this: • http://an.yonsei.ac.kr/corpus/example_corpus.txt From https://radimrehurek.com/gensim/models/word2vec.html Practice with Python 6 Gensim for Word Embedding • If the input is in multiple files or file size is large: – Use custom iterator and yield From https://rare-technologies.com/word2vec-tutorial/ Practice with Python 7 Gensim for Word Embedding • gensim.models.Word2Vec Parameters – min_count:
    [Show full text]