DOCSLIB.ORG

California State University, Northridge a Comparison of Lexicographical and Machine Learning Approaches to Sentiment Analysi

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
California State University, Northridge a Comparison of Lexicographical and Machine Learning Approaches to Sentiment Analysi California State University, Northridge A Comparison of Lexicographical and Machine Learning Approaches to Sentiment Analysis A thesis submitted in partial fulfillment of the requirements For the degree of Master of Science in Computer Science By Jeffrey Yoshida May 2019 The thesis of Jeffrey Yoshida is approved: ___________________________________ ____________________ Robert McIlhenny, Ph.D. Date ___________________________________ ____________________ Kyle Dewey, Ph.D. Date ___________________________________ ____________________ George Wang, Ph.D., Chair Date California State University, Northridge ii Acknowledgments I would like to thank Dr. George Wang for being my committee chair and for all his help throughout the entire thesis process. Thank you also to Dr. Robert McIlhenney for making time for me despite being on countless other thesis committees and Dr. Kyle Dewey for giving me tremendous feedback on my paper. Finally, I would like to thank my parents for the support they have given me all these years. I would not be where I am today without your constant support. iii Table of Contents SIGNATURES ....................................................................................................... ii ACKNOWLEDGEMENTS …………………………………………………….. iii LIST OF FIGURES ............................................................................................... vi LIST OF TABLES ............................................................................................... vii ABSTRACT ........................................................................................................ viii 1. INTRODUCTION ............................................................................................ 1 1.1 Background …………………………….…………………………………..1 1.2 Approaches to Sentiment Analysis …………………………………………2 2. RELATED WORKS…………......................................................................... 4 3. TECHNICAL APPROACH.............................................................................. 5 3.1 Data Exploration ........................................................................................... 5 3.2 General Workflow ........................................................................................ 7 3.3 Hardware ...................................................................................................... 8 3.4 Software …………………………………………………………………....9 4. DATA PREPROCESSING ........................................................................... 10 5. WORD EMBEDDINGS ................................................................................. 13 5.1 Count-Based Methods …………………………………………………… 13 5.2 Prediction-Based Methods ………………………………………………. 15 5.3 Word Embedding Implementations ……………………………………... 17 6. MODELS ........................................................................................................ 19 6.1 SentiWordNet ……………………………………………………………. 20 6.2 AFINN …………………………………………………………….…..…. 20 6.3 Logistic Regression …………………………………………...…………. 21 6.4 Support Vector Machine …………………………………………………. 21 iv 6.5 Random Forest …………………………………..……………………….. 23 6.6 Naïve Bayes Classifier …………………………...………………………. 24 6.7 Multilayer Perceptron …………………….………...……………………. 25 6.8 Convolutional Neural Network …………………………………..………. 27 7. RESULTS ....................................................................................................... 29 7.1 Metrics ……………………………………………………...……………. 29 7.2 Model Results ……………………………………………………………. 31 8. CONCLUSION ............................................................................................... 35 REFERENCES ………………………………………………………………….36 v List of Figures Figure 1 – Histogram of Review Scores in the Amazon Fine Foods Dataset ................... 6 Figure 2 – Histogram of Positive and Negative Reviews in the Amazon Fine Foods Dataset ………………………………………………….................................................... 6 Figure 3 – Workflow for Project …………………............................................................ 7 Figure 4 – Example of Discrete Representation of Words .............................................. 13 Figure 5 – Word2vec Example …………………………................................................ 16 Figure 6 – SVM with Linearly Separable Data …………………………....................... 22 Figure 7 – Example of Linearly Inseparable Data ………………………....................... 23 Figure 8 – Perceptron Diagram ........................................................................................ 26 Figure 9 – Multilayer Perceptron Diagram ...................................................................... 27 Figure 10 –True Positives, False Positives, False Negatives, and True Negatives …..... 29 vi List of Tables Table 1 – Sample Count Vectorization ............................................................................ 14 Table 2 –10 Most Similar Words to “tasty” in the Amazon Fine Foods Reviews Dataset Word2vec Model.............................................................................................................. 17 Table 3 – All Predictive Model and Word Embedding Combinations Used …............... 19 Table 4 – Overall Results ……………………................................................................. 31 Table 5 – Single-Class Classification Results …………...……...................................... 31 Table 6 – Domain-Specific Words Similar to Good ........................................................ 34 Table 7 – Pre-trained Model Words Similar to Good ...................................................... 34 vii Abstract A Comparison of Lexicographical and Machine Learning Approaches to Sentiment Analysis By Jeffrey Yoshida Master of Science in Computer Science Sentiment analysis is an area of computer science research that deals with extracting subjective information such as opinions, attitudes and emotions from text data. Although sentiment analysis is a technically challenging task, the potential benefits it can yield are great. An industry particularly interested in sentiment analysis is the e- commerce industry where major companies often receive far more product reviews than can be handled manually. These reviews if analyzed accurately, can become an invaluable source of consumer insights that go beyond numerical review scores. The goal of this thesis is to analyze and compare the performance of different methods of sentiment analysis on a dataset of Amazon product reviews. viii 1. Introduction 1.1 Background Due to rapid improvements in internet technology the amount of data being produced, consumed and stored has skyrocketed. In 2013 it was estimated that 90% of the worlds stored data was generated in only the two years prior [1]. What is more astounding is that an estimated 95% of this data is unstructured data such as text, videos and audio [2]. Because of this rich abundance of unstructured data, there has never been a greater need for methods of extracting valuable insights from data. This increased need to generate insights from unstructured data has fueled interest in research areas such as sentiment analysis. Sentiment analysis is generally defined as an area of computer science research that deals with extracting subjective information such as opinions, attitudes and emotions from text data [3]. For the purposes of this paper, sentiment analysis is the task of computationally determining whether a body of text has a positive or negative tone. Whether it be understanding the social sentiment regarding a clothing brand or political opinions on a contentious specific topic, sentiment analysis can provide crucial insights that are impractical to obtain by manual inspection of data. With the growth of e- commerce, the need for in-depth consumer insights has never been greater. A study of online consumer-generated reviews by Comscore, a large media analytics company, found that consumers were willing to play at least 20% more for products and services that received a 5-star rating when compared to the same service that had received a 4-star rating [4]. In food, legal and hotel services, consumers were willing to pay between 40% and 99% more [4]. The influence of product reviews is further supported by a 2018 study 1 done by a site called brightlocal which found that 95% of people between the age of 18- 34 read reviews of local businesses [5]. More importantly this survey also found that 57% of consumers will only use a business if it has 4 or more stars. Because of this, understanding product sentiment is valuable. 1.2 Approaches to Sentiment Analysis There are two main approaches to sentiment analysis: the lexicographical approach and the machine learning approach [6]. In the lexicographical approach to sentiment analysis, the overall attitude of a body of text is determined by analyzing individual words or phrases. The polarity of each individual word is determined using a sentiment dictionary, a specialized dictionary that gives the word a sentiment polarity score. The sentiment of the entire body of text is computed as the sum of the polarity scores of the individual words or phrases in the text. There are many different sentiment dictionaries available such as SentiWordNet, AFINN and Opinion Lexicon [7,8, 9, 10, 11]. The other common method of performing sentiment analysis is by using machine learning. Like many other research fields, the field of sentiment analysis has been influenced by the rapid growth of machine learning [12, 13]. By creating a training dataset consisting of pieces of text labeled as positive or negative, a model can
Load more

Recommended publications
  • Malware Classification with BERT
    San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-25-2021 Malware Classification with BERT Joel Lawrence Alvares Follow this and additional works at: https://scholarworks.sjsu.edu/etd_projects Part of the Artificial Intelligence and Robotics Commons, and the Information Security Commons Malware Classification with Word Embeddings Generated by BERT and Word2Vec Malware Classification with BERT Presented to Department of Computer Science San José State University In Partial Fulfillment of the Requirements for the Degree By Joel Alvares May 2021 Malware Classification with Word Embeddings Generated by BERT and Word2Vec The Designated Project Committee Approves the Project Titled Malware Classification with BERT by Joel Lawrence Alvares APPROVED FOR THE DEPARTMENT OF COMPUTER SCIENCE San Jose State University May 2021 Prof. Fabio Di Troia Department of Computer Science Prof. William Andreopoulos Department of Computer Science Prof. Katerina Potika Department of Computer Science 1 Malware Classification with Word Embeddings Generated by BERT and Word2Vec ABSTRACT Malware Classification is used to distinguish unique types of malware from each other. This project aims to carry out malware classification using word embeddings which are used in Natural Language Processing (NLP) to identify and evaluate the relationship between words of a sentence. Word embeddings generated by BERT and Word2Vec for malware samples to carry out multi-class classification. BERT is a transformer based pre- trained natural language processing (NLP) model which can be used for a wide range of tasks such as question answering, paraphrase generation and next sentence prediction. However, the attention mechanism of a pre-trained BERT model can also be used in malware classification by capturing information about relation between each opcode and every other opcode belonging to a malware family.
    [Show full text]
  • Fun with Hyperplanes: Perceptrons, Svms, and Friends
    Perceptrons, SVMs, and Friends: Some Discriminative Models for Classification Parallel to AIMA 18.1, 18.2, 18.6.3, 18.9 The Automatic Classification Problem Assign object/event or sequence of objects/events to one of a given finite set of categories. • Fraud detection for credit card transactions, telephone calls, etc. • Worm detection in network packets • Spam filtering in email • Recommending articles, books, movies, music • Medical diagnosis • Speech recognition • OCR of handwritten letters • Recognition of specific astronomical images • Recognition of specific DNA sequences • Financial investment Machine Learning methods provide one set of approaches to this problem CIS 391 - Intro to AI 2 Universal Machine Learning Diagram Feature Things to Magic Vector Classification be Classifier Represent- Decision classified Box ation CIS 391 - Intro to AI 3 Example: handwritten digit recognition Machine learning algorithms that Automatically cluster these images Use a training set of labeled images to learn to classify new images Discover how to account for variability in writing style CIS 391 - Intro to AI 4 A machine learning algorithm development pipeline: minimization Problem statement Given training vectors x1,…,xN and targets t1,…,tN, find… Mathematical description of a cost function Mathematical description of how to minimize/maximize the cost function Implementation r(i,k) = s(i,k) – maxj{s(i,j)+a(i,j)} … CIS 391 - Intro to AI 5 Universal Machine Learning Diagram Today: Perceptron, SVM and Friends Feature Things to Magic Vector
    [Show full text]
  • Learned in Speech Recognition: Contextual Acoustic Word Embeddings
    LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS Shruti Palaskar∗, Vikas Raunak∗ and Florian Metze Carnegie Mellon University, Pittsburgh, PA, U.S.A. fspalaska j vraunak j fmetze [email protected] ABSTRACT model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13]. Using this model, we jointly learn to End-to-end acoustic-to-word speech recognition models have re- automatically segment and classify input speech into individual cently gained popularity because they are easy to train, scale well to words, hence getting rid of the problem of chunking or requiring large amounts of training data, and do not require a lexicon. In addi- pre-defined word boundaries. As our A2W model is trained at the tion, word models may also be easier to integrate with downstream utterance level, we show that we can not only learn acoustic word tasks such as spoken language understanding, because inference embeddings, but also learn them in the proper context of their con- (search) is much simplified compared to phoneme, character or any taining sentence. We also evaluate our contextual acoustic word other sort of sub-word units. In this paper, we describe methods embeddings on a spoken language understanding task, demonstrat- to construct contextual acoustic word embeddings directly from a ing that they can be useful in non-transcription downstream tasks. supervised sequence-to-sequence acoustic-to-word speech recog- Our main contributions in this paper are the following: nition model using the learned attention distribution. On a suite 1. We demonstrate the usability of attention not only for aligning of 16 standard sentence evaluation tasks, our embeddings show words to acoustic frames without any forced alignment but also for competitive performance against a word2vec model trained on the constructing Contextual Acoustic Word Embeddings (CAWE).
    [Show full text]
  • Introduction to Machine Learning
    Introduction to Machine Learning Perceptron Barnabás Póczos Contents History of Artificial Neural Networks Definitions: Perceptron, Multi-Layer Perceptron Perceptron algorithm 2 Short History of Artificial Neural Networks 3 Short History Progression (1943-1960) • First mathematical model of neurons ▪ Pitts & McCulloch (1943) • Beginning of artificial neural networks • Perceptron, Rosenblatt (1958) ▪ A single neuron for classification ▪ Perceptron learning rule ▪ Perceptron convergence theorem Degression (1960-1980) • Perceptron can’t even learn the XOR function • We don’t know how to train MLP • 1963 Backpropagation… but not much attention… Bryson, A.E.; W.F. Denham; S.E. Dreyfus. Optimal programming problems with inequality constraints. I: Necessary conditions for extremal solutions. AIAA J. 1, 11 (1963) 2544-2550 4 Short History Progression (1980-) • 1986 Backpropagation reinvented: ▪ Rumelhart, Hinton, Williams: Learning representations by back-propagating errors. Nature, 323, 533—536, 1986 • Successful applications: ▪ Character recognition, autonomous cars,… • Open questions: Overfitting? Network structure? Neuron number? Layer number? Bad local minimum points? When to stop training? • Hopfield nets (1982), Boltzmann machines,… 5 Short History Degression (1993-) • SVM: Vapnik and his co-workers developed the Support Vector Machine (1993). It is a shallow architecture. • SVM and Graphical models almost kill the ANN research. • Training deeper networks consistently yields poor results. • Exception: deep convolutional neural networks, Yann LeCun 1998. (discriminative model) 6 Short History Progression (2006-) Deep Belief Networks (DBN) • Hinton, G. E, Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554. • Generative graphical model • Based on restrictive Boltzmann machines • Can be trained efficiently Deep Autoencoder based networks Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H.
    [Show full text]
  • Lecture 26 Word Embeddings and Recurrent Nets
    CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Where we’re at Lecture 25: Word Embeddings and neural LMs Lecture 26: Recurrent networks Lecture 26 Lecture 27: Sequence labeling and Seq2Seq Lecture 28: Review for the final exam Word Embeddings and Lecture 29: In-class final exam Recurrent Nets Julia Hockenmaier [email protected] 3324 Siebel Center CS447: Natural Language Processing (J. Hockenmaier) !2 What are neural nets? Simplest variant: single-layer feedforward net For binary Output unit: scalar y classification tasks: Single output unit Input layer: vector x Return 1 if y > 0.5 Recap Return 0 otherwise For multiclass Output layer: vector y classification tasks: K output units (a vector) Input layer: vector x Each output unit " yi = class i Return argmaxi(yi) CS447: Natural Language Processing (J. Hockenmaier) !3 CS447: Natural Language Processing (J. Hockenmaier) !4 Multi-layer feedforward networks Multiclass models: softmax(yi) We can generalize this to multi-layer feedforward nets Multiclass classification = predict one of K classes. Return the class i with the highest score: argmaxi(yi) Output layer: vector y In neural networks, this is typically done by using the softmax N Hidden layer: vector hn function, which maps real-valued vectors in R into a distribution … … … over the N outputs … … … … … …. For a vector z = (z0…zK): P(i) = softmax(zi) = exp(zi) ∕ ∑k=0..K exp(zk) Hidden layer: vector h1 (NB: This is just logistic regression) Input layer: vector x CS447: Natural Language Processing (J. Hockenmaier) !5 CS447: Natural Language Processing (J. Hockenmaier) !6 Neural Language Models LMs define a distribution over strings: P(w1….wk) LMs factor P(w1….wk) into the probability of each word: " P(w1….wk) = P(w1)·P(w2|w1)·P(w3|w1w2)·…· P(wk | w1….wk#1) A neural LM needs to define a distribution over the V words in Neural Language the vocabulary, conditioned on the preceding words.
    [Show full text]
  • Audio Event Classification Using Deep Learning in an End-To-End Approach
    Audio Event Classification using Deep Learning in an End-to-End Approach Master thesis Jose Luis Diez Antich Aalborg University Copenhagen A. C. Meyers Vænge 15 2450 Copenhagen SV Denmark Title: Abstract: Audio Event Classification using Deep Learning in an End-to-End Approach The goal of the master thesis is to study the task of Sound Event Classification Participant(s): using Deep Neural Networks in an end- Jose Luis Diez Antich to-end approach. Sound Event Classifi- cation it is a multi-label classification problem of sound sources originated Supervisor(s): from everyday environments. An auto- Hendrik Purwins matic system for it would many applica- tions, for example, it could help users of hearing devices to understand their sur- Page Numbers: 38 roundings or enhance robot navigation systems. The end-to-end approach con- Date of Completion: sists in systems that learn directly from June 16, 2017 data, not from features, and it has been recently applied to audio and its results are remarkable. Even though the re- sults do not show an improvement over standard approaches, the contribution of this thesis is an exploration of deep learning architectures which can be use- ful to understand how networks process audio. The content of this report is freely available, but publication (with reference) may only be pursued due to agreement with the author. Contents 1 Introduction1 1.1 Scope of this work.............................2 2 Deep Learning3 2.1 Overview..................................3 2.2 Multilayer Perceptron...........................4
    [Show full text]
  • 4 Nonlinear Regression and Multilayer Perceptron
    4 Nonlinear Regression and Multilayer Perceptron In nonlinear regression the output variable y is no longer a linear function of the regression parameters plus additive noise. This means that estimation of the parameters is harder. It does not reduce to minimizing a convex energy functions { unlike the methods we described earlier. The perceptron is an analogy to the neural networks in the brain (over-simplified). It Pd receives a set of inputs y = j=1 !jxj + !0, see Figure (3). Figure 3: Idealized neuron implementing a perceptron. It has a threshold function which can be hard or soft. The hard one is ζ(a) = 1; if a > 0, ζ(a) = 0; otherwise. The soft one is y = σ(~!T ~x) = 1=(1 + e~!T ~x), where σ(·) is the sigmoid function. There are a variety of different algorithms to train a perceptron from labeled examples. Example: The quadratic error: t t 1 t t 2 E(~!j~x ; y ) = 2 (y − ~! · ~x ) ; for which the update rule is ∆!t = −∆ @E = +∆(yt~! · ~xt)~xt. Introducing the sigmoid j @!j function rt = sigmoid(~!T ~xt), we have t t P t t t t E(~!j~x ; y ) = − i ri log yi + (1 − ri) log(1 − yi ) , and the update rule is t t t t ∆!j = −η(r − y )xj, where η is the learning factor. I.e, the update rule is the learning factor × (desired output { actual output)× input. 10 4.1 Multilayer Perceptrons Multilayer perceptrons were developed to address the limitations of perceptrons (introduced in subsection 2.1) { i.e.
    [Show full text]
  • Arxiv:2007.00183V2 [Eess.AS] 24 Nov 2020
    WHOLE-WORD SEGMENTAL SPEECH RECOGNITION WITH ACOUSTIC WORD EMBEDDINGS Bowen Shi, Shane Settle, Karen Livescu TTI-Chicago, USA fbshi,settle.shane,[email protected] ABSTRACT Segmental models are sequence prediction models in which scores of hypotheses are based on entire variable-length seg- ments of frames. We consider segmental models for whole- word (“acoustic-to-word”) speech recognition, with the feature vectors defined using vector embeddings of segments. Such models are computationally challenging as the number of paths is proportional to the vocabulary size, which can be orders of magnitude larger than when using subword units like phones. We describe an efficient approach for end-to-end whole-word segmental models, with forward-backward and Viterbi de- coding performed on a GPU and a simple segment scoring function that reduces space complexity. In addition, we inves- tigate the use of pre-training via jointly trained acoustic word embeddings (AWEs) and acoustically grounded word embed- dings (AGWEs) of written word labels. We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional Fig. 1. Whole-word segmental model for speech recognition. (smaller) gains can be obtained by pre-training the word pre- Note: boundary frames are not shared. diction layer with AGWEs. Our final models improve over segmental models, where the sequence probability is com- prior A2W models. puted based on segment scores instead of frame probabilities. Index Terms— speech recognition, segmental model, Segmental models have a long history in speech recognition acoustic-to-word, acoustic word embeddings, pre-training research, but they have been used primarily for phonetic recog- nition or as phone-level acoustic models [11–18].
    [Show full text]
  • Overview of Lstms and Word2vec and a Bit About Compositional Distributional Semantics If There’S Time
    Overview of LSTMs and word2vec and a bit about compositional distributional semantics if there’s time Ann Copestake Computer Laboratory University of Cambridge November 2016 Outline RNNs and LSTMs Word2vec Compositional distributional semantics Some slides adapted from Aurelie Herbelot. Outline. RNNs and LSTMs Word2vec Compositional distributional semantics Motivation I Standard NNs cannot handle sequence information well. I Can pass them sequences encoded as vectors, but input vectors are fixed length. I Models are needed which are sensitive to sequence input and can output sequences. I RNN: Recurrent neural network. I Long short term memory (LSTM): development of RNN, more effective for most language applications. I More info: http://neuralnetworksanddeeplearning.com/ (mostly about simpler models and CNNs) https://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Sequences I Video frame categorization: strict time sequence, one output per input. I Real-time speech recognition: strict time sequence. I Neural MT: target not one-to-one with source, order differences. I Many language tasks: best to operate left-to-right and right-to-left (e.g., bi-LSTM). I attention: model ‘concentrates’ on part of input relevant at a particular point. Caption generation: treat image data as ordered, align parts of image with parts of caption. Recurrent Neural Networks http://colah.github.io/posts/ 2015-08-Understanding-LSTMs/ RNN language model: Mikolov et al, 2010 RNN as a language model I Input vector: vector for word at t concatenated to vector which is output from context layer at t − 1. I Performance better than n-grams but won’t capture ‘long-term’ dependencies: She shook her head.
    [Show full text]
  • Feedforward Neural Networks and Word Embeddings
    Feedforward Neural Networks and Word Embeddings Fabienne Braune1 1LMU Munich May 14th, 2017 Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 1 Outline 1 Linear models 2 Limitations of linear models 3 Neural networks 4 A neural language model 5 Word embeddings Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 2 Linear Models Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 3 Binary Classification with Linear Models Example: the seminar at < time > 4 pm will Classification task: Do we have an < time > tag in the current position? Word Lemma LexCat Case SemCat Tag the the Art low seminar seminar Noun low at at Prep low stime 4 4 Digit low pm pm Other low timeid will will Verb low Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 4 Feature Vector Encode context into feature vector: 1 bias term 1 2 -3 lemma the 1 3 -3 lemma giraffe 0 ... ... ... 102 -2 lemma seminar 1 103 -2 lemma giraffe 0 ... ... ... 202 -1 lemma at 1 203 -1 lemma giraffe 0 ... ... ... 302 +1 lemma 4 1 303 +1 lemma giraffe 0 ... ... ... Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 5 Dot product with (initial) weight vector 2 3 2 3 x0 = 1 w0 = 1:00 6 x = 1 7 6 w = 0:01 7 6 1 7 6 1 7 6 x = 0 7 6 w = 0:01 7 6 2 7 6 2 7 6 ··· 7 6 ··· 7 6 7 6 7 6x = 17 6x = 0:017 6 101 7 6 101 7 6 7 6 7 6x102 = 07 6x102 = 0:017 T 6 7 6 7 h(X )= X · Θ X = 6 ··· 7 Θ = 6 ··· 7 6 7 6 7 6x201 = 17 6x201 = 0:017 6 7
    [Show full text]
  • 4 Perceptron Learning
    4 Perceptron Learning 4.1 Learning algorithms for neural networks In the two preceding chapters we discussed two closely related models, McCulloch–Pitts units and perceptrons, but the question of how to find the parameters adequate for a given task was left open. If two sets of points have to be separated linearly with a perceptron, adequate weights for the comput- ing unit must be found. The operators that we used in the preceding chapter, for example for edge detection, used hand customized weights. Now we would like to find those parameters automatically. The perceptron learning algorithm deals with this problem. A learning algorithm is an adaptive method by which a network of com- puting units self-organizes to implement the desired behavior. This is done in some learning algorithms by presenting some examples of the desired input- output mapping to the network. A correction step is executed iteratively until the network learns to produce the desired response. The learning algorithm is a closed loop of presentation of examples and of corrections to the network parameters, as shown in Figure 4.1. network test input-output compute the examples error fix network parameters Fig. 4.1. Learning process in a parametric system R. Rojas: Neural Networks, Springer-Verlag, Berlin, 1996 78 4 Perceptron Learning In some simple cases the weights for the computing units can be found through a sequential test of stochastically generated numerical combinations. However, such algorithms which look blindly for a solution do not qualify as “learning”. A learning algorithm must adapt the network parameters accord- ing to previous experience until a solution is found, if it exists.
    [Show full text]
  • An Unsupervised Method for OCR Post-Correction and Spelling
    An Unsupervised method for OCR Post-Correction and Spelling Normalisation for Finnish Quan Duong,♣ Mika Hämäläinen,♣,♦ Simon Hengchen,♠ firstname.lastname@{helsinki.fi;gu.se} ♣University of Helsinki, ♦Rootroo Ltd, ♠Språkbanken Text – University of Gothenburg Abstract Historical corpora are known to contain errors introduced by OCR (optical character recognition) methods used in the digitization process, often said to be degrading the performance of NLP systems. Correcting these errors manually is a time-consuming process and a great part of the automatic approaches have been relying on rules or supervised machine learning. We build on previous work on fully automatic unsupervised extraction of parallel data to train a character- based sequence-to-sequence NMT (neural machine translation) model to conduct OCR error correction designed for English, and adapt it to Finnish by proposing solutions that take the rich morphology of the language into account. Our new method shows increased performance while remaining fully unsupervised, with the added benefit of spelling normalisation. The source code and models are available on GitHub1 and Zenodo2. 1 Introduction As many people dealing with digital humanities study historical data, the problem that researchers continue to face is the quality of their digitized corpora. There have been large scale projects in the past focusing on digitizing parts of cultural history using OCR models available in those times (see Benner (2003), Bremer-Laamanen (2006)). While OCR methods have substantially developed and im- proved in recent times, it is still not always possible to re-OCR text. Re-OCRing documents is also not a priority when not all collections are digitised, as is the case in many national libraries.
    [Show full text]