DOCSLIB.ORG

Chapter 6: Vector Semantics, Part II

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Chapter 6: Vector Semantics, Part II Dan Jurafsky and James Martin Speech and Language Processing Chapter 6: Vector Semantics, Part II Tf-idf and PPMI are sparse representations tf-idf and PPMI vectors are ◦long (length |V|= 20,000 to 50,000) ◦sparse (most elements are zero) Alternative: dense vectors vectors which are ◦ short (length 50-1000) ◦ dense (most elements are non-zero) 3 Sparse versus dense vectors Why dense vectors? ◦ Short vectors may be easier to use as features in machine learning (less weights to tune) ◦ Dense vectors may generalize better than storing explicit counts ◦ They may do better at capturing synonymy: ◦ car and automobile are synonyms; but are distinct dimensions ◦ a word with car as a neighbor and a word with automobile as a neighbor4 should be similar, but aren't ◦ In practice, they work better Dense embeddings you can download! Word2vec (Mikolov et al.) https://code.google.com/archive/p/word2vec/ Fasttext http://www.fasttext.cc/ Glove (Pennington, Socher, Manning) http://nlp.stanford.edu/projects/glove/ Word2vec Popular embedding method Very fast to train Code available on the web Idea: predict rather than count Word2vec ◦Instead of counting how often each word w occurs near "apricot" ◦Train a classifier on a binary prediction task: ◦Is w likely to show up near "apricot"? ◦We don’t actually care about this task ◦But we'll take the learned classifier weights as the word embeddings Brilliant insight: Use running text as implicitly supervised training data! • A word s near apricot • Acts as gold ‘correct answer’ to the question • “Is word w likely to show up near apricot?” • No need for hand-labeled supervision • The idea comes from neural language modeling • Bengio et al. (2003) • Collobert et al. (2011) Word2Vec: Skip-Gram Task Word2vec provides a variety of options. Let's do ◦ "skip-gram with negative sampling" (SGNS) Skip-gram algorithm 1. Treat the target word and a neighboring context word as positive examples. 2. Randomly sample other words in the lexicon to get negative samples 3. Use logistic regression to train a classifier to distinguish those two cases 4. Use the weights as the embeddings 10 9/7/18 Skip-Gram Training Data Training sentence: ... lemon, a tablespoon of apricot jam a pinch ... c1 c2 target c3 c4 Asssume context words are those in +/- 2 word window 11 9/7/18 Skip-Gram Goal Given a tuple (t,c) = target, context ◦(apricot, jam) ◦(apricot, aardvark) Return probability that c is a real context word: P(+|t,c) P(−|t,c) = 1−P(+|t,c) 12 9/7/18 How to compute p(+|t,c)? Intuition: ◦ Words are likely to appear near similar words ◦ Model similarity with dot-product! ◦ Similarity(t,c) ∝ t · c Problem: ◦Dot product is not a probability! ◦ (Neither is cosine) 16 CHAPTER 6 VECTOR SEMANTICS • 6.7.1 The classifier Let’s start by thinking about the classification task, and then turn to how to train. Imagine a sentence like the following, with a target word apricot and assume we’re using a window of 2 context words: ± ... lemon, a [tablespoon of apricot jam, a] pinch ... c1 c2 t c3 c4 Our goal is to train a classifier such that, given a tuple (t,c) of a target word t paired with a candidate context word c (for example (apricot, jam), or perhaps (apricot, aardvark) it will return the probability that c is a real context word (true for jam, false for aardvark): P(+ t,c) (6.15) | The probability that word c is not a real context word for t is just 1 minus Eq. 6.15: P( t,c)=1 P(+ t,c) (6.16) −| − | How does the classifier compute the probability P? The intuition of the skip- gram model is to base this probability on similarity: a word is likely to occur near the target if its embedding is similar to the target embedding. How can we compute similarity between embeddings? Recall that two vectors are similar if they have a high dot product (cosine, the most popular similarity metric, is just a normalized dot product). In other words: Similarity(t,c) t c (6.17) ⇡ · Of course, the dot product t c is not a probability, it’s just a number ranging · from 0 to •. (Recall, for that matter, that cosine isn’t a probability either). To turn the dot product into aTurning probability, dot we’ll product use the intologistic a or sigmoid function s(x), the fundamental core ofprobability logistic regression: The sigmoid lies between 0 and 1: 1 s(x)= x (6.18) 1 + e− The probability that word c is a real context word for target word t is thus com- puted as: 1 P(+ t,c)= t c (6.19) | 1 + e− · The sigmoid function just returns a number between 0 and 1, so to make it a probability we’ll need to make sure that the total probability of the two possible events (c being a context word, and c not being a context word) sum to 1. The probability that word c is not a real context word for t is thus: P( t,c)=1 P(+ t,c) −| − t c | e− · = t c (6.20) 1 + e− · 16 16CHAPTERCHAPTER6 6 VECTORVECTORSEMANTICSSEMANTICS • • 6.7.16.7.1 The The classifier classifier Let’s start by thinking about the classification task, and then turn to how to train. Let’sImagine start by a sentence thinking like about the the following, classification with a target task, word andapricot then turnand to assume how to we’re train. Imagineusing a a sentence window of like2 the context following, words: with a target word apricot and assume we’re using a window of 2± context words: ... lemon,± a [tablespoon of apricot jam, a] pinch ... ... lemon, a [tablespoonc1 of c2 apricot t jam, c3 c4 a] pinch ... c1 c2 t c3 c4 Our goal is to train a classifier such that, given a tuple (t,c) of a target word Ourt paired goal with is to a train candidate a classifier context such word that,c (for given example a tuple (apricot(t,c, )jamof), a or target perhaps word t paired(apricot with, aardvark a candidate) it will context return word the probabilityc (for example that c is (apricot a real context, jam), word or perhaps (true (apricotfor jam, aardvark, false for) itaardvark will return): the probability that c is a real context word (true for jam, false for aardvark): P(+ t,c) (6.15) | The probability that word c isP not(+ at, realc) context word for t is just 1 minus(6.15) Eq. 6.15: | The probability that word c is not a real context word for t is just 1 minus Eq. 6.15: P( t,c)=1 P(+ t,c) (6.16) −| − | How does the classifierP compute( t,c)= the1 probabilityP(+ t,c)P? The intuition of the skip-(6.16) gram model is to base this probability−| on− similarity:| a word is likely to occur near Howthe target does if the its embedding classifier compute is similar the to the probability target embedding.P? The How intuition can we of compute the skip- gramsimilarity model is between to base embeddings? this probability Recall on that similarity: two vectors a word are similar is likely if to they occur have near a the targethigh dot if its product embedding (cosine, is the similar most popular to the target similarity embedding. metric, is How just a can normalized we compute dot similarityproduct). between In other embeddings? words: Recall that two vectors are similar if they have a high dot product (cosine, the most popular similarity metric, is just a normalized dot product). In other words: Similarity(t,c) t c (6.17) ⇡ · Of course, the dot product t c is not a probability, it’s just a number ranging · from 0 to •. (Recall, for thatSimilarity matter, that(t,c cosine) t isn’tc a probability either). To turn(6.17) ⇡ · the dot product into a probability, we’ll use the logistic or sigmoid function s(x), Ofthe course, fundamental the dot core product of logistict regression:c is not a probability, it’s just a number ranging · from 0 to •. (Recall, for that matter, that cosine isn’t a probability either). To turn the dot product into a probability, we’ll use1 the logistic or sigmoid function s(x), the fundamental core of logistic regression:s(x)= x (6.18) 1 + e− The probability that word c is a real context word for target word t is thus com- puted as: 1 s(x)= x (6.18) 1 + e− The probability that wordTurningc is a dot real contextproduct word1 into for a target word t is thus com- P(+ t,c)= t c (6.19) puted as: probability| 1 + e− · The sigmoid function just returns a number between 0 and 1, so to make it a probability we’ll need to make sure that the total1 probability of the two possible events (c being a context word,P(+ andt,cc)=not being a context word) sum to 1. (6.19) | 1 + e t c The probability that word c is not a real context− · word for t is thus: The sigmoid function just returns a number between 0 and 1, so to make it a ( , )= (+ , ) probability we’ll need to makeP suret c that the1 totalP t probabilityc of the two possible −| − t c | events (c being a context word, and c not beinge− a· context word) sum to 1. = t c (6.20) The probability that word c is not a real1 context+ e− · word for t is thus: P( t,c)=1 P(+ t,c) −| − t c | e− · = t c (6.20) 1 + e− · 6.7 WORD2VEC 17 • Equation 6.19 give us the probability for one word, but we need to take account of the multiple context words in the window.
Load more

Recommended publications
  • Malware Classification with BERT
    San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-25-2021 Malware Classification with BERT Joel Lawrence Alvares Follow this and additional works at: https://scholarworks.sjsu.edu/etd_projects Part of the Artificial Intelligence and Robotics Commons, and the Information Security Commons Malware Classification with Word Embeddings Generated by BERT and Word2Vec Malware Classification with BERT Presented to Department of Computer Science San José State University In Partial Fulfillment of the Requirements for the Degree By Joel Alvares May 2021 Malware Classification with Word Embeddings Generated by BERT and Word2Vec The Designated Project Committee Approves the Project Titled Malware Classification with BERT by Joel Lawrence Alvares APPROVED FOR THE DEPARTMENT OF COMPUTER SCIENCE San Jose State University May 2021 Prof. Fabio Di Troia Department of Computer Science Prof. William Andreopoulos Department of Computer Science Prof. Katerina Potika Department of Computer Science 1 Malware Classification with Word Embeddings Generated by BERT and Word2Vec ABSTRACT Malware Classification is used to distinguish unique types of malware from each other. This project aims to carry out malware classification using word embeddings which are used in Natural Language Processing (NLP) to identify and evaluate the relationship between words of a sentence. Word embeddings generated by BERT and Word2Vec for malware samples to carry out multi-class classification. BERT is a transformer based pre- trained natural language processing (NLP) model which can be used for a wide range of tasks such as question answering, paraphrase generation and next sentence prediction. However, the attention mechanism of a pre-trained BERT model can also be used in malware classification by capturing information about relation between each opcode and every other opcode belonging to a malware family.
    [Show full text]
  • Learned in Speech Recognition: Contextual Acoustic Word Embeddings
    LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS Shruti Palaskar∗, Vikas Raunak∗ and Florian Metze Carnegie Mellon University, Pittsburgh, PA, U.S.A. fspalaska j vraunak j fmetze [email protected] ABSTRACT model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13]. Using this model, we jointly learn to End-to-end acoustic-to-word speech recognition models have re- automatically segment and classify input speech into individual cently gained popularity because they are easy to train, scale well to words, hence getting rid of the problem of chunking or requiring large amounts of training data, and do not require a lexicon. In addi- pre-defined word boundaries. As our A2W model is trained at the tion, word models may also be easier to integrate with downstream utterance level, we show that we can not only learn acoustic word tasks such as spoken language understanding, because inference embeddings, but also learn them in the proper context of their con- (search) is much simplified compared to phoneme, character or any taining sentence. We also evaluate our contextual acoustic word other sort of sub-word units. In this paper, we describe methods embeddings on a spoken language understanding task, demonstrat- to construct contextual acoustic word embeddings directly from a ing that they can be useful in non-transcription downstream tasks. supervised sequence-to-sequence acoustic-to-word speech recog- Our main contributions in this paper are the following: nition model using the learned attention distribution. On a suite 1. We demonstrate the usability of attention not only for aligning of 16 standard sentence evaluation tasks, our embeddings show words to acoustic frames without any forced alignment but also for competitive performance against a word2vec model trained on the constructing Contextual Acoustic Word Embeddings (CAWE).
    [Show full text]
  • Lecture 26 Word Embeddings and Recurrent Nets
    CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Where we’re at Lecture 25: Word Embeddings and neural LMs Lecture 26: Recurrent networks Lecture 26 Lecture 27: Sequence labeling and Seq2Seq Lecture 28: Review for the final exam Word Embeddings and Lecture 29: In-class final exam Recurrent Nets Julia Hockenmaier [email protected] 3324 Siebel Center CS447: Natural Language Processing (J. Hockenmaier) !2 What are neural nets? Simplest variant: single-layer feedforward net For binary Output unit: scalar y classification tasks: Single output unit Input layer: vector x Return 1 if y > 0.5 Recap Return 0 otherwise For multiclass Output layer: vector y classification tasks: K output units (a vector) Input layer: vector x Each output unit " yi = class i Return argmaxi(yi) CS447: Natural Language Processing (J. Hockenmaier) !3 CS447: Natural Language Processing (J. Hockenmaier) !4 Multi-layer feedforward networks Multiclass models: softmax(yi) We can generalize this to multi-layer feedforward nets Multiclass classification = predict one of K classes. Return the class i with the highest score: argmaxi(yi) Output layer: vector y In neural networks, this is typically done by using the softmax N Hidden layer: vector hn function, which maps real-valued vectors in R into a distribution … … … over the N outputs … … … … … …. For a vector z = (z0…zK): P(i) = softmax(zi) = exp(zi) ∕ ∑k=0..K exp(zk) Hidden layer: vector h1 (NB: This is just logistic regression) Input layer: vector x CS447: Natural Language Processing (J. Hockenmaier) !5 CS447: Natural Language Processing (J. Hockenmaier) !6 Neural Language Models LMs define a distribution over strings: P(w1….wk) LMs factor P(w1….wk) into the probability of each word: " P(w1….wk) = P(w1)·P(w2|w1)·P(w3|w1w2)·…· P(wk | w1….wk#1) A neural LM needs to define a distribution over the V words in Neural Language the vocabulary, conditioned on the preceding words.
    [Show full text]
  • Arxiv:2007.00183V2 [Eess.AS] 24 Nov 2020
    WHOLE-WORD SEGMENTAL SPEECH RECOGNITION WITH ACOUSTIC WORD EMBEDDINGS Bowen Shi, Shane Settle, Karen Livescu TTI-Chicago, USA fbshi,settle.shane,[email protected] ABSTRACT Segmental models are sequence prediction models in which scores of hypotheses are based on entire variable-length seg- ments of frames. We consider segmental models for whole- word (“acoustic-to-word”) speech recognition, with the feature vectors defined using vector embeddings of segments. Such models are computationally challenging as the number of paths is proportional to the vocabulary size, which can be orders of magnitude larger than when using subword units like phones. We describe an efficient approach for end-to-end whole-word segmental models, with forward-backward and Viterbi de- coding performed on a GPU and a simple segment scoring function that reduces space complexity. In addition, we inves- tigate the use of pre-training via jointly trained acoustic word embeddings (AWEs) and acoustically grounded word embed- dings (AGWEs) of written word labels. We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional Fig. 1. Whole-word segmental model for speech recognition. (smaller) gains can be obtained by pre-training the word pre- Note: boundary frames are not shared. diction layer with AGWEs. Our final models improve over segmental models, where the sequence probability is com- prior A2W models. puted based on segment scores instead of frame probabilities. Index Terms— speech recognition, segmental model, Segmental models have a long history in speech recognition acoustic-to-word, acoustic word embeddings, pre-training research, but they have been used primarily for phonetic recog- nition or as phone-level acoustic models [11–18].
    [Show full text]
  • Overview of Lstms and Word2vec and a Bit About Compositional Distributional Semantics If There’S Time
    Overview of LSTMs and word2vec and a bit about compositional distributional semantics if there’s time Ann Copestake Computer Laboratory University of Cambridge November 2016 Outline RNNs and LSTMs Word2vec Compositional distributional semantics Some slides adapted from Aurelie Herbelot. Outline. RNNs and LSTMs Word2vec Compositional distributional semantics Motivation I Standard NNs cannot handle sequence information well. I Can pass them sequences encoded as vectors, but input vectors are fixed length. I Models are needed which are sensitive to sequence input and can output sequences. I RNN: Recurrent neural network. I Long short term memory (LSTM): development of RNN, more effective for most language applications. I More info: http://neuralnetworksanddeeplearning.com/ (mostly about simpler models and CNNs) https://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Sequences I Video frame categorization: strict time sequence, one output per input. I Real-time speech recognition: strict time sequence. I Neural MT: target not one-to-one with source, order differences. I Many language tasks: best to operate left-to-right and right-to-left (e.g., bi-LSTM). I attention: model ‘concentrates’ on part of input relevant at a particular point. Caption generation: treat image data as ordered, align parts of image with parts of caption. Recurrent Neural Networks http://colah.github.io/posts/ 2015-08-Understanding-LSTMs/ RNN language model: Mikolov et al, 2010 RNN as a language model I Input vector: vector for word at t concatenated to vector which is output from context layer at t − 1. I Performance better than n-grams but won’t capture ‘long-term’ dependencies: She shook her head.
    [Show full text]
  • Feedforward Neural Networks and Word Embeddings
    Feedforward Neural Networks and Word Embeddings Fabienne Braune1 1LMU Munich May 14th, 2017 Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 1 Outline 1 Linear models 2 Limitations of linear models 3 Neural networks 4 A neural language model 5 Word embeddings Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 2 Linear Models Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 3 Binary Classification with Linear Models Example: the seminar at < time > 4 pm will Classification task: Do we have an < time > tag in the current position? Word Lemma LexCat Case SemCat Tag the the Art low seminar seminar Noun low at at Prep low stime 4 4 Digit low pm pm Other low timeid will will Verb low Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 4 Feature Vector Encode context into feature vector: 1 bias term 1 2 -3 lemma the 1 3 -3 lemma giraffe 0 ... ... ... 102 -2 lemma seminar 1 103 -2 lemma giraffe 0 ... ... ... 202 -1 lemma at 1 203 -1 lemma giraffe 0 ... ... ... 302 +1 lemma 4 1 303 +1 lemma giraffe 0 ... ... ... Fabienne Braune (CIS) Feedforward Neural Networks and Word Embeddings May 14th, 2017 · 5 Dot product with (initial) weight vector 2 3 2 3 x0 = 1 w0 = 1:00 6 x = 1 7 6 w = 0:01 7 6 1 7 6 1 7 6 x = 0 7 6 w = 0:01 7 6 2 7 6 2 7 6 ··· 7 6 ··· 7 6 7 6 7 6x = 17 6x = 0:017 6 101 7 6 101 7 6 7 6 7 6x102 = 07 6x102 = 0:017 T 6 7 6 7 h(X )= X · Θ X = 6 ··· 7 Θ = 6 ··· 7 6 7 6 7 6x201 = 17 6x201 = 0:017 6 7
    [Show full text]
  • An Unsupervised Method for OCR Post-Correction and Spelling
    An Unsupervised method for OCR Post-Correction and Spelling Normalisation for Finnish Quan Duong,♣ Mika Hämäläinen,♣,♦ Simon Hengchen,♠ firstname.lastname@{helsinki.fi;gu.se} ♣University of Helsinki, ♦Rootroo Ltd, ♠Språkbanken Text – University of Gothenburg Abstract Historical corpora are known to contain errors introduced by OCR (optical character recognition) methods used in the digitization process, often said to be degrading the performance of NLP systems. Correcting these errors manually is a time-consuming process and a great part of the automatic approaches have been relying on rules or supervised machine learning. We build on previous work on fully automatic unsupervised extraction of parallel data to train a character- based sequence-to-sequence NMT (neural machine translation) model to conduct OCR error correction designed for English, and adapt it to Finnish by proposing solutions that take the rich morphology of the language into account. Our new method shows increased performance while remaining fully unsupervised, with the added benefit of spelling normalisation. The source code and models are available on GitHub1 and Zenodo2. 1 Introduction As many people dealing with digital humanities study historical data, the problem that researchers continue to face is the quality of their digitized corpora. There have been large scale projects in the past focusing on digitizing parts of cultural history using OCR models available in those times (see Benner (2003), Bremer-Laamanen (2006)). While OCR methods have substantially developed and im- proved in recent times, it is still not always possible to re-OCR text. Re-OCRing documents is also not a priority when not all collections are digitised, as is the case in many national libraries.
    [Show full text]
  • Poor Man's OCR Post-Correction
    Poor Man’s OCR Post-Correction Unsupervised Recognition of Variant Spelling Applied to a Multilingual Document Collection Harald Hammarström Shafqat Mumtaz Virk Markus Forsberg Department of Linguistics and Språkbanken Språkbanken Philology University of Gothenburg University of Gothenburg Uppsala University P.O. Box 200 P.O. Box 200 P.O. Box 635 Gothenburg 405 30, Sweden Gothenburg 405 30, Sweden Uppsala 751 26, Sweden [email protected] [email protected] [email protected] ABSTRACT 1 INTRODUCTION The accuracy of Optical Character Recognition (OCR) is sets the In search engines and digital libraries, more and more documents limit for the success of subsequent applications used in text analyz- become available from scanning of legacy print collections ren- ing pipeline. Recent models of OCR post-processing significantly dered searchable through optical character recognition (OCR). Ac- improve the quality of OCR-generated text but require engineering cess to these texts is often not satisfactory due to the mediocre work or resources such as human-labeled data or a dictionary to performance of OCR on historical texts and imperfectly scanned perform with such accuracy on novel datasets. In the present pa- or physically deteriorated originals [19, 21]. Historical texts also per we introduce a technique for OCR post-processing that runs often contain more spelling variation which turns out to be a re- off-the-shelf with no resources or parameter tuning required. In lated obstacle. essence, words which are similar in form that are also distribu- Given this need, techniques for automatically improving OCR tionally more similar than expected at random are deemed OCR- output have been around for decades.
    [Show full text]
  • Recent Improvements on Error Detection for Automatic Speech Recognition
    Recent improvements on error detection for automatic speech recognition Yannick Esteve` and Sahar Ghannay and Nathalie Camelin1 Abstract. Error detection can be performed in three steps: first, generating a Automatic speech recognition(ASR) offers the ability to access the set of features that are based on ASR system or gathered from other semantic content present in spoken language within audio and video source of knowledge. Then, based on these features, estimating cor- documents. While acoustic models based on deep neural networks rectness probabilities (confidence measures). Finally, a decision is have recently significantly improved the performances of ASR sys- made by applying a threshold on these probabilities. tems, automatic transcriptions still contain errors. Errors perturb the Many studies focus on the ASR error detection. In [14], authors exploitation of these ASR outputs by introducing noise to the text. have applied the detection capability for filtering data for unsuper- To reduce this noise, it is possible to apply an ASR error detection in vised learning of an acoustic model. Their approach was based on order to remove recognized words labelled as errors. applying two thresholds on the linear combination of two confi- This paper presents an approach that reaches very good results, dence measures. The first one, was derived from language model and better than previous state-of-the-art approaches. This work is based takes into account backoff behavior during the ASR decoding. This on a neural approach, and more especially on a study targeted to measure is different from the language model score, because it pro- acoustic and linguistic word embeddings, that are representations of vides information about the word context.
    [Show full text]
  • Unsupervised Learning of Cross-Modal Mappings Between
    Unsupervised Learning of Cross-Modal Mappings between Speech and Text by Yu-An Chung B.S., National Taiwan University (2016) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2019 c Massachusetts Institute of Technology 2019. All rights reserved. Author.............................................................. Department of Electrical Engineering and Computer Science May 3, 2019 Certified by. James R. Glass Senior Research Scientist in Computer Science Thesis Supervisor Accepted by . Leslie A. Kolodziejski Professor of Electrical Engineering and Computer Science Chair, Department Committee on Graduate Students 2 Unsupervised Learning of Cross-Modal Mappings between Speech and Text by Yu-An Chung Submitted to the Department of Electrical Engineering and Computer Science on May 3, 2019, in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering and Computer Science Abstract Deep learning is one of the most prominent machine learning techniques nowadays, being the state-of-the-art on a broad range of applications in computer vision, nat- ural language processing, and speech and audio processing. Current deep learning models, however, rely on significant amounts of supervision for training to achieve exceptional performance. For example, commercial speech recognition systems are usually trained on tens of thousands of hours of annotated data, which take the form of audio paired with transcriptions for training acoustic models, collections of text for training language models, and (possibly) linguist-crafted lexicons mapping words to their pronunciations. The immense cost of collecting these resources makes applying state-of-the-art speech recognition algorithm to under-resourced languages infeasible.
    [Show full text]
  • Multi-Modal Speech Emotion Recognition Using Speech Embeddings and Audio Features
    Multi-Modal Speech Emotion Recognition Using Speech Embeddings and Audio Features Krishna D N, Sai Sumith Reddy YouPlus India [email protected], [email protected] Abstract volution neural networks are used in the paper by Dario Bertero et. al [10] for speech emotion recognition. Work by Abdul Ma- In this work, we propose a multi-modal emotion recognition lik Badshah et. al [11] shows how deep convolutional networks model to improve the speech emotion recognition system per- can be used for speech emotion recognition and also they show formance. We use two parallel Bidirectional LSTM networks how pretrained Alex-net model can be used as means of transfer called acoustic encoder (ENC1) and speech embedding encoder learning for the emotion recognition task. Recently many works (ENC2). The acoustic encoder is a Bi-LSTM which takes se- have been done on improving speech emotion recognition us- quence of speech features as inputs and speech embedding en- ing multi-modal techniques. Researchers have shown that we coder is also a Bi-LSTM which takes sequence of speech em- can improve the accuracy of an emotion recognition model if beddings as input and the output hidden representation at the we have text data of an utterance. If we provide text data as last time step of both the Bi-LSTM are concatenated and passed a guiding signal along with speech features to our model, the into a classification which predicts emotion label for that par- model performance can be greatly improved. Since face emo- ticular utterance. The speech embeddings are learned using the tion and speech emotions are correlated with each other, work encoder-decoder framework as described in [1] using skipgram by Samuel Albanie et.
    [Show full text]
  • Audio Word2vec: Unsupervised Learning of Audio Segment Representations Using Sequence-To-Sequence Autoencoder
    INTERSPEECH 2016 September 8–12, 2016, San Francisco, USA Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-Yi Lee, Lin-Shan Lee College of Electrical Engineering and Computer Science, National Taiwan University {b01902040, b01902038, r04921047, hungyilee}@ntu.edu.tw, [email protected] Abstract identification [4], and several approaches have been success- fully used in STD [10, 6, 7, 8]. But these vector representa- The vector representations of fixed dimensionality for tions may not be able to precisely describe the sequential pho- words (in text) offered by Word2Vec have been shown to be netic structures of the audio segments as we wish to have in this very useful in many application scenarios, in particular due paper, and these approaches were developed primarily in more to the semantic information they carry. This paper proposes a heuristic ways, rather than learned from data. Deep learning has parallel version, the Audio Word2Vec. It offers the vector rep- also been used for this purpose [12, 13]. By learning Recurrent resentations of fixed dimensionality for variable-length audio Neural Network (RNN) with an audio segment as the input and segments. These vector representations are shown to describe the corresponding word as the target, the outputs of the hidden the sequential phonetic structures of the audio segments to a layer at the last few time steps can be taken as the representation good degree, with very attractive real world applications such as of the input segment [13]. However, this approach is supervised query-by-example Spoken Term Detection (STD).
    [Show full text]