DOCSLIB.ORG

Relating Articles Textually and Visually

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Relating Articles Textually and Visually Nachum Dershowitz Daniel Labenski Adi Silberpfennig School of Computer Science School of Computer Science School of Electrical Engineering Tel Aviv University Tel Aviv University Tel Aviv University Tel Aviv, Israel Tel Aviv, Israel Tel Aviv, Israel [email protected] [email protected] [email protected] Lior Wolf Yaron Tsur School of Computer Science Department of Jewish History Tel Aviv University Tel Aviv University Tel Aviv, Israel Tel Aviv, Israel [email protected] [email protected] Abstract— Historical documents have been undergoing The task we tackle is topic detection – specifically, large-scale digitization over the past years, placing massive looking for newspaper articles that discuss the same image collections online. Optical character recognition ongoing event. Identifying related articles in historical (OCR) often performs poorly on such material, which makes searching within these resources problematic and archives can help scholars explore historical stories from textual analysis of such documents difficult. a broad perspective, can provide visitors of archival We present two approaches to overcome this obstacle, websites with effective navigation – as in modern news one textual and one visual. We show that, for tasks like sites, and can help compare the coverage of events finding newspaper articles related by topic, poor-quality between different publications.2 Unfortunately, optical OCR text suffices. An ordinary vector-space model is used to represent articles. Additional improvements obtain by character recognition (OCR) for older documents is not adding words with similar distributional representations. yet satisfactory, making it quite challenging to search As an alternative to OCR-based methods, one can within those images for something particular, limiting perform image-based search, using word spotting. Synthetic research options.3 images are generated for every word in a lexicon, and word- We adopt the following nomenclature [1]: An event is spotting is used to compile vectors of their occurrences. “something that happens at a specific time and location”. Retrieval is by means of a usual nearest-neighbor search. The results of this visual approach are comparable to those A story is “a topically cohesive segment of news that obtained using noisy OCR. includes two or more declarative independent clauses We report on experiments applying both methods, sepa- about a single event”; in our case this is always a single rately and together, on historical Hebrew newspapers, with newspaper article. A topic is “a set of news stories that their added problem of rich morphology. are strongly related by some seminal real world event”. We use subject to refer to thematically related topics. I. INTRODUCTION For example [2], the eruption of Mount Pinatubo on June Recent large-scale digitization and preservation efforts 15th, 1991 is an event, whereas the ensuing crisis and have made a huge quantity of images of historical books, the aftermaths caused by the cataclysmic event are part manuscripts, and other media readily available over the of the larger topic, and the subject is “natural disasters”. Internet, opening doors to new opportunities, like distant The following is one of the stories that day (New York 1 reading and data mining. Times): Research supported in part by Grant #I-145-101.3-2013 from the 2A list of archives may be found at https://en.wikipedia.org/wiki/ German-Israeli Foundation for Scientific Research and Development Wikipedia:LOONA. (GIF), Grant #01019841 from the Deutsch-Israelische Projektkooper- 3See, for example, the famous newspaper article by Craig Clai- ation (DIP), Grant #1330/14 of the Israel Science Foundation (ISF), borne (The New York Times, November 14, 1975), “Just a quiet and a grant from the Blavatnik Family Fund. It forms part of D.L.s dinner for two in Paris” (https://ia801009.us.archive.org/18/items/ and A.S.’s M.Sc. theses at Tel Aviv University. 354644-claiborne/354644-claiborne.pdf, last item), which read as gib- 1http://www.nytimes.com/2011/06/26/books/review/ berish not too long ago (https://assets.documentcloud.org/documents/ the-mechanic-muse-what-is-distant-reading.html. 354644/claiborne.txt). Mount Pinatubo rumbled today with explosions that Vector Representations: For solving such tasks, it hurled ash and gas more than 12 miles high. is common to represent each story as a vector in a President Corazon C. Aquino dismissed a British conventional vector space [3]. Various term weighting newspaper report that the Americans had warned her of possible radioactive contamination if the schemes for combining term frequency (tf ) and inverse volcano damaged nuclear storage sites at Clark Air document frequency (idf ) can be employed [7]. We use Base. She said the story was “baseless and purely this standard method for document representation and fabricated.” . measuring similarity. We first consider the case where printed texts have Allan and Harding [2], [8] used a story representation been digitized, resulting in noisy OCR text. We approach of tf-idf weights of the 1000 most weighted terms in the problem traditionally and then enhance it by using the document, computed cosine similarity of the story word embeddings to augment document representations. to every previous story and assign a story to the nearest This helps solve the sparsity problem, due to language neighbor cluster if the similarity was above a threshold variation, morphology, and OCR noise. Augmentation otherwise it creates a new cluster containing the story. proved to be helpful in finding more related articles We take this basic approach. Other document represen- and proved to be even better than language-specific tation and similarity measures have been suggested; for standardization methods. example, [9], [10] use language models for clusters to As an alternative to working with OCR results, we compute the probability that a new story is related to a propose a novel image-based approach for retrieval. cluster and use that as a similarity measure. Given a query image of a word, one seeks all occurrences Lexical Expansions: Petrovic´ and Osborne [4] use of that same word within the dataset. To that end, we lexical paraphrases from various sources of paraphrases, utilize a word-spotting engine that is both simple to including WordNet, to expand document vectors and implement and fast to run, making it extremely easy to overcome lexical variation in documents with different incorporate for a wide range of resources. The method is terms discussing the same topic. Moran and McCreadie completely unsupervised: words in the document images [5] show how to use word embedding to improve ac- are unsegmented and unlabeled. curacy by expanding tweets with semantically related The next section summarizes some recent approaches phrases. Their method does not depend on complex to topic detection and to word spotting. It is followed linguistic resources like WordNet and still gets better by a section on the newspaper corpus we experimented results. This resource independence allows one to run it with, its problematics, and the additional linguistic/visual on low-resource languages. We use a similar approach resources and tools that we used. SectionIV describes to augment the document bag of words. the textual method, and SectionV describes the visual Morphological Expansions: one. They are followed by experimental results for the Avraham and Goldberg two methods. We conclude with a brief discussion. [11] reported that, when using word embedding in Hebrew, which is a morphologically rich inflectional II. RELATED WORK language, word vectors bear a mix of semantic and Topic Detection: The task we study is related to morphological properties. Many neighboring word vec- the field of topic detection and tracking (TDT). This tors are of inflected forms, rather than having semantic research program, which began in 1997, aims to find similarity. We will leverage morphological properties of core technologies for organizing streams of news arti- word embeddings to expand terms with others sharing cles by topic [3]. Event detection has recently regained the same base form. popularity with the emergence of social media [4], [5]. Noisy Texts: Agarwal et al. [12] experimented with We look for stories in a retrospective fashion on a static texts from different resource types in order to understand database (Retrospective Event Detection [RED]), unlike how different levels of noise in the text harm text TDT, where one wishes “to identify new events as they classification success. They developed a spelling error occur, based on an on-line stream of stories” (New simulator and created synthetic texts with several levels Event Detection [NED]). For example, a common way of noise. They found that with up to 40% word error rate, of approaching detection is by representing stories via the accuracy of the classification was not significantly a set of features and assigning every new story to the affected. Vinciarelly [13], [14] reported similar results cluster of the most similar past one. Yang et al. [6] found in text clustering and categorization (classification) tasks. that retrospective event detection can obtain much better We were encouraged by these results to work on topic results than the online setup. detection despite the noisy OCR text. 2 Word Spotting: Whereas topic tracking in the field The average word error rate for this newspaper is of NLP is widely researched, we suggest here a novel about 20% [Ido Kissos, private communication], but is approach using images as sole input. This appears to somewhat worse for the test suite. (For example, the be the first attempt to take an image-based approach error rate in the first two paragraphs of the lead article for solving the topic-detection problem. The work of of the February 1 issue is 30=85 = 35%, and may even Wilkinson et al. [15] is somewhat related. They use be considerably worse.) Only about 45% of the tokens a word-spotting engine for computing word clouds, produced by OCR of the test corpus are found in the whereas we are applying spotting to topic detection and MILA lexicon of modern Hebrew,7 on account of OCR categorization.
Recommended publications
  • Malware Classification with BERT

    Malware Classification with BERT

    San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-25-2021 Malware Classification with BERT Joel Lawrence Alvares Follow this and additional works at: https://scholarworks.sjsu.edu/etd_projects Part of the Artificial Intelligence and Robotics Commons, and the Information Security Commons Malware Classification with Word Embeddings Generated by BERT and Word2Vec Malware Classification with BERT Presented to Department of Computer Science San José State University In Partial Fulfillment of the Requirements for the Degree By Joel Alvares May 2021 Malware Classification with Word Embeddings Generated by BERT and Word2Vec The Designated Project Committee Approves the Project Titled Malware Classification with BERT by Joel Lawrence Alvares APPROVED FOR THE DEPARTMENT OF COMPUTER SCIENCE San Jose State University May 2021 Prof. Fabio Di Troia Department of Computer Science Prof. William Andreopoulos Department of Computer Science Prof. Katerina Potika Department of Computer Science 1 Malware Classification with Word Embeddings Generated by BERT and Word2Vec ABSTRACT Malware Classification is used to distinguish unique types of malware from each other. This project aims to carry out malware classification using word embeddings which are used in Natural Language Processing (NLP) to identify and evaluate the relationship between words of a sentence. Word embeddings generated by BERT and Word2Vec for malware samples to carry out multi-class classification. BERT is a transformer based pre- trained natural language processing (NLP) model which can be used for a wide range of tasks such as question answering, paraphrase generation and next sentence prediction. However, the attention mechanism of a pre-trained BERT model can also be used in malware classification by capturing information about relation between each opcode and every other opcode belonging to a malware family.
  • Arxiv:2002.06235V1

    Arxiv:2002.06235V1

    Semantic Relatedness and Taxonomic Word Embeddings Magdalena Kacmajor John D. Kelleher Innovation Exchange ADAPT Research Centre IBM Ireland Technological University Dublin [email protected] [email protected] Filip Klubickaˇ Alfredo Maldonado ADAPT Research Centre Underwriters Laboratories Technological University Dublin [email protected] [email protected] Abstract This paper1 connects a series of papers dealing with taxonomic word embeddings. It begins by noting that there are different types of semantic relatedness and that different lexical representa- tions encode different forms of relatedness. A particularly important distinction within semantic relatedness is that of thematic versus taxonomic relatedness. Next, we present a number of ex- periments that analyse taxonomic embeddings that have been trained on a synthetic corpus that has been generated via a random walk over a taxonomy. These experiments demonstrate how the properties of the synthetic corpus, such as the percentage of rare words, are affected by the shape of the knowledge graph the corpus is generated from. Finally, we explore the interactions between the relative sizes of natural and synthetic corpora on the performance of embeddings when taxonomic and thematic embeddings are combined. 1 Introduction Deep learning has revolutionised natural language processing over the last decade. A key enabler of deep learning for natural language processing has been the development of word embeddings. One reason for this is that deep learning intrinsically involves the use of neural network models and these models only work with numeric inputs. Consequently, applying deep learning to natural language processing first requires developing a numeric representation for words. Word embeddings provide a way of creating numeric representations that have proven to have a number of advantages over traditional numeric rep- resentations of language.
  • Transfer Learning Using CNN for Handwritten Devanagari Character

    Transfer Learning Using CNN for Handwritten Devanagari Character

    Accepted for publication in IEEE International Conference on Advances in Information Technology (ICAIT), ICAIT - 2019, Please refer IEEE Explore for final version Transfer Learning using CNN for Handwritten Devanagari Character Recognition Nagender Aneja∗ and Sandhya Aneja ∗ ∗Universiti Brunei Darussalam Brunei Darussalam fnagender.aneja, sandhya.aneja [email protected] Abstract—This paper presents an analysis of pre-trained models to recognize handwritten Devanagari alphabets using transfer learning for Deep Convolution Neural Network (DCNN). This research implements AlexNet, DenseNet, Vgg, and Inception ConvNet as a fixed feature extractor. We implemented 15 epochs for each of AlexNet, DenseNet 121, DenseNet 201, Vgg 11, Vgg 16, Vgg 19, and Inception V3. Results show that Inception V3 performs better in terms of accuracy achieving 99% accuracy with average epoch time 16.3 minutes while AlexNet performs fastest with 2.2 minutes per epoch and achieving 98% accuracy. Index Terms—Deep Learning, CNN, Transfer Learning, Pre-trained, handwritten, recognition, Devanagari I. INTRODUCTION Handwriting identification is one of the challenging research domains in computer vision. Handwriting character identification systems are useful for bank signatures, postal code recognition, and bank cheques, etc. Many researchers working in this area tend to use the same database and compare different techniques that either perform better concerning the accuracy, time, complexity, etc. However, the handwritten recognition in non-English is difficult due to complex shapes. Fig. 1: Devanagari Alphabets Many Indian Languages like Hindi, Sanskrit, Nepali, Marathi, Sindhi, and Konkani uses Devanagari script. The alphabets of Devanagari Handwritten Character Dataset (DHCD) [1] is shown in Figure 1. Recognition of Devanagari is particularly challenging due to many groups of similar characters.
  • Combining Word Embeddings with Semantic Resources

    Combining Word Embeddings with Semantic Resources

    AutoExtend: Combining Word Embeddings with Semantic Resources Sascha Rothe∗ LMU Munich Hinrich Schutze¨ ∗ LMU Munich We present AutoExtend, a system that combines word embeddings with semantic resources by learning embeddings for non-word objects like synsets and entities and learning word embeddings that incorporate the semantic information from the resource. The method is based on encoding and decoding the word embeddings and is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The obtained embeddings live in the same vector space as the input word embeddings. A sparse tensor formalization guar- antees efficiency and parallelizability. We use WordNet, GermaNet, and Freebase as semantic resources. AutoExtend achieves state-of-the-art performance on Word-in-Context Similarity and Word Sense Disambiguation tasks. 1. Introduction Unsupervised methods for learning word embeddings are widely used in natural lan- guage processing (NLP). The only data these methods need as input are very large corpora. However, in addition to corpora, there are many other resources that are undoubtedly useful in NLP, including lexical resources like WordNet and Wiktionary and knowledge bases like Wikipedia and Freebase. We will simply refer to these as resources. In this article, we present AutoExtend, a method for enriching these valuable resources with embeddings for non-word objects they describe; for example, Auto- Extend enriches WordNet with embeddings for synsets. The word embeddings and the new non-word embeddings live in the same vector space. Many NLP applications benefit if non-word objects described by resources—such as synsets in WordNet—are also available as embeddings.
  • Word-Like Character N-Gram Embedding

    Word-Like Character N-Gram Embedding

    Word-like character n-gram embedding Geewook Kim and Kazuki Fukui and Hidetoshi Shimodaira Department of Systems Science, Graduate School of Informatics, Kyoto University Mathematical Statistics Team, RIKEN Center for Advanced Intelligence Project fgeewook, [email protected], [email protected] Abstract Table 1: Top-10 2-grams in Sina Weibo and 4-grams in We propose a new word embedding method Japanese Twitter (Experiment 1). Words are indicated called word-like character n-gram embed- by boldface and space characters are marked by . ding, which learns distributed representations FNE WNE (Proposed) Chinese Japanese Chinese Japanese of words by embedding word-like character n- 1 ][ wwww 自己 フォロー grams. Our method is an extension of recently 2 。␣ ！！！！ 。␣ ありがと proposed segmentation-free word embedding, 3 ！␣ ありがと ][ wwww 4 .. りがとう 一个 ！！！！ which directly embeds frequent character n- 5 ]␣ ございま 微博 めっちゃ grams from a raw corpus. However, its n-gram 6 。。 うござい 什么 んだけど vocabulary tends to contain too many non- 7 ，我 とうござ 可以 うござい 8 ！！ ざいます 没有 line word n-grams. We solved this problem by in- 9 ␣我 がとうご 吗？ 2018 troducing an idea of expected word frequency. 10 了， ください 哈哈 じゃない Compared to the previously proposed meth- ods, our method can embed more words, along tion tools are used to determine word boundaries with the words that are not included in a given in the raw corpus. However, these segmenters re- basic word dictionary. Since our method does quire rich dictionaries for accurate segmentation, not rely on word segmentation with rich word which are expensive to prepare and not always dictionaries, it is especially effective when the available.
  • Learned in Speech Recognition: Contextual Acoustic Word Embeddings

    Learned in Speech Recognition: Contextual Acoustic Word Embeddings

    LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS Shruti Palaskar∗, Vikas Raunak∗ and Florian Metze Carnegie Mellon University, Pittsburgh, PA, U.S.A. fspalaska j vraunak j fmetze [email protected] ABSTRACT model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13]. Using this model, we jointly learn to End-to-end acoustic-to-word speech recognition models have re- automatically segment and classify input speech into individual cently gained popularity because they are easy to train, scale well to words, hence getting rid of the problem of chunking or requiring large amounts of training data, and do not require a lexicon. In addi- pre-defined word boundaries. As our A2W model is trained at the tion, word models may also be easier to integrate with downstream utterance level, we show that we can not only learn acoustic word tasks such as spoken language understanding, because inference embeddings, but also learn them in the proper context of their con- (search) is much simplified compared to phoneme, character or any taining sentence. We also evaluate our contextual acoustic word other sort of sub-word units. In this paper, we describe methods embeddings on a spoken language understanding task, demonstrat- to construct contextual acoustic word embeddings directly from a ing that they can be useful in non-transcription downstream tasks. supervised sequence-to-sequence acoustic-to-word speech recog- Our main contributions in this paper are the following: nition model using the learned attention distribution. On a suite 1. We demonstrate the usability of attention not only for aligning of 16 standard sentence evaluation tasks, our embeddings show words to acoustic frames without any forced alignment but also for competitive performance against a word2vec model trained on the constructing Contextual Acoustic Word Embeddings (CAWE).
  • Knowledge-Powered Deep Learning for Word Embedding

    Knowledge-Powered Deep Learning for Word Embedding

    Knowledge-Powered Deep Learning for Word Embedding Jiang Bian, Bin Gao, and Tie-Yan Liu Microsoft Research {jibian,bingao,tyliu}@microsoft.com Abstract. The basis of applying deep learning to solve natural language process- ing tasks is to obtain high-quality distributed representations of words, i.e., word embeddings, from large amounts of text data. However, text itself usually con- tains incomplete and ambiguous information, which makes necessity to leverage extra knowledge to understand it. Fortunately, text itself already contains well- defined morphological and syntactic knowledge; moreover, the large amount of texts on the Web enable the extraction of plenty of semantic knowledge. There- fore, it makes sense to design novel deep learning algorithms and systems in order to leverage the above knowledge to compute more effective word embed- dings. In this paper, we conduct an empirical study on the capacity of leveraging morphological, syntactic, and semantic knowledge to achieve high-quality word embeddings. Our study explores these types of knowledge to define new basis for word representation, provide additional input information, and serve as auxiliary supervision in deep learning, respectively. Experiments on an analogical reason- ing task, a word similarity task, and a word completion task have all demonstrated that knowledge-powered deep learning can enhance the effectiveness of word em- bedding. 1 Introduction With rapid development of deep learning techniques in recent years, it has drawn in- creasing attention to train complex and deep models on large amounts of data, in order to solve a wide range of text mining and natural language processing (NLP) tasks [4, 1, 8, 13, 19, 20].
  • Lecture 26 Word Embeddings and Recurrent Nets

    Lecture 26 Word Embeddings and Recurrent Nets

    CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Where we’re at Lecture 25: Word Embeddings and neural LMs Lecture 26: Recurrent networks Lecture 26 Lecture 27: Sequence labeling and Seq2Seq Lecture 28: Review for the final exam Word Embeddings and Lecture 29: In-class final exam Recurrent Nets Julia Hockenmaier [email protected] 3324 Siebel Center CS447: Natural Language Processing (J. Hockenmaier) !2 What are neural nets? Simplest variant: single-layer feedforward net For binary Output unit: scalar y classification tasks: Single output unit Input layer: vector x Return 1 if y > 0.5 Recap Return 0 otherwise For multiclass Output layer: vector y classification tasks: K output units (a vector) Input layer: vector x Each output unit " yi = class i Return argmaxi(yi) CS447: Natural Language Processing (J. Hockenmaier) !3 CS447: Natural Language Processing (J. Hockenmaier) !4 Multi-layer feedforward networks Multiclass models: softmax(yi) We can generalize this to multi-layer feedforward nets Multiclass classification = predict one of K classes. Return the class i with the highest score: argmaxi(yi) Output layer: vector y In neural networks, this is typically done by using the softmax N Hidden layer: vector hn function, which maps real-valued vectors in R into a distribution … … … over the N outputs … … … … … …. For a vector z = (z0…zK): P(i) = softmax(zi) = exp(zi) ∕ ∑k=0..K exp(zk) Hidden layer: vector h1 (NB: This is just logistic regression) Input layer: vector x CS447: Natural Language Processing (J. Hockenmaier) !5 CS447: Natural Language Processing (J. Hockenmaier) !6 Neural Language Models LMs define a distribution over strings: P(w1….wk) LMs factor P(w1….wk) into the probability of each word: " P(w1….wk) = P(w1)·P(w2|w1)·P(w3|w1w2)·…· P(wk | w1….wk#1) A neural LM needs to define a distribution over the V words in Neural Language the vocabulary, conditioned on the preceding words.
  • Local Homology of Word Embeddings

    Local Homology of Word Embeddings

    Local Homology of Word Embeddings Tadas Temcinasˇ 1 Abstract Intuitively, stratification is a decomposition of a topologi- Topological data analysis (TDA) has been widely cal space into manifold-like pieces. When thinking about used to make progress on a number of problems. stratification learning and word embeddings, it seems intu- However, it seems that TDA application in natural itive that vectors of words corresponding to the same broad language processing (NLP) is at its infancy. In topic would constitute a structure, which we might hope this paper we try to bridge the gap by arguing why to be a manifold. Hence, for example, by looking at the TDA tools are a natural choice when it comes to intersections between those manifolds or singularities on analysing word embedding data. We describe the manifolds (both of which can be recovered using local a parallelisable unsupervised learning algorithm homology-based algorithms (Nanda, 2017)) one might hope based on local homology of datapoints and show to find vectors of homonyms like ‘bank’ (which can mean some experimental results on word embedding either a river bank, or a financial institution). This, in turn, data. We see that local homology of datapoints in has potential to help solve the word sense disambiguation word embedding data contains some information (WSD) problem in NLP, which is pinning down a particular that can potentially be used to solve the word meaning of a word used in a sentence when the word has sense disambiguation problem. multiple meanings. In this work we present a clustering algorithm based on lo- cal homology, which is more relaxed1 that the stratification- 1.
  • Word Embedding, Sense Embedding and Their Application to Word Sense Induction

    Word Embedding, Sense Embedding and Their Application to Word Sense Induction

    Word Embeddings, Sense Embeddings and their Application to Word Sense Induction Linfeng Song The University of Rochester Computer Science Department Rochester, NY 14627 Area Paper April 2016 Abstract This paper investigates the cutting-edge techniques for word embedding, sense embedding, and our evaluation results on large-scale datasets. Word embedding refers to a kind of methods that learn a distributed dense vector for each word in a vocabulary. Traditional word embedding methods first obtain the co-occurrence matrix then perform dimension reduction with PCA. Recent methods use neural language models that directly learn word vectors by predicting the context words of the target word. Moving one step forward, sense embedding learns a distributed vector for each sense of a word. They either define a sense as a cluster of contexts where the target word appears or define a sense based on a sense inventory. To evaluate the performance of the state-of-the-art sense embedding methods, I first compare them on the dominant word similarity datasets, then compare them on my experimental settings. In addition, I show that sense embedding is applicable to the task of word sense induction (WSI). Actually we are the first to show that sense embedding methods are competitive on WSI by building sense-embedding-based systems that demonstrate highly competitive performances on the SemEval 2010 WSI shared task. Finally, I propose several possible future research directions on word embedding and sense embedding. The University of Rochester Computer Science Department supported this work. Contents 1 Introduction 3 2 Word Embedding 5 2.1 Skip-gramModel................................
  • A Novel Approach to On-Line Handwriting Recognition Based on Bidirectional Long Short-Term Memory Networks

    A Novel Approach to On-Line Handwriting Recognition Based on Bidirectional Long Short-Term Memory Networks

    A Novel Approach to On-Line Handwriting Recognition Based on Bidirectional Long Short-Term Memory Networks Marcus Liwicki1 Alex Graves2 Horst Bunke1 Jurgen¨ Schmidhuber2,3 1 Inst. of Computer Science and Applied Mathematics, University of Bern, Neubruckstr.¨ 10, 3012 Bern, Switzerland 2 IDSIA, Galleria 2, 6928 Manno-Lugano, Switzerland 3 TU Munich, Boltzmannstr. 3, 85748 Garching, Munich, Germany Abstract underlying this paper we aim at developing a handwriting recognition system to be used in a smart meeting room sce- In this paper we introduce a new connectionist approach nario [17], in our case the smart meeting room developed in to on-line handwriting recognition and address in partic- the IM2 project [11]. Smart meeting rooms usually have ular the problem of recognizing handwritten whiteboard multiple acquisition devices, such as microphones, cam- notes. The approach uses a bidirectional recurrent neu- eras, electronic tablets, and a whiteboard. In order to allow ral network with long short-term memory blocks. We use for indexing and browsing [18], automatic transcription of a recently introduced objective function, known as Connec- the recorded data is needed. tionist Temporal Classification (CTC), that directly trains In this paper, we introduce a novel approach to on-line the network to label unsegmented sequence data. Our new handwriting recognition, using a single recurrent neural net- system achieves a word recognition rate of 74.0 %, com- work (RNN) to transcribe the data. The key innovation is a pared with 65.4 % using a previously developed HMM- recently introduced RNN objective function known as Con- based recognition system. nectionist Temporal Classification (CTC) [5].
  • Arxiv:2007.00183V2 [Eess.AS] 24 Nov 2020

    Arxiv:2007.00183V2 [Eess.AS] 24 Nov 2020

    WHOLE-WORD SEGMENTAL SPEECH RECOGNITION WITH ACOUSTIC WORD EMBEDDINGS Bowen Shi, Shane Settle, Karen Livescu TTI-Chicago, USA fbshi,settle.shane,[email protected] ABSTRACT Segmental models are sequence prediction models in which scores of hypotheses are based on entire variable-length seg- ments of frames. We consider segmental models for whole- word (“acoustic-to-word”) speech recognition, with the feature vectors defined using vector embeddings of segments. Such models are computationally challenging as the number of paths is proportional to the vocabulary size, which can be orders of magnitude larger than when using subword units like phones. We describe an efficient approach for end-to-end whole-word segmental models, with forward-backward and Viterbi de- coding performed on a GPU and a simple segment scoring function that reduces space complexity. In addition, we inves- tigate the use of pre-training via jointly trained acoustic word embeddings (AWEs) and acoustically grounded word embed- dings (AGWEs) of written word labels. We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional Fig. 1. Whole-word segmental model for speech recognition. (smaller) gains can be obtained by pre-training the word pre- Note: boundary frames are not shared. diction layer with AGWEs. Our final models improve over segmental models, where the sequence probability is com- prior A2W models. puted based on segment scores instead of frame probabilities. Index Terms— speech recognition, segmental model, Segmental models have a long history in speech recognition acoustic-to-word, acoustic word embeddings, pre-training research, but they have been used primarily for phonetic recog- nition or as phone-level acoustic models [11–18].