DOCSLIB.ORG

Word Embedding Subspaces

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Word Embedding Subspaces Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) Getting in Shape: Word Embedding SubSpaces Tianyuan Zhou1 , Joao˜ Sedoc2 and Jordan Rodu1∗ 1Department of Statistics, University of Virginia 2Department of Computer and Information Science, University of Pennsylvania [email protected], [email protected], [email protected] Abstract Despite a large body of work in each of the two areas, the link between subspace alignment and manifold isotropy Many tasks in natural language processing require has not been fully explored. Given that word embeddings the alignment of word embeddings. Embedding contain some form of “meaning”, presumably common to all alignment relies on the geometric properties of the word embeddings, different representations should align to manifold of word vectors. This paper focuses on some degree. But alignment necessarily prioritizes resolving supervised linear alignment and studies the relation- disparity in larger singular values, which could be problematic ship between the shape of the target embedding. We for two word embeddings that encode information differently assess the performance of aligned word vectors on across their spectrums. When two word embeddings represent semantic similarity tasks and find that the isotropy information similarly, they can be successfully aligned. For of the target embedding is critical to the alignment. instance, Artetxe [2016] shows orthogonal transformations for Furthermore, aligning with an isotropic noise can alignment are superior for retaining performance on analogy deliver satisfactory results. We provide a theoretical tasks. However, orthogonal transformations do not allow for framework and guarantees which aid in the under- the alignment of more disparate methods such as distributional standing of empirical results. and non-distributional word embeddings. In this paper, we present a theoretical framework for un- 1 Introduction derstanding the alignment between word embeddings- when they can work, and when they might fail. Our theoretical Mono-lingual and multi-lingual alignment of word embed- results show that the singular value structure of the source dings is important for domain adaptation, word embedding embeddings is completely discarded, and when information is assessment, and machine translation [Ben-David et al., 2007; encoded differently in the source and target embeddings, the Blitzer et al., 2011; Tsvetkov et al., 2015; Lample et al., distortion of the spectrum from source to aligned embeddings 2018]. Fundamentally, this is a subspace alignment problem could drastically effect downstream results. which has seen much interest in machine learning commu- nities, including computer vision and nature language pro- cessing (NLP) [Fernando et al., 2013; Xing et al., 2015; 2 Theoretical Results Wang and Mahadevan, 2013; Lample et al., 2018; Mikolov In this section we provide some theoretical underpinnings for et al., 2013a] and can be either supervised or unsupervised, the phenomena observed in our experiments. Lemma 1 shows depending on the setting. that when a source representation is aligned to a target repre- Simultaneously, some work has focused on uncovering the sentation using a linear transformation, the column space of structure of word embedding manifolds [Mimno and Thomp- the aligned representation is determined by the column space son, 2017a; Hartmann et al., 2018; Shin et al., 2018; Mu et al., of the source, but the singular value structure of the source 2017], where in this paper we focus their isotropy/anisotropy. is entirely discarded. Theorem 1 guarantees the existence of Word embeddings are known to not be isotropic [Andreas a lower-bounded singular value. Since the summation of the and Klein, 2015], but Arora [2015] argue that isotropic singular values is upper-bounded by the target embedding, if word embeddings mitigate the effect of approximation er- the lower bound of the largest singular value is relatively large ror. Indeed, recent work in post-processing word embed- in comparison, then the singular value structure will have high dings has shown that increasing isotropy increases seman- eccentricity. [ tic task performance Mu et al., 2017; Liu et al., 2019b; Finally, proposition 1 shows that the correlation of measure- ] Liu et al., 2019a . ments between words (Euclidean distance between vectors or the cosine of their angles) in a vector cloud, and those mea- ∗Corresponding Author surements after the vector cloud has been stretched by unequal Link of Supplementary Materials and Source codes: https:// amounts in different directions, is small. Combined with the github.com/NoahZhouTianyuan/ConceptorOnNondistEmbedding results of theorem 1, this implies that alignment can greatly 5478 Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) impact the performance of aligned word representations on Further, for a subset I ⊂ f1; : : : ; pg define Xi;I to be the semantic similarity tasks. We expand on this below. vector Xi with indices not in I set to 0. Then We include the proof of the theorem here, but defer the 2 2 proof of the lemma and the proposition to the supplementary corr(jjXiS−XjSjj2; jjXi;I − Xj;I jj2) material. We also provide additional theorems in the supple- P p i2I si= jIj mentary material that, while not critical for the main focus = q 2 2 2 of this work, provide justification for some minor experimen- s1 + s2 + ··· + sp tal results and contribute more broadly to our fundamental theoretical understanding of word representation alignment. To see the implications of this proposition, consider the We begin by stating our word representation alignment alignment process loosely as 1) establishing the directions of model. the aligned singular vectors (which could be by an orthogo- nal rotation of the source singular vectors, or through linear Alignment Model: Let Y 2 Rn×p be the target word combinations of the source embeddings if the information representations, and X 2 Rn×p the source representations. content is not encoded similarly in the source and target em- To align X to Y , we seek a linear mapping W such that beddings), 2) stretching the singular vectors according to the Y = XW + . The least squares solution for W yields ∗ T −1 T T singular values of the source embeddings (or functions of the Yb = XW = X(X X) X Y = UX UX Y . singular values that distribute their information appropriately Lemma 1. Let UX and UY be the left singular vectors of X if the transformation is not orthogonal), and 3) adjusting the and Y respectively, and ΣY a diagonal matrix containing the stretched singular vectors through the S matrix of proposition T 1 according to the spectrum of the target embeddings (see singular values of Y , then σ(Yb) = σ(UX UY ΣY ). lemma 1). For two word embeddings that encode information ^ Corollary 1. If UX = UY , then σ(Y ) = σ(Y ). similarly (say two distributional word embeddings) the entries th For the following theorem, define σi(M) to be the i sin- of the S matrix will all be roughly equal. However, for two gular value of the matrix M, and UMi to be the singular vector word embeddings that do not, it is likely that significant ad- associated with the ith singular value of M. justment will be required, with some entries in the S large to stretch the singular vectors, and some small to shrink them. Rn×p ∗ Theorem 1. Suppose X; Y 2 , and let Yb = XW be The results of our experiments support this idea. the least squares solution of Y = XW + . For singular value σi(Y ) and corresponding left singular vector Uyi, let > 3 Empirical Results ci = jjUX Uyijj2, then there exists a singular value in Yb at least as large as ciσi(Y ). In this section, we show some empirical results of word rep- resentation alignment. Our key finding suggests that isotropy ^ T Proof. By Lemma 1, σ(Y ) = σ(UX UY ΣY ). Recall- is important to successful alignment. In fact, aligning with T ing the definition of singular values, σ1(UX UY ΣY ) = isotropic noise can even yield satisfactory intrinsic evaluation T T maxjjujj2=1;jjvjj2=1 u UX UY ΣY v where u; v are unit vec- results. T tors. Define ei = (0; 0; ··· ; 0; 1; 0; ··· ; 0) where Conceptor Negation (CN). It is worth noting that the ge- ei is a unit vector whose elements are 0 except the th ometry of the distributional word embedding has been stud- i element which is 1. Then for any 1 ≤ i ≤ ied carefully [Mimno and Thompson, 2017b]. Mimno et p, σ (U >U Σ ) ≥ max uT U T U Σ e = 1 X Y Y jjujj2=1 X Y Y i al. [2017b] note that for word2vec word embedding point > > > maxjjujj2=1 u UX σi(Y )Uyi = σi(Y )jjUX Uyijj2 = clouds are concentrated within a narrow cone, which may lead ciσi(Y ). Therefore, the largest singular value must be greater to bias. ABTT [Mu et al., 2017] and conceptor negation [Liu than or equal to ciσi(Y ), for all i. et al., 2019b] are two methods used to correct this bias by Proposition 1. Suppose X 2 Rp is a random vector with all damping the larger singular values. Hence we suggest that entries distributed i.i.d with mean zero and a bounded fourth conceptor negation should be used post alignment in order to p p p control the eccentricity of the resulting aligned representation. moment. Let S be a matrix with s1; s2; ··· ; sp) along the diagonal and 0 everywhere else. Then for realizations X i 3.1 Experimental Setup and Xj of X, we have the following two results: We perform multiple experiments using distributional word 2 2 corr(jjXi−Xjjj2; jjXiS − XjSjj2) representations (each 300-dimensional) including word2vec p (s1 + s2 + ··· + sp)= p [Mikolov et al., 2013b] (Google News), GloVe [Pennington = q et al., 2014] (840 billion Common Crawl) and FastText [Bo- 2 2 2 s1 + s2 + ··· + sp janowski et al., 2017] (Common Crawl without subword), as our source embeddings, and align them through linear re- and gression to various target representations.
Load more

Recommended publications
  • Malware Classification with BERT
    San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-25-2021 Malware Classification with BERT Joel Lawrence Alvares Follow this and additional works at: https://scholarworks.sjsu.edu/etd_projects Part of the Artificial Intelligence and Robotics Commons, and the Information Security Commons Malware Classification with Word Embeddings Generated by BERT and Word2Vec Malware Classification with BERT Presented to Department of Computer Science San José State University In Partial Fulfillment of the Requirements for the Degree By Joel Alvares May 2021 Malware Classification with Word Embeddings Generated by BERT and Word2Vec The Designated Project Committee Approves the Project Titled Malware Classification with BERT by Joel Lawrence Alvares APPROVED FOR THE DEPARTMENT OF COMPUTER SCIENCE San Jose State University May 2021 Prof. Fabio Di Troia Department of Computer Science Prof. William Andreopoulos Department of Computer Science Prof. Katerina Potika Department of Computer Science 1 Malware Classification with Word Embeddings Generated by BERT and Word2Vec ABSTRACT Malware Classification is used to distinguish unique types of malware from each other. This project aims to carry out malware classification using word embeddings which are used in Natural Language Processing (NLP) to identify and evaluate the relationship between words of a sentence. Word embeddings generated by BERT and Word2Vec for malware samples to carry out multi-class classification. BERT is a transformer based pre- trained natural language processing (NLP) model which can be used for a wide range of tasks such as question answering, paraphrase generation and next sentence prediction. However, the attention mechanism of a pre-trained BERT model can also be used in malware classification by capturing information about relation between each opcode and every other opcode belonging to a malware family.
    [Show full text]
  • Arxiv:2002.06235V1
    Semantic Relatedness and Taxonomic Word Embeddings Magdalena Kacmajor John D. Kelleher Innovation Exchange ADAPT Research Centre IBM Ireland Technological University Dublin [email protected] [email protected] Filip Klubickaˇ Alfredo Maldonado ADAPT Research Centre Underwriters Laboratories Technological University Dublin [email protected] [email protected] Abstract This paper1 connects a series of papers dealing with taxonomic word embeddings. It begins by noting that there are different types of semantic relatedness and that different lexical representa- tions encode different forms of relatedness. A particularly important distinction within semantic relatedness is that of thematic versus taxonomic relatedness. Next, we present a number of ex- periments that analyse taxonomic embeddings that have been trained on a synthetic corpus that has been generated via a random walk over a taxonomy. These experiments demonstrate how the properties of the synthetic corpus, such as the percentage of rare words, are affected by the shape of the knowledge graph the corpus is generated from. Finally, we explore the interactions between the relative sizes of natural and synthetic corpora on the performance of embeddings when taxonomic and thematic embeddings are combined. 1 Introduction Deep learning has revolutionised natural language processing over the last decade. A key enabler of deep learning for natural language processing has been the development of word embeddings. One reason for this is that deep learning intrinsically involves the use of neural network models and these models only work with numeric inputs. Consequently, applying deep learning to natural language processing first requires developing a numeric representation for words. Word embeddings provide a way of creating numeric representations that have proven to have a number of advantages over traditional numeric rep- resentations of language.
    [Show full text]
  • Combining Word Embeddings with Semantic Resources
    AutoExtend: Combining Word Embeddings with Semantic Resources Sascha Rothe∗ LMU Munich Hinrich Schutze¨ ∗ LMU Munich We present AutoExtend, a system that combines word embeddings with semantic resources by learning embeddings for non-word objects like synsets and entities and learning word embeddings that incorporate the semantic information from the resource. The method is based on encoding and decoding the word embeddings and is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The obtained embeddings live in the same vector space as the input word embeddings. A sparse tensor formalization guar- antees efficiency and parallelizability. We use WordNet, GermaNet, and Freebase as semantic resources. AutoExtend achieves state-of-the-art performance on Word-in-Context Similarity and Word Sense Disambiguation tasks. 1. Introduction Unsupervised methods for learning word embeddings are widely used in natural lan- guage processing (NLP). The only data these methods need as input are very large corpora. However, in addition to corpora, there are many other resources that are undoubtedly useful in NLP, including lexical resources like WordNet and Wiktionary and knowledge bases like Wikipedia and Freebase. We will simply refer to these as resources. In this article, we present AutoExtend, a method for enriching these valuable resources with embeddings for non-word objects they describe; for example, Auto- Extend enriches WordNet with embeddings for synsets. The word embeddings and the new non-word embeddings live in the same vector space. Many NLP applications benefit if non-word objects described by resources—such as synsets in WordNet—are also available as embeddings.
    [Show full text]
  • Word-Like Character N-Gram Embedding
    Word-like character n-gram embedding Geewook Kim and Kazuki Fukui and Hidetoshi Shimodaira Department of Systems Science, Graduate School of Informatics, Kyoto University Mathematical Statistics Team, RIKEN Center for Advanced Intelligence Project fgeewook, [email protected], [email protected] Abstract Table 1: Top-10 2-grams in Sina Weibo and 4-grams in We propose a new word embedding method Japanese Twitter (Experiment 1). Words are indicated called word-like character n-gram embed- by boldface and space characters are marked by . ding, which learns distributed representations FNE WNE (Proposed) Chinese Japanese Chinese Japanese of words by embedding word-like character n- 1 ][ wwww 自己 フォロー grams. Our method is an extension of recently 2 。␣ ！！！！ 。␣ ありがと proposed segmentation-free word embedding, 3 ！␣ ありがと ][ wwww 4 .. りがとう 一个 ！！！！ which directly embeds frequent character n- 5 ]␣ ございま 微博 めっちゃ grams from a raw corpus. However, its n-gram 6 。。 うござい 什么 んだけど vocabulary tends to contain too many non- 7 ，我 とうござ 可以 うござい 8 ！！ ざいます 没有 line word n-grams. We solved this problem by in- 9 ␣我 がとうご 吗？ 2018 troducing an idea of expected word frequency. 10 了， ください 哈哈 じゃない Compared to the previously proposed meth- ods, our method can embed more words, along tion tools are used to determine word boundaries with the words that are not included in a given in the raw corpus. However, these segmenters re- basic word dictionary. Since our method does quire rich dictionaries for accurate segmentation, not rely on word segmentation with rich word which are expensive to prepare and not always dictionaries, it is especially effective when the available.
    [Show full text]
  • Learned in Speech Recognition: Contextual Acoustic Word Embeddings
    LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS Shruti Palaskar∗, Vikas Raunak∗ and Florian Metze Carnegie Mellon University, Pittsburgh, PA, U.S.A. fspalaska j vraunak j fmetze [email protected] ABSTRACT model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13]. Using this model, we jointly learn to End-to-end acoustic-to-word speech recognition models have re- automatically segment and classify input speech into individual cently gained popularity because they are easy to train, scale well to words, hence getting rid of the problem of chunking or requiring large amounts of training data, and do not require a lexicon. In addi- pre-defined word boundaries. As our A2W model is trained at the tion, word models may also be easier to integrate with downstream utterance level, we show that we can not only learn acoustic word tasks such as spoken language understanding, because inference embeddings, but also learn them in the proper context of their con- (search) is much simplified compared to phoneme, character or any taining sentence. We also evaluate our contextual acoustic word other sort of sub-word units. In this paper, we describe methods embeddings on a spoken language understanding task, demonstrat- to construct contextual acoustic word embeddings directly from a ing that they can be useful in non-transcription downstream tasks. supervised sequence-to-sequence acoustic-to-word speech recog- Our main contributions in this paper are the following: nition model using the learned attention distribution. On a suite 1. We demonstrate the usability of attention not only for aligning of 16 standard sentence evaluation tasks, our embeddings show words to acoustic frames without any forced alignment but also for competitive performance against a word2vec model trained on the constructing Contextual Acoustic Word Embeddings (CAWE).
    [Show full text]
  • Knowledge-Powered Deep Learning for Word Embedding
    Knowledge-Powered Deep Learning for Word Embedding Jiang Bian, Bin Gao, and Tie-Yan Liu Microsoft Research {jibian,bingao,tyliu}@microsoft.com Abstract. The basis of applying deep learning to solve natural language process- ing tasks is to obtain high-quality distributed representations of words, i.e., word embeddings, from large amounts of text data. However, text itself usually con- tains incomplete and ambiguous information, which makes necessity to leverage extra knowledge to understand it. Fortunately, text itself already contains well- defined morphological and syntactic knowledge; moreover, the large amount of texts on the Web enable the extraction of plenty of semantic knowledge. There- fore, it makes sense to design novel deep learning algorithms and systems in order to leverage the above knowledge to compute more effective word embed- dings. In this paper, we conduct an empirical study on the capacity of leveraging morphological, syntactic, and semantic knowledge to achieve high-quality word embeddings. Our study explores these types of knowledge to define new basis for word representation, provide additional input information, and serve as auxiliary supervision in deep learning, respectively. Experiments on an analogical reason- ing task, a word similarity task, and a word completion task have all demonstrated that knowledge-powered deep learning can enhance the effectiveness of word em- bedding. 1 Introduction With rapid development of deep learning techniques in recent years, it has drawn in- creasing attention to train complex and deep models on large amounts of data, in order to solve a wide range of text mining and natural language processing (NLP) tasks [4, 1, 8, 13, 19, 20].
    [Show full text]
  • Lecture 26 Word Embeddings and Recurrent Nets
    CS447: Natural Language Processing http://courses.engr.illinois.edu/cs447 Where we’re at Lecture 25: Word Embeddings and neural LMs Lecture 26: Recurrent networks Lecture 26 Lecture 27: Sequence labeling and Seq2Seq Lecture 28: Review for the final exam Word Embeddings and Lecture 29: In-class final exam Recurrent Nets Julia Hockenmaier [email protected] 3324 Siebel Center CS447: Natural Language Processing (J. Hockenmaier) !2 What are neural nets? Simplest variant: single-layer feedforward net For binary Output unit: scalar y classification tasks: Single output unit Input layer: vector x Return 1 if y > 0.5 Recap Return 0 otherwise For multiclass Output layer: vector y classification tasks: K output units (a vector) Input layer: vector x Each output unit " yi = class i Return argmaxi(yi) CS447: Natural Language Processing (J. Hockenmaier) !3 CS447: Natural Language Processing (J. Hockenmaier) !4 Multi-layer feedforward networks Multiclass models: softmax(yi) We can generalize this to multi-layer feedforward nets Multiclass classification = predict one of K classes. Return the class i with the highest score: argmaxi(yi) Output layer: vector y In neural networks, this is typically done by using the softmax N Hidden layer: vector hn function, which maps real-valued vectors in R into a distribution … … … over the N outputs … … … … … …. For a vector z = (z0…zK): P(i) = softmax(zi) = exp(zi) ∕ ∑k=0..K exp(zk) Hidden layer: vector h1 (NB: This is just logistic regression) Input layer: vector x CS447: Natural Language Processing (J. Hockenmaier) !5 CS447: Natural Language Processing (J. Hockenmaier) !6 Neural Language Models LMs define a distribution over strings: P(w1….wk) LMs factor P(w1….wk) into the probability of each word: " P(w1….wk) = P(w1)·P(w2|w1)·P(w3|w1w2)·…· P(wk | w1….wk#1) A neural LM needs to define a distribution over the V words in Neural Language the vocabulary, conditioned on the preceding words.
    [Show full text]
  • Local Homology of Word Embeddings
    Local Homology of Word Embeddings Tadas Temcinasˇ 1 Abstract Intuitively, stratification is a decomposition of a topologi- Topological data analysis (TDA) has been widely cal space into manifold-like pieces. When thinking about used to make progress on a number of problems. stratification learning and word embeddings, it seems intu- However, it seems that TDA application in natural itive that vectors of words corresponding to the same broad language processing (NLP) is at its infancy. In topic would constitute a structure, which we might hope this paper we try to bridge the gap by arguing why to be a manifold. Hence, for example, by looking at the TDA tools are a natural choice when it comes to intersections between those manifolds or singularities on analysing word embedding data. We describe the manifolds (both of which can be recovered using local a parallelisable unsupervised learning algorithm homology-based algorithms (Nanda, 2017)) one might hope based on local homology of datapoints and show to find vectors of homonyms like ‘bank’ (which can mean some experimental results on word embedding either a river bank, or a financial institution). This, in turn, data. We see that local homology of datapoints in has potential to help solve the word sense disambiguation word embedding data contains some information (WSD) problem in NLP, which is pinning down a particular that can potentially be used to solve the word meaning of a word used in a sentence when the word has sense disambiguation problem. multiple meanings. In this work we present a clustering algorithm based on lo- cal homology, which is more relaxed1 that the stratification- 1.
    [Show full text]
  • Word Embedding, Sense Embedding and Their Application to Word Sense Induction
    Word Embeddings, Sense Embeddings and their Application to Word Sense Induction Linfeng Song The University of Rochester Computer Science Department Rochester, NY 14627 Area Paper April 2016 Abstract This paper investigates the cutting-edge techniques for word embedding, sense embedding, and our evaluation results on large-scale datasets. Word embedding refers to a kind of methods that learn a distributed dense vector for each word in a vocabulary. Traditional word embedding methods first obtain the co-occurrence matrix then perform dimension reduction with PCA. Recent methods use neural language models that directly learn word vectors by predicting the context words of the target word. Moving one step forward, sense embedding learns a distributed vector for each sense of a word. They either define a sense as a cluster of contexts where the target word appears or define a sense based on a sense inventory. To evaluate the performance of the state-of-the-art sense embedding methods, I first compare them on the dominant word similarity datasets, then compare them on my experimental settings. In addition, I show that sense embedding is applicable to the task of word sense induction (WSI). Actually we are the first to show that sense embedding methods are competitive on WSI by building sense-embedding-based systems that demonstrate highly competitive performances on the SemEval 2010 WSI shared task. Finally, I propose several possible future research directions on word embedding and sense embedding. The University of Rochester Computer Science Department supported this work. Contents 1 Introduction 3 2 Word Embedding 5 2.1 Skip-gramModel................................
    [Show full text]
  • Learning Word Meta-Embeddings by Autoencoding
    Learning Word Meta-Embeddings by Autoencoding Cong Bao Danushka Bollegala Department of Computer Science Department of Computer Science University of Liverpool University of Liverpool [email protected] [email protected] Abstract Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks. However, their performances vary significantly across different tasks, implying that the word embeddings learnt by those methods capture complementary aspects of lexical semantics. Therefore, we believe that it is important to combine the existing word em- beddings to produce more accurate and complete meta-embeddings of words. We model the meta-embedding learning problem as an autoencoding problem, where we would like to learn a meta-embedding space that can accurately reconstruct all source embeddings simultaneously. Thereby, the meta-embedding space is enforced to capture complementary information in differ- ent source embeddings via a coherent common embedding space. We propose three flavours of autoencoded meta-embeddings motivated by different requirements that must be satisfied by a meta-embedding. Our experimental results on a series of benchmark evaluations show that the proposed autoencoded meta-embeddings outperform the existing state-of-the-art meta- embeddings in multiple tasks. 1 Introduction Representing the meanings of words is a fundamental task in Natural Language Processing (NLP). A popular approach to represent the meaning of a word is to embed it in some fixed-dimensional
    [Show full text]
  • A Comparison of Word Embeddings and N-Gram Models for Dbpedia Type and Invalid Entity Detection †
    information Article A Comparison of Word Embeddings and N-gram Models for DBpedia Type and Invalid Entity Detection † Hanqing Zhou *, Amal Zouaq and Diana Inkpen School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa ON K1N 6N5, Canada; [email protected] (A.Z.); [email protected] (D.I.) * Correspondence: [email protected]; Tel.: +1-613-562-5800 † This paper is an extended version of our conference paper: Hanqing Zhou, Amal Zouaq, and Diana Inkpen. DBpedia Entity Type Detection using Entity Embeddings and N-Gram Models. In Proceedings of the International Conference on Knowledge Engineering and Semantic Web (KESW 2017), Szczecin, Poland, 8–10 November 2017, pp. 309–322. Received: 6 November 2018; Accepted: 20 December 2018; Published: 25 December 2018 Abstract: This article presents and evaluates a method for the detection of DBpedia types and entities that can be used for knowledge base completion and maintenance. This method compares entity embeddings with traditional N-gram models coupled with clustering and classification. We tackle two challenges: (a) the detection of entity types, which can be used to detect invalid DBpedia types and assign DBpedia types for type-less entities; and (b) the detection of invalid entities in the resource description of a DBpedia entity. Our results show that entity embeddings outperform n-gram models for type and entity detection and can contribute to the improvement of DBpedia’s quality, maintenance, and evolution. Keywords: semantic web; DBpedia; entity embedding; n-grams; type identification; entity identification; data mining; machine learning 1. Introduction The Semantic Web is defined by Berners-Lee et al.
    [Show full text]
  • Arxiv:2007.00183V2 [Eess.AS] 24 Nov 2020
    WHOLE-WORD SEGMENTAL SPEECH RECOGNITION WITH ACOUSTIC WORD EMBEDDINGS Bowen Shi, Shane Settle, Karen Livescu TTI-Chicago, USA fbshi,settle.shane,[email protected] ABSTRACT Segmental models are sequence prediction models in which scores of hypotheses are based on entire variable-length seg- ments of frames. We consider segmental models for whole- word (“acoustic-to-word”) speech recognition, with the feature vectors defined using vector embeddings of segments. Such models are computationally challenging as the number of paths is proportional to the vocabulary size, which can be orders of magnitude larger than when using subword units like phones. We describe an efficient approach for end-to-end whole-word segmental models, with forward-backward and Viterbi de- coding performed on a GPU and a simple segment scoring function that reduces space complexity. In addition, we inves- tigate the use of pre-training via jointly trained acoustic word embeddings (AWEs) and acoustically grounded word embed- dings (AGWEs) of written word labels. We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional Fig. 1. Whole-word segmental model for speech recognition. (smaller) gains can be obtained by pre-training the word pre- Note: boundary frames are not shared. diction layer with AGWEs. Our final models improve over segmental models, where the sequence probability is com- prior A2W models. puted based on segment scores instead of frame probabilities. Index Terms— speech recognition, segmental model, Segmental models have a long history in speech recognition acoustic-to-word, acoustic word embeddings, pre-training research, but they have been used primarily for phonetic recog- nition or as phone-level acoustic models [11–18].
    [Show full text]