2009 IEEE International Conference on Semantic Computing Generalizing Latent Semantic Analysis Andrew M. Olney Institute for Intelligent Systems University of Memphis Memphis, USA Email: [email protected] Abstract—Latent Semantic Analysis (LSA) is a vector space for each “document,” usually a paragraph. The cells cij of technique for representing word meaning. Traditionally, LSA the matrix consist of a simple count of the number of times consists of two steps, the formation of a word by document word appeared in document . Since many words do not matrix followed by singular value decomposition of that ma- i j trix. However, the formation of the matrix according to the appear in any given document, the matrix is often sparse. dimensions of words and documents is somewhat arbitrary. Weightings are applied to the cells that take into account This paper attempts to reconceptualize LSA in more general the frequency of wordi in documentj and the frequency of terms, by characterizing the matrix as a feature by context wordi across all documents, such that distinctive words that matrix rather than a word by document matrix. Examples appear infrequently are given the most weight. of generalized LSA utilizing n-grams and local context are presented and compared with traditional LSA on paraphrase Using the vector space model, two collections of words comparison tasks. of arbitrary size are compared by creating two vectors. Each word is associated with a row vector in the matrix, and Keywords-latent semantic analysis; vector space; n-gram; paraphrase; the vector of a collection is simply the sum of all the row vectors of words in that collection. Vectors are compared I. INTRODUCTION geometrically by the cosine of the angle between them. From this description it should be clear that the vector space At the most basic level, models of semantics try to capture model is both unsupervised and generative, meaning that the meanings of words. This somewhat then begs the ques- new collections of words, i.e. sentences or documents, can tion of where the words themselves get their meanings. A be turned into vectors and then compared with any other common example of this problem can be seen in dictionary collection of words. entries. Any entry defines the meaning of a word in terms Latent semantic analysis (LSA) [4]–[7] is an extension of . other words! Therefore there is a certain amount of the vector space model that uses singular value de- of circularity in defining the meanings of words that is composition (SVD). SVD is a technique that creates an inescapable when working only with other words. approximation of the original word by document matrix. A natural way of looking at word meanings is whether After SVD, the original matrix is equal to the product two words have the same or opposite meanings, i.e. are of three matrices, word by singular value, singular value synonyms or antonyms [1]. Words with these relations have by singular value, and singular value by document. The the property that they may occur in the same contexts: size of each singular value corresponds to the amount of John felt happy vs. John felt sad variance captured by a particular dimension of the matrix. Mary walked across the street vs. Mary ambled Because the singular values are ordered in decreasing size, across the street it is possible to remove the smaller dimensions and still Therefore some aspects of word meaning can be defined account for most of the variance. The approximation to the by looking at the contexts in which words occur. Words original matrix is optimal, in the least squares sense, for any with similar contexts may have meaning relationships like number of dimensions one would choose [8]. In addition, synonym or antonym. This is particularly relevant in in- the removal of smaller dimensions introduces linear depen- formation retrieval, where there may be multiple ways of dencies between words that are distinct only in dimensions expressing the same query. that account for the least variance. Consequently, two words The vector space model is a statistical technique that that were distant in the original space can be near in the represents the similarity between collections of words as a compressed space, causing the inductive machine learning cosine between vectors [2], [3]. As such it can be used as the and knowledge acquisition effects reported in the literature basis for a computational approach to word meaning or to [6]. This inductive property of LSA is what makes it useful information retrieval. The process begins by collecting text for many applications, including approximating vocabulary into a corpus. A matrix is created from the corpus, having acquisition in children [6], cohesion detection [9], grading one row for each unique word in the corpus and one column essays [10], and understanding student contributions in tu- 978-0-7695-3800-6/09 $26.00 © 2009 IEEE 40 DOI 10.1109/ICSC.2009.89 torial dialogue [11], [12], entailment detection [13], and applied to both unsupervised part of speech induction [21]– dialogue segmentation [14], amongst many others. [25] and unsupervised grammar induction [26]–[29]. LSA Recently several attempts have been made to generalize can be thought of as a kind of distributional analysis, but semantic spaces and related distributional approaches. Al- with a different definition of context. With LSA, context is though LSA is a kind of semantic space, these previous defined in the most relaxed sense, because left and right attempts do not specifically address matrix representation environments are equivalent: all that counts is that the and construction. Weeds & Weir [15] present a framework word appears in the document at all. Again, this example for distributional similarity metrics, derived from the infor- suggests that other dimensions besides documents might be mation retrieval metrics of precision and recall. Thus their productive for some tasks. paper focuses on calculating distributional similarity, rather The examples of n-grams and distributional contexts as than matrix representation. Pado´ & Lapata [16] describe a replacements for words and documents respectively are methodology for constructing dependency-based semantic just a few out of countless possibilities. What is common spaces that extends a previous formulation [17]. This work amongst all these matrices is that they are constructed is complementary to the work presented here, in that it according to the following procedure. First, some feature attempts to extend beyond the descriptive formalism of of interest is selected for which a vector representation is [17] to a constructive/generative methodology. However, the desired. In LSA, this feature is a word. However, it could differences between the present work and [16] are threefold: be n-grams, part of speech, etc. Second, a data stream is the present work does not focus solely on syntax but rather identified that contains the feature of interest, and a sampling attempts to generalize beyond syntax, is strongly focused protocol for the data stream is defined. In LSA, the data on matrix construction, and specifically addresses questions stream is a corpus and the sampling protocol is tokenization raised by the LSA community regarding generalization and into sentence, paragraph, or multi-page units. However, the word order [7]. sampling protocol could be a moving window as in the distributional analysis literature mentioned above and also in II. GENERALIZING LATENT SEMANTIC ANALYSIS Burgess et al. [30]. Finally, this sampling protocol must be mapped to a schema representing the columns of the matrix. We argue that the traditional word by document matrix is In LSA the schema is transparent: a sample (document) is somewhat arbitrary, and that other matrix forms should be a column. However, in a distributional schema representing considered. These matrix forms may differ on one or both left and right neighbor words, the mapping is slightly less dimensions, either word or document. Consider the word obvious. Let α be the position before the target word (row dimension (rows). In LSA, a “bag of words” approach, the word) and β be the position following the target word. If sentences “John called Mary” and “Mary called John” are there are n unique words in the corpus, then both α and equivalent, i.e. they are the same vector. This is so because β have size n. Then there are 2n columns in the matrix the individual word vectors from each sentence are added for both of these positions: the first n columns for all together to make the sentence vectors, and addition in vector words that occur before the target word, and the second n spaces is commutative, i.e. 1+4 = 4+1 [18]. However, columns for all the words that occur after the target word. the semantic equivalence of “John called Mary” and “Mary For example, the frequency of a specific word (“likes”) at called John” is clearly undesirable. a specific position (α) relative to a target word (“John”), One proposal for incorporating word order into LSA is to across the entire corpus, is in the cell (“John”,αlikes). create an n-gram by document matrix instead of a word by Assuming a sorted order on the columns for α and β, the document matrix. In this model, a bigram vector for “tall corresponding frequency for “likes” in the β position is in man” is atomic, rather than the combination of “tall” and the cell (“John”,βlikes = αlikes+n). Therefore, rather than a “man.” By incorporating word order at the atomic level of a “word by document matrix,” an appropriate description for vector, word order is included without violating the vector generalized LSA would be a “feature by context” matrix, space property of LSA that requires commutative addition where context is dependent both on the sampling protocol of vectors.