An Application of Latent Semantic Analysis to Word Sense Discrimination for Words with Related and Unrelated Meanings Juan Pino and Maxine Eskenazi (jmpino, max)@cs.cmu.edu Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA Abstract We present an application of Latent Semantic Analysis to word sense discrimination within a tutor for English vocabulary learning. We attempt to match the meaning of a word in a document with the meaning of the same word in a fill-in-the-blank question. We compare the performance of the Lesk algorithm to La- tent Semantic Analysis. We also compare the Figure 1: Example of cloze question. performance of Latent Semantic Analysis on a set of words with several unrelated meanings and on a set of words having both related and To define the problem formally, given a target unrelated meanings. word w, a string r (the reading) containing w and n strings q1, ..., qn (the sentences used for the ques- 1 Introduction tions) each containing w, find the strings qi where In this paper, we present an application of Latent the meaning of w is closest to its meaning in r. Semantic Analysis (LSA) to word sense discrimi- We make the problem simpler by selecting only one nation (WSD) within a tutor for English vocabu- question. lary learning for non-native speakers. This tutor re- This problem is challenging because the context trieves documents from the Web that contain tar- defined by cloze questions is short. Furthermore, get words a student needs to learn and that are at a word can have only slight variations in meaning an appropriate reading level (Collins-Thompson and that even humans find sometimes difficult to distin- Callan, 2005). It presents a document to the student guish. LSA was originally applied to Information and then follows the document reading with practice Retrieval (Dumais et al., 1988). It was shown to be questions that measure how the student’s knowledge able to match short queries to relevant documents has evolved. It is important that the fill-in-the-blank even when there were no exact matches between the questions (also known as cloze questions) that we words. Therefore LSA would seem to be an appro- ask to the students allow us to determine their vocab- priate technique for matching a short context, such ulary knowledge accurately. An example of cloze as a question, with a whole document. question is shown in Figure 1. So we are looking to first discriminate between Some words have more than one meaning and so the meanings of words, such as “compound”, that the cloze question we give could be about a different have several very different meanings (a chemical meaning than the one that the student learned in the compound or a set of buildings) and then to dis- document. This is something that can lead to confu- ambiguate words that have senses that are closely sion and must be avoided. To do this, we need to use related such as “comprise” (“be composed of” or some automatic measure of semantic similarity. “compose”). In the following sections, we present 43 Proceedings of the NAACL HLT Workshop on Innovative Use of NLP for Building Educational Applications, pages 43–46, Boulder, Colorado, June 2009. c 2009 Association for Computational Linguistics LSA and some of its applications, then we present 3 Experimental Setup some experimental results that compare a baseline to We used a database of 62 manually generated cloze the use of LSA for both tasks we have just described. 1 We expect the task to be easier on words with unre- questions covering 16 target words . We manually lated meanings. In addition, we expect that LSA will annotated the senses of the target words in these perform better when we use context selection on the questions using WordNet senses (Fellbaum, 1998). documents. For each word and for each sense, we manually gath- ered documents from the Web containing the target 2 Related Work word with the corresponding sense. There were 84 documents in total. We added 97 documents ex- LSA was originally applied to Information Retrieval tracted from the tutor database of documents that (Dumais et al., 1988) and called Latent Semantic In- contained at least one target word but we did not an- dexing (LSI). It is based on the singular value de- notate their meaning. composition (SVD) theorem. A m n matrix X We wanted to evaluate the performances of LSA × with m n can be written as X = U S V T where for WSD for words with unrelated meanings and for ≥ · · U is a m n matrix such that U T U = I ; S is words with both related and unrelated meanings. For × · m a n n diagonal matrix whose diagonal coefficients the first type of evaluation, we retained four target × are in decreasing order; and V is a n n matrix such words. For the second type of evaluation, all 16 × that V T V = I . words were included. We also wanted to evaluate · n X is typically a term-document matrix that repre- the influence of the size of the context of the tar- sents the occurrences of vocabulary words in a set of get words. We therefore considered two matrices: documents. LSI uses truncated SVD, that is it con- a term-document matrix and a term-context matrix siders the first r columns of U (written Ur), the r where context designates five sentences around the highest coefficients in S (Sr) and the first r columns target word in the document. In both cases each of V (Vr). Similarity between a query and a docu- cell of the matrix had a tf-idf weight. Finally, we ment represented by vectors d and q is performed by wanted to investigate the influence of the dimension computing the cosine similarity between S 1 U T d reduction on performance. In our experiments, we r− · r · and S 1 U T q. The motivation for computing sim- explored these three directions. r− · r · ilarity in a different space is to cope with the sparsity of the vectors in the original space. The motivation 4 Results for truncating SVD is that only the most meaning- 4.1 Baseline ful semantic components of the document and the query are represented after this transformation and We first used a variant of the Lesk algorithm (Lesk, that noise is discarded. 1986), which is based on word exact match. This al- gorithm seems well suited for the unsupervised ap- LSA was subsequently applied to number of prob- proach we took here since we were dealing with lems, such as synonym detection (Landauer et al., discrimination rather than disambiguation. Given 1998), document clustering (Song and Park, 2007), a document and a question , we computed the vocabulary acquisition simulation (Landauer and d q number of word tokens that were shared between Dumais, 1997), etc. d and q, excluding the target word. The words were Levin and colleagues (2006) applied LSA to word lower cased and stemmed using the Porter stem- sense discrimination. They clustered documents mer. Stop words and punctuation were discarded; containing ambiguous words and for a test instance we used the standard English stopword list. Finally, of a document, they assigned the document to its we selected a window of nw words around the tar- closest cluster. Our approach is to assign to a doc- get word in the question q and a window of ns ument the question that is closest. In addition, we sentences around the target word in the document examine the cases where a word has several unre- d. In order to detect sentence boundaries, we used lated meanings and where a word has several closely related meanings. 1available at: www.cs.cmu.edu/ jmpino/questions.xls 44 the OpenNLP toolkit (Baldridge et al., 2002). With nw = 10 and ns = 2, we obtained an accuracy of 61% for the Lesk algorithm. This can be compared to a random baseline of 44% accuracy. Accuracy vs. Dimension Reduction for Related Meanings with Different Contexts 1 whole document selected context Lesk baseline 4.2 LSA 0.8 We indexed the document database using the Lemur 0.6 toolkit (Allan et al., 2003). The database contained Accuracy both the manually annotated documents and the doc- 0.4 uments used by the tutor and containing the target words. The Colt package (Binko et al., ) was used 0.2 to perform singular value decomposition and matrix operations because it supports sparse matrix oper- 0 60 80 100 120 140 160 180 ations. We explored three directions in our analy- Truncation Parameter sis. We investigated how LSA performs for words with related meanings and for words with unrelated Figure 2: Accuracy vs. r, the truncation parameter, for words with related and unrelated meanings and with meanings. We also explored the influence of the whole document or selected context (95% confidence for truncation parameter r. Finally, we examined if re- whole document: [0.59; 0.65], 95% confidence for se- ducing the document to a selected context of the tar- lected context: [0.52; 0.67]) get word improved performance. Figures 2 and 3 plot accuracy versus dimension reduction in different cases. In all cases, LSA out- performs the baseline for certain values of the trun- cation parameter and when context selection was used. This shows that LSA is well suited for measur- ing semantic similarity between two contexts when at least one of them is short. In general, using the full Accuracy vs. Dimension Reduction for Unrelated Meanings with Different Contexts 1 whole document dimension in SVD hurts the performances.