JUNITMZ at SemEval-2016 Task 1: Identifying Semantic Similarity Using Levenshtein Ratio Sandip Sarkar Dipankar Das Computer Science and Engineering Computer Science and Engineering Jadavpur University, Kolkata Jadavpur University, Kolkata [email protected] [email protected] Partha Pakray Alexander Gelbukh Computer Science and Engineering Center for Computing Research NIT Mizoram, Mizoram Instituto Politcnico Nacional, Mexico [email protected] [email protected] Abstract across a variety of different domains. Research systems developed specifically for the STS 1 In this paper we describe the JUNITMZ sys- task have resulted in a progression of systems that tem that was developed for participation in Se- achieve increasing levels of performance but that are mEval 2016 Task 1: Semantic Textual Simi- larity. Methods for measuring the textual sim- often also increasingly more complex. Complex ap- ilarity are useful to a broad range of applica- proaches may be difficult if not impossible to incor- tions including: text mining, information re- porate as a component of larger applied NLP sys- trieval, dialogue systems, machine translation tems. and text summarization. However, many sys- The system described in this paper explores an al- tems developed specifically for STS are com- ternative approach based on three simple and ro- plex, making them hard to incorporate as a bust textual similarity features. Our features are module within a larger applied system. simple enough that they can be easily incorporated In this paper, we present an STS system based on three simple and robust similarity into larger applied systems that could benefit from features that can be easily incorporated into textual similarity scores. The first feature simply more complex applied systems. The shared counts the number of words common to the pair task results show that on most of the shared of sentences being assessed. The second provides tasks evaluation sets, these signals achieve a the length normalized edit distance to transform one strong (>0.70) level of correlation with hu- sentence into another. The final feature scores the man judgements. Our system’s three features two sentences using the METEOR machine transla- are: unigram overlap count, length normalized edit distance and the score computed by the tion metric. The latter allows the reuse of the lin- METEOR machine translation metric. Fea- guistic analysis modules developed within the ma- tures are combined to produces a similarity chine translation community to assess translation prediction using both a feedforward and recur- quality. METEOR’s implementation of these mod- rent neural network. ules is lightweight and efficient, making it not overly cumbersome to incorporate features based on ME- 1 Introduction TEOR into larger applied systems. The remainder of this paper is structured as follows. Semantic similarity plays important role in many Section 2 provides an overview of our system archi- natural language processing (NLP) applications. tecture and Section 3 describes our feature set. Sec- The semantic textual similarity (STS) shared task tion 4 reviews the neural network models we use to has been held annually since 2012 in order to assess predict the STS scores. Section 5 describes the eval- different approaches to computing textual similarity uation data followed by our results on the evaluation 1This work was supported by the Project No. data in Section 6. YSS/2015/000988 of National Institute of Technology Mizoram. 702 Proceedings of SemEval-2016, pages 702–705, San Diego, California, June 16-17, 2016. c 2016 Association for Computational Linguistics 2 System Framework Sentence Pair Score TSA drops effort to allow small knives As shown in Figure 1, our system performs on a neu- on planes. 6 ral network based regression over our three textual TSA drops plan to allow small knives similarity features. As described in the next sec- on planes. tion, the three similarity features we use are: uni- Table 1: Unigram matching, ignoring stopwords gram overlap count, editdistance and the METEOR score from the machine translation evaluation met- of individual characters that are required to trans- ric research community. The three features are com- form one sentence into another. Commonly known bined using a neural network in order to predict a as the Levenshtein distance (Levenshtein, 1966), pair’s final STS score. this string similarity metric captures both similarity in the overall structure of the two sentences being compared as well as some similarity between dif- ferent word forms (e.g., ”California” vs. ”Califor- nian”). As shown in equation (1) , we normalize the raw edit distance by the length of the two sentences. The score is then inverted such that a perfect match will have a score of 1.0, and completely dissimilar strings will be assigned a value of 0.0. EditDistance(a, b) EditRatio(a, b) = 1 (1) − a + b | | | | An example of the edit distance ratio feature is given in Table 2. Levenshtein Edit Distance Sentence Pair Distance Ratio Figure 1: JUNITMZ STS System Architecture TSA drops effort to allow small knives on planes. 6 .8958 TSA drops plan to allow 3 Features small knives on planes. Table 2: Edit Distance Ratio 3.1 Unigram matching without stop-word The unigram overlap count feature indicates the 3.3 Meteor number of non stop-words that appear in both sen- tence pairs. 2 Table 1 illustrates the operation of this METEOR is a well known evaluation metric from feature on an STS sentence pair. the machine translation community (Denkowski The words ”to” and ”on” are present in both sen- and Lavie, 2014). The method incorporates lin- tences, but we excluded them as stopwords for the guistic analysis modules but in a manner that is purposes of the unigram overlap count. lightweight, efficient and robust to the noisy data generated by machine translation systems. The 3.2 Edit Distance Ratio method operates by first computing an alignment be- tween the individual words in a sentence pair. In ad- We compute the minimum number of edit opera- ditional to matching identical words with each other, tions involving the insertion, deletion or substitution METEOR also supports matching words based on 2We obtain our stop word list from synonymy relationships in WordNet, entries in a http://www.nltk.org/book/ch02.html paraphrase database or by word stem. The metric 703 then computes a weighted F score based on the uni- Sentence Pairs Score gram alignments that is then scaled by a word scram- Two green and white trains bling penalty. The synonym matching is computed sitting on the tracks. 4.4 Two green and white trains using WordNet. We use the METEOR 1.5 system on tracks. for our STS Task. A cat standing on tree branches. 3.6 A black and white cat is high 4 Neural Network Framework up on tree branches. A woman riding a brown horse. 3.8 We predict STS scores based on three similarity fea- A young girl riding a brown horse. tures as described above using Matlab toolkit con- Table 3: Examples of sentence pairs with their gold scores (on taining modules for three different neural networks. a 5-point rating scale) The neural networks have been used with respect to each of the corresponding runs submitted by our Type Sentence Pair team to the shared task. The inputs of those network answer-answer 1572 were the feature set along with the gold standard headlines 1498 similarity scores extracted from the training data plagiarism 1271 whereas the outputs produce the semantic scores for postediting 3287 question-question 1555 the test dataset. In Run1, we use two-layer feedfor- ward network with 10 neurons in the hidden layer Table 4: Statistics of STS-2016 Test Data and trained using the Levenberg-Marquardt algo- rithm. 3 Run2 uses the same network but trained scores are aggregated to assign a final gold standard using Resilient Backpropagation algorithm (Ried- similarity score to each pair. miller and Braun, 1992). 4 In case of Run 3, we use Table 3 provides example sentence pairs with their the framework of Layer Recurrent Network which corresponding gold standard similarity scores. Sys- can be seen as a generalization of simple recurrent tems are assessed on each data set based on the Pear- networks (Elman, 1990). 5. The inputs of this re- son correlation between the scores they produce and current neural network were similar like the other 2 the gold standard. The detailed statistics of the STS- neural network with default parameter. 2016 Test datasets are given in Table 4. For the training process we used all gold standard training 5 Dataset and test data of year 2012 to 2015 resulting in 12500 sentence pairs. The 2016 STS shared task includes sentence pairs from a number of different data sources organized 6 Result into five evaluation sets: News Headlines, Plagia- rism, Postediting, Q&A Answer-Answer and Q&A For our training dataset we use trail, training and test Question-Question. The sentence pairs are assigned data from previous STS competitions. As shown in similarity scores by multiple crowdsourced annota- Table 5, we used different subsets of the data from tors on a scale ranging from 0 to 5 with the scores prior STS evaluations to train different models for having the following interpretations: (5) complete the 2016 evaluation sets. equivalence, (4) equivalent but differing in minor Table 7 illustrates the performance of our three details, (3) roughly equivalent but differing in im- system submission on each of the STS 2016 eval- portant details (2) not equivalent but sharing some uation sets as assessed by their correlation with details (1) not equivalent but on the same topic (0) the gold standard similarity scores. Overall perfor- completely dissimilar. The individual crowdsourced mance is reported as the weighted mean correlation across all five data sets.