EMNLP 2011 TextInfer 2011 Workshop on Textual Entailment Proceedings of the Workshop July 30, 2011 Edinburgh, Scotland, UK c 2011 The Association for Computational Linguistics Order copies of this and other ACL proceedings from: Association for Computational Linguistics (ACL) 209 N. Eighth Street Stroudsburg, PA 18360 USA Tel: +1-570-476-8006 Fax: +1-570-476-0860 [email protected] ISBN 978-1-937284-15-2 / 1-937284-15-8 ii Introduction Textual inference and paraphrase have attracted a significant amount of attention in recent years. Many NLP tasks, including question answering, information extraction, and text summarization, can be mapped at least partially onto the recognition of textual entailments and the detection of semantic equivalence between texts. Robust and accurate algorithms and resources for inference and paraphrasing can be beneficial for a broad range of NLP applications, and have stimulated research in the area of applied semantics over the last years. The success of the Recognizing Textual Entailment challenges and the high participation in previous workshops on textual inference and paraphrases – Empirical Modeling of Semantic Equivalence and Entailment (ACL 2005), Textual Entailment and Paraphrasing (ACL/PASCAL 2007), and TextInfer 2009 (ACL) – show that there is substantial interest in the area among the research community. TextInfer 2011 follows these workshops and aims to provide a common forum for researchers to discuss and compare novel ideas, models and tools for textual inference and paraphrasing. One particular goal is to broaden the workshop to invite both theoretical and applied research contributions on the joint topic of “inference.” We aim to bring together empirical approaches, which have tended to dominate previous textual entailment events, with formal approaches to inference, which are more often presented at events like ICoS or IWCS. We feel that the time is ripe for researchers from both groups to join for this event, with the goal of establishing a discussion on how the two approaches relate to one another, and how to define interfaces between the two methodologies. We would like to thank all the people that made this event possible: the authors of submitted papers, the reviewers, and the participants. Enjoy the workshop! The workshop organizers, Peter Clark, Vulcan Inc. Ido Dagan, Bar-Ilan University Katrin Erk, University of Texas at Austin Sebastian Pado, Heidelberg University (Program Co-chair) Stefan Thater, Saarland University (Program Co-chair) Fabio Massimo Zanzotto, University of Rome “Tor Vergata” iii Organizers: Peter Clark, Vulcan Inc. Ido Dagan, Bar-Ilan University Katrin Erk, University of Texas at Austin Sebastian Pado, Heidelberg University (Program Co-chair) Stefan Thater, Saarland University (Program Co-chair) Fabio Massimo Zanzotto, University of Rome “Tor Vergata” Program Committee: Richard Bergmair, University of Cambridge (UK) Johan Bos, University of Groningen (Netherlands) Aljoscha Burchardt, DFKI (Germany) Chris Callison-Burch, John Hopkins University (USA) Phillip Cimiano, Bielefeld University (Germany) David Clausen, Stanford University (USA) Ann Copestake, Cambridge University (UK) Kees van Deemter, Aberdeen University (UK) Bill Dolan, Microsoft Research (USA) Mark Dras, Macquarie University (Australia) Markus Egg, HU Berlin (Germany) Anette Frank, Heidelberg University (Germany) Claire Gardent, LORIA (France) Andy Hickl, Extractiv/Swingly (USA) Graeme Hirst, University of Toronto (Canada) Jerry Hobbs, USC/ISI (USA) Kentaro Inui Tohoku University (Japan) Hans-Ulrich Krieger, DFKI (Germany) Piroska Lendvai, Hungarian Academy of Sciences (Hungary) Bill MacCartney, Google (USA) Bernardo Magnini, Fondazione Bruno Kessler (Italy) Marie-Catherine de Marneffe, Stanford University (USA) Erwin Marsi, NTNU (Norway) Yashar Mehdad, University of Trento (Italy) Detmar Meurers, Tuebingen University (Germany) Shachar Mirkin, Bar-Ilan University (Israel) Michael Mohler, University of North Texas (USA) Dan Moldovan, University of Texas at Dallas (USA) Robero Navigli, University of Rome (Italy) Patrick Pantel, Microsoft Research (USA) Marco Pennacchiotti, Yahoo! (USA) Ian Pratt-Hartmann, Manchester University (UK) v Dan Roth, University of Illinois at Urbana-Champaign (USA) Satoshi Sato, Nagoya University (Japan) Satoshi Sekine, New York University (USA) Idan Szpektor, Yahoo! (USA) Ivan Titov, Saarland University (Germany) Antonio Toral, Dublin City University (Ireland) Kentaro Torisawa, NICT (Japan) Annie Zaenen, PARC (USA) Invited Speaker: Bill Dolan, Microsoft Research (USA) vi Table of Contents Evaluating Answers to Reading Comprehension Questions in Context: Results for German and the Role of Information Structure Detmar Meurers, Ramon Ziai, Niels Ott and Janina Kopp . 1 Towards a Probabilistic Model for Lexical Entailment Eyal Shnarch, Jacob Goldberger and Ido Dagan. .10 Classification-based Contextual Preferences Shachar Mirkin, Ido Dagan, Lili Kotlerman and Idan Szpektor . 20 Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner Milen Kouylekov, Yashar Mehdad and Matteo Negri . 30 Diversity-aware Evaluation for Paraphrase Patterns Hideki Shima and Teruko Mitamura . 35 Representing and resolving ambiguities in ontology-based question answering Christina Unger and Philipp Cimiano . 40 Strings over intervals Tim Fernando . 50 Discovering Commonsense Entailment Rules Implicit in Sentences Jonathan Gordon and Lenhart Schubert . 59 vii Workshop Program Saturday, July 30, 2011 8:45–9:00 Opening Remarks 9:00–10:00 Invited Talk by Bill Dolan: “Broad-domain Paraphrasing” 10:00–10:30 Evaluating Answers to Reading Comprehension Questions in Context: Results for German and the Role of Information Structure Detmar Meurers, Ramon Ziai, Niels Ott and Janina Kopp 10:30–11:00 Coffee break 11:00–11:30 Towards a Probabilistic Model for Lexical Entailment Eyal Shnarch, Jacob Goldberger and Ido Dagan 11:30–12:00 Classification-based Contextual Preferences Shachar Mirkin, Ido Dagan, Lili Kotlerman and Idan Szpektor 12:00–12:20 Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner Milen Kouylekov, Yashar Mehdad and Matteo Negri 12:20–12:40 Diversity-aware Evaluation for Paraphrase Patterns Hideki Shima and Teruko Mitamura 12:40–14:20 Lunch 14:20–14:50 Representing and resolving ambiguities in ontology-based question answering Christina Unger and Philipp Cimiano 14:50–15:20 Strings over intervals Tim Fernando 15:20–15:40 Discovering Commonsense Entailment Rules Implicit in Sentences Jonathan Gordon and Lenhart Schubert 15:40–16:10 Coffee break ix Saturday, July 30, 2011 (continued) 16:10–17:00 Final Panel and Discussion x Evaluating Answers to Reading Comprehension Questions in Context: Results for German and the Role of Information Structure Detmar Meurers Ramon Ziai Niels Ott Janina Kopp Seminar fur¨ Sprachwissenschaft / SFB 833 Universitat¨ Tubingen¨ Wilhelmstraße 19 / Nauklerstraße 35 72074 Tubingen,¨ Germany {dm,rziai,nott,jkopp}@sfs.uni-tuebingen.de Abstract teaching, making it possible to use activities which are authentic and for which the language teachers Reading comprehension activities are an au- provide the gold standard judgements. thentic task including a rich, language-based context, which makes them an interesting real- Apart from the availability of authentic exercises life challenge for research into automatic con- and independently motivated gold standard judge- tent analysis. For textual entailment research, ments, there are two further reasons for putting read- content assessment of reading comprehension ing comprehension tasks into the spotlight for au- exercises provides an interesting opportunity tomatic meaning analysis. Firstly, such activities for extrinsic, real-purpose evaluation, which include a text as an explicit context on the basis of also supports the integration of context and which the questions are asked. Secondly, answers to task information into the analysis. reading comprehension questions in foreign language In this paper, we discuss the first results for teaching typically are between a couple of words and content assessment of reading comprehension several sentences in length – too short to rely purely activities for German and present results which are competitive with the current state of the on the distribution of lexical material (as, e.g., in art for English. Diving deeper into the results, LSA, Landauer et al., 1998). The answers also ex- we provide an analysis in terms of the differ- hibit a significant variation in form, including a high ent question types and the ways in which the number of form errors, which makes it necessary to information asked for is encoded in the text. develop an approach which is robust enough to de- We then turn to analyzing the role of the ques- termine meaning correspondences in the presence of tion and argue that the surface-based account errors yet flexible enough to support the rich vari- of information that is given in the question ation in form which language offers for expressing should be replaced with a more sophisticated, related meanings. linguistically informed analysis of the informa- tion structuring of the answer in the context of There is relatively little research on content assess- the question that it is a response to. ment for reading comprehension tasks and it so far has focused exclusively on English, including both reading comprehension questions answered by na- 1 Introduction tive speakers (Leacock and Chodorow, 2003; Nielsen Reading