Learning Narrative Structure from Annotated Folktales by Mark Alan Finlayson B.S.E., University of Michigan (1998) S.M., Massachusetts Institute of Technology (2001) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY February 2012 c Massachusetts Institute of Technology 2012. All rights reserved. Author............................................................................ Department of Electrical Engineering and Computer Science December 20, 2011 Certified by........................................................................ Patrick H. Winston Ford Professor of Artificial Intelligence and Computer Science Thesis Supervisor Accepted by....................................................................... Leslie A. Kolodziejski Chair of the Department Committee on Graduate Students 2 Learning Narrative Structure from Annotated Folktales by Mark Alan Finlayson Submitted to the Department of Electrical Engineering and Computer Science on December 20, 2011, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical Engineering and Computer Science Abstract Narrative structure is an ubiquitous and intriguing phenomenon. By virtue of structure we recognize the presence of Villainy or Revenge in a story, even if that word is not actually present in the text. Narrative structure is an anvil for forging new artificial intelligence and machine learning techniques, and is a window into abstraction and conceptual learning as well as into culture and its influence on cognition. I advance our understanding of narrative structure by describing Analogical Story Merging (ASM), a new machine learning algorithm that can extract culturally-relevant plot patterns from sets of folktales. I demonstrate that ASM can learn a substantive portion of Vladimir Propp's influential theory of the structure of folktale plots. The challenge was to take descriptions at one semantic level, namely, an event timeline as de- scribed in folktales, and abstract to the next higher level: structures such as Villainy, Stuggle- Victory, and Reward. ASM is based on Bayesian Model Merging, a technique for learning regular grammars. I demonstrate that, despite ASM's large search space, a carefully-tuned prior allows the algorithm to converge, and furthermore it reproduces Propp's categories with a chance-adjusted Rand index of 0.511 to 0.714. Three important categories are identified with F-measures above 0.8. The data are 15 Russian folktales, comprising 18,862 words, a subset of Propp's original tales. This subset was annotated for 18 aspects of meaning by 12 annotators using the Story Workbench, a general text-annotation tool I developed for this work. Each aspect was doubly-annotated and adjudicated at inter-annotator F-measures that cluster around 0.7 to 0.8. It is the largest, most deeply-annotated narrative corpus assembled to date. The work has significance far beyond folktales. First, it points the way toward important ap- plications in many domains, including information retrieval, persuasion and negotiation, natural language understanding and generation, and computational creativity. Second, abstraction from natural language semantics is a skill that underlies many cognitive tasks, and so this work provides insight into those processes. Finally, the work opens the door to a computational understanding of cultural influences on cognition and understanding cultural differences as captured in stories. Dissertation Supervisor: Patrick H. Winston Professor, Electrical Engineering and Computer Science Dissertation Committee: Whitman A. Richards Professor, Brain & Cognitive Sciences Peter Szolovits Professor, Electrical Engineering and Computer Science & Harvard-MIT Division of Health Sciences and Technology Joshua B. Tenenbaum Professor, Brain & Cognitive Sciences 3 4 to LDF 5 6 Acknowledgments Without generous funding by the government this work would not have been possible. My initial explorations of intermediate-feature-based analogical retrieval were supported by National Science Foundation grant IIS-0218861, \Reinventing Artificial Intelligence." It was then that I began to con- sider the challenge of breaking stories into their constituents. Development of the Story Workbench was begun under National Science Foundation grant IIS-0413206, \Representationally Complete Analogical Reasoning Systems and Steps Toward Their Application in Political Science." I co-wrote that proposal with Patrick Winston, my advisor. It was the first of the many deeply educational experiences he provided me over my graduate career. He was an armamentarium of sage advice, and he was surely a sine qua non of this work and my doctorate. The Air Force Office of Scientific Research, under contract A9550-05-1-032, \Computational Models for Belief Revision, Group Decisions, and Cultural Shifts," supported further development of the Story Workbench as well as the early implementations of Analogical Story Merging. The AFOSR funding was particularly influential on my work, primarily via Whitman Richards, who early on pointed out to me the importance of stories in culture, and guided my work in that direction; he later joined my committee and, as attested by the excellent advice and support he provided, became my de facto co-advisor. The Defense Advanced Research Projects Agency, under contract FA8750-10-1-0076, “Defining and Demonstrating Capabilities for Experience-Based Narrative Memory," provided the support for annotation of the Russian folktales that are the data underlying this work. The remainder of my graduate career, including the final experiments and writing of the dissertation itself, was supported by the Office of the Secretary of Defense, under Minerva contract N00014-09-1-0597, \Explorations in Cyber-International Relations." Without all of these various funds this work would have been impossible to complete, even within the generous timeline it has so far consumed. Brett van Zuiden, my especially bright and motivated undergraduate research assistant during the Summer and Fall of 2010, was responsible for a significant portion of the implementation described in Section 3.4.3. Finally, my annotation team was critical to the success of this work. They were Julia Arnous, Saam Batmanghelidj, Paul Harold, Beryl Lipton, Matt Lord, Sharon Mozgai, Patrick Rich, Jared Sprague, Courtney Thomas, Geneva Trotter, and Ashley Turner. 7 Contents 1 Motivation 13 1.1 Outline . 14 1.2 Artificial Intelligence & Machine Learning . 14 1.3 Cognition & Abstraction . 15 1.4 Culture & Folktales . 16 2 Propp's Morphology 19 2.1 Three Levels . 19 2.1.1 Gross Tale Structure: Moves . 19 2.1.2 Intermediate Tale Structure: Functions . 20 2.1.3 Fine Tale Structure: Subtypes . 21 2.2 Learning Strategy . 23 2.2.1 Grammatical Inference . 23 2.2.2 Learning Strategy for Propp's Data . 24 2.3 Texts and Translations . 25 3 Analogical Story Merging 27 3.1 Bayesian Model Merging . 27 3.2 ASM Augmentations . 28 3.2.1 Filtering . 28 3.2.2 Analogical Mapping . 29 3.3 ASM Parameters . 31 3.3.1 Initial Model Construction . 31 3.3.2 Merge Operations . 31 3.3.3 Prior Distribution . 32 3.4 Examples . 33 3.4.1 Toy Example . 33 3.4.2 Shakespearean Plays . 35 3.4.3 Lehnert-style Plot Units . 37 4 Corpus Annotation 41 4.1 Text Annotation . 41 4.1.1 Automatic Annotation . 42 4.1.2 Manual Annotation . 43 4.1.3 Semi-Automatic Annotation: The Story Workbench . 43 4.2 Representations . 43 4.2.1 Syntax . 43 4.2.2 Senses . 47 4.2.3 Entities . 48 4.2.4 Timeline . 49 4.2.5 New Representations . 52 4.2.6 Propp's Morphology . 54 4.3 Annotation Evaluation . 56 4.4 The Story Workbench: Design Details . 58 8 4.4.1 Design Principles & Desiderata . 58 4.4.2 Annotation Process . 61 4.4.3 Story Workbench Functionality . 62 5 Results 67 5.1 Data Preparation . 67 5.1.1 Timeline . 67 5.1.2 Semantic Roles . 68 5.1.3 Referent Structure . 70 5.1.4 Filtering the Timeline . 70 5.2 Merging . 71 5.2.1 Stage One: Semantics . 71 5.2.2 Stage Two: Valence . 72 5.2.3 Filtering . 72 5.3 Gold Standard . 73 5.4 Analysis . 73 5.4.1 Event Clustering . 75 5.4.2 Function Categories . 75 5.4.3 Transition Structure . 76 5.4.4 Data Variation . 76 5.4.5 Parameter Variation . 78 5.5 Shortcomings . 78 5.5.1 Performance . 78 5.5.2 Prior . 80 5.5.3 Representations . 81 5.5.4 Gold Standard . 82 6 Related Work 83 6.1 Structural Analyses of Narrative . 83 6.1.1 Universal Plot Catalogs . 83 6.1.2 Story Grammars . 84 6.1.3 Scripts, Plans, & Goals . 85 6.1.4 Motifs & Tale Types . 85 6.1.5 Other Morphologies . 86 6.2 Extracting Narrative Structure . 87 6.3 Corpus Annotation . 89 6.3.1 Tools & Formats . ..