Monte Carlo Simulation Generation Through Operationalization of Spatial Primitives A Dissertation Presented to The Faculty of the Graduate School of Arts and Sciences Brandeis University Department of Computer Science Dr. James Pustejovsky, Advisor In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy by Nikhil Krishnaswamy August, 2017 The signed version of this form is on file in the Graduate School of Arts and Sciences. This dissertation, directed and approved by Nikhil Krishnaswamy’s committee, has been accepted and approved by the Graduate Faculty of Brandeis University in partial fulfillment of the requirements for the degree of: DOCTOR OF PHILOSOPHY Eric Chasalow, Dean of Arts and Sciences Dissertation Committee: Dr. James Pustejovsky, Chair Dr. Kenneth D. Forbus, Dept. of Electrical Eng. and Comp. Sci., Northwestern University Dr. Timothy J. Hickey, Dept. of Computer Science, Brandeis University Dr. Marc Verhagen, Dept. of Computer Science, Brandeis University c Copyright by Nikhil Krishnaswamy 2017 For my father Acknowledgments Like a wizard, a thesis never arrives late, or early, but precisely when it means to; but it would never arrive at all without the people who helped it along the way. First and foremost, I would like to thank Prof. James Pustejovsky for taking a chance on a crazy idea and tirelessly pursuing opportunities for the topic and for me in particular, for always taking time to discuss ideas and applications any time, any place (during the week, on the weekend, with beer, without beer, on three continents). And for, when I requested to stay on at Brandeis after completing my Master’s degree, writing a letter of recommendation on my behalf, to himself. That story usually kills. I would like to thank my committee members: Dr. Marc Verhagen, for hours of stimulating discussions; Prof. Tim Hickey, for letting me poach his animation students to help me develop the simulation software; and Prof. Ken Forbus, with whom it is an honor to have the opportunity to share my research. Additionally, thank you all for your perceptive, thorough, and insightful feedback in molding the draft copy of this thesis into its final form. To my friends and family, thank you; particularly to my wife, Heather, for letting this shady roommate of an idea move in with us; to my mother and stepfather, Uma Krishnaswami and Satish Shrikhande, for their unwavering faith and support—didn’t I say not to worry? I promised I could handle it. To the student workers who contributed many enthusiastic hours developing VoxSim, thank v you; particularly to Jessica Huynh, Paul Kang, Subahu Rayamajhi, Amy Wu, Beverly Lum, and Victoria Tran. I’d probably have quit long ago without you. To the community of Unity developers, whose collective knowledge I spent hours reading online. Special thanks to Neil Mehta of LaunchPoint Games for prompt response and service helping me debug his plugin. To the faculty and staff of the Brandeis Master’s program in Computational Linguistics, partic- ularly to Prof. Lotus Goldberg for her consistent good wishes, cheer, advice, and ongoing interest in this thesis, and to all the students I’ve had the pleasure of getting to know along the way. Clearly there’s something special going on here because I’ve done everything I can to avoid leaving. Without Dr. Paul Cohen and the Communicating with Computers project at DARPA, this research would likely never have gotten off the ground. Working on the program I have enjoyed, and continue to enjoy, many fruitful collaborations that drove, and sometimes forced, development on VoxSim and have really put the software through a stress test. In this regard, I’d like to give particular thanks to Prof. Bruce Draper and his group at Colorado State University. Additional thanks also goes to Eric Burns at Lockheed Martin, for his advice and encourage- ment during the early part of my Ph.D., where I learned to spot opportunities as they arose, and seize them before time ran out. This work was supported by Contract W911NF-15-C-0238 with the U.S. Defense Advanced Research Projects Agency (DARPA) and the Army Research Office (ARO). Approved for Public Release, Distribution Unlimited. The views expressed herein are mine and do not reflect the official policy or position of the Department of Defense or the U.S. Government. All errors and mistakes are, of course, my own. I would like to dedicate this research and its culmination, this dissertation, to the memories of those family who passed on during my time in graduate school: my grandfather, Mr. V. Krishna Swami Iyengar; my grandmother, Mrs. Hema Krishnaswamy; my stepbrother, Kedar Shrikhande; vi and my father, Dr. Sumant Krishnaswamy. vii Abstract Monte Carlo Simulation Generation Through Operationalization of Spatial Primitives A dissertation presented to the Faculty of the Graduate School of Arts and Sciences of Brandeis University, Waltham, Massachusetts by Nikhil Krishnaswamy Much existing work in text to scene generation focuses on generating static scenes, which leaves aside entire word classes such as motion verbs. This thesis introduces a system for generating animated visualizations of motion events by integrating dynamic semantics into a formal model of events, resulting in a simulation of an event described in natural language. Visualization, herein defined as a dynamic three-dimensional simulation and rendering that satisfies the constraints of an associated minimal model, provides a framework for evaluating the properties of spatial predicates in real-time, but requires the specification of values and parameters that can be left underspecified in the model. Thus, there remains the matter of determining what, if any, the “best” values of those parameters are. This research explores a method of using a three-dimensional simulation and visualization interface to determine prototypical values for underspecified param- eters of motion predicates, built on a game engine-based platform that allows the development of semantically-grounded reasoning components in areas in the intersection of theoretical reasoning and AI. viii Contents Abstract viii 1 Introduction 1 1.1 Background . .3 1.2 Information-Theoretic Foundations . .4 1.3 Linguistic Underspecification in Motion Events . .5 1.4 Related Prior Work . .6 2 Framework 8 2.1 VoxML: Visual Object Concept Modeling Language . 12 2.2 VoxSim . 20 2.3 Spatial Reasoning . 23 2.4 Object Model . 28 2.5 Action Model . 29 2.6 Event Model . 29 2.7 Event Composition . 31 2.8 Specification Methods . 36 3 Methodology and Experimentation 38 3.1 Preprocessing . 40 3.2 Operationalization . 48 3.3 Monte Carlo Simulation . 52 3.4 Evaluation . 57 4 Results and Discussion 65 4.1 Human Evaluation Task 1 Results . 66 4.2 Human Evaluation Task 2 Results . 80 4.3 Automatic Evaluation Task Results . 92 4.4 Mechanical Turk Worker Response . 106 4.5 Summary . 108 ix CONTENTS 5 Future Directions 112 5.1 Extensions to Methdology . 114 5.2 VoxML and Robotics . 114 5.3 Information-Theoretic Implications . 115 A VoxML Structures 117 A.1 Objects . 117 A.2 Programs . 131 A.3 Relations . 136 A.4 Functions . 138 B Underspecifications 139 C [[TURN]]: Complete Operationalization 141 D Sentence Test Set 147 E Data Tables 166 E.1 DNN with Unweighted Features . 167 E.2 DNN with Weighted Features . 170 E.3 DNN with Weighted Discrete Features . 173 E.4 DNN with Feature Weights Only . 176 E.5 Combined Linear-DNN with Unweighted Features . 179 E.6 Combined Linear-DNN with Weighted Features . 182 E.7 Combined Linear-DNN with Weighted Discrete Features . 185 E.8 Combined Linear-DNN with Feature Weights Only . 188 F Publication History 191 x List of Tables 2.1 Example voxeme properties . 10 2.2 VoxML OBJECT attributes . 13 2.3 VoxML OBJECT HEAD types . 13 2.4 VoxML PROGRAM attributes . 16 2.5 VoxML PROGRAM HEAD types . 16 2.6 Example VoxML ATTRIBUTE scalar types . 18 3.1 Test set of verbal programs and objects . 39 3.2 Program test set with underspecified parameters . 48 3.3 Number of videos captured per motion predicate . 56 4.1 Acceptability judgments and statistical metrics for “move x” visualizations, condi- tioned on respecification predicate . 66 4.2 Acceptability judgments and statistical metrics for “turn x” visualizations, condi- tioned on respecification predicate . 67 4.3 Acceptability judgments and statistical metrics for unrespecified “turn x” visual- izations, conditioned on rotation angle . 68 4.4 Acceptability judgments and statistical metrics for “roll x” visualizations, condi- tioned on path length . 69 4.5 Acceptability judgments and statistical metrics for “slide x” visualizations, condi- tioned on translocation speed . 69 4.6 Acceptability judgments and statistical metrics for “spin x” visualizations respeci- fied as “roll x,” conditioned on path length . 70 4.7 Acceptability judgments for unrespecified “spin x” visualizations, conditioned on rotation axis . 70 4.8 Acceptability judgments and statistical metrics for “lift x” visualizations, condi- tioned on translocation speed and distance traversed . 71 4.9 Acceptability judgments and statistical metrics for “put x touching y” visualiza- tions, conditioned on relations between x and y at event start and completion . 72 4.10 Acceptability judgments and statistical metrics for “put x touching y” visualiza- tions, conditioned on x movement relative to y .................... 73 xi LIST OF TABLES 4.11 Acceptability judgments and statistical metrics for “put x near y” visualizations, conditioned on distance between x and y at event start and completion . 73 4.12 Acceptability judgments and statistical metrics for “put x near y” visualizations, conditioned on start and end distance intervals between x and y ........... 74 4.13 Acceptability judgments and statistical metrics for “put x near y” visualizations, conditioned on distance between x and y and POV-relative orientation at event completion . 75 4.14 Acceptability judgments and statistical metrics for “lean x” visualizations, condi- tioned on rotation angle .