UNIVERSITY OF CALIFORNIA SANTA CRUZ PLAYING WITH WORDS: FROM INTUITION TO EVALUATION OF GAME DIALOGUE INTERFACES A dissertation submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in COMPUTER SCIENCE Serdar Sali December 2012 The Dissertation of Serdar Sali is approved: _______________________________ Professor Michael Mateas, Chair _______________________________ Associate Professor Noah Wardrip-Fruin _______________________________ Associate Professor Sri Kurniawan _______________________________ Professor Marilyn Walker _____________________________ Tyrus Miller Vice Provost and Dean of Graduate Studies Copyright © by Serdar Sali 2012 TABLE OF CONTENTS CHAPTER 1. INTRODUCTION ........................................................................................ 1 OBJECTIVES ........................................................................................................................... 3 CONTRIBUTIONS ................................................................................................................... 4 ORGANIZATION .................................................................................................................... 5 CHAPTER 2. RELATED WORK ......................................................................................... 7 DIALOGUE IN TASK-BASED SYSTEMS ................................................................................... 7 DIALOGUE SYSTEMS FOR VIRTUAL AND BELIEVABLE AGENTS ........................................... 9 DIALOGUE SYSTEMS IN GAMES AND INTERACTIVE STORIES ............................................ 11 EVALUATING DIALOGUE SYSTEMS .................................................................................... 27 Evaluation methods for task-based systems: The PARADISE framework ..................... 28 Evaluating believable agents ........................................................................................... 30 Evaluating dialogue systems in games ............................................................................ 30 CHAPTER 3. A PRIMER ON FAÇADE .......................................................................... 38 FAÇADE ............................................................................................................................... 38 FAÇADE’S DIALOGUE SYSTEM ............................................................................................ 41 PAST USER STUDIES ON FAÇADE ........................................................................................ 42 NLU Accuracy ................................................................................................................ 42 Coping with failures ........................................................................................................ 46 Effects of mediation on presence and engagement .......................................................... 48 iii IMPLEMENTING DIFFERENT DIALOGUE SYSTEMS ON FAÇADE ......................................... 50 General Implementation Details ..................................................................................... 50 Sentence-selection ............................................................................................................ 51 Abstract-response ............................................................................................................ 52 Reactive-pause ................................................................................................................. 53 Prompt-pause .................................................................................................................. 54 STRUCTURES AND SUPPORT FOR METRIC COLLECTION AND AI LOGS ............................ 54 TERMINOLOGY OF STATISTICAL METHODS USED ............................................................ 55 Cronbach’s alpha ............................................................................................................. 55 Factor analysis ................................................................................................................. 55 Non-parametric statistical significance tests .................................................................. 56 DISCUSSION & CONCLUSION ............................................................................................. 56 CHAPTER 4. STUDYING EFFECTS OF DIFFERENT INTERFACES ...................... 58 STUDY GOALS ..................................................................................................................... 58 METHODOLOGY .................................................................................................................. 59 Stimulus material ............................................................................................................ 59 Recruitment ..................................................................................................................... 60 Measurement Instruments .............................................................................................. 60 RESULTS .............................................................................................................................. 63 Qualitative results ........................................................................................................... 63 Quantitative results ........................................................................................................ 79 DISCUSSION & CONCLUSION ............................................................................................. 87 iv CHAPTER 5. STUDYING EFFECTS OF PACING ....................................................... 89 DESIGN GOALS .................................................................................................................... 89 METHODOLOGY .................................................................................................................. 90 Stimulus material ............................................................................................................ 90 Recruitment ..................................................................................................................... 91 RESULTS .............................................................................................................................. 92 Engagement ..................................................................................................................... 92 Sense of control ................................................................................................................ 95 Difficulty of use ............................................................................................................... 97 Story Involvement ......................................................................................................... 100 Presence & immersion ................................................................................................... 101 Enjoyment ..................................................................................................................... 102 DISCUSSION & CONCLUSION ........................................................................................... 103 CHAPTER 6. USING INFORMATION VISUALIZATION TO UNDERSTAND INTERACTIVE NARRATIVE ........................................................................................ 107 RELATED WORK ............................................................................................................... 109 FAÇADE LOG ANALYSIS AND VISUALIZATION TOOL ..................................................... 112 Visualization techniques employed ............................................................................... 114 CASE STUDY I: FAÇADE .................................................................................................. 117 Discourse act coverage .................................................................................................. 118 Story space coverage ...................................................................................................... 121 Average Time spent in different beats ........................................................................... 122 v Story timelines ............................................................................................................... 123 CASE STUDY II: PROM WEEK ........................................................................................... 126 Strategy Driven Play .................................................................................................... 133 Story Goal Completion .................................................................................................. 134 DISCUSSION & CONCLUSION ........................................................................................... 136 CHAPTER 7. EXPLORING QUANTITATIVE METRICS OF PLAYER SATISFACTION IN DIALOGUE SYSTEMS .............................................................. 138 STUDY DESIGN .................................................................................................................. 139 Measurement instrument .............................................................................................. 139 Collected metrics ........................................................................................................... 139 RESULTS ...........................................................................................................................