ANNEE´ 2016 THESE` / UNIVERSITE´ DE RENNES 1 sous le sceau de l’Universit´e Bretagne Loire pour le grade de DOCTEUR DE L’UNIVERSITE´ DE RENNES 1 Mention : Informatique Ecole doctorale Matisse pr´esent´ee par Hui-Yin Wu pr´epar´ee a` l’unit´e de recherche UMR 6074 IRISA et au centre INRIA - Rennes Bretagne Atlantique ISTIC These` soutenue a` Rennes le 7 octobre 2016 devant le jury compos´e de : Cinematic Discourse for Ulrike SPIERLING Professeur, Hochschule RheinMain / rapporteur Michael YOUNG Interactive 3D Storytelling Professeur, University of Utah / rapporteur Tim SMITH Senior lecturer, Birbeck University of London / examinateur Bruno ARNALDI Professeur, INSA / examinateur Stephane´ DONIKIAN Directeur de recherche, INRIA / directeur de th`ese Marc CHRISTIE Maˆıtre de conf´erence, Universit´e de Rennes 1 / co-directeur de th`ese Acknowledgements Many thanks to my thesis supervisors, Stéphane Donikian and Marc Christie for offering me this thesis on developing and understanding technologies for interactive storytelling, topics that are very dear to me. I would like to thank Marc in particular, who apart from offering advice and guidance, never hesitated to send me anywhere in the world, if only to capture new ideas, present our work, or ignite possible future collaborations. During the thesis, I was lucky to continue working with Prof. Tsai-Yen Li from IMLab (National Chengchi University) on many aspects following my master’s work. I am also deeply grateful to Prof. Michael Young for hosting me in the Liquid Narrative Group and our continued collaboration and publication of a paper. It was also a great pleasure to work alongside colleagues in the MimeTIC and Hybrid teams in INRIA, members of the Liquid Narrative Group, and members of IMLab, who have been nothing but enthusiastic and encouraging in work and happy to share ideas about languages, cultures, and various lifestyles. Finally, this thesis is for my family, for Mom, Dad and Yoyo, for Ola and Tata, and for David who was there to support me, and there as we set out on our next journey. I love you all. iii Contents List of Figures ................................... ix List of Tables .................................... xi Resumé en français ............................... 1 1 Présentation .................................. 1 2 Problématique ................................ 3 3 Contributions ................................. 5 1 Introduction ................................... 7 1 Overview ................................... 7 2 Problems ................................... 9 3 Contributions ................................. 10 2 State of the Art ................................. 13 1 Film Theory and Cognition .......................... 14 1.1 Visual Storytelling .......................... 14 1.2 Framing and Cognition ........................ 15 1.3 Editing ................................. 16 1.4 Summary ............................... 18 2 Interactive Storytelling ............................ 19 2.1 Interactive Storytelling Systems ................... 19 2.2 Authorial Control for Logic ...................... 23 2.3 Temporal Structures ......................... 24 2.4 Summary ............................... 24 3 Film Analysis ................................. 25 3.1 Film Annotation ............................ 26 3.2 Automated Analysis ......................... 27 3.3 Summary ............................... 27 4 Virtual Cinematographic Storytelling .................... 28 4.1 Camera Positioning and Movement ................. 29 4.2 Editing ................................. 32 4.3 AI Cinematography Systems .................... 36 4.4 Summary ............................... 39 3 Temporal Properties of Interactive Stories ............. 41 1 Controlling Logic in Temporally Rearranged Stories ............ 42 v CONTENTS 1.1 Background and Contributions ................... 42 1.2 Overview ............................... 44 1.3 Story Representation ......................... 44 1.4 Logic Control ............................. 46 1.5 The Theater .............................. 50 1.6 Evaluation ............................... 52 1.7 Summary and Future Work ..................... 53 2 Evaluating Viewer Perceptions of Temporally Rearranged Storylines .. 54 2.1 Background .............................. 55 2.2 Overview ............................... 56 2.3 Story Representation ......................... 56 2.4 Model for Evaluating Flashbacks .................. 59 2.5 Evaluation ............................... 63 2.6 Summary and Future Work ..................... 66 3 Chapter Conclusion ............................. 67 4 Patterns: Analysis of Cinematographic Storytelling ...... 69 1 Background .................................. 70 2 Visual Features of Film ............................ 72 2.1 Framing Constraints ......................... 72 2.2 Shot Relation ............................. 74 3 ECP Definition ................................ 76 3.1 Sub-Sequences ............................ 77 3.2 Ranges ................................ 77 4 Insight Annotation Tool ............................ 79 5 Solver ..................................... 81 5.1 Verifying constraints ......................... 81 5.2 Search Algorithm ........................... 82 6 Potential for Film Analysis .......................... 83 6.1 Extensive Case Study: The Lord of the Rings ........... 84 6.2 Selected Findings .......................... 86 7 Chapter Conclusion ............................. 89 5 Data-Driven Cameras for Interactive Storytelling ........ 91 1 ECP-Based Camera for Interactive Stories ................. 92 1.1 Overview ............................... 93 1.2 Virtual Director ............................ 94 1.3 Virtual Cinematographer ....................... 95 1.4 Demonstration ............................ 96 1.5 Summary and Future Work ..................... 98 2 Using ECPs for Smart Cinematography Editing .............. 98 2.1 Overview ............................... 99 2.2 Pattern Interface Solver ....................... 100 2.3 Interface Concept Design ...................... 103 2.4 Summary and Future Work ..................... 105 3 Chapter Conclusion ............................. 106 vi CONTENTS 6 Conclusion ................................... 107 1 Contributions ................................. 107 2 Future Work .................................. 108 2.1 User Evaluations ........................... 108 2.2 Virtual Cinematography ....................... 109 2.3 Assisted Creativity .......................... 110 3 Summary ................................... 111 Bibliography .................................... 112 Film Bibliography ................................. 123 Publications .................................... 124 vii List of Figures 1 Placer une caméra .............................. 1 2 Montage .................................... 2 3 Panocam ................................... 2 1.1 Placing a camera ............................... 7 1.2 Editing ..................................... 8 1.3 Panocam ................................... 8 2.1 The 180 degree rule ............................. 17 2.2 Example of matching between shots .................... 18 2.3 Plan vs. Graph ................................ 21 2.4 Scenejo editing interface for story graphs ................. 23 2.5 Insight annotation tool ............................ 26 2.6 Hidden Markov Models for cinematography style analysis ........ 27 2.7 Virtual camera control overview ....................... 28 2.8 Camera positioning and movement overview ............... 30 2.9 The Toric Space for camera placement ................... 30 2.10 The Cambot blockings system ........................ 31 2.11 Editing techniques overview ......................... 32 2.12 The DCCL language for virtual cameras .................. 34 2.13 Automated film editing ............................ 36 2.14 AI cinematography systems overview .................... 37 2.15 The Darshak discourse planner ....................... 38 3.1 Interactive storytelling system overview .................. 43 3.2 Embedding in a plotpoint story graph. ................... 46 3.3 Graph traversal algorithm for logic control ................. 47 3.4 Annotating a Theater scenario ....................... 51 3.5 Example generated plotlines ......................... 53 3.6 Model of events and beliefs in the viewer’s memory ............ 59 3.7 Model of the Establishing flashback ..................... 60 3.8 Model of the Changing flashback ...................... 61 3.9 Model of Reinforcing flashback ....................... 62 3.10 A computed variation of “The Interview” with flashbacks ......... 63 3.11 User Evaluation: Relation between the normalised memory retention for a scene and percentage of participants who chose a scene as the salient cause. ..................................... 65 ix LIST OF FIGURES 3.12 User Evaluation: Distribution of participant responses compared against four types of flashbacks calculated by our algorithm. ........... 66 4.1 Example of intensify sequence in two movies ............... 71 4.2 Shot sizes in Patterns ............................ 72 4.3 Shot angles in Patterns ........................... 73 4.4 Framing regions in Patterns ......................... 74 4.5 Screenshot of Insight annotation tool for Patterns ............. 79 4.6 Framing features extracted from the film database ............ 80 4.7 ECP solver: framing constraints ....................... 82 4.8 ECP solver: relation constraints ....................... 82 4.9 Example ECPs over Lord of