Facilitating Real-Time Sketch-Based Storyboards for Stereoscopic and Virtual Reality Environments by Rorik Henrikson A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Computer Science University of Toronto © Copyright by Rorik Henrikson 2017 Abstract Facilitating Real-Time Sketch-Based Storyboards for Stereoscopic and Virtual Reality Environments Rorik Henrikson Doctor of Philosophy, 2017 Department of Computer Science University of Toronto Recently, we have seen a resurgence in stereoscopic movies, and an explosion of interest with stories for virtual reality (VR). Though these art forms share many similarities with traditional film, there are numerous differences. Some differences result in aspects that are more challenging to discuss, and are therefore often ignored, frequently resulting in bad decisions. This latest resurgence with stereoscopic cinema has resulted in a better understanding of the artistic challenges and opportunities provided by this form of storytelling. There has been a realization that creating stereoscopic movies does not mean simply adding depth, but rather involves a complex set of considerations to lead the audience through a pleasant viewing experience; one that engages them as consumers of the visual story, rather than using stereopsis as a gimmick that disconnects the viewer from the show. To facilitate these differences, the ii stereoscopic community encourages directors and producers to think of projects in stereo as early as possible, to make good design decisions. However, there are few early stage design tools to help support this design process. Likewise, with the consumerization of VR through devices such as Oculus Rift, Samsung GearVR, and Google Cardboard, directors have the ability to connect with audiences in new ways, and have started creating stories for virtual reality. However, directors are unsure how to direct a VR movie, or plan for these experiences. Due to factors such as the stereoscopic nature, 360˚ surrounding view, and uncertainty of how to perform transitions, directors have difficulty planning for this environment using traditional means. This work explores different approaches, and feedback to allow artists to sketch on tablets, leveraging existing drawing skills, yet allowing for quick creation of stereoscopic and virtual reality storyboards in real-time. The presented approaches allow artists to create stereoscopic and VR storyboards with minimal pre-planning, and almost no increase to the exerted effort by the artist. These techniques when applied to VR, allow artists to work in more immersive environments, and collaborate more easily. These techniques can help directors and producers plan films more effectively and easily with these new, unexplored stereoscopic and virtual sketched worlds. iii Acknowledgments A body of work such as this happens over many months with much help and support from many people. I am very grateful to the many people that helped ensure that this thesis reached its conclusion. First, I need to thank my family for bearing with me through this whole process; the understanding and acceptance of my many absent hours – days and nights, distractions, and erratic schedules. A huge special thank you to my wife, Julia, for being patient while I toiled away on my projects and being an ear for me to air my thoughts. I strongly believe a supportive partner is critical to achieving a task such as this one, and Julia, you have been there to support me regardless of the challenges, frustrations and elations that comprise the rollercoaster that is involved while researching and writing a thesis. Thank you! My children, Rhys and Seren, thank you for being a continuous joy and reminder of how exciting and how much fun life can be…that there is still magic in the world! As well, my parents, Steven and Lilian – for your patience, encouragement, and beliefs through the years as I found my way on my educational journey. You provided an environment that has taught me to think and question, formulate and explore, yet remain creative through the whole process. Further, this project would not have been possible without the support and guidance from my advisor Prof. Ravin Balakrishnan, and Prof. Karan Singh (interim advisor) who provided an environment that allowed for me to pursue my area of interest, and direction to achieve results that have garnered much excitement by individuals and professionals within the film industry. As well, a special thank you to Prof. Daniel Wigdor for being both part of my committee and providing input and insights through brainstorming and challenging my assumptions. Prof. Khai iv Truong, you have also been invaluable through the many discussions and insights you were able to provide during our multiple discussions. Dr. Bruno De Araujo, for helping me with both modern C++ concepts and software engineering approaches. As well, along with Dr. Fanny Chevalier, helping me adjust my research approach and crafting my conference endeavors into the works that they eventually became. A special thank you to both of you for helping me figure out how to present my work and prepare it in a way that the community has found acceptable. Finally, the DGP lab: a large thank you to John Hancock who helped me with all things network and hardware related throughout my many years working at the lab. Your patience, understanding, and willingness to learn about new challenges helped make the process much easier to navigate. Dr. Richardo Jota, thank you for helping put me back on track and refocusing my energies when I was losing my way! Jonathan Deber, thank you for being an ear, sounding- board, un-biased observer and friend through this process. Dustin Freeman, Pif Edwards, Mike Glueck, Haijun Xia, Chris De Paoli, Jian Zhao, Peter Hamilton and all the members of the DGP lab with whom I have had the fortune to interact, discuss, brainstorm and banter. Your support, thoughts, and energy have helped me immensely through the years in achieving my goals. Thank you all! v Association for Computing Machinery Copyright Notice Several sections of this document have previously appeared in ACM publications. Permission from ACM has been granted for these works to appear in this thesis. Where appropriate, each chapter is marked with references to the relevant publication, each of which is listed below: Chapter 4 & 5 Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Storeoboard: Sketching Stereoscopic Storyboards. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4587-4598. DOI: https://dx.doi.org/10.1145/2858036.2858079 Chapter 6 & 7 Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Multi- Device Storyboards for Cinematic Narratives in VR. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST'16). ACM, New York, NY, USA. 787-796. DOI: https://doi.org/10.1145/2984511.2984539 ACM COPYRIGHT NOTICE. Copyright © 2005, 2006, 2007 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or [email protected]. © 2005, 2006, 2007 ACM, Inc. Included here by permission. vi Table of Contents Abstract ........................................................................................................................................... ii Acknowledgments.......................................................................................................................... iv Association for Computing Machinery Copyright Notice ............................................................. vi Table of Contents .......................................................................................................................... vii List of Figures .............................................................................................................................. xiii List of Appendices ...................................................................................................................... xvii FUNDAMENTALS Chapter 1. Introduction ....................................................................................................................2 1.1 Understanding the problem ..................................................................................................4 1.1.1 Stereoscopic Film.....................................................................................................4 1.1.2 Virtual Reality ..........................................................................................................9 1.2 Thesis Statement ................................................................................................................12 1.3 Dissertation Outline and Contributions .............................................................................13 Chapter 2. Overview of