FLUID INTERACTION FOR HIGH RESOLUTION WALL-SIZE DISPLAYS A DISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY François Guimbretière January 2002 2002 by François Guimbretière All Rights Reserved ii I certify that I have read this dissertation and that in my opinion it is fully ade- quate, in scope and quality, as a dissertation for the degree of Doctor of Philoso- phy. __________________________________ Terry Winograd (Principal Advisor) I certify that I have read this dissertation and that in my opinion it is fully ade- quate, in scope and quality, as a dissertation for the degree of Doctor of Philoso- phy. __________________________________ Pat Hanrahan I certify that I have read this dissertation and that in my opinion it is fully ade- quate, in scope and quality, as a dissertation for the degree of Doctor of Philoso- phy. __________________________________ David Kelley I certify that I have read this dissertation and that in my opinion it is fully ade- quate, in scope and quality, as a dissertation for the degree of Doctor of Philoso- phy. __________________________________ Maureen Stone (StoneSoup Consulting) Approved for the University Committee on Graduate Studies: __________________________________ iii Abstract As computers become more ubiquitous, direct interaction with wall-size, high-resolution dis- plays will become commonplace. The familiar desktop computer interface is ill-suited to the affor- dances of these screens, such as size, and capacity for using pen or finger as primary input device. Current Graphical User Interfaces (GUIs) do not take into account the cost of reaching for a far- away menu bar, for example, and they rely heavily on the keyboard for rapid interactions. GUIs are extremely powerful, but their interaction style contrasts sharply with the casual interaction style available with traditional wall-size displays such as whiteboards and bulletin boards. This thesis explores how to bridge the gap between the power provided by current desktop com- puter interfaces and the fluid use of whiteboards and pin-boards. Based on our observations of fluid expert interactions from everyday life, such as driving a car or playing a violin, we have designed and built a fluid interaction framework which encourages gesture memory, reduces the need for dialog with the user, and provides a scoping mechanism for modes. Together, these fea- tures progressively make the cognitive load of using the interface disappear. The user becomes free to focus on other tasks, the same way one can drive a car while conversing with a passenger. To validate our design, we built the Stanford Interactive Mural, a 9 Mpixel whiteboard-size screen, evaluated the performance of our proposed menu system FlowMenu, and implemented two applications using our framework. The Geometer's Workbench allows one to explore differential geometry; PostBrainstorm is a brainstorm tool that lets users gather and organize sketches, snap- shots of physical documents, and a variety of digital documents on the Interactive Mural. Post- Brainstorm was tested in brainstorming sessions by professional designers. It demonstrates the feasibility of fluid, transparent interactions for complex, real life applications. iv Acknowledgments I would like to thank the many people who helped me as I did the work described in this disser- tation. This work would not have been possible without their support. It was a real privilege to work with my advisor, Terry Winograd. We had many long and insight- ful discussions exploring how human computer interactions can be improved. Even though we sometimes disagreed on the specifics, I believe we share the same vision in which humans and computers will be engaged in a synergetic relationship, each leveraging the strength of the other while minimizing the other’s weaknesses. Terry was always ready to discuss new ideas and was always very supportive during the four or so years of our collaboration. I next thank the other members of my committee for their support. Pat Hanrahan brought the vision of ubiquitous, large, high-resolution displays to my attention after his visit to AT&T Research Labs. All through this project, he was an invaluable support for bringing this vision closer to reality. Maureen Stone and I spent endless hours together—first designing, then assem- bling, then calibrating the version of the Stanford Interactive Mural presented in Chapter 4. Her insight and support during the interactions design phase of what was to become PostBrainstorm were invaluable. David Kelley introduced me to the world of product design, IDEO style. This work would not have been possible without his support. With his help, I was able to observe many brainstorming sessions inside IDEO, and I was able to draw from a population of skilled brain- stormers to run more realistic user studies. Many designers contributed to this project and I would like to thank them here. I would like to thank the numerous designers from IDEO who helped me. It was a great pleasure to interact with them and gather their feedback at different stages of the project. Among them, Chris Kurjan was v always very supportive and went out of her way to help me find my way inside IDEO. David Law, principal at Speck Product Design, broadened my perspective on how brainstorming could be con- ducted and was adventurous enough to trust one of his design meetings to our system. Phil Weaver and his team from Electronics for Imaging provided us with an early version of the Ebeam system and modified it to better fit our needs. Thanks to Greg Wolff from Ricoh Innovations for giving us an early version of Ricoh’s RDC-7 and its control software. Finally the Mural may never have been built without the skills of David Mallard and Adrian James, who built many custom parts for us. Many fellow graduate students made working in the Graphics Lab a great experience. Among them Tamara Munzner was probably the most influential. Tamara introduced me to the field of information visualization, and our endless nights discussing what a good visualization is all about were key to establishing my research agenda. Her comments on many drafts of my papers and this dissertation were always insightful, helped me improve greatly the quality of my work, and encouraged me to go further. I owe her most of what I know about making an interesting video. Ian Buck, Matthew Eldridge, and Greg Humphreys, all members of the Distributed Rendering Group at Stanford, provided invaluable support and were very patient with my bug reports. I hope that Brad Johanson (and the iRoom team) will forgive me for coming into his office almost every day to try to convince him to implement a feature I needed. James Davis was always there in time of need, and I believe he is the only one to truly understand the video rack. I thank my officemates for accommodating my very special desk management strategy, and members of offices 360 and 376 in the Gates building for letting me invade their offices and babble away about the latest idea I had at the time. I shall not forget John Gerth, whose technical expertise saved my day many times, or the com- puter science administrative staff, who made it a snap to order equipment, get reimbursed, and ful- fill administrative requirements. Finally, I owe a lot to my family and my friends, including Corinna Löckenhoff, Elena Krasnop- erova, Tamara Munzner, Maureen Stone, James Vera and his wife Jennifer Ochs, and Walt Mann and his wife Becky Jennings who edited this document. Their unconditional support was needed all along the way. Special thanks to Nicolas Scapel who drew the sketches reproduced in Figures 1 and 2. vi A mon père vii Table of Contents Abstract iv Acknowledgments v 1 Introduction 1 1.1 Contributions ................................................................................................................... 4 1.2 Dissertation organization ................................................................................................ 5 2 Previous Work 7 2.1 Large wall displays .........................................................................................................7 2.1.1 Visualization-oriented project ............................................................................ 7 2.1.2 Digital whiteboard .............................................................................................. 8 2.2 Beyond mouse and keyboard .......................................................................................... 8 2.2.1 Pen-based command mechanism ....................................................................... 9 2.2.2 Parameter manipulation ..................................................................................... 9 2.3 Applications .................................................................................................................. 10 2.3.1 Differential geometry ....................................................................................... 10 2.3.2 Brainstorming ................................................................................................... 11 2.3.2.1 Strokes-only systems .......................................................................... 11 2.3.2.2 Mixing digital and non-digital content ............................................... 11 2.3.2.3 Space management .............................................................................