UNDERSTANDING INTERACTION MECHANICS IN TOUCHLESS TARGET SELECTION Debaleena Chattopadhyay Submitted to the faculty of the University Graduate School in partial fulfillment of the requirements for the degree Doctor of Philosophy in the School of Informatics and Computing Indiana University September 2016 Accepted by the Graduate Faculty, Indiana University, in partial fulfillment of the requirements for the degree of Doctor of Philosophy. _________________________________________ Davide Bolchini, Ph.D., Chair _________________________________________ Karl F. MacDorman, Ph.D. Doctoral Committee July 28, 2016 _________________________________________ Stephen Voida, Ph.D. _________________________________________ Erik Stolterman, Ph.D. ii © 2016 Debaleena Chattopadhyay iii Dedication To my parents iv Acknowledgements This dissertation was made possible by the perseverance, guidance, and encouragement of my mentors. In what follows, I thank those who were crucial to my doctoral training. However, a doctorate is a terminal degree, culminating one’s education in a particular field of study. So I first take this opportunity to thank all my teachers who came before and equipped me to undertake this training effectively. I would like to thank my advisor, Davide, for his unwavering support toward my research career. He continues to invigorate my half-baked ideas—even when I am half- heartedly pursuing them. As goes the adage, recognizing a good idea is as important as having a good idea—if not more. He gave me the utmost freedom to pursue my research goals while investing countless hours in training me to become a skilled researcher. He believed in my vision when I did not have the skills to undertake my research and helped me in acquiring those skills. Enumerating Davide’s role in my doctoral training would befit a longer story and this acknowledgment, by no means, measures up to that. Nevertheless, Davide is a great teacher, both in and out of the classroom—and among those rare ones, who can inspire greatness. I doubt if a doctoral student could ask for an advisor with any more positivity. In 2012, I had earned an A+ in my graduate research design course; the course instructor’s feedback read: “Your significance section is inadequate. Significance means, if you are lucky, and your experiment goes as planned, and you collect the results anticipated and publish them, how does that change the world?” Earlier in that course, I had got my first B in a class assignment—an annotated bibliography. In research (or otherwise), Karl never settles for anything less than perfect; his obsession with high standards has significantly contributed to the groundwork of my doctoral training. Karl continues to be my touchstone for excellence in research. I would like to thank him for all the different hats he had put on during my Ph.D. life—a teacher, collaborator, mentor, dissertation committee member, and above all, a ruthless critic. Without his unnerving demand for excellence, I wouldn’t be the researcher that I am today. Thank you for everything, and a day. Sincere thanks go out to my committee members, Stephen Voida and Erik Stolterman, who advised this dissertation from proposal to defense—ensuring my plans are feasible and diligently curbing my underestimation of time and effort required for a successful completion. Steve’s course on ubiquitous computing—that I took in the Spring of 2014—was instrumental in bringing the different pieces of my research v together, under the umbrella of embodied interaction. I owe Steve another round of thanks for introducing me to Heidegger, and to Karl (rather, some of his old papers that he doesn’t cite any more) for introducing me to Gibson’s affordances. A doctorate may be a terminal degree, but it also resides at the lowest rung of academia or a career in research. Indeed, a doctorate is at best the license to conduct independent research. My dreams of furthering that coveted research career continue to be supported by mentors outside the school, and Kenton, with whom I spent the summer of 2015 at Microsoft Research Cambridge (MSRC), deserves special mention. I would like to thank him for giving me an eye for simplicity and a taste of how sociotechnical systems can (and should) dissolve into the fabric of a familiar social milieu. My internship at the Human Experience and Design (HxD) lab at MSRC will always be a cherished memory—professionally and personally. I am grateful for receiving research support during my Ph.D. from Indiana University-Purdue University Indianapolis (IUPUI) Graduate Office, IUPUI Office of the Vice Chancellor for Research, IUPUI School of Informatics and Computing, Xerox Foundation, Microsoft, Association for Computing Machinery (ACM), and National Science Foundation (NSF). I would like to thank Chauncey Frend, Jeff Rogers, and Michael Boyles of the Advanced Visualization Lab (AVL) for their generous support toward conducting my research studies. Many thanks go out to Elizabeth Bunge and Nancy Barker for their administrative assistance throughout my doctoral studies. Doctoral research (or any research) is a poignantly lonesome pursuit. I would like to thank my friends and family, who put up with my perennial absence—yet being there at times I needed them. Particularly, heartfelt thanks go out to Manisha Deogharia (for listening to my computing research problems amid her HeLa cell cultures) and Raquel Perales (for always being open to trading her doctorate in Mathematics with mine). Finally, my most sincere gratitude to my parents: During their college lives, my father wanted to inspire an auditorium full of students, and my mother wanted to live in a lab around pipettes and burners; I owe my love of knowledge to them. vi Debaleena Chattopadhyay UNDERSTANDING INTERACTION MECHANICS IN TOUCHLESS TARGET SELECTION We use gestures frequently in daily life—to interact with people, pets, or objects. But interacting with computers using mid-air gestures continues to challenge the design of touchless systems. Traditional approaches to touchless interaction focus on exploring gesture inputs and evaluating user interfaces. I shift the focus from gesture elicitation and interface evaluation to touchless interaction mechanics. I argue for a novel approach to generate design guidelines for touchless systems: to use fundamental interaction principles, instead of a reactive adaptation to the sensing technology. In five sets of experiments, I explore visual and pseudo-haptic feedback, motor intuitiveness, handedness, and perceptual Gestalt effects. Particularly, I study the interaction mechanics in touchless target selection. To that end, I introduce two novel interaction techniques: touchless circular menus that allow command selection using directional strokes and interface topographies that use pseudo-haptic feedback to guide steering–targeting tasks. Results illuminate different facets of touchless interaction mechanics. For example, motor-intuitive touchless interactions explain how our sensorimotor abilities inform touchless interface affordances: we often make a holistic oblique gesture instead of several orthogonal hand gestures while reaching toward a distant display. Following the Gestalt theory of visual perception, we found similarity between user interface (UI) components decreased user accuracy while good continuity made users faster. Other findings include hemispheric asymmetry affecting transfer of training between dominant and nondominant hands and pseudo-haptic feedback improving touchless accuracy. The results of this dissertation contribute design guidelines for future touchless systems. Practical applications of this work include the use of touchless interaction techniques in various domains, such as entertainment, consumer appliances, surgery, patient-centric health settings, smart cities, interactive visualization, and collaboration. Davide Bolchini, Ph.D., Chair vii Table of Contents Chapter 1. Introduction ................................................................................................ 1 Chapter 2. Background, scope, and significance ........................................................ 8 2.1. The use of touchless systems across different domains .............................. 8 2.1.1. Touchless interaction with large displays ........................................... 9 2.2. Interaction mechanics ................................................................................. 12 2.3. An embodied interaction perspective: The tool, or lack thereof .................. 14 2.4. Emerging problems .................................................................................... 18 2.5. Scope of the work ....................................................................................... 21 2.6. Significance of this research ....................................................................... 22 Chapter 3. Understanding touchless interaction mechanics ..................................... 23 3.1. Related work ............................................................................................... 23 3.2. Interaction mechanics ................................................................................ 24 3.2.1. Sensing ............................................................................................. 24 3.2.2. Input, feedback, and affordances ..................................................... 26 3.3. Target selection ............................................................................................ 27 Chapter 4. Visual