Improving Command Selection In Smart Environments By Exploiting Spatial Constancy A Thesis Submitted to the College of Graduate Studies and Research In Partial Fulfillment of the Requirements For the Degree of Doctor of Philosophy In the Department of Computer Science University of Saskatchewan Saskatoon By Adrian Reetz Copyright Adrian Reetz, November 2015. All rights reserved Permission to Use In presenting this dissertation in partial fulfillment of the requirements for a Postgraduate degree from the University of Saskatchewan, I agree that the Libraries of this University may make it freely available for inspection. I further agree that permission for copying of this dissertation in any manner, in whole or in part, for scholarly purposes may be granted by the professor or professors who supervised my thesis/dissertation work or, in their absence, by the Head of the Department or the Dean of the College in which my thesis work was done. It is understood that any copying or publication or use of this dissertation or parts thereof for financial gain shall not be allowed without my written permission. It is also understood that due recognition shall be given to me and to the University of Saskatchewan in any scholarly use which may be made of any material in my thesis/dissertation. Disclaimer Reference in this thesis/dissertation to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not constitute or imply its endorsement, recommendation, or favoring by the University of Saskatchewan. The views and opinions of the author expressed herein do not state or reflect those of the University of Saskatchewan, and shall not be used for advertising or product endorsement purposes. Requests for permission to copy or to make other uses of materials in this thesis/dissertation in whole or part should be addressed to: Head of the Department of Computer Science University of Saskatchewan 110 Science Place Saskatoon, Saskatchewan S7N 5C9 Canada or Dean College of Graduate Studies and Research University of Saskatchewan 107 Administration Place Saskatoon, Saskatchewan S7N 5A2 Canada i Abstract With the a steadily increasing number of digital devices, our environments are becoming increasingly smarter: we can now use our tablets to control our TV, access our recipe database while cooking, and remotely turn lights on and off. Currently, this Human-Environment Interaction (HEI) is limited to in-place interfaces, where people have to walk up to a mounted set of switches and buttons, and navigation-based interaction, where people have to navigate on- screen menus, for example on a smart-phone, tablet, or TV screen. Unfortunately, there are numerous scenarios in which neither of these two interaction paradigms provide fast and convenient access to digital artifacts and system commands. People, for example, might not want to touch an interaction device because their hands are dirty from cooking: they want device-free interaction. Or people might not want to have to look at a screen because it would interrupt their current task: they want system-feedback-free interaction. Currently, there is no interaction paradigm for smart environments that allows people for these kinds of interactions. In my dissertation, I introduce Room-based Interaction to solve this problem of HEI. With room- based interaction, people associate digital artifacts and system commands with real-world objects in the environment and point toward these real-world proxy objects for selecting the associated digital artifact. The design of room-based interaction is informed by a theoretical analysis of navigation- and pointing-based selection techniques, where I investigated the cognitive systems involved in executing a selection. An evaluation of room-based interaction in three user studies and a comparison with existing HEI techniques revealed that room-based interaction solves many shortcomings of existing HEI techniques: the use of real-world proxy objects makes it easy for people to learn the interaction technique and to perform accurate pointing gestures, and it allows for system-feedback-free interaction; the use of the environment as flat input space makes selections fast; the use of mid-air full-arm pointing gestures allows for device-free interaction and increases awareness of other’s interactions with the environment. Overall, I present an alternative selection paradigm for smart environments that is superior to existing techniques in many common HEI-scenarios. This new paradigm can make HEI more user-friendly, broaden the use cases of smart environments, and increase their acceptance for the average user. ii Acknowledgements I would like to thank my supervisors Carl Gutwin for his unconditional support and for sacrificing so much of his time in order to guide me for the last seven years. I would also like to thank the members of my committee for their interest, guidance, and encouragement. Thank you to Ralph Deters, Ian Stavness, Scott Bell (cognate), and to my external examiner Derek Reilly. The work in this dissertation has benefited from many great collaborations. I would like to thank Regan Mandryk and Miguel Nacenta for providing me with fantastic guidance on several projects. I would also like to thank the members of the Interaction Lab for providing an amazing environment in which to work and learn. I could not have made it through the last seven years without the encouragement and support of my friends. Thank you to Zoe Zhou, Malcolm Lucy, Andre Doucette, and Nelson Wong. Finally, I want to thank my parents, Eva and Hartmut for their unconditional love and support throughout my entire life. Ich bedanke mich bei meinen Eltern Eva und Hartmut für Ihre bedingungslose Liebe un Unterstüzung. iii Dedication I dedicate my dissertation to my parents: my mother, Eva Reetz, and my dad, Hartmut Reetz, who is not with us anymore but will forever live on in my heart and thoughts. Ich widme meine Dissertation meinen Eltern: meiner Mutter Eva Reetz und meinem Vater Hartmut Reetz, der nicht mehr unter uns weilt aber für immer in meinem Herzen und meinen Gedanken weiterleben wird. iv Publications from this Dissertation Adrian Reetz and Carl Gutwin. 2014. Making big gestures: effects of gesture size on observability and identification for co-located group awareness. In Proceedings of the 32nd conference on Human Factors in Computing Systems – CHI ‘14. ACM Press, New York, NY, USA, 4087–4096. http://doi.org/10.1145/2556288.2557219 v Permissions Figure 3-1: Reprinted from Ishii and Ullmer, 1997, with permission from ACM. Figure 3-2: Reprinted from reactable.com, with kind permission from Reactable Systems, SL. Figure 4-1: Reprinted from Baudel and Beaudouin-Lafon, 1993, with permission from ACM. Figure 4-2: Reprinted from Gustafson et al., 2010, with permission from ACM. Figure 5-1: Reprinted from Wilson and Shafer, 2003, with permission from ACM. Figure 5-2: Reprinted from Wilson and Pham, 2003, with kind permission from IOS Press. Figure 6-1: Reprinted from Li et al., 2009, with permission from ACM. Figure 6-2: Reprinted from Cockburn et al., 2011, with kind permission from Elsevier. Figure 10-1: Reprinted from Hinckley et al., 1994, with permission from ACM. Figure 10-2: Reprinted from Robertson et al., 1998, with permission from ACM. vi Table of Contents Chapter 1 Introduction ................................................................................................................ 1 1.1 The Problem with Existing Human-Environment Interaction Techniques............. 2 1.1.1 Performance ........................................................................................................ 2 1.1.2 No Device-free Interaction .................................................................................. 3 1.1.3 Visibility of Interaction ....................................................................................... 4 1.2 Room-based Interaction as Solution ....................................................................... 4 1.3 Scoping of this Dissertation .................................................................................... 6 1.4 Outline of this Dissertation ..................................................................................... 7 1.5 Contributions of my Dissertation ............................................................................ 9 Chapter 2 Related Work ............................................................................................................ 10 2.1 Ubiquitous Computing and Smart Domestic Environments ................................. 10 2.1.1 Human-Computer Interaction in Smart Domestic Environments ..................... 11 2.1.2 Primary Tasks and Supporting Tasks ................................................................ 12 2.2 Pointing-based Selection Mechanisms, Awareness, and Selection Proxies ......... 12 2.2.1 Anatomy of a Selection Technique ................................................................... 13 2.2.2 A Brief History of Pointing Devices and Graphical User Interfaces ................ 13 2.2.3 Manipulation-based Full-arm Pointing Techniques .......................................... 18 2.2.4 Awareness of People and Their Actions ........................................................... 27 2.2.5 Static Real-world Proxy-based Selection Techniques ...................................... 31 2.3 Pointing Gestures in Human–Human Communication......................................... 36 2.3.1 Types, Functions,