Direct Pen Input and Hand Occlusion
Total Page:16
File Type:pdf, Size:1020Kb
Direct Pen Input and Hand Occlusion by Daniel Vogel A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Department of Computer Science University of Toronto © Copyright 2010 Daniel Vogel Direct Pen Input and Hand Occlusion Daniel Vogel Doctor of Philosophy Department of Computer Science, University of Toronto 2010 Abstract We investigate, model, and design interaction techniques for hand occlusion with direct pen input. Our focus on occlusion follows from a qualitative and quantitative study of direct pen usability with a conventional graphical user interface (GUI). This study reveals overarching problems relating to poor precision, ergonomics, cognitive differences, limited input, and problems resulting from occlusion. To investigate occlusion more closely, we conduct three formal experiments to examine its area and shape, its affect on performance, and compensatory postures. We find that the shape of the occluded area varies across participants with some common characteristics. Our results provide evidence that occlusion affects target selection performance: especially for continuous tasks or when the goal is initially hidden. We observe how users contort their wrist posture during a simultaneous monitoring task, and show this can increase task time. Based on these investigations, we develop a five parameter geometric model to represent the shape of the occluded area and extend this to a user configurable, real-time version. To evaluate our model, we introduce a novel analytic testing methodology using optimization for geometric fitting and precision- recall statistics for comparison; as well as conducting a user study. To address problems with occlusion, we introduce the notion of occlusion-aware interfaces: techniques which can use ii our configurable model to track currently occluded regions and then counteract potential problems and/or utilize the occluded area. As a case study, we present the Occlusion-Aware Viewer: an interaction technique which displays otherwise missed previews and status messages in a non-occluded area. Within this thesis we also present a number of methodology contributions for quantitative and qualitative study design, multi-faceted study logging using synchronized video, qualitative analysis, image-based analysis, task visualization, optimization-based analytical testing, and user interface image processing. iii Acknowledgements I feel incredibly lucky to have Ravin Balakrishnan as an advisor, whose guidance and encouragement made the completion of this dissertation possible. I was also fortunate to have an outstanding and well-rounded committee: Ron Baecker, Khai Truong, Karan Singh, and my external examiner, Brad Myers. There are many students, faculty members, and administrative staff at the University of Toronto who contributed directly and indirectly. John Hancock deserves specific mention for technical assistance; and Géry Casiez, whom I met when he was a Postdoctoral Researcher at the University of Toronto, developed an initial version of the real-time occlusion model. There are several individuals at Mount Allison University who made it much easier for me to complete this work after I moved to Sackville, New Brunswick: Liam Keliher, Anna Sheridan-Jonah, Jeff Ollerhead, Laurie Ricker, and Ron Beattie in particular. Matthew Cudmore’s assistance with programming and facilitation of experiments, in addition to video analysis, was especially valuable. Of course, I owe a debt of gratitude to Jennifer, family, and friends who encouraged and supported me throughout. iv Copyright Notices and Disclaimers Sections of this document have appeared in publications or are forthcoming (at the time of writing). In all cases, permission has been granted by the publisher for these works to appear here. Below, the publisher’s copyright notice and/or disclaimer is given, with thesis chapter(s) and corresponding publication(s) noted. Taylor and Francis Copyright © 2010 Taylor & Francis. Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf. This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material. portions of chapters 2 and 3 Vogel, D., and Balakrishnan, R. (forthcoming). Direct Pen Interaction with a Conventional Graphical User Interface. Human-Computer Interaction. Taylor and Francis. Association for Computing Machinery Copyright © 2009, 2010 by the Association for Computing Machinery, Inc. (ACM). Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page in print or the first screen in digital media. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Send written requests for republication to ACM Publications, Copyright & Permissions at the address above or fax +1 (212) 869-0481 or email [email protected]. Copyright © 2009, 2010 ACM Inc. Included here by permission. v portions of chapters 4 and 5 Vogel, D., Cudmore, M., Casiez, G., Balakrishnan, R., and Keliher, L. (2009). Hand occlusion with tablet-sized direct pen input. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 557-566. portions of chapters 5 and 6 Vogel, D., and, Balakrishnan, R. Occlusion-Aware Interfaces. (2010). In Proceedings of the 28th international Conference on Human Factors in Computing Systems (Atlanta, GA, USA, April 10 - 15, 2010). CHI '10. ACM, New York, NY, 263-272. vi Table of Contents 1 Introduction ..............................................................................................................1 1.1 Research Objectives and Overview ................................................................5 1.2 Contributions...................................................................................................8 1.3 Dissertation Outline ......................................................................................12 2 Background Literature ...........................................................................................13 2.1 The Hand and the Pen ...................................................................................14 2.2 The Pen as a Computer Input Device ...........................................................27 2.3 Pen Input Performance and Capabilities .......................................................34 2.4 Pen Interaction Paradigms ............................................................................46 2.5 Summary .......................................................................................................50 3 Observational Study of Pen Input ..........................................................................51 3.1 Related Work ................................................................................................52 3.2 Study .............................................................................................................55 3.3 Analysis.........................................................................................................66 3.4 Results ...........................................................................................................75 3.5 Interactions of Interest ..................................................................................95 3.6 Discussion ...................................................................................................109 3.7 Summary .....................................................................................................116 4 Investigating Occlusion .......................................................................................119 4.1 Related Work ..............................................................................................120 4.2 Experiment 4-1: Area and Shape ................................................................126 4.3 Experiment 4-2: Performance .....................................................................144 4.4 Experiment 4-3: Influence on Hand and Arm Posture................................172 4.5 Design Implications ....................................................................................183 4.6 Summary .....................................................................................................184 5 Modelling Occlusion ............................................................................................187 5.1 Related Work ..............................................................................................189