Haptic Issues for Virtual Manipulation A Dissertation Presented to the Faculty of the School of Engineering and Applied Science at the University of Virginia In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy (Computer Science) by Ken Hinckley December 1996 © Copyright by Ken Hinckley All Rights Reserved December 1996 APPROVAL SHEET This dissertation is submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science) Ken Hinckley This dissertation has been read and approved by the Examining Committee: Randy Pausch (Dissertation Advisor) Worthy Martin (Committee Chair) Dennis Proffitt (Minor Representative) Andrew S. Grimshaw John C. Knight John W. Snell Accepted for the School of Engineering and Applied Science: Dean Richard W. Miksad School of Engineering and Applied Science December 1996 Dedication I dedicate this work to my father, my mother, and my brother; my grandparents Harold and Mildred, who passed away during the course of my graduate work; and to Kerrie, who has been my anchor for the past two years. Abstract The Windows-Icons-Menus-Pointer (WIMP) interface paradigm dominates modern computing systems. Yet these interaction techniques were originally developed for machines that are now 10, 15, or nearly 20 years old. Human-computer interaction currently faces the challenge of getting past this “WIMP plateau” and introducing new techniques which take advantage of the capabilities of today’s computing systems and which more effectively match human capabilities. Two-handed spatial interaction techniques form one possible candidate for the post-WIMP interface in application areas such as scientific visualization, computer aided design, and medical applications. The literature offers many examples of point design, offering only a description of the thing (what the artifact is) and not the process. But point design only provides a hit-or- miss coverage of the design space and does not tie the multiplicity of efforts into a common understanding of fundamental issues. To get past the WIMP plateau, we need to understand the nature of human-computer interaction as well as the underlying human capabilities. v vi My research contributes a working system which has undergone extensive informal usability testing in the context of real domain experts doing real work, and it also presents the results of experimental evaluations which illustrate human behavioral principles. Together, these approaches make a decisive statement that using both hands for virtual manipulation can result in improved user productivity. I contribute to the field by: (1) Showing that virtual manipulation needs to study the feel of the interface, and not just the graphical look of the interface; (2) Applying virtual manipulation technology to a volume visualization application which has been well received by neurosurgeons; (3) Demonstrating two-handed virtual manipulation techniques which take advantage of the highly developed motor skill of both hands; (4) Contributing basic knowledge about how two hands are used; (5) Showing that two hands are not just faster than one hand, but that two hands together provide information which one hand alone cannot, and can change how users think about a task; and finally (6) Providing an overall case study for an interdisciplinary approach to the design and evaluation of new human-computer interaction techniques. Acknowledgments My research has necessarily been an interdisciplinary effort, and as such, there are many persons to whom I am indebted. In particular, I owe great thanks to the Department of Neurological Surgery for supporting the work and participating in the design effort. I would like to thank the members of my committee, and in particular, Randy Pausch, Dennis Proffitt, and John Snell, who have all worked closely with me over the past few years. I also owe thanks to Joe Tullio for assistance with experiments, design, and implementation of testing software, and James Patten, for assistance with running experimental subjects. I would like to acknowledge external colleagues including Yves Guiard, currently at the Cambridge Applied Psychology Unit, Bill Buxton, of the University of Toronto and Alias | Wavefront, and Rob Jacob, at Tufts University, for enlightening discussions about some of the issues raised by my work. I have been assisted by the friendly disposition and skill of machinists Bob Frazier and Bob Bryant who helped to design and build the physical devices and apparatus which made the interface tools and the experimental equipment work. vii viii I would like to thank the present members, graduates, and associates of the Neurosurgical Visualization Lab, including Hunter Downs, Sean Graves, Adrian Filipi- Martin, Dave Schlesinger, Dave Moore, Michelle Plantec, Minyan Shi, Greg Harrington, Delia McGarry, Bill Katz, Ted Jackson, Will McClennan, Marc Pilipuf, Jim Bertolina, Dr. Neal Kassell, Dr. Dheerendra Prasad, and Dr. Mark Quigg. Each of these individuals participated in design discussions, assisted implementation efforts, participated in pilot experiments, or provided general support, assistance, and advice in the course of my dissertation work. I would like to thank the general support and many valuable discussions provided by members of the user interface group, including current members Steve Audia, Tommy Burnette, Matt Conway, Kevin Christiansen, Dennis Cosgrove, Beth Mallory, Kristen Monkaitis, Jeff Pierce, Joe Shochet, Brian Stearns, Chris Sturgill, John Viega, and George Williams; as well as the many graduates with whom I have worked, including Robert Allen, Jonathan Ashton, Brian Cadieux, AC Capehart, Roddy Collins, Thomas Crea, Rob DeLine, Jim Durbin, Rich Gossweiler, Simeon Fitch, Drew Kessler, Shuichi Koga, Jan Hoenisch, Chris Long, Steve Miale, Kimberly Passarella Jones, Anne Shackelford, Richard Stoakley, Jeff White, and Scott Witherell. I also owe thanks to colleagues in the Biomedical Engineering, Radiology, and other departments who helped along the way, including Jim Brookeman, Sam Dwyer, Linda Young, and Doug Kraus. I would like to thank members of Dennis Proffit’s Perception Lab who have assisted my efforts, shared valuable experimental space with me, and offered advice on working with subjects, including Sarah Creem, Steve Jacquot, Jane Joseph, MJ Wraga, and Tyrone Yang. Finally, I thank the many students who participated in my experiments. Contents List of Figures xiv 1 Introduction 1 1.1 Problem motivation. 2 1.2 Virtual manipulation . 5 1.3 Passive haptic issues . 6 1.4 Humans have two hands . 8 1.5 Thesis statement . 9 1.6 Contributions and overview . 10 1.6.1 Interdisciplinary approach . 10 1.6.2 Revisiting haptic issues . 10 1.6.3 Application to neurosurgical visualization . 11 1.6.4 Two-handed virtual manipulation techniques . 11 1.6.5 Basic knowledge about two hands. 12 1.6.6 Two hands are not just faster than one hand . 12 1.7 Organization . 12 2 Related Work 14 2.1 Introduction. 14 2.2 Two-dimensional approaches for 3D manipulation . 14 2.2.1 3D Widgets . 15 2.2.2 SKETCH . 16 2.3 Using the hand itself for input . 17 2.3.1 Gloves and Gesture for virtual manipulation. 17 2.3.1.1 The Virtual Wind Tunnel. 18 2.3.2 Electric field sensing . 19 2.3.3 VIDEODESK and VIDEOPLACE . 19 ix Contents x 2.3.4 Hands and voice: multimodal input . 20 2.3.4.1 Hauptmann’s behavioral study . 20 2.3.4.2 Put-That-There. 21 2.3.4.3 Two hands and voice . 22 2.4 One-handed spatial interaction techniques . 22 2.4.1 Schmandt’s stereoscopic workspace . 22 2.4.2 Ware’s investigations of the “bat” . 23 2.4.3 High Resolution Virtual Reality . 24 2.4.4 JDCAD. 24 2.4.5 Butterworth’s 3DM (Three-Dimensional Modeler). 26 2.5 Two-handed spatial interaction techniques. 26 2.5.1 3Draw. 26 2.5.2 Interactive Worlds-in-Miniature . 27 2.5.3 The Virtual Workbench . 28 2.5.4 The Reactive Workbench and ImmersaDesk. 29 2.5.5 Other notable systems . 30 2.6 Theory and experiments for two hands. 31 2.6.1 Guiard’s Kinematic Chain Model . 31 2.6.2 Formal experiments . 35 2.6.2.1 Buxton and Myers experiments . 35 2.6.2.2 Kabbash experiments. 36 2.6.2.3 Leganchuk’s area selection experiment. 38 2.7 Summary. 39 3 System Description 40 3.1 Overview. 40 3.2 The application domain: neurosurgery and neurosurgeons . 41 3.2.1 Traditional practice . 41 3.2.2 Computer-assisted surgery. 43 3.2.3 Some system requirements . 46 3.3 System design philosophy . 47 3.4 Real-time interaction. 49 3.5 Props for neurosurgical visualization . 50 3.5.1 Viewing patient data with a head prop . 50 3.5.2 Slicing the patient data with a cutting-plane prop . 52 3.5.3 Indicating surgical paths with a trajectory prop. 55 3.6 Two-handed interaction . 56 3.6.1 Two-handed input and the task hierarchy . 58 3.6.2 The natural central object. 60 3.7 Interactive volume cross-sectioning . 63 3.7.1 Frames-of-reference for the cross section display . 63 Contents xi 3.7.2 Texture mapping hardware . 65 3.7.3 Disappearing object problem . 67 3.8 Clutching mechanisms . 67 3.9 Touchscreens for hybrid 2D and 3D input . 69 3.9.1 Previous techniques for hybrid input . 70 3.9.2 Description of the touchscreen interface . 71 3.9.3 Limitations and proposed enhancements. 76 3.10 Informal evaluation: notes on user acceptance . 78 3.10.1 User observations. 78 4 Design Issues in Spatial Input 81 4.1 Introduction. 81 4.2 Understanding 3D space vs. experiencing 3D space . 82 4.3 Spatial references . 83 4.4 Relative gesture vs. absolute gesture . 84 4.5 Two-handed interaction . 85 4.5.1 Working volume of the user’s hands . 86 4.6 Multisensory feedback . 86 4.7 Physical constraints and affordances .