MULTI-TOUCH FOR GENERAL-PURPOSE COMPUTING: AN EXAMINATION OF TEXT ENTRY by PAUL DAVID VARCHOLIK B.S. Valdosta State University, 1998 M.S. University of Central Florida, 2008 A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Modeling & Simulation in the College of Engineering & Computer Science at the University of Central Florida Orlando, Florida Spring Term 2011 Major Professors: Charles E. Hughes Joseph J. LaViola © 2011 Paul David Varcholik ii ABSTRACT In recent years, multi-touch has been heralded as a revolution in human- computer interaction. Multi-touch provides features such as gestural interaction, tangible interfaces, pen-based computing, and interface customization – features embraced by an increasingly tech-savvy public. However, multi-touch platforms have not been adopted as “everyday” computer interaction devices; that is, multi-touch has not been applied to general-purpose computing. The questions this thesis seeks to address are: Will the general public adopt these systems as their chief interaction paradigm? Can multi-touch provide such a compelling platform that it displaces the desktop mouse and keyboard? Is multi-touch truly the next revolution in human-computer interaction? As a first step toward answering these questions, we observe that general- purpose computing relies on text input, and ask: “Can multi-touch, without a text entry peripheral, provide a platform for efficient text entry? And, by extension, is such a platform viable for general-purpose computing?” We investigate these questions through four user studies that collected objective and subjective data for text entry and word processing tasks. The first of these studies establishes a benchmark for text entry performance on a multi-touch platform, across a variety of input modes. The second study attempts to improve this performance by iii examining an alternate input technique. The third and fourth studies include mouse- style interaction for formatting rich-text on a multi-touch platform, in the context of a word processing task. These studies establish a foundation for future efforts in general-purpose computing on a multi-touch platform. Furthermore, this work details deficiencies in tactile feedback with modern multi-touch platforms, and describes an exploration of audible feedback. Finally, the thesis conveys a vision for a general-purpose multi-touch platform, its design and rationale. iv ACKNOWLEDGMENTS I wish to acknowledge the assistance of a variety of groups and individuals, without whose support, this work would not have been possible. I am sincerely grateful to my family, friends, and co-workers at the University of Central Florida (UCF), the Institute for Simulation & Training (IST), the Florida Interactive Entertainment Academy (FIEA), the Media Convergence Lab (MCL), the Interactive Systems and User Experiences Lab (ISUE) and the Applied Cognitive Training & Immersive Virtual Environments Lab (ACTIVE). I also wish to thank the esteemed members of my dissertation committee: Charles Hughes, Joseph LaViola, Stacey Scott, Brian Goldiez, and Michael Moshell. These efforts have been funded, in part, by: IARPA, under US Air Force Research Lab award FA87500820202; SAIC, under Science Applications International Corporation award 4400157271; and by the National Science Foundation under award number DRL0638977 v TABLE OF CONTENTS LIST OF FIGURES ....................................................................................................... xvii LIST OF TABLES ...........................................................................................................xx LIST OF ACRONYNMS/ABBREVIATIONS ................................................................. xxiii CHAPTER ONE: INTRODUCTION ................................................................................. 1 1.1 Motivation ............................................................................................................ 1 1.2 Multi-Touch .......................................................................................................... 2 1.3 General-Purpose Computing ............................................................................... 3 1.4 Research Question .............................................................................................. 5 1.5 Research ............................................................................................................. 8 1.5.1 Examination of Existing Text Entry Methods ................................................... 8 1.5.2 Empirical Evaluations of Text Entry and Word Processing Methods ............... 9 1.5.2.1 Text Entry Study ....................................................................................... 9 1.5.2.2 Text Entry Improvement Study ............................................................... 11 1.5.2.3 Word Processing Technique Study ........................................................ 12 1.5.2.4 Word Processing Platform Study ............................................................ 14 1.5.3 Additional Investigations ................................................................................ 15 vi 1.5.3.1 General-Purpose Multi-Touch Platform (GPMT) ..................................... 15 1.5.3.2 Tactile Feedback .................................................................................... 16 1.5.3.3 Audible Feedback ................................................................................... 16 1.6 Contributions ...................................................................................................... 17 1.7 Summary ........................................................................................................... 18 CHAPTER TWO: BACKGROUND ON MULTI-TOUCH COMPUTING ......................... 21 2.1 Introduction ........................................................................................................ 21 2.1.1 Origins ........................................................................................................... 21 2.1.2 Hollywood, the Media, and Multi-Touch ......................................................... 23 2.1.3 Multi-touch Components ................................................................................ 24 2.2 Hardware ........................................................................................................... 24 2.2.1 Display ........................................................................................................... 25 2.2.2 Input ............................................................................................................... 27 2.2.2.1 Frustrated Total Internal Reflection (FTIR) ............................................. 28 2.2.2.2 Diffused Illumination (DI) ........................................................................ 30 2.2.2.3 Diffused Surface Illumination (DSI) ......................................................... 31 2.2.2.4 Optical Occlusion (OO) ........................................................................... 31 2.2.2.5 Laser Light Plane (LLP) .......................................................................... 32 vii 2.2.2.6 Shadow Detection (SD) .......................................................................... 32 2.2.2.7 Capacitance ............................................................................................ 33 2.2.2.8 Resistance .............................................................................................. 35 2.2.2.9 Interpolating Force-Sensitive Resistance (IFSR) .................................... 35 2.2.2.10 Input Summary ..................................................................................... 36 2.3 Software ............................................................................................................. 37 2.4 Multi-Touch for General-Purpose Computing..................................................... 41 2.4.1 Examination of Existing Text Entry Methods for Multi-Touch Application ...... 43 2.4.1.1 Evaluative Criteria .................................................................................. 43 2.4.1.2 Identifying Viable Existing Text Entry Methods ....................................... 48 2.5 Summary ........................................................................................................... 53 CHAPTER THREE: TEXT ENTRY STUDY ................................................................... 54 3.1 Introduction ........................................................................................................ 54 3.2 Experiment ......................................................................................................... 54 3.2.1 Materials and Methods .................................................................................. 55 3.2.1.1 Participants ............................................................................................. 55 3.2.1.2 Apparatus ............................................................................................... 56 3.2.1.3 Software ................................................................................................. 58 viii 3.2.1.4 Procedure ............................................................................................... 62 3.2.1.5 Environment ........................................................................................... 63 3.2.1.6 Design ...................................................................................................