Discrete and Continuous Shape Writing for Text Entry and Control
Total Page:16
File Type:pdf, Size:1020Kb
Linköping Studies in Science and Technology Dissertation No. 1106 Discrete and Continuous Shape Writing for Text Entry and Control Per Ola Kristensson Department of Computer and Information Science Linköping University 581 83 Linköping, Sweden Linköping 2007 ISBN 978-91-85831-77-7 ISSN 0345-7524 Printed by LiU-Tryck, Linköping 2007 Abstract Mobile devices gain increasing computational power and storage capabilities, and there are already mobile phones that can show movies, act as digital music players and offer full-scale web browsing. The bottleneck for information flow is however limited by the inefficient communication channel between the user and the small device. The small mobile phone form factor has proven to be surprisingly difficult to overcome and limited text entry capabilities are in effect crippling mobile devices’ use experience. The desktop keyboard is too large for mobile phones, and the keypad too limited. In recent years, advanced mobile phones have come equipped with touch- screens that enable new text entry solutions. This dissertation explores how software keyboards on touch-screens can be improved to provide an efficient and practical text and command entry experience on mobile devices. The central hypothesis is that it is possible to combine three elements: software keyboard, language redundancy and pattern recognition, and create new effective interfaces for text entry and control. These are collectively called “shape writing” interfaces. Words form shapes on the software keyboard layout. Users write words by articulating the shapes for words on the software keyboard. Two classes of shape writing interfaces are developed and analyzed: discrete and continuous shape writing. The former recognizes users’ pen or finger tapping motion as discrete patterns on the touch-screen. The latter recognizes users’ continuous motion patterns. Experimental results show that novice users can write text with an average entry rate of 25 wpm and an error rate of 1% after 35 minutes of practice. An accelerated novice learning experiment shows that users can exactly copy a single well-practiced phrase with an average entry rate of 46.5 wpm, with individual phrase entry rate measurements up to 99 wpm. When used as a control interface, users can send commands to applications 1.6 times faster than using de- facto standard linear pull-down menus. Visual command preview leads to significantly less errors and shorter gestures for unpracticed commands. Taken together, the quantitative results show that shape writing is among the fastest mobile interfaces for text entry and control, both initially and after practice, that are currently known. i Discrete and Continuous Shape Writing for Text Entry and Control ii Acknowledgements Although my name stands alone on this dissertation, it would never have been completed without the support of many other people. In particular, I would like to express my gratitude to Dr. Shumin Zhai for being an outstanding advisor, collaborator and friend. I could write pages on why Shumin is an excellent thesis advisor. In fact I did exactly that in 2005 for a nomination of best technical co-op advisor at IBM Almaden Research Center (he won). Shumin not only taught me the principles of human-computer interaction research, he taught me the importance of focus, logical thinking and setting priorities straight. I can see now that I matured tremendously after these years of research. I am at this point in time Shumin’s only graduate student and as a result I have benefited from essentially an unlimited stream of advice, ideas and inspiration from him, both when I was working across the hallway from his office at IBM Almaden and at Linköping University across half of the globe. The opportunity to become a human-computer interaction scientist through his expert guidance is a once in a lifetime opportunity and something I will be eternally grateful for. I also want thank Prof. Sture Hägglund for his enthusiasm, advice and his generous funding of my doctoral research tenure at Linköping University. Without Sture’s support this dissertation would not have been a reality. I also want to thank my research group’s leader Prof. Henrik Eriksson for his invaluable advice. I also really appreciate Prof. Erik Sandewall at the artificial intelligence and integrated computer systems division, who generously offered financial support for the first semester before I was admitted into the graduate program in computer science. Erik was one of the first who came to my “office” (bunker!) after I finished my undergraduate thesis (D-uppsats) and explicitly congratulated me on having written an excellent thesis. I am very thankful for his truly unselfish initiative to support my research at that initial stage when the technology was yet unproven. I’d like to thank, in advance, Prof. Bill Buxton to agree to serve as my thesis opponent, and Prof. Lars Ahrenberg, Prof. Jan Gulliksen and Docent Mikael Goldstein to serve as my thesis examination committee members. Thank you for not only taking your valuable time to read and examine my thesis but also travel all the way to Linköping from various parts of the world. iii Discrete and Continuous Shape Writing for Text Entry and Control At Linköping University I particularly want to thank Prof. Lars Ahrenberg, Prof. Nils Dahlbäck, Maria Holmqvist, Magnus Ingmarsson, Magnus Rimbark and Dr. Pernilla Qvarfordt who offered advice that directly helped my research. In addition I thank Imad-Eldin Ali Abugessaisa, Lise-Lott Andersson, Dr. Mattias Arvola, Dr. Erik Berglund, Prof. Patrick Doherty, Arne Fäldt, Fredrik Heintz, Dr. Stefan Holmlid, Dr. Björn Johansson, Prof. Arne Jönsson, Britt-Inger Karlsson, Stefan Karlsson, Anders Larsson, Ola Leifler, Dr. Hanjing Li, Dr. Jonas Lundberg, Jalal Maleki, Docent Magnus Merkel, Rolf Nilsson, Piotr Rudol, Sonia Sangari, Dr. Annika Silvervarg, Mustapha Skhiri, Sara Stymne, Per Sökjer, Eva Ragnemalm, Jiri Trnka, Lillemor Wallgren, Mariusz Wzorek and the faculty, technical and administrative staff, and my fellow graduate students for providing a vibrant work environment. During my four years of research I was offered the fantastic opportunity to work for almost two years at IBM Almaden Research Center in sunny San Jose, California, USA. I had a wonderful time in California and my graduate internship at IBM Research gave me the opportunity to interact with many world class scientists. The following researchers there directly affected my research direction and generously shared their experiences and offered invaluable advice: Dr. Johnny Accot, Dr. Arnon Amir, Dr. Tue Haste Andersen, Dr. Sreeram Balakrishnan, Dr. John Barton, Dr. Allen Cypher, Dr. Ronald Fagin, Jon Graham, John Karidis, Min Lin, Dr. Paul Maglio, Dr. Robert Morris, Dr. Wayne Niblack, Dr. John Pitrelli, Dr. Barton Smith, Dr. Jayashree Subrahmonia and Alison Sue. I also want to thank Dr. David Beymer, Dr. Christopher Campbell, Dr. Steve Cousins, Dr. Alex Cozzi, Dr. Andreas Dieberger, Clemens Drews, Stephen Farell, Dr. Myron Flickner, Dr. Eben Haber, Dr. Beverly Harrison, Dr. Eser Kandogan, Dr. Stina Nylander, Dr. Daniel Russel, Dr. John Tang, Vladimir Zbarsky and everyone else I have interacted with at the research center. California and especially San Francisco is an amazing place and I got the opportunity to hang out with some really cool people such as Kristoffer Hallgren, Dr. Tue Haste Andersen, Dr. Jhilmil Jain, Sophia Liu, Eric Perlman and Lubomira Stoilova, and all you others. I also want to thank the generous support of the ACM SIGCHI community particularly by way of admitting and sponsoring me to the doctoral symposia at UIST 2004 and CHI 2005. At UIST 2004 I was given invaluable advice from a panel consisting of Dr. Ken Hinckley, Prof. Scott Hudson and Prof. Beth Mynatt, and the other graduate students that participated. At CHI 2005 the panel consisted of Prof. Gilbert Cockton, Prof. Joëlle Coutaz, Dr. Mary Czerwinski Prof. Jan Gulliksen, Dr. Henry Lieberman, Prof. Philippe Palanque, Prof. Fabio Paternò, Prof. Raquel Prates and Prof. Janet Wesson. Needless to say, the panel members and participating graduate students were invaluable to my continued research. I was very fortunate to receive a “Best doctoral consortium contribution” award at CHI 2005 that certainly boosted my confidence when pursuing my research direction. iv At the CHI, IUI and UIST conferences I have attended I have had the pleasure to interact with many insightful researchers. In particular I would like to thank Prof. James Fogarty, Dr. Antti Oulasvirta, Jingtao Wang and Prof. Kari-Jouko Räihä (who also was my opponent for the licentiate thesis). The Gesture & Dynamics workshop at Glasgow University in 2006 was very inspiring. I am grateful to Prof. Roderick Murray-Smith who arranged the workshop, invited me to attend and managed to fund my expenses. The talks at the workshop were excellent and the conversations afterwards even better. I thank Prof. Poika Isokoski, Prof. David MacKay, Prof. Benoît Martin, Prof. Roderick Murray-Smith, Morten Proschowsky and Dr. John Williamson for stimulating conversations. I also want to take the opportunity to thank some of my fellow text entry researchers: Prof. Scott MacKenzie, William Soukoreff, Prof. Poika Isokoski, Prof. Brad Myers and Prof. Jacob Wobbrock. Your research papers have been inspiring and I hope you feel the same of my own and my co-authors work. Unfortunately I have only had the opportunity to interact briefly with some of you as of yet. I want to thank all participants that volunteered to participate in my experiments. Aside from providing me with invaluable experimental data, many of you generously provided insightful comments that affected my research direction. I thank my parents Eva and Sven Kristensson and my brother Lars Johan Kristensson, my family and friends in Malmö and everywhere else, for your support, understanding and patience with me when I have insisted on traveling back and forth between the USA and Sweden for the last four years.