The Fundamental Issues of Pen-Based Interaction with Tablet Devices by Michelle Kathryn Annett A thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Computing Science University of Alberta © Michelle Kathryn Annett, 2014 1 Abstract Although pens and paper are pervasive in the analog world, their digital counterparts, styli and tablets, have yet to achieve the same adoption or frequency of use. Digital styli should provide a natural, intuitive method to take notes, annotate, and sketch, but have yet to reach their full potential. There has been surprisingly little research focused on understanding why inking experiences differ so vastly between analog and digital media and amongst various styli themselves. To enrich our knowledge on the stylus experience, this thesis contributes a foundational understanding of the factors implicated in the varied experiences found within the stylus ecosystem today. The thesis first reports on an exploratory study utilizing traditional pen and paper and tablets and styli that observed quantitative and behavioural data, in addition to preferential opinions, to understand current inking experiences. The exploration uncovered the significant impact latency , unintended touch, and stylus accuracy have on the user experience, whilst also determining the increasing importance of stylus and device aesthetics and stroke beautification . The observed behavioural adaptations and quantitative measurements dictated the direction of the research presented herein. A systematic approach was then taken to gather a deeper understanding of device latency and stylus accuracy. A series of experiments garnered insight into latency and accuracy, examining the underlying elements that result in the lackluster experiences found today. The results underscored the importance of visual feedback, user expectations, and perceptual limitations on user performance and satisfaction. The proposed Latency Perception ii Model has provided a cohesive understanding of touch- and pen-based latency perception, and a solid foundation upon which future explorations of latency can occur. The thesis also presents an in-depth exploration of unintended touch. The data collection and analysis underscored the importance of stylus information and the use of additional data sources for solving unintended touch. The behavioral observations reemphasized the importance of designing devices and interfaces that support natural, fluid interaction and suggested hardware and software advancements necessary in the future. The commentary on the interaction - rejection dichotomy should be of great value to developers of unintended touch solutions along with designers of next-generation interaction techniques and styli. The thesis then concludes with a commentary on the areas of the stylus ecosystem that would benefit from increased attention and focus in the years to come and future technological advancements that could present interesting challenges in the future. iii Preface This thesis is an original work by Michelle Annett. The research projects, of which this thesis is a part, received research ethics approval from the University of Alberta Research Ethics Board, as “Gestures and Interactive Devices”, Pro00027121 on January 16, 2012, and as “Evaluation of Human-Computer Interaction Devices and Techniques”, Pro00033336 on September 30, 2012. Some of the research conducted for this thesis was performed while I was an intern at Microsoft Research, in Redmond, Washington, USA. Throughout this time, Dr. Walter Bischof was my University of Alberta supervisor, Dr. Anoop Gupta and Dr. Paul Dietz were my Microsoft Research mentors, and Albert Ng worked with Dr. Paul Dietz to design and test the hardware presented in Chapter 3. Chapter 2 of this thesis has been published as Michelle Annett, Fraser Anderson, Walter F. Bischof, and Anoop Gupta, “The Pen is Mightier: Understanding Stylus Behaviour While Inking on Tablets” in the proceedings of the 2014 Graphics Interface conference. I was responsible for designing the experiment, developing the hardware apparatus, recruiting participants, running the experiment and collecting data, analyzing and interpreting the results and writing the manuscript. Walter and Anoop served as advisors and assisted in the manuscript editing process. Fraser also assisted with various revisions of the manuscript. Chapter 3 of this thesis has been published as Albert Ng, Michelle Annett, Paul Dietz, Anoop Gupta, and Walter F. Bischof, “In the Blink of an Eye: Investigating Latency Perception during Stylus Interaction” in the proceedings of the 2014 SIGCHI Conference on Human Factors in Computing and in Michelle Annett, Albert Ng, Paul Dietz, Walter F. Bischof, and Anoop Gupta, “How Low Should We Go? Understanding the Perception of Latency While Inking” in the proceedings of the 2014 Graphics Interface conference. For this research project, I was responsible for designing and testing the stylus used in the experiment, designing the experiments, recruiting participants, running the experiments and collecting data, analyzing and interpreting the results, and developing the Latency Perception Model. I also wrote the Graphics Interface manuscript and worked with Albert and Paul to write the SIGCHI manuscript. Albert and Paul designed and implemented the hardware apparatus and Walter assisted in the initial design of the user studies. Walter, Anoop, and Paul were advisors and also assisted in the manuscript editing processes. iv Chapter 4 of this thesis has been accepted for publication in ACM Transactions on Computer-Human Interaction as Michelle Annett, Walter Bischof, and Anoop Gupta “Exploring and Understanding Unintended Touch during Direct Pen Interaction”. I was responsible for developing the hardware apparatus, designing the experiment, recruiting participants, running the experiments and collecting data, analyzing and interpreting the results, and writing the manuscript. Walter and Anoop were my advisors and assisted in the manuscript editing process. Chapter 5 of this thesis is in preparation for publication as Michelle Annett and Walter Bischof as “Hands, Hover, and Nibs: Exploring the Other Factors Influencing Stylus Accuracy on Tablets”. I was responsible for designing the experiments, recruiting participants, running the experiments and collecting data, analyzing and interpreting the results, and writing the manuscript. Walter was my advisor and assisted in the manuscript editing process. v “The art challenges the technology, and the technology inspires the art” - John Lasseter vi Acknowledgements First off, my advisor Walter F. Bischof deserves my utmost appreciation and gratitude. Walter, you have shared your research experience and psychological knowledge with me on a daily basis, teaching me everything I know about statistics and experimental methodologies (ahem, effect sizes!) Although I still despise your red pen, your attention to detail and inquisitive nature helped shape the writer and researcher that I am today. I know that I would not be anywhere close to where I am if it were not for your high standards and constant pursuit of research excellence. Regardless of how wacky or non-traditional Fraser’s and my ideas were (e.g., “We want to build a talking monkey Walter”, “Walter, we’re going to build a Smell-o-Vision interface”), you never wavered in your support of Fraser and me. You let me spread my wings and go experience industry, first in Toronto, at Autodesk Research, and then again in Redmond, at Microsoft Research. You have always encouraged me to take research risks and immerse myself in projects I enjoyed, no matter how many times I changed my dissertation topic! Although Fraser and I still love our mac and cheese, we will always cherish the culinary adventures you took us on and the magic of a Swiss Raclette! Eine Minute auf der Lippe, ein Leben lang auf der Hüfte! I also would like to extend my utmost thanks to my industry mentors. Dr. George Fitzmaurice and Dr. Tovi Grossman, you both took a chance on an unknown student from the University of Alberta and let her run wild with hot glue and proximity sensors. My time at Autodesk Research opened my eyes to the excitement of industry and underscored the importance of working with great people to me. Thank-you. Dr. Anoop Gutpa and Dr. Paul Dietz, like Tovi and George, you both took a chance on a little Canadian girl and gave her the project and experience of a lifetime. I will never forget my time in Redmond and the opportunities you provided. Paul, thanks for indulging a Disney fan in your many stories and tales about Imagineering and Disney. There are also many people who I have worked with behind the scenes that have been instrumental throughout my PhD. First off, Dr. Pierre Boulanger, thank you for ensuring that I always had new toys and technology to tinker with and for letting Fraser and me continually rearrange the lab, stock it full of interns, and turn the spring and summer terms into chaos for the sake of “science”. I will miss calling the AMMI Lab and CSC 363 my home. I would also like to thank Wei Li, Justin Matejka, Dr. Daniel Wigdor, and John Yee, who were incredibly helpful during my time at Autodesk Research and ensured that Medusa was a success (sorry for disrupting your workspace Justin!). I also owe very big thanks to the members of the Microsoft Research Hardware Lab, vii including Dr. Tom Blank, Patrick Therien, Chris O’Dowd, and Bruce Cleary. I am very grateful that you