Mobile Pen-And-Paper Interaction
Total Page:16
File Type:pdf, Size:1020Kb
Mobile Pen-and-Paper Interaction Infrastructure Design, Conceptual Frameworks of Interaction and Inter- action Theory Dissertation genehmigt zur Erlangung des akademischen Grades eines Doktor-Ingenieurs (Dr.-Ing.) vorgelegt von Dipl.-Inform. Farid Hamza Felix Heinrichs geb. in Frankfurt am Main am Fachbereich Informatik der Technischen Universit at¨ Darmstadt Gutachter Prof. Dr. Max M uhlh¨ auser¨ (TU Darmstadt) Prof. Dr. Johannes Sch oning¨ (Universiteit Hasselt) Tag der Einreichung 26. August 2015 Tag der Disputation 12. November 2015 Darmstadt 2015 D17 ÕækQË @ á ÔgQË @ é <Ë @ Õ æ . ”In the name of God, the Most Gracious, the Most Merciful” Abstract Although smartphones, tablets and other mobile devices become increasingly popu- lar, pen and paper continue to play an important role in mobile settings, such as note taking or creative discussions. However, information on paper documents remains static and usage practices involving sharing, researching, linking or in any other way digitally processing information on paper are hindered by the gap between the digi- tal and physical worlds. A considerable body of research has leveraged digital pen technology in order to overcome this problem with respect to static settings, however, systematically neglecting the mobile domain. Only recently, several approaches began exploring the mobile domain and develop- ing initial insights into mobile pen-and-paper interaction (mPPI), e.g., to publish digi- tal sketches, [Cowan et al., 2011], link paper and digital artifacts, [Pietrzak et al., 2012] or compose music, [Tsandilas, 2012]. However, applications designed to integrate the most common mobile tools pen, paper and mobile devices, thereby combining the benefits of both worlds in a hybrid mPPI ensemble, are hindered by the lack of sup- porting infrastructures and limited theoretical understanding of interaction design in the domain. This thesis advances the field by contributing a novel infrastructural approach to- ward supporting mPPI. It allows applications employing digital pen technology in controlling interactive functionality while preserving mobile characteristics of pen and paper. In addition, it contributes a conceptual framework of user interaction in the domain suiting to serve as basis for novel mPPI toolkits. Such toolkits ease devel- opment of mPPI solutions by focusing on expressing interaction rather than designing user interfaces by means of rigid widget sets. As such, they provide the link between infrastructure and interaction in the domain. Lastly, this thesis presents a novel, em- pirically substantiated theory of interaction in hybrid mPPI ensembles. This theory informs interaction design of mPPI, ultimately allowing to develop compelling and engaging interactive systems employing this modality. i Zusammenfassung Obwohl Smartphones, Tablets und andere mobile Ger ate¨ mittlerweile weit verbreitet sind, spielen Stift und Papier weiterhin eine wichtige Rolle in mobilen Situationen, beispielsweise beim Anfertigen von Notizen, oder in kreativen Diskussionen. In- formation auf Papierdokumenten bleibt dabei jedoch statisch und Verwendungsprak- tiken, wie beispielsweise teilen, recherchieren, verlinken oder in einer anderen Art Information digital weiterverarbeiten werden durch den Bruch zwischen digitaler und physischer Welt erschwert. Eine ganze Reihe von Forschungsarbeiten nutzt digitale Stifttechnologie um dieses Problem in statischen Situationen zu adressieren, mobile Situationen werden jedoch bislang systematisch vernachl assigt.¨ Erst in letzter Zeit begannen einige neue Ans atze¨ die mobile Domane¨ zu explo- rieren und initiale Einsichten in mobile Stift-und-Papier Interaktion , im Englischen mobile pen-and-paper interaction (mPPI), zu entwickeln, beispielweise um digitale Sketche zu ver offentlichen,¨ [Cowan et al., 2011], Papier und digitale Artefakte zu ver- linken, [Pietrzak et al., 2012], oder Musik zu komponieren, [Tsandilas, 2012]. Derar- tige Ansatze,¨ entworfen um die meist verbreiteten mobilen Werkzeuge Stift, Papier und mobile Gerate¨ zu integrieren und dabei die Vorteile beider Welten in einem hy- briden mPPI ensemble zu kombinieren, werden jedoch durch einen Mangel an un- terstutzenden¨ Infrastrukturen und eingeschranktes¨ theoretisches Verst andnis¨ im Hin- blick auf Interaktionsdesign in der Dom ane¨ gehindert. Die vorliegende Dissertation stellt einen neuartigen Infrastrukturansatz vor, welcher mPPI direkt unterstutzt,¨ und leistet damit einen wichtigen Beitrag zum Voranschreiten dieses Forschungsfelds. Dieser Ansatz erlaubt Applikationen, digitale Stifttechnolo- gie zum Steuern von interaktiver Funktionalit at¨ einzusetzen und erh alt¨ gleichzeitig mobile Charakteristiken von Stift und Papier. Weiterhin tr agt¨ diese Dissertation einen konzeptuellen Rahmen f ur¨ Stift-und-Papier basierte Nutzerinteraktion bei, welcher dazu geeignet ist als Basis von mPPI Toolkits zu dienen. Derartige Toolkits verein- fachen die Entwicklung von mPPI L osungen¨ durch Fokussierung auf das Beschreiben von Interaktion, im Gegensatz zur klassischen Fokussierung auf rigide Widget sets. Somit stellen sie das Bindeglied zwischen Infrastruktur und Interaktion dar. Ab- schließend, stellt diese Dissertation eine neuartige, empirisch fundierte Theorie der Interaktion in hybriden mPPI ensembles vor. Diese Theorie dient zur Unterst utzung¨ des Interaktionsdesigns und erlaubt es letztlich, spannende und einfach zu bedienende, interaktive, Stift-und-Papier basierte Systeme zu entwickeln. iii Acknowledgements I am deeply indebted to those people who have made this thesis and its finalization possible. It is owed to the continuous and relentless support of my family, colleagues, advisors, friends and students, who never gave up on me even when I almost did. I want to take this chance to say ”thank you”. Although I cannot name everyone here who had a part, I want to extend my special gratitude to Prof. Dr. Max M uhlh¨ auser¨ for giving me the chance to embark on this journey, advising me throughout it and putting up the patience for me to finish it. Also, I want to thank Prof. Dr. Johannes Sch oning¨ for reviewing this thesis and making its finalization possible. Further, I want to thank my former colleagues and students at the Telecooperation group TU Darmstadt, es- pecially the support team around Elke Halla. Here, another special and warm hearted ”thank you” goes to Dr. Daniel Schreiber, who accompanied me throughout the long journey, taught me the skill of ”just doing it” (among others) and put up incredible patience listening to all my doubts, continuously motivating me. Thanks Daniel, also for reading through the drafts although you are a busy family man and work some place else now, I really appreciate it. Finally, I want to thank my family, in particular my mother, for always being there, always listening to the endless stream of weird technical jargon while at the same time motivating and supporting me. And I want to thank my wife Nadia, without whom this thesis never would have been finished. Thank you for trusting in me, thank you for your patience and support, thank you my sunshine. v Contents 1 Introduction 1 1.1 Hybrid mPPI Ensemble Research . 4 1.1.1 Mobile Devices vs. Paper . 4 1.1.2 Case Study: Digital Grocery Shopping Support . 9 1.1.3 Characteristics of Hybrid mPPI Ensembles . 15 1.2 Research Questions and Contributions . 17 1.2.1 Methodology . 20 1.2.2 Publications Related to this Thesis . 22 1.3 Structure of this Thesis . 22 2 Background and Related Work 25 2.1 Background Pen-and-Paper Interaction . 26 2.1.1 Essential Terminology . 27 2.1.2 Underlying Technology: Digital Pens and Paper . 28 2.2 Application Domains: Categorization and Related Work . 34 2.2.1 Note-taking . 34 2.2.2 Knowledge Work: Active Reading and Education . 39 2.2.3 Document Editing and Form Filling . 41 2.2.4 Other Application Domains . 45 2.3 Infrastructures for Mobile Pen-and-Paper Interaction . 50 2.3.1 Infrastructures: Functional Decomposition and Requirements 50 2.3.2 Existing Infrastructures for Pen-and-Paper Interaction . 55 2.3.3 Deficits in Existing Infrastructures . 60 2.4 Conceptual Frameworks of (Mobile) Pen-and-Paper Interaction . 62 2.4.1 Pen-and-Paper Interaction Techniques . 63 2.4.2 Conceptual Frameworks: Requirements for Toolkit Design . 67 2.4.3 Existing Conceptual Frameworks . 68 2.4.4 Deficits in Existing Conceptual Frameworks . 71 2.5 Theories of Mobile Pen-and-Paper Interaction . 72 2.5.1 Theoretical Impact of Hybrid mPPI Ensemble Components . 73 2.5.2 Existing Theories of PPI and mPPI . 77 2.5.3 Deficits in Existing Theories . 79 vii Contents 2.6 Summary and Conclusion . 80 3 Infrastructure for Mobile Pen-and-Paper Interaction 83 3.1 Mobile PPI Processing . 84 3.1.1 The PPI Processing Pipeline . 85 3.1.2 Distributed PPI Processing . 88 3.1.3 Interactive Region Publishing and Discovery . 93 3.2 Distributed mPPI Processing Infrastructure . 97 3.2.1 Basic Platform Concepts . 97 3.2.2 Digital Ink Processing . 103 3.2.3 Interactive Region Publishing and Discovery . 113 3.3 Reference Implementation: The Letras Platform . 115 3.3.1 Generic Distributed Deployment . 116 3.3.2 Mobile Deployment: MobiLetras . 121 3.4 Evaluation . 122 3.4.1 Analytic Comparison: Monolithic vs. Distributed Pipeline . 123 3.4.2 Prototype Applications . 128 3.4.3 Case Study: The Digital Grocery List . 135 3.5 Summary and Conclusion . 139 4 Conceptual Framework of (Mobile) Pen-and-Paper Interaction 141 4.1 The W 5 Conceptual Framework . 143 4.1.1 Requirements Reviewed . 143 4.1.2 Basic Framework Structure . 145 4.1.3 Semantics and Notation . 150 4.2 Design Space Analysis: Design Vocabulary . 155 4.2.1 Pidget Interaction: PaperPoint and PaperProof . 156 4.2.2 Gesture Systems: PapierCraft . 159 4.2.3 Cross-media Techniques: CoScribe . 162 4.2.4 Coverage of Derived Interaction Predicates . 163 4.3 Evaluation . 165 4.3.1 Analytic Evaluation . 165 4.3.2 Proof-of-Concept Evaluation: mPPI Toolkit based on W 5 . 168 4.3.3 Proof-of-Concept Evaluation: Interaction Techniques . 171 4.4 Summary and Conclusion . 174 5 Theory of Mobile Pen-and-Paper Interaction 177 5.1 Developing Theory . 178 5.1.1 Understanding the Domain . 178 5.1.2 Extending Existing Theories . 180 viii Contents 5.1.3 Research Method .