SHOW ME WHAT YOU MEAN Gestures and Drawings on Physi

Thesis in Master’s Degree Programme in Industrial and Strategic Design SHOW ME WHAT YOU MEAN Gestures and drawings on physical objects as means for remote collaboration and guidance Andrei Savolainen Aalto University | School of Art and Design 2011 Abstract This thesis presents findings based on the study of remote projected interaction and guidance on physical objects. First, the results are based on the study of literature and previous research in the fields of ubiqutious computing and environment, augmented reality, remote collaboration and guidance. Second, the results are based on findings through testing projector technology in remote interaction and guidance with users with the help of prototype. Previous studies indicate that guidance on physical objects is seen as valuable and in such interaction, the focus should be shifted to the actual object. This thesis contributes to previous research and suggest better integration of hand gestures and drawings into remote collaboration and guidance. Projected interaction model, described in this thesis, enhances the feeling of together- ness between remote users (expert and novice), and provides critical help in conversa- tional grounding in remote collaboration and guidance with physical objects. Keywords Pico projector (also handheld projector, micro projector, mobile projector), remote interaction, remote collaboration, remote guidance, augmented reality, contextual sensitivity, context-aware, ubiquitous computing, ubiquitous environment, gestures in remote collaboration and guidance. Acknowledgements I would like to express my appreciation to my tutor, professor Turkka Keinonen from Aalto University and Lutz Abe, a supervisor from Nokia. Thanks for the valuable and con- structive feedback and advises during this process. Also thanks goes to Jussi Mikkonen, Laboratory Manager at Aalto University for the technical expertise with pico projectors and valuable insights. I would also like to express my gratitude to my family and friends, and most of all to my girlfriend - Taru for the unconditional support, patience and feedback. This would not have been possible without you. Table of contents Abstract Acknowledgements Chapter 1: 1 Introduction 3 Background 4 Framing the topic 5 Chapter 2: 7 Ubiquitous computing 9 Ubiquitous environment 12 Definition of a ubiquitous environment 13 The vision of a ubiquitous environment 13 Ubiquitous computing 14 Advanced interaction 14 Algorithmic intelligence 15 Summing up 15 Mobile technology in ubiquitous environment 15 Natural interaction in physical environment 16 Context sensitive interaction 16 Challenges of a ubiquitous environment 17 Summing up 18 Chapter 3: 19 Pico projecting technology as enabler for augmenting environment 20 Pico projector 21 Augmented reality 22 Augmented reality and projectors 24 Summing up 25 Remote collaboration 26 Video conferencing 27 Remote collaboration on physical objects 29 Research related to remote expertise and guidance 28 Use gestures and drawings in remote guidance and help 29 Chapter 4: 34 Research questions 34 The vision of ubiquitous environment 35 Projected augmented reality in physical environment 35 Table of contents Remote collaboration and guidance in physical environment 35 Research methods 37 Chapter 5: Experimental setup 45 Prototype 39 Test objectives, goals and structure 42 Roles 43 Description of the structure of test setup 43 User tests results 46 Stakeholder description 47 Mind map 49 Block test 53 Tilted plug table 56 Origami 59 Chapter 6: Findings 62 Analysis methods 63 Practical limitations 65 Evaluation by designer group 68 Gestures and drawings 71 Chapter 7: 81 Answers to research questions 82 Discussion 87 Future work and research 90 Conclusions 92 Appendix 94 Prototype specifications Mapping test References 1 Chapter 1: 2 “A picture is worth a thousand words” (unknown) 3 Introduction Pico projector technology has been on the the physical environment should not be market for years where display technology forgotten. People use everyday applianc- is the dominating technology. While pico es and objects, experience problems with projector technology is constantly being them and seek for help to fix these prob- developed and improved, it still remains to lems (e.g. by calling to a service). This the- be a niche product. This thesis tackles this sis focuses to on addressing this problem, problem by bringing forward advantages by providing a projected interaction model of pico projector technology in remote in- with gesture support to be used in remote teraction, collaboration and guidance with guidance. physical objects. This paper presents a novel approach to Real-time remote interaction and collabo- enhance remote collaboration and guid- ration is based on voice and video confer- ance especially with the help of pico pro- encing and methodologies lack the con- jecting technology. ventions of physicality. “In such approaches interactions are limited to visual and audi- tory media, and shared environments are confined to digital world.” [1] While pro- jection lacks the ability to be touched, it can be used in remote collaboration and guidance on physical objects. As previous research suggests, the focus in such interaction should be on the object, rather the facial expressions of participants. Remote collaboration with the physical world and with physical objects needs different approaches than what simple video conferencing can offer. While people use digital environments for a number of activities, 4 Background This thesis was done in collaboration with Remote interaction and collaboration has Nokia. I have been working at Nokia since been a subject for research for years, 2007 as an User Interface Graphic Design- where the most interesting results and er and I was excited when I found out that concepts in terms of this thesis are related there was an opportunity for thesis collab- to tangible interfaces - decreasing the gap oration with pico projector technology as a between digital and physical or merging theme. After initial discussions and mutu- these two. al agreement that pico projecting technology provides a good ground for research This Thesis emphasizes tridimensional- in Master thesis - I begun to explore pos- ity of the physical environment. The main sibilities to approach this technology. goal is to present the physical environment with its objects as a a layer on top The main goal at first was to find a suit- of which the interaction is built on, in a able approach which would provide the similar way as augmented reality does. Re- “big picture”, so that possibilities for this mote interaction model as it is presented technology would be seen as a part of a in this paper is built upon pico projecting larger context. Initial inspiration and a big technology and it takes place in a physical spark for this thesis was found through environment where it augments physical ubiquitous computing and ubiquitous en- objects with digital content. vironment. These subjects provided a high level of theoretical ground for this thesis. This model provides a new way to partici- pate and guide remotely in physical envi- Theories and concepts behind ubiquitous ronment and with physical objects. While environment provide approaches to the co-located collaboration is by far most ef- smarter use of technology as an enabler fective, the model presented in this thesis for more natural and meaningful interac- provides an immersive approach to make tion between the human and the environ- the remote collaboration and guidance ment. less ‘remote’. 5 Framing the topic Pico projecting technology in ubiq- physical environment is based on haptic senses, we manipulate objects directly, uitous environment by moving and touching objects and use I believe that pico projecting technology pointing and other gestures to help out in has significant potential but that potential communicating with others, when we col- can be found only by taking a fresh look at laborate locally on shared physical object. this technology with keeping in mind ‘the big picture’. In this thesis, this means look- Local collaboration, taking place in single, ing at this technology as a part of a ubiq- real-world over a physical object relies uitous environment. The ‘vision of ubiqui- not only on direct manipulation of an ob- tous environment’ [2]provides wireframe ject by all users, but also on various sets for better understanding of relationships of gestures. When students are working between user, environment and the role of in a group task involving constructing a technology. mechanical structure for instance, not all of them are working simultaneously. The process requires for some of them to Remote (tangible) interaction and step back, observe, give feedback along guidance direct manipulation of pieces of mechani- Current systems for real-time interac- cal structure. Giving feedback is not only tion and collaboration contribute to great based on verbal input, but it is also sup- extent to voice and video conferencing ported by hand gestures: “take this”, “put alone and provide access to shared digital it there”, “replace this piece by that”. environments. Such interfaces fail to ad- dress our sense of touch and physicality When collaboration takes place remotely, and gesture based interaction. Our life in current systems are not supporting hand 6 gestures, which is after all seen a natural Theoretical wireframe part of human interaction. The Vision of Ubiquitous Environment with This thesis addresses physicality

Load more