A Natural User Interface and Touchless Interaction Approach on Web Browsing Master of Science Thesis in the Master Degree Programme, Interaction Design & Technologies GEORGIOS LAFKAS Report No: 2013:143 ISSN: 1651-4769 Department of Applied I.T. CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden, 2013 Abstract The aim of this Master’s Thesis is to explore the principles of Natural Interaction and suggest an implementation of a web browsing application that uses them. The research that takes place is examining the ingredients of Natural User Interfaces and Touchless Interaction with a focus on Gestures and Voice. How can those be combined to control an interface? What are the most important things to mind when designing Natural Interfaces and Interactions? These are questions that this report attempts to answer. The final result of the thesis is a prototype web browser application that works only with gestures and voice. The prototype simulates browsing on a number of created web pages. It is implemented using the Microsoft Kinect for XBox sensor and Kinect SDK v1.7 with all the included tools and interaction controls. The browser is undergoing a usability test and the participants are observed and later interviewed to describe their experience. Based on that, the conclusions drawn are that designing an application that uses a Natural Interface needs to provide short and comfortable Gesture and Speech Vocabularies and Feedback mechanisms that inform the user at every point of the interaction. Furthermore, it is concluded that the design of the content for such an application should be different than the traditional design of a webpage for optimal use while responding to implicit input (context- awareness) adds to the naturalness of the interaction. Finally, the setting for using such an application is recommended to be part of a home entertainment system (e.g. smart TV) or public space installation rather than a desktop computer. Keywords: human-computer interaction, natural user interfaces, touchless interaction, gestures, speech recognition, touchless web browsing. 2 Table of Contents Introduction................................................................................................................................................5 Problem................................................................................................................................................5 Goal and aim........................................................................................................................................5 Research question.................................................................................................................................5 Structure...............................................................................................................................................6 Involved Parties....................................................................................................................................6 Background................................................................................................................................................7 User Interfaces......................................................................................................................................7 Computer Interfaces (CI)................................................................................................................7 Command Line Interfaces (CLI)....................................................................................................8 Graphical User Interfaces (GUI)....................................................................................................9 Touch User Interfaces (TUI)...........................................................................................................9 Natural User Interfaces (NUI)......................................................................................................10 Motion Sensing Technology...............................................................................................................11 Mechanical Sensing......................................................................................................................11 Inertial Sensing.............................................................................................................................12 Acoustic Sensing..........................................................................................................................12 Magnetic Sensing..........................................................................................................................12 Optical Sensing.............................................................................................................................12 Radio and Microwave Sensing.....................................................................................................12 Current motion tracking solutions......................................................................................................13 Nintendo Wii.................................................................................................................................13 Playstation Move..........................................................................................................................13 Kinect............................................................................................................................................14 Leap Motion..................................................................................................................................15 Omek Interactive..........................................................................................................................15 Related Work......................................................................................................................................16 Theory......................................................................................................................................................19 Gestures..............................................................................................................................................19 Gesture Recognition.....................................................................................................................21 Speech................................................................................................................................................21 Speech recognition........................................................................................................................22 Semiotics............................................................................................................................................23 Natural Interaction concepts...............................................................................................................24 Intuition and intuitive behavior....................................................................................................24 Channels of communication and multimodality...........................................................................25 Context..........................................................................................................................................25 Ergonomics.........................................................................................................................................26 Methodology............................................................................................................................................27 Overview............................................................................................................................................27 Ideation.........................................................................................................................................27 Requirements................................................................................................................................27 Prototyping...................................................................................................................................28 User test........................................................................................................................................29 Execution.................................................................................................................................................30 Ideation...............................................................................................................................................30 Concept Definition.............................................................................................................................30 Literature Study..................................................................................................................................31 Benchmarking....................................................................................................................................31 Requirements......................................................................................................................................31 Low-fidelity Prototype.......................................................................................................................32 Concept Refinement...........................................................................................................................32