Full text available at: http://dx.doi.org/10.1561/1100000026 Tangible User Interfaces: Past, Present, and Future Directions Full text available at: http://dx.doi.org/10.1561/1100000026 Tangible User Interfaces: Past, Present, and Future Directions Orit Shaer Wellesley College Wellesley, MA 02481 USA [email protected] Eva Hornecker University of Strathclyde Scotland, G1 1XH UK [email protected] Boston { Delft Full text available at: http://dx.doi.org/10.1561/1100000026 Foundations and Trends R in Human{Computer Interaction Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 USA Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is O. Shaer and E. Hornecker, Tangible User Interfaces: Past, Present, and Future Directions, Foundations and Trends R in Human{Computer Interaction, vol 3, nos 1{2, pp 1{137, 2009 ISBN: 978-1-60198-328-2 c 2010 O. Shaer and E. Hornecker All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Cen- ter, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC). The `services' for users can be found on the internet at: www.copyright.com For those organizations that have been granted a photocopy license, a separate system of payment has been arranged. Authorization does not extend to other kinds of copy- ing, such as that for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. In the rest of the world: Permission to pho- tocopy must be obtained from the copyright owner. Please apply to now Publishers Inc., PO Box 1024, Hanover, MA 02339, USA; Tel. +1-781-871-0245; www.nowpublishers.com; [email protected] now Publishers Inc. has an exclusive license to publish this material worldwide. Permission to use this content must be obtained from the copyright license holder. Please apply to now Publishers, PO Box 179, 2600 AD Delft, The Netherlands, www.nowpublishers.com; e-mail: [email protected] Full text available at: http://dx.doi.org/10.1561/1100000026 Foundations and Trends R in Human{Computer Interaction Volume 3 Issues 1{2, 2009 Editorial Board Editor-in-Chief: Ben Bederson Human{Computer Interaction Lab University of Maryland 3171 A. V. Williams Bldg 20742, College Park, MD Editors Gregory Abowd (Georgia Institute of Technology) Jonathan Grudin (Microsoft Research) Clayton Lewis (University of Colorado) Jakob Nielsen (Nielsen Norman Group) Don Norman (Nielsen Norman Group and Northwestern University) Dan Olsen (Brigham Young University) Gary Olson (UC Irvine) Full text available at: http://dx.doi.org/10.1561/1100000026 Editorial Scope Foundations and Trends R in Human{Computer Interaction will publish survey and tutorial articles in the following topics: • History of the research Community • Online communities • Design and Evaluation • Games • Ergonomics/Human Factors • Communication technologies • Cognitive engineering and • Interdisciplinary influence performance models • The role of the social sciences in HCI • Predictive models of interaction • MIS and HCI • User-centered design processes • Graphic design • Participatory design • Artificial intelligence and the user • Graphic design interface • Discount evaluation techniques • Architecture and the role of the • Design and interaction physical environment • Ethnography • Advanced topics and tends • Theory • Information visualization • Models of cognition • Web design • Empirical methods of evaluation • Assistive technologies • Qualitative methods of design • Multimodal interaction and evaluation • Perception and the user interface • Technology • Specific user groups (children, elders, • Programming the graphical user etc.) interface • Sensor-based or tangible interaction • Input technologies • Ubiquitous computing • Output technologies • Virtual reality • Computer supported cooperative • Augmented reality work • Wearable computing • History of CSCW in HCI • Design and fashion • Organizational issues • Privacy and social implications Information for Librarians Foundations and Trends R in Human{Computer Interaction, 2009, Volume 3, 4 issues. ISSN paper version 1551-3955. ISSN online version 1551-3963. Also avail- able as a combined paper and online subscription. Full text available at: http://dx.doi.org/10.1561/1100000026 Foundations and Trends R in Human{Computer Interaction Vol. 3, Nos. 1{2 (2009) 1{137 c 2010 O. Shaer and E. Hornecker DOI: 10.1561/1100000026 Tangible User Interfaces: Past, Present, and Future Directions Orit Shaer1 and Eva Hornecker2 1 Wellesley College, 106 Central St., Wellesley, MA, 02481, USA, [email protected] 2 University of Strathclyde, 26 Richmond Street, Glasgow, Scotland, G1 1XH, UK, [email protected] Abstract In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in order to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This monograph examines the existing body of work on Tangible User Interfaces. We start by sketching the history of tangible user inter- faces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frameworks and taxonomies. We also discuss conceptual foundations Full text available at: http://dx.doi.org/10.1561/1100000026 of TUIs including perspectives from cognitive sciences, psychology, and philosophy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limitations of TUIs and chart directions for future research. Full text available at: http://dx.doi.org/10.1561/1100000026 Contents 1 Introduction 1 2 Origins of Tangible User Interfaces 5 2.1 Graspable User Interface 6 2.2 Tangible Bits 7 2.3 Precursors of Tangible User Interfaces 9 3 Tangible Interfaces in a Broader Context 13 3.1 Related Research Areas 13 3.2 Unifying Perspectives 16 3.3 Reality-Based Interaction 18 4 Application Domains 21 4.1 TUIs for Learning 22 4.2 Problem Solving and Planning 26 4.3 Information Visualization 30 4.4 Tangible Programming 32 4.5 Entertainment, Play, and Edutainment 35 4.6 Music and Performance 38 4.7 Social Communication 42 4.8 Tangible Reminders and Tags 43 5 Frameworks and Taxonomies 45 5.1 Properties of Graspable User Interfaces 46 ix Full text available at: http://dx.doi.org/10.1561/1100000026 5.2 Conceptualization of TUIs and the MCRit Interaction Model 47 5.3 Classifications of TUIs 48 5.4 Frameworks on Mappings: Coupling the Physical with the Digital 50 5.5 Tokens and Constraints 53 5.6 Frameworks for Tangible and Sensor-Based Interaction 55 5.7 Domain-Specific Frameworks 58 6 Conceptual Foundations 61 6.1 Cuing Interaction: Affordances, Constraints, Mappings and Image Schemas 61 6.2 Embodiment and Phenomenology 63 6.3 External Representation and Distributed Cognition 65 6.4 Two-Handed Interaction 68 6.5 Semiotics 69 7 Implementation Technologies 73 7.1 RFID 74 7.2 Computer Vision 75 7.3 Microcontrollers, Sensors, and Actuators 77 7.4 Comparison of Implementation Technologies 79 7.5 Tool Support for Tangible Interaction 81 8 Design and Evaluation Methods 89 8.1 Design and Implementation 89 8.2 Evaluation 94 9 Strengths and Limitations of Tangible User Interfaces 97 9.1 Strengths 98 9.2 Limitations 106 Full text available at: http://dx.doi.org/10.1561/1100000026 10 Research Directions 111 10.1 Actuation 111 10.2 From Tangible User Interfaces to Organic User Interfaces 113 10.3 From Tangible Representation to Tangible Resources for Action 114 10.4 Whole-Body Interaction and Performative Tangible Interaction 116 10.5 Aesthetics 117 10.6 Long-Term Interaction Studies 117 11 Summary 121 Acknowledgments 123 References 125 Full text available at: http://dx.doi.org/10.1561/1100000026 1 Introduction \We live in a complex world, filled with myriad objects, tools, toys, and people. Our lives are spent in diverse interaction with this environment. Yet, for the most part, our computing takes place sitting in front of, and staring at, a single glowing screen attached to an array of buttons and a mouse." [253] For a long time, it seemed as if the human{computer interface was to be limited to working on a desktop computer, using a mouse and a key- board to interact with windows, icons, menus, and pointers (WIMP). While the detailed design was being refined with ever more polished graphics, WIMP interfaces seemed undisputed and no alternative inter- action styles existed. For any application domain, from productivity tools to games, the same generic input devices were employed. Over the past two decades, human{computer interaction (HCI) researchers have developed a wide range of interaction styles and inter- faces that diverge from the WIMP interface. Technological advance- ments and a better understanding of the psychological and social aspects of HCI have lead to a recent explosion of new post-WIMP 1 Full text available at: http://dx.doi.org/10.1561/1100000026 2 Introduction interaction styles. Novel input devices that draw on users' skill of inter- action with the real non-digital world gain increasing popularity (e.g., the Wii Remote controller, multi-touch surfaces).