Full text available at: http://dx.doi.org/10.1561/1100000049 A Survey of Augmented Reality Mark Billinghurst, Adrian Clark, and Gun Lee The Human Interface Technology Laboratory New Zealand University of Canterbury Christchurch, New Zealand {mark.billinghurst, adrian.clark, gun.lee}@hitlabnz.org Boston — Delft Full text available at: http://dx.doi.org/10.1561/1100000049 R Foundations and Trends in Human-Computer Interaction Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 United States Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is M. Billinghurst, A. Clark, and G. Lee. ASurveyofAugmentedReality. Foundations R and Trends• in Human-Computer Interaction, vol. 8, no. 2-3, pp. 73–272, 2014. R This Foundations and Trends• issue was typeset in LATEXusingaclassfiledesigned by Neal Parikh. Printed on acid-free paper. ISBN: 978-1-60198-921-5 c 2015 M. Billinghurst, A. Clark, and G. Lee • All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Cen- ter, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC). The ‘services’ for users can be found on the internet at: www.copyright.com For those organizations that have been granted a photocopy license, a separate system of payment has been arranged. Authorization does not extend to other kinds of copy- ing, such as that for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. In the rest of the world: Permission to pho- tocopy must be obtained from the copyright owner. Please apply to now Publishers Inc., PO Box 1024, Hanover, MA 02339, USA; Tel. +1 781 871 0245; www.nowpublishers.com; [email protected] now Publishers Inc. has an exclusive license to publish this material worldwide. Permission to use this content must be obtained from the copyright license holder. Please apply to now Publishers, PO Box 179, 2600 AD Delft, The Netherlands, www.nowpublishers.com; e-mail: [email protected] Full text available at: http://dx.doi.org/10.1561/1100000049 R Foundations and Trends in Human-Computer Interaction Volume 8, Issue 2-3, 2014 Editorial Board Editor-in-Chief Desney S. Tan Microsoft Research United States Editors Gregory Abowd Gary Klein Georgia Institute of Technology The MITRE Corporation Ben Bederson Joe Konstan University of Maryland University of Minnesota Batya Friedman Chris North University of Washington Virginia Tech Jon Froehlich Yvonne Rogers University of Maryland University College London Jonathan Grudin Orit Shaer Microsoft Research Wel lesley Col lege Jason Hong Desney Tan Carnegie Mellon University Microsoft Research Juan Pablo Hourcade Kentaro Toyama University of Iowa UC Berkeley Karrie Karahalios Jacob Wobbrock University of Illinois University of Washington at Urbana-Champaign Full text available at: http://dx.doi.org/10.1561/1100000049 Editorial Scope Topics R Foundations and Trends• in Human-Computer Interaction publishes surveys and tutorials on the foundations of human-computer interac- tion. The scope is broad. The list of topics below is meant to illustrate some of the coverage, and is not intended to be an exhaustive list. History of the research Computer supported • • community cooperative work Design and evaluation • Interdisciplinary influence Theory • • Technology Advanced topics and trends • • Information for Librarians R Foundations and Trends• in Human-Computer Interaction, 2014, Vol- ume 8, 4 issues. ISSN paper version 1551-3955. ISSN online version 1551-3963. Also available as a combined paper and online subscription. Full text available at: http://dx.doi.org/10.1561/1100000049 R Foundations and Trends• in Human-Computer Interaction Vol. 8, No. 2-3 (2014) 73–272 c 2015 M. Billinghurst, A. Clark, and G. Lee • DOI: 10.1561/1100000049 A Survey of Augmented Reality Mark Billinghurst, Adrian Clark, and Gun Lee The Human Interface Technology Laboratory New Zealand University of Canterbury Christchurch, New Zealand {mark.billinghurst, adrian.clark, gun.lee}@hitlabnz.org Full text available at: http://dx.doi.org/10.1561/1100000049 Contents 1 Introduction 2 2 Definition and Taxonomy 5 3 History 13 4 AR Tracking Technology 31 4.1 Magnetic Tracking . 32 4.2 Vision Based Tracking . 33 4.3 Inertial Tracking . 50 4.4 GPS Tracking . 50 4.5 Hybrid Tracking . 51 4.6 Summary . 53 5 AR Display Technology 54 5.1 Combining Real and Virtual View Images . 54 5.2 Eye-to-World spectrum . 67 5.3 Other Sensory Displays . 72 5.4 Summary . 74 6 AR Development Tools 75 6.1 Low Level software libraries and frameworks . 76 ii Full text available at: http://dx.doi.org/10.1561/1100000049 iii 6.2 Rapid Prototyping/Development Tools . 81 6.3 Plug-ins to Existing Developer Tools . 84 6.4 Stand Alone AR Authoring Tools . 86 6.5 Summary . 91 7 AR Input and Interaction Technologies 93 7.1 AR Information Browsers . 94 7.2 3D User Interfaces in AR . 95 7.3 Tangible User Interface in AR . 97 7.4 Natural User Interfaces in AR . 100 7.5 Multimodal Interaction in AR . 102 7.6 Other Interaction Methods . 104 7.7 Summary . 105 8 Design Guidelines and Interface Patterns 107 8.1 Case Study: levelHead . 111 8.2 Using Design Patterns . 112 8.3 Designing for Special Needs . 115 8.4 Summary . 116 9 Evaluation of AR Systems 117 9.1 Types of User Studies . 119 9.2 Evaluation Methods . 131 9.3 Summary . 134 10 AR Applications Today 135 10.1 Education . 135 10.2 Architecture . 139 10.3 Marketing . 145 11 Research Directions 147 11.1 Tracking . 148 11.2 Interaction . 151 11.3 Displays . 157 11.4 Social Acceptance . 159 11.5 Summary . 163 Full text available at: http://dx.doi.org/10.1561/1100000049 iv 12 Conclusion 164 Acknowledgements 166 References 167 Full text available at: http://dx.doi.org/10.1561/1100000049 Abstract This survey summarizes almost 50 years of research and development in the field of Augmented Reality (AR). From early research in the 1960’s until widespread availability by the 2010’s there has been steady progress towards the goal of being able to seamlessly combine real and virtual worlds. We provide an overview of the common definitions of AR, and show how AR fits into taxonomies of other related technolo- gies. A history of important milestones in Augmented Reality is fol- lowed by sections on the key enabling technologies of tracking, display and input devices. We also review design guidelines and provide some examples of successful AR applications. Finally, we conclude with a summary of directions for future work and a review of some of the areas that are currently being researched. M. Billinghurst, A. Clark, and G. Lee. ASurveyofAugmentedReality. Foundations R and Trends• in Human-Computer Interaction, vol. 8, no. 2-3, pp. 73–272, 2014. DOI: 10.1561/1100000049. Full text available at: http://dx.doi.org/10.1561/1100000049 1 Introduction In 1977 many moviegoers were amazed as a small robot projected a three-dimensional image of a woman in mid air. With the words "Help me Obiwan-Kenobi, you’re my only hope", a recording of Princess Leia delivered a message that would change Luke Skywalker’s life forever. In this Star Wars1 scene, special effects were used to create the magi- cal impression that three-dimensional virtual content was appearing as part of the real world. The movie forecast a future where people could interact with computers as easily as interacting with the real world around them, with digital and physical objects existing in the same space. Thirty years later, in the 2008 US presidential campaign, a ver- sion of technology was shown for real. During the CNN election cov- erage reporter Wolf Blitzer turned to an empty studio and suddenly a life sized three-dimensional virtual image of reporter Jessica Yellin appeared beamed in live from Chicago2. Just like Princess Leia, she appeared to be part of the real world, but this time it was real and not through movie special effects. Wolf was able to talk to her as easily 1http://www.starwars.com 2http://edition.cnn.com/2008/TECH/11/06/hologram.yellin/ 2 Full text available at: http://dx.doi.org/10.1561/1100000049 3 as if there was there face to face, even though she was thousands of miles away. It had taken only thirty years for the Star Wars fantasy to become reality. The CNN experience is an example of technology known as Aug- mented Reality (AR), which aims to create the illusion that virtual images are seamlessly blended with the real world. AR is one of the most recent developments in human computer interaction technology. Ever since the creation of the first interactive computers there has been a drive to create intuitive interfaces. Beginning in the 1960’s, computer input has changed from punch cards, to teletype, then mouse and key- board, and beyond. One overarching goal is to make the computer interface invisible and make interacting with the computer as natural as interacting with real world objects, removing the separation between the digital and physical. Augmented Reality is one of the first technolo- gies that makes this possible. Star Wars and CNN showed how the technology could enhance communication and information presentation, but like many enabling technologies, AR can be used in a wide variety of application domains.