Image Retrieval within Augmented Reality Philip Manja May 5, 2017 Technische Universität Dresden Fakultät Informatik Institut für Software und Multimediatechnik Professur für Multimedia-Technologie Master’s Thesis Image Retrieval within Augmented Reality Philip Manja 1. Reviewer Prof. Raimund Dachselt Fakultät Informatik Technische Universität Dresden 2. Reviewer Dr. Annett Mitschick Fakultät Informatik Technische Universität Dresden Supervisors Dr. Annett Mitschick and Wolfgang Büschel (M.Sc.) May 5, 2017 Philip Manja Image Retrieval within Augmented Reality Master’s Thesis, May 5, 2017 Reviewers: Prof. Raimund Dachselt and Dr. Annett Mitschick Supervisors: Dr. Annett Mitschick and Wolfgang Büschel (M.Sc.) Technische Universität Dresden Professur für Multimedia-Technologie Institut für Software und Multimediatechnik Fakultät Informatik Nöthnitzer Straße 46 01187 Dresden Abstract The present work investigates the potential of augmented reality for improving the image retrieval process. Design and usability challenges were identified for both fields of research in order to formulate design goals for the development of concepts. A taxonomy for image retrieval within augmented reality was elaborated based on research work and used to structure related work and basic ideas for interaction. Based on the taxonomy, application scenarios were formulated as further requirements for concepts. Using the basic interaction ideas and the requirements, two comprehensive concepts for image retrieval within augmented reality were elaborated. One of the concepts was implemented using a Microsoft HoloLens and evaluated in a user study. The study showed that the concept was rated generally positive by the users and provided insight in different spatial behavior and search strategies when practicing image retrieval in augmented reality. Abstract (deutsch) Die vorliegende Arbeit untersucht das Potenzial von Augmented Reality zur Verbes- serung von Image Retrieval Prozessen. Herausforderungen in Design und Gebrauchs- tauglichkeit wurden für beide Forschungsbereiche dargelegt und genutzt, um De- signziele für Konzepte zu entwerfen. Eine Taxonomie für Image Retrieval in Aug- mented Reality wurde basierend auf der Forschungsarbeit entworfen und eingesetzt, um verwandte Arbeiten und generelle Ideen für Interaktionsmöglichkeiten zu struk- turieren. Basierend auf der Taxonomie wurden Anwendungsszenarien als weitere Anforderungen für Konzepte formuliert. Mit Hilfe der generellen Ideen und An- forderungen wurden zwei umfassende Konzepte für Image Retrieval in Augmented Reality ausgearbeitet. Eins der Konzepte wurde auf einer Microsoft HoloLens umge- setzt und in einer Nutzerstudie evaluiert. Die Studie zeigt, dass das Konzept grund- sätzlich positiv aufgenommen wurde und bietet Erkenntnisse über unterschiedliches Verhalten im Raum und verschiedene Suchstrategien bei der Durchführung von Image Retrieval in der erweiterten Realität. v Contents 1 Introduction 1 1.1 Motivation and Problem Statement . 1 1.1.1 Augmented Reality and Head-Mounted Displays . 1 1.1.2 Image Retrieval . 2 1.1.3 Image Retrieval within Augmented Reality . 3 1.2 Thesis Structure . 4 2 Foundations of Image Retrieval and Augmented Reality 5 2.1 Foundations of Image Retrieval . 5 2.1.1 Definition of Image Retrieval . 5 2.1.2 Classification of Image Retrieval Systems . 6 2.1.3 Design and Usability in Image Retrieval . 10 2.2 Foundations of Augmented Reality . 16 2.2.1 Definition of Augmented Reality . 16 2.2.2 Augmented Reality Design and Usability . 17 2.3 Taxonomy for Image Retrieval within Augmented Reality . 22 2.3.1 Session Parameters . 23 2.3.2 Interaction Process . 26 2.3.3 Summary of the Taxonomy . 30 3 Concepts for Image Retrieval within Augmented Reality 33 3.1 Related Work . 33 3.1.1 Natural Query Specification . 33 3.1.2 Situated Result Visualization . 38 3.1.3 3D Result Interaction . 41 3.1.4 Summary of Related Work . 43 3.2 Basic Interaction Concepts for Image Retrieval in Augmented Reality 44 3.2.1 Natural Query Specification . 44 3.2.2 Situated Result Visualization . 46 3.2.3 3D Result Interaction . 48 3.3 Requirements for Comprehensive Concepts . 50 3.3.1 Design Goals . 50 3.3.2 Application Scenarios . 52 3.4 Comprehensive Concepts . 55 vii 3.4.1 Tangible Query Workbench . 55 3.4.2 Situated Photograph Queries . 57 3.4.3 Conformance of Concept Requirements . 59 4 Prototypic Implementation of Situated Photograph Queries 63 4.1 Implementation Design . 63 4.1.1 Implementation Process . 63 4.1.2 Structure of the Implementation . 67 4.2 Developer and User Manual . 71 4.2.1 Setup of the Prototype . 71 4.2.2 Usage of the Prototype . 72 4.3 Discussion of the Prototype . 73 5 Evaluation of Prototype and Concept by User Study 75 5.1 Design of the User Study . 75 5.1.1 Usability Testing . 75 5.1.2 Questionnaire . 78 5.2 Results . 81 5.2.1 Logging of User Behavior . 81 5.2.2 Rating through Likert Scales . 84 5.2.3 Free Text Answers and Remarks during the Study . 88 5.2.4 Observations during the Study . 90 5.2.5 Discussion of Results . 91 6 Conclusion 93 6.1 Summary of the Present Work . 93 6.2 Outlook on Further Work . 95 Bibliography 97 viii Introduction 1 The aim of the present work is to investigate the possibility of improving the image retrieval process with the help of augmented reality (AR). Being an entry to the thesis, section 1.1 motivates why such a connection between the two fields has potential. Section 1.2 is then giving an overview on the methodical approach and the resulting structure of the present work. 1.1 Motivation and Problem Statement Connecting image retrieval with AR can provide advantages for the image retrieval process by making use of novel interaction techniques. On one side, head-mounted displays (HMDs) are currently on the rise with several devices introduced during the last years. Such devices offer a totally different way of interacting with data (see section 1.1.1). On the other side, the importance of image retrieval is rising with the growing number of images. A new form of interaction provides the possibility for image retrieval to adopt to the changing needs of people (see section 1.1.2). A conjunction of both field conforms to a general trend towards natural search interfaces and can provide a direct way of retrieving data when information needs are triggered by the surroundings (see section 1.1.3). 1.1.1 Augmented Reality and Head-Mounted Displays Despite of the rapid development of technology and computer devices in the last decades there have only been minor changes in the way we interact with them. Desktop and laptop devices mostly still rely on the usage of a keyboard and a mouse, both of which have been introduced a long time ago. Only recently, with the intro- duction of powerful mobile devices, other interaction techniques like multi-touch were brought into our everyday life. This development shows that a fundamental change of interaction will probably rather likely be closely related to an introduction of new devices. Mobile devices are not only an addition to the range of such devices. They rather depict an ongoing trend towards mobile and everywhere computing, with mobile usage of digital media overtaking desktops in 2013 [LL14]. A part of this develop- ment is the growing popularity of wearable devices such as smart watches and the recent introduction of HMD. While virtual reality (VR) headsets like Oculus Rift and 1 HTC Vive are mostly aimed at gaming and entertainment, AR outfits like Meta 2 or Microsoft HoloLens claim to change the way of everyday interaction with computer devices. This vision implies a shift of our computer activities to holographic devices similar to the one from stationary to mobile devices during the last decade. That shift would take interaction from surfaces into space. 3D interaction has been a topic in research for decades [Han97], however there are still no well-known and proven sets of gestures as there are for multi-touch surfaces. Therefore, in order to help establish such proven ways of interaction, the field of AR, especially the usage of HMDs, is an important recent field of research. 1.1.2 Image Retrieval Humans are very visual beings. Whole social media platforms like YouTube, Insta- gram and Snapchat are built around the presentation and sharing of visual content. Nowadays, every owner of a modern mobile phone is able to shoot photos and videos in nearly every situation. The ever increasing amount of images, on the Internet as well as in every single personal collection, inherently makes the process of searching and retrieving more important than ever. Additionally, because different content is often represented by some kind of visual cue, research conducted in image retrieval can also be transfered to other domains. The way we retrieve images today does not differ greatly from what it used to be 15 years ago. When searching for pictures on the Internet, keyword-based search is still the main kind of interaction. Browsing personal images is mostly done with pictures organized in folders. On the one hand, this might show that other forms of search are simply not sufficient enough. As Marti A. Hearst points out, search interfaces are used by nearly everyone, therefore having to rely on simple and understandable kinds of interaction [Hea09]. However there are some shortcomings of those classical ways of interaction. One of them is the semantic gap, which describes the discrepancy between the understanding of a picture from a user point of view and a system point of view. This problem has been a part of research for many years [Sme+00]. One central approach to reduce the gap is to improve the system’s understanding of the pictures, for example in the form of content-based image retrieval (CBIR). Today’s advanced computer vision techniques are making such systems relatively reliable and robust.