LUDWIG-MAXIMILIANS-UNIVERSITÄT MÜNCHEN Department “Institut für Informatik” Lehr- und Forschungseinheit Medieninformatik Prof. Dr. Heinrich Hußmann Diplomarbeit Informed Browsing of Digital Image Collections Dominikus Baur [email protected]fi.lmu.de Bearbeitungszeitraum: 4. 4. 2007 bis 28. 9. 2007 Betreuer: Dipl. Inf. Otmar Hilliges Verantw. Hochschullehrer: Prof. Dr. Andreas Butz Abstract The proliferation of digital cameras and their low operating costs lead to an abundance of digital photos that are not only laborious to organize but also lack the tangibility of their physical coun- terparts. Informed Browsing tries to ease the organization of a large media collection by applying automatic analysis methods. Tabletop displays allow for a more direct handling of virtual objects and facilitate multi-user interaction and communication. In the course of my diploma thesis I developed and implemented an application, flux, that takes concepts from Informed Browsing and puts them in a tabletop interface to combine a convenient, physical treatment of photos with the advantages of digital information: manipulability and easier organization. This work presents the basic motivation, the idea of Informed Browsing and its roots in the pho- towork process, related work in the same field, the design process with multiple iterations and user feedback in form of a focus group and the implementation with selected aspects and the hardware and technology side. Zusammenfassung Durch die weite Verbreitung von Digitalkameras und ihre niedrigen Betriebskosten entstehen eine Unmenge von digitalen Fotos, die nicht nur aufwändig zu organisieren sind, sondern auch die Greifbarkeit ihrer physikalischen Gegenstücke missen lassen. Informed Browsing versucht, die Organisation großer Mediensammlungen zu vereinfachen, indem es automatische Analysever- fahren anwendet. Tabletop Bildschirme erlauben eine direktere Handhabung von virtuellen Ob- jekten und erleichtern Kommunikation und Zusammenspiel zwischen mehreren Benutzern. Im Zuge meiner Diplomarbeit habe ich eine Anwendung, flux, entworfen und implementiert, die Konzepte aus Informed Browsing nimmt und sie auf Tabletop Computer portiert, um einen an- genehmen, physikalischen Umgang mit Fotos mit den Vorteilen von digitalen Daten, Anpass- barkeit und einfachere Organisation, zu verbinden. Diese Arbeit stellt die grundlegende Motivation, die Idee hinter Informed Browsing und seine Wurzeln im Arbeiten mit Fotos, verwandte Arbeiten im gleichen Bereich, den Designprozess, der in mehreren Stufen stattfand, und Nutzerresonanz dazu in Form einer Fokusgruppe und schließlich die Implementierung mit ausgewählten Aspekten und die benutzte Hardware und andere ver- wandte Technologien dar. Aufgabenstellung The Fluidum project investigates interaction with different types of information units in instru- mented environments. In order to represent and manipulate potentially large collections of data, interactive visual representations are needed. These visualizations should support searching, grouping, categorizing and sharing of data and should work in different contexts of usage (single user, co-located collaboration on a shared display, remote collaboration, etc.). Recently more and more technology is on the leap from our working environments into our living rooms. Large screen flat TVs, internet enabled set top boxes and media-center PCs are only the beginning. With more screen real estate and more content available at our finger tips we need to think about new ways to organize, access and interact with digital media. To browse large photo collections efficiently a new technology to gain an overview of the collections’ content has to be developed. One possible approach would be to use state of the art information retrieval technologies to pre-cluster image collections (e.g., by object, person, event) in order to reduce visual clutter. Other IR techniques could be used to inform following information selection processes, such as deciding which images to keep vs. which images to delete. The student has to develop visualization of a digital image collection that allows, browsing, organizing, and sharing (as in story telling) and of course viewing of digital photos. Further a matching interaction technique, on the interactive table, has to be developed. Finally an evaluation of the prototype needs to be conducted. Ich erkläre hiermit, dass ich die vorliegende Arbeit selbstständig angefertigt, alle Zitate als solche kenntlich gemacht sowie alle benutzten Quellen und Hilfsmittel angegeben habe. München, July 18, 2008 ......................................... Acknowledgements I would like to thank all the people who made this work possible: Many thanks go to my tutor, Otmar Hilliges, who I could always rely on and who was a perma- nent source of creative and constructive input and never grew tired of discussing everything from theoretical background to implementation details. I would also like to thank Peter Kunath and Alexey Pryakhin for their preparation and help with the image analysis tools and for sacrificing their Friday evening to investigate weird OPTICS re- sults. Andreas Butz, for good ideas and undismayed help when I wondered about odd things, like how to best draw metaballs. The members of the focus group, who bore with me for an exhausting two and a half hours on a hot August afternoon and all the time stayed in good humour and had productive ideas. Additional thanks go to the members of the Fluidum team and the other members of the Media Informatics Group for their comments and input on my project. A great Thank You to my family who supported me not only financially and morally, but also helped me keep things in perspective. And last but not least I want to thank Bettina Zech for helping me through difficult times and, of course, for building piles. Contents 1 Introduction1 1.1 Motivation......................................2 1.2 Short overview of the thesis.............................3 2 Informed Browsing5 2.1 Photowork......................................5 2.2 Approaches for filing................................6 2.3 Informed Browsing.................................7 3 Related Work 11 3.1 Desktop Photo applications............................. 11 3.1.1 Applications for hobbyists......................... 11 3.1.2 Academic background........................... 12 3.1.3 Software for professionals......................... 13 3.1.4 Disadvantages............................... 14 3.2 Tabletop Photo applications............................. 14 3.2.1 Photo applications............................. 15 3.2.2 Disadvantages............................... 16 3.3 Informed Browsing applications.......................... 16 4 Design of flux 18 4.1 Hardware prerequisites and target group...................... 18 4.2 Usage scenarios................................... 18 4.2.1 Alone at home to organize or randomly browse through a collection... 19 4.2.2 Showing photos to friends and family................... 19 4.2.3 Exchanging photos with a friend...................... 20 4.3 Requirements.................................... 20 4.3.1 Support the photowork tasks........................ 20 4.3.2 Support multiple concurrent users..................... 21 4.3.3 Work with the characteristics of the hardware............... 21 4.3.4 Support a regular-sized photo collection.................. 21 4.3.5 Prevent the users from getting lost..................... 21 4.3.6 Reduce visual clutter............................ 22 4.4 First Design..................................... 22 4.4.1 Initial thoughts............................... 22 4.4.2 Design evolution.............................. 25 4.5 Main concepts, screen elements and interaction.................. 28 4.5.1 Overall interaction concepts........................ 29 4.5.2 Types of objects and specific interaction.................. 33 4.5.3 Additional features............................. 36 4.6 Focus Group..................................... 37 4.6.1 Outcome.................................. 38 4.6.2 Focus Group Prototype........................... 39 4.6.3 Feedback from the participants....................... 40 4.6.4 Lessons learned............................... 42 4.7 Second, refined and final design.......................... 42 4.7.1 Conceptual changes............................ 42 4.7.2 Features dropped due to time constraints.................. 45 4.8 Flux in the context of Informed Browsing..................... 46 I 5 Implementation 48 5.1 Hardware...................................... 48 5.2 Prototype...................................... 49 5.2.1 Interaction................................. 49 5.2.2 Results................................... 49 5.3 Basic program structure of flux........................... 50 5.3.1 Class structure............................... 50 5.3.2 Important aspects of the implementation.................. 53 5.3.3 Data flow in concrete situations...................... 59 5.3.4 Interaction with persistent data....................... 65 5.4 Used Technology.................................. 66 5.4.1 Microsoft .NET framework and Direct3D................. 66 5.4.2 Physical simulation via Ageia PhysX................... 67 5.4.3 Automated Feature Extraction and Quality analysis............ 70 5.4.4 Other technology.............................. 71 5.5 Selected algorithms................................. 71 5.5.1 Moving objects............................... 71 5.5.2 Metaballs.................................. 72 5.5.3 Arrangements...............................