Design and Implementation of Post-WIMP Interactive Spaces with the ZOIL Paradigm Dissertation zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) vorgelegt von Hans-Christian Jetter an der Universität Konstanz Mathematisch-Naturwissenschaftliche Sektion Fachbereich Informatik und Informationswissenschaft Tag der mündlichen Prüfung: 22. März 2013 1. Referent: Prof. Dr. Harald Reiterer (Universität Konstanz) 2. Referent: Prof. Dr. Andreas Butz (LMU München) Prüfungsvorsitzender: Prof. Dr. Daniel Keim (Universität Konstanz) Konstanz, Dezember 2013 Design and Implementation of Post-WIMP Interactive Spaces with the ZOIL Paradigm Dissertation submitted for the degree of Doctor of Natural Sciences (Dr. rer. nat.) Presented by Hans-Christian Jetter at the University of Konstanz Faculty of Sciences Department of Computer and Information Science Date of the oral examination: March 22nd, 2013 1st Supervisor: Prof. Dr. Harald Reiterer (University of Konstanz) 2nd Supervisor: Prof. Dr. Andreas Butz (University of Munich, LMU) Chief Examiner: Prof. Dr. Daniel Keim (University of Konstanz) Konstanz, December 2013 i Zusammenfassung Post-WIMP (post-“Windows, Icons, Menu, Pointer”) interaktive Räume sind physische Umgebungen oder Räume für die kollaborative Arbeit, die mit Technologien des Ubiquitous Computing angereichert sind. Ihr Zweck ist es, eine computer-unterstützte Kollaboration mehrerer Benutzer zu ermöglichen, die auf einer nahtlosen Benutzung mehrerer Geräte und Bildschirme mittels „natürlicher“ post-WIMP Interaktion basiert. Diese Dissertation beantwortet die Forschungsfrage, wie Gestalter und Entwickler solcher Umgebungen dabei unterstützt werden können, gebrauchstaugliche interaktive Räume für mehrere Benutzer mit mehreren Geräten zu erschaffen, die eine kollaborative Wissensarbeit ermöglichen. Zu diesem Zweck werden zunächst Konzepte wie post-WIMP Interaktion, interaktive Räume und Wissensarbeit definiert. Die Arbeit formuliert dann das neue technologische Paradigma ZOIL (Zoomable Object-Oriented Information Landscape). Dieses ZOIL Paradigma ist der Hauptbeitrag dieser Arbeit und besteht aus drei Komponenten: 1.) Die sechs ZOIL Gestaltungsprinzipien, welche die Art der Benutzerinteraktion mit ZOIL Benutzungsschnittstellen definieren und Interaktionsdesigner mit „goldenen Regeln“ unterstützen. 2.) Das ZOIL Software Framework, das Entwickler während der Implementierung von post-WIMP interaktiven Räumen für die kollaborative Wissensarbeit unterstützt und die praktische Umsetzung der ZOIL Gestaltungsprinzipien ermöglicht. 3.) Die vier auf ZOIL basierenden Beispiel-Prototypen, die Gestaltern und Entwicklern gleichermaßen als Anschauungsobjekte dienen können. Jedes der sechs ZOIL Gestaltungsprinzipien ist aus existierender Literatur hergeleitet. Dazu wurde Literatur aus den Disziplinen Mensch-Computer Interaktion, Ubiquitous Computing, Informationsvisualisierung, Computerunterstützte Gruppenarbeit, Kognitionswissenschaft, Persönliches Informationsmanagement und Software Engineering herangezogen. Die Formulierung der Prinzipien wird durch eigene Erkenntnisse während der Erstellung der ZOIL Beispiel-Prototypen für verschiedene Anwendungsdomänen (z.B. e- Science, kollaborative Suche, Produktdesign) und durch die Ergebnisse von eigenen Benutzerstudien empirisch validiert und erweitert. Das neue quelloffene ZOIL Software Framework dient dem Zweck der Implementierung von post-WIMP Interaktiven Räumen entsprechend den ZOIL Gestaltungsprinzipien. Dieses Software Framework wird anhand seiner Architektur und Design Patterns vorgestellt und mit Hinblick auf seine Gebrauchstauglichkeit und seinen praktischen Nutzen für Entwickler im Rahmen einer API-Gebrauchstauglichkeits-Studie evaluiert. ii iii Abstract Post-WIMP (post-“Windows, Icons, Menu, Pointer”) interactive spaces are physical environments or rooms for collaborative work that are augmented with ubiquitous computing technology. Their purpose is to enable a computer-supported collaboration of multiple co-located users that is based on a seamless use of and “natural” interaction with multiple devices and displays. This thesis answers the research question how the designers and developers of such ubiquitous computing environments can be supported to create more usable multi-user and multi-device post-WIMP interactive spaces for co-located collaborative knowledge work. For this purpose, it first defines concepts such as post-WIMP interaction, interactive spaces, and knowledge work. Then it formulates the novel ZOIL (Zoomable Object- Oriented Information Landscape) technological paradigm. The ZOIL paradigm is the main contribution of this thesis and consists of three components: 1.) The six ZOIL design principles that define ZOIL’s interaction style and provide “golden rules” to support interaction designers. 2.) The ZOIL software framework that supports developers during the implementation of post-WIMP interactive spaces for collaborative knowledge work and enables the realization of ZOIL’s design principles in practice. 3.) The four example prototypes based on ZOIL that can serve as exemplars for designers and developers likewise. Each of the six ZOIL design principles is derived from literature of disciplines related to Human-Computer Interaction including Ubiquitous Computing, Information Visualization, Computer-supported Cooperative Work, Cognitive Science, Personal Information Management, and Software Engineering. The formulation of the ZOIL design principles is empirically validated and extended based on the experiences and the findings from own user studies during creating and deploying the four ZOIL example prototypes for different application domains (e.g., e- Science, collaborative search, creative design). The new open-source ZOIL software framework serves the implementation of post-WIMP interactive spaces that follow the ZOIL design principles. This software framework is described in its architecture and software design patterns and is evaluated in terms of its usability and practical value for developers in an API usability evaluation study. iv v Acknowledgements It feels almost impossible to name all the different people who helped to shape this work and directly or indirectly contributed to it. During my years in the Human-Computer Interaction Group at the University of Konstanz, I met and collaborated with so many different inspiring persons in so many different contexts that it is now hard, if not impossible, to remember and name them all or to order them by their importance. Here, I want to thank all of them, including those who I might have forgotten. First of all, I would like to thank Michael Zöllner and Andreas Engl without whom the ZOIL prototypes and the ZOIL software framework would not have become reality. I was very fortunate to convince them to work in my project when they were Bachelor students, because they understood my conceptual goals for ZOIL from the beginning and how I wanted these goals to be reflected in software architecture and program code. In their code, they were always willing to go the extra mile of not just making things work but sharing or exceeding my standards for readablility, accessibility, and maintainability. This is particularly true for Michael and his excellent implementation work on ZOIL’s data backend, MVVM pattern, and the Facet-Streams prototype. In connection with Facet-Streams, I also would like to thank my fellow PhD candidate Jens Gerken. I cannot remember a time as a researcher during which I have been more productive than during the time that Jens, Michael, and I spent at Microsoft Research Cambridge in Winter 2009. Facet-Streams would not have become reality without the many days and nights that we have spent discussing, designing, coding, and testing Facet-Streams together – and still being able to enjoy Cambridge’s pub culture and pub food now and then. Furthermore, throughout my PhD, Jens was always a constant source of invaluable input concerning user studies and experimental designs. Thanks to him, I could study the collaborative use of Facet-Streams and ZOIL’s API usability under realistic conditions. In a similar fashion, Svenja Leifert was irreplaceable and helped me to greatly refine the initial experimental setup and data analysis of Sören Schubert for my ZUI navigation study. Therefore Svenja and Sören deserve my great gratitude. There were also many people who helped me to demonstrate ZOIL and its software framework with great prototypes, most notably Mathias Heilig, Mischa Demarmels, Sebastian Rexhausen, Stephan Huber, Oliver Runge, and Benjamin Frantzen. They used the ZOIL design principles and an early version of the ZOIL software framework to create their own impressive prototypes (e.g., MedioVis 2.0) in their Bachelor’s, Master’s or PhD theses. Furthermore, they helped to create the Media Seminar Room prototype. I would also like to thank Jonas Schweizer for his design and implementation of a multi-focus ZUI for ZOIL as part of his Bachelor’s thesis that I supervised. Later in the PhD project, it was Florian Geyer and his team of student developers suc as Anita Höchtl or Jochen Budzinski who used the ZOIL design principles and pushed the ZOIL software framework to its limits and created award-winning ZOIL prototypes such as Affinity Table. vi Furthermore, I would like to especially thank Toni Schmidt for the implementation of the DeskPiles prototype that we deployed at the University of Cambridge and for having the patience for realizing the countless changes that