Graz University of Technology Institute for Computer Graphics and Vision Dissertation WIMP Interfaces for Emerging Display Environments Manuela Waldner Graz, Austria, June 2011 Thesis supervisor Prof. Dr. Dieter Schmalstieg Referee Prof. Dr. Andreas Butz To Martin Abstract With the availability of affordable large-scale monitors and powerful projector hardware, an increasing variety of display configurations can be found in our everyday environments, such as office spaces and meeting rooms. These emerging display environments combine conventional monitors and projected displays of different size, resolution, and orientation into a common interaction space. However, the commonly used WIMP (windows, icons, menus, and pointers) user interface metaphor is still based on a single pointer operating multiple overlapping windows on a single, rectangular screen. This simple concept cannot easily capture the complexity of heterogeneous display settings. As a result, the user cannot facilitate the full potential of emerging display environments using these interfaces. The focus of this thesis is to push the boundaries of conventional WIMP interfaces to enhance information management in emerging display environments. Existing and functional interfaces are extended to incorporate knowledge from two layers: the physical environment and the content of the individual windows. The thesis first addresses the tech- nical infrastructure to construct spatially aware multi-display environments and irregular displays. Based on this infrastructure, novel WIMP interaction and information presenta- tion techniques are demonstrated, exploiting the system's knowledge of the environment and the window content. These techniques cover two areas: spatially-aware cross-display navigation techniques for efficient information access on remote displays and window man- agement techniques incorporating knowledge of display form factors and window content to support information discovery, manipulation, and sharing. Results of user studies indicate that environment- and content-awareness of WIMP interfaces indeed improves information management in many varieties of emerging display environments. Awareness of the environment and the window content thereby can be facilitated without an obtrusive and expensive hardware and software infrastructure. Keywords. WIMP Interfaces; Multi-Display Environments; Irregular Displays; Large- Scale Displays; Information Management; Human-Computer Interaction; Computer- Supported Cooperative Work; Window Management; Information Presentation; Cross- Display Navigation; Windowing Systems v vii Statutory Declaration I declare that I have authored this thesis independently, that I have not used other than the declared sources / resources, and that I have explicitly marked all material which has been quoted either literally or by content from the used sources. Place Date Signature Eidesstattliche Erkl¨arung Ich erkl¨are an Eides statt, dass ich die vorliegende Arbeit selbstst¨andigverfasst, andere als die angegebenen Quellen/Hilfsmittel nicht benutzt, und die den benutzten Quellen w¨ortlich und inhaltlich entnommene Stellen als solche kenntlich gemacht habe. Ort Datum Unterschrift Acknowledgments There is a large number of people without whom this thesis would not have been pos- sible and many people that directly or indirectly supported my work through technical, scientific, or personal support. First of all, I want to acknowledge my supervisor Dieter Schmalstieg who gave me the opportunity to join his research group at Graz University of Technology. I particularly want to thank him for letting me pursue my own goals { despite often going far away from the actual research agenda of this institute { while still guiding and supporting my work. I also want to thank my referee Andreas Butz from Ludwig- Maximilian University in Munich for his spontaneous and uncomplicated agreement to support this thesis and our precedent discussions at conferences. In addition, I want to acknowledge all my colleagues of the Deskotheque project. First of all, I want to thank Christian Pirchheim for teaching me so many things about Linux, for the countless shell scripts, for doing so much \dirty" work in our lab, and for endless discussions { not necessarily limited to the project. I also want to point out the short but fruitful collaboration with Werner Puff who made valuable contributions to the visual links infrastructure and was also a great help for porting the Deskotheque framework. Finally, Markus Steinberger was more than just a \coding monkey" for this project. Still, I first want to point out his stunning ability to implement high-performance solutions to complex optimization problems in minimum time. I also want to thank him for his general support in the last year of the project. Apart from the project members, there are many other colleagues at ICG that deserve being mentioned at this point, for discussions and support: Alexander Lex and Marc Streit, Albert Walzer, Raphael Grasset and Ernst Kruijff, Mark Dokter, Markus Murschitz, Werner Trobin, Matthias R¨uther,and Christopher Zach, Erick Mendez, Markus Tatzgern, Andreas Wurm, and many more. A special thank, of course, goes to all the participants of our user studies, their patience and all their valuable feedback! This is probably also the right spot to also acknowledge all the Linux open source developers { especially the Compiz and multi-pointer X developers. Finally, I want to thank my family who made this work possible by believing in me all these years. I want to gratefully mention my partner Martin for always being there for me, for all his support and endless patience. The least I can do is dedicate this thesis to you. ix Contents I Overview 1 1 Introduction 3 1.1 Emerging Display Factors . 5 1.2 WIMP Interfaces . 6 1.3 Information Management . 7 1.4 Research Hypothesis . 13 1.5 Contribution . 13 1.6 Collaboration Statement . 14 2 Background and Related Work 21 2.1 Visual Information Management . 21 2.2 Display Environments . 36 2.3 Enabling Technologies . 54 II Multi-Display Framework 57 3 System Infrastructure of the Deskotheque Environment 59 3.1 Requirements and Design Principles . 59 3.2 Distributed Software Infrastructure . 61 3.3 Discussion . 65 4 Spatial Awareness in the Deskotheque Environment 67 4.1 Spatial Model Creation . 68 4.2 Compensation of Projected Displays . 79 4.3 User Location Estimation . 81 4.4 Multi-Display Coordinate Systems . 82 4.5 Discussion . 88 xi xii CONTENTS III Multi-Display Navigation 91 5 Cross-Display Navigation Techniques 93 5.1 Design Space . 94 5.2 Mouse Pointer Navigation Infrastructure . 96 5.3 Exemplary Navigation Techniques . 101 5.4 Cross-Display Information Sharing . 104 5.5 Discussion . 105 6 Evaluations of Multi-Display Navigation 107 6.1 Comparison of Cross-Display Navigation Techniques . 107 6.2 Comparison of Pointer Warping and Seamless Navigation . 119 6.3 Comparison of Outcome Positions for Pointer Warping . 128 6.4 Discussion . 133 IV Window Management for Emerging Display Environments 135 7 Window Manager Extensions 137 7.1 Multi-User Interaction and Identification . 137 7.2 Accessing Window Content . 138 7.3 Display Adaptivity . 143 7.4 Importance-Driven Compositing Window Management . 147 7.5 Discussion . 152 8 Window Management Techniques 155 8.1 Visual Links . 155 8.2 Uncovering Windows . 164 8.3 Display-Adaptive Window Management . 166 8.4 Polarization-Based Interfaces . 174 8.5 Discussion . 179 9 Evaluations of Window Management Techniques 183 9.1 Exploratory Evaluation of Visual Links across Applications . 184 9.2 Exploratory Evaluation of Collaborative Information Linking . 187 9.3 Comparison of Techniques to Discover and Access Occluded Windows . 195 9.4 Exploratory Evaluation of Window Management on Irregular Displays . 201 9.5 Discussion . 212 CONTENTS xiii V Summary 215 10 Discussion and Future Directions 217 10.1 Information Discovery . 217 10.2 Information Access . 220 10.3 Information Manipulation . 221 10.4 Information Sharing . 223 11 Conclusion 225 Bibliography 228 Part I Overview 1 Chapter 1 Introduction In recent years, display hardware has become more affordable and more powerful, in terms of resolution, size, and brightness. As a result, we can see a variety of display configurations in our everyday working environments, living rooms, meeting rooms, control rooms, and public spaces. Examples range from already quite common dual monitor configurations connected to a work PC to complex control room setups combining multiple personal and shared display spaces into a common interaction space. Of particular interest are office environments leveraging projection technologies to augment multiple isolated single-user workspaces to a shared interaction space (Figure 1.1). Figure 1.1: Multi-display environments combine projected displays and monitors of different size, resolution, and orientation into a common interaction space. Yet, the WIMP (windows, icons, menus, and pointers) metaphor has remained largely unchanged since its introduction more than 30 years ago. The Smalltalk system (1979) [89] and the Xerox Star (1981) [132] were the first systems allowing users to show multiple pieces of information (views [89]) simultaneously on the screen (Figure 1.2(a)). Today, WIMP interfaces mainly rely on the same fundamental concepts: they provide multi- 3 4 Chapter 1. Introduction ple overlapping, rectangular windows to physically separate different applications on the screen (Figure 1.2(b)) and a single pointer to access and manipulate their