Umar Rashid Phd Thesis
Total Page:16
File Type:pdf, Size:1020Kb
CROSS-DISPLAY ATTENTION SWITCHING IN MOBILE INTERACTION WITH LARGE DISPLAYS Umar Rashid A Thesis Submitted for the Degree of PhD at the University of St. Andrews 2012 Full metadata for this item is available in Research@StAndrews:FullText at: http://research-repository.st-andrews.ac.uk/ Please use this identifier to cite or link to this item: http://hdl.handle.net/10023/3193 This item is protected by original copyright This item is licensed under a Creative Commons License Cross-Display Attention Switching in Mobile Interaction with Large Displays PhD Thesis by Umar Rashid School of Computer Science University of St Andrews March 2012 Abstract Mobile devices equipped with features (e.g., camera, network connectivity and media player) are increasingly being used for different tasks such as web browsing, document reading and photography. While the portability of mobile devices makes them desirable for pervasive access to information, their small screen real-estate often imposes restrictions on the amount of information that can be displayed and manipulated on them. On the other hand, large displays have become commonplace in many outdoor as well as indoor environ- ments. While they provide an efficient way of presenting and disseminating information, they provide little support for digital interactivity or physical accessibility. Researchers argue that mobile phones provide an efficient and portable way of interacting with large displays, and the latter can overcome the limitations of the small screens of mobile devices by providing a larger presentation and interaction space. However, distributing user inter- face (UI) elements across a mobile device and a large display can cause switching of visual attention and that may affect task performance. This thesis specifically explores how the switching of visual attention across a handheld mobile device and a vertical large display can affect a single user’s task performance du- ring mobile interaction with large displays. It introduces a taxonomy based on the factors associated with the visual arrangement of Multi Display User Interfaces (MDUIs) that can influence visual attention switching during interaction with MDUIs. It presents an em- pirical analysis of the effects of different distributions of input and output across mobile and large displays on the user’s task performance, subjective workload and preference in the multiple-widget selection task, and in visual search tasks with maps, texts and photos. Experimental results show that the selection of multiple widgets replicated on the mobile device as well as on the large display, versus those shown only on the large display, is faster despite the cost of initial attention switching in the former. On the other hand, a hybrid UI configuration where the visual output is distributed across the mobile and large displays is the worst, or equivalent to the worst, configuration in all the visual search tasks. A mo- bile device-controlled large display configuration performs best in the map search task and equal to best (i.e., tied with a mobile-only configuration) in text- and photo-search tasks. Keywords: multi-display environment, distributed user interfaces, multi-device use, device interoperability, smartphones, large displays. Declaration I, Umar Rashid, hereby certify that this thesis, which is approximately 40,000 words in length, has been written by me, that it is the record of work carried out by me, and that it has not been submitted in any previous application for a higher degree. I was admitted as a research student in September 2010 and as a candidate for the degree of Doctor of Philosophy in September 2010; the higher study of which this is a record was carried out in the University of St Andrews between 2010 and 2012, (and in University College Dublin between 2007 and 2010). date signature of candidate I hereby certify that the candidate has fulfilled the conditions of the Resolution and Regu- lations appropriate for the degree of Doctor of Philosophy in the University of St Andrews and that the candidate is qualified to submit this thesis in application for that degree. date signature of supervisor In submitting this thesis to the University of St Andrews we understand that we are gi- ving permission for it to be made available for use in accordance with the regulations of the University Library for the time being in force, subject to any copyright vested in the work not being affected thereby. We also understand that the title and the abstract will be published, and that a copy of the work may be made and supplied to any bona fide library or research worker, that my thesis will be electronically accessible for personal or research use unless exempt by award of an embargo as requested below, and that the library has the right to migrate my thesis into new electronic forms as required to ensure continued access to the thesis. We have obtained any third-party copyright permissions that may be required in order to allow such access and migration, or have requested the appropriate embargo below. The following is an agreed request by candidate and supervisor regarding the electronic publication of this thesis: Access to Printed copy and electronic publication of thesis through the University of St An- drews. signature of candidate signature of supervisor date Acknowledgements I have had the opportunity of interacting and working with some remarkable people during my doctoral research. I would like to express my thanks to all of them whose contribution and encouragement has played a crucial role in the accomplishment of this work. First of all, I am highly indebted to my advisor Prof. Aaron Quigley for giving me the opportunity to pursue my doctoral research under his supervision. Working with him im- mensely helped refine my ideas and hone my presentation skills. His inquisitiveness, en- thusiasm and ambition has always been a source of inspiration for me. Similarly, I was really blessed to have Dr. Miguel A. Nacenta as my co-advisor whose meticulous approach to problem solving I found both challenging and enlightening. I have learnt a great deal about the pitfalls of experiment design and data analysis from him. I want to thank Lucia Terrenghi and Thomas Lang who supervised my internship at Vo- dafone Group Research in Munich (Germany). I am grateful to Jonna Hakkil¨ a¨ for giving me the opportunity to do an internship with her team at Nokia Research Center, Tampere (Finland). Special thanks to my former colleagues at Nokia Research Center, Jarmo Kauko and Antti Virolainen. Big thanks to Prof. Patrick Baudisch, Dr. Hrvoje Benko and Prof. Sriram Subramanian for being part of the doctoral panel of ITS 2010 conference and sharing their insightful comments and critiques for driving my research. I am grateful to Dr. Per Ola Kristensson and Dr. Adrian Friday for being part of my doctoral examination committee and providing valuable suggestions for my thesis. Thanks to my other colleagues at St Andrews Human Computer Interaction (SACHI) lab: Tristan Henderson, Jakub Dostal, Luke Hutton, Shyam Reyal. It has been a pleasure to work with all of you. Much credit is due to Dr. Katharine Preedy of Mathematics and Statistics Support Net- work at University of St Andrews for helping me comprehend the intricacies of statistical analyses. And to Dr. Marie Robinson for proof-reading parts of my thesis. I want to express my thanks to my parents, Jamila and Arshad Mir, sister, Emmara, and brother, Usman, who stood by me through thick and thin, and always supported me in my ambitions. I am also thankful to my better half, Aimen Zeb, for being a wonderful companion and a valuable addition to my life. Publications Some ideas presented in this thesis have already been published in some of the following papers. [1]R ASHID, U., KAUKO, J., HAKKIL¨ A¨ , J., AND QUIGLEY, A. Proximal and dis- tal selection of widgets: Designing distributed UI for mobile interaction with large display. In Proceedings of the International Conference on Human Computer Inter- action with Mobile devices and services (New York, NY, USA, 2011), MobileHCI ’11, ACM, pp. 495–498. [2]R ASHID, U., NACENTA, M. A., AND QUIGLEY, A. Factors influencing visual attention switch in multi-display user interfaces: A survey. In Proceedings of the 2012 International Symposium on Pervasive Displays (New York, NY, USA, 2012), PerDis ’12, ACM, pp. 1–6. [3]R ASHID, U., NACENTA, M. A., AND QUIGLEY, A. The cost of display switching: A comparison of mobile, large display and hybrid distributed UIs. In Proceedings of the International Working Conference on Advanced Visual Interfaces (New York, NY, USA, 2012), AVI ’12, ACM, pp. 99–106. Contents List of Figures vi List of Tables ix 1 Introduction 1 1.1 Motivation . .2 1.1.1 Mobile Phones: More than Telecommunication . .3 1.1.2 Limitations of Mobile Displays . .4 1.1.3 Large Displays: More than Household Items . .4 1.1.4 Coupling Mobile Displays with Large Displays . .5 1.1.5 Distribution of User Interfaces . .6 1.1.6 Attention Switching in Distributed User Interfaces . .6 1.2 Problem Statement . .7 1.3 Research Approach . .8 1.4 Thesis Statement . .9 1.5 Scope of Thesis . .9 1.6 Thesis Contributions . 10 1.7 Thesis Organisation . 11 2 Background 14 2.1 Definitions . 15 2.1.1 Multi-Device Environment . 16 2.1.2 Multi-Display Environment . 16 i ii 2.1.3 Device Interoperability . 17 2.1.4 Multi-Display User Interface . 17 2.2 Usage of Multiple Devices . 18 2.3 Computing Paradigms for Device Interoperability . 19 2.3.1 Ubiquitous Computing . 19 2.3.2 Activity-Based Computing . 20 2.3.3 Recombinant Computing . 21 2.3.4 Synergy of Multiple Devices . 22 2.4 Multi-Display User Interfaces .