Processing Desktop Work on a Large High-Resolution Display: Studies and Designs

Xiaojun Bi

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Department of Computer Science University of Toronto

Xiaojun Bi

Doctor of Philosophy

Department of Computer Science University of Toronto

2011

Abstract

With the ever increasing amount of digital information, information workers desire more screen real estate to process their daily desktop work. Thanks to the quick advance in display technology, big screens are increasingly affordable and have been gradually adopted in desktop computing environments. A large wall-size high resolution display, a recent emerging class of display which possesses a huge visualization surface, could potentially benefit information processing work. In this dissertation we investigate such a large display as the primary working space for information processing work.

We firstly conducted a longitudinal diary study and three control experiments investigating effects of a large display on information processing work. The longitudinal diary study investigates large display use in a personal desktop computing context by comparing it with single- and dual-monitor. The three controlled experiments further investigate the effects of two factors determining resolution of a display—physical size and pixel-density on users’ performance and behaviors. The diary study reveals the distinct behavior patterns of large

display users in partitioning screen space and managing windows, while the control experiments deeply reveal the effects of the physical size and pixel density of a display on different information processing tasks. Aside from studying a continuous large display, we also articulate how interior bezels within a tiled-monitor large display affect users’ performance and behaviors in basic visual search and action tasks via a series of controlled experiments. Based on the understanding of large display effects and users’ behavior patterns, we then design new interaction techniques to address a big challenge of working on a large display: managing overflowing windows. We design and implement a large display oriented window management system prototype: WallTop. It includes a set of interaction techniques that provide greater flexibility for managing windows. Usability tests show that users can quickly and easily learn the new techniques and apply them to realistic window management tasks with increased efficiency on a large display.

iii

Acknowledgments

There is an old saying in Chinese “Yu Rui Zhe Xu, Yu Zhi Zhe Xing”(talk to the smart, walk with the intelligent). Over the course of my Ph.D. study, I am very grateful that many smart and intelligent minds walk with me on this journey. I could not have completed this dissertation without the help of these great people.

First and foremost, I would like to thank my Ph.D. supervisor, Professor Ravin Balakrishnan, who has taught me how to conduct research with high standards, and trained me to become a successful researcher. I am very grateful to all his contributions of ideas, criticisms, time, and funding throughout my Ph.D. study. The experience of working with him is the most valuable asset in my life. I would like to express my sincere gratitude to the rest of my Ph.D. committee:

Professor Khai Truong and Professor Karan Singh. I really appreciate their comments, criticisms, suggestions throughout my Ph.D. study. I am also grateful to Professor Sriram Subramanian and

Professor Daniel Wigdor for reading my thesis, offering insightful comments, and serving as extra examiners.

I have been privileged to work with outstanding researchers outside the University of Toronto, including Dr. Shumin Zhai (former IBM, now Google), Dr. Barton Smith (IBM), Dr. Ken

Hinckley (Microsoft), Dr. Michel Pahud (Microsoft), Dr. Tovi Grossman (Autodesk), Dr.

George Fitzmaurice (Autodesk), and Dr. Yang Li (Google). I would like to thank them for taking me as an intern at world-class research institutes. These experiences not only broadened my research interests, but also offered me great opportunities to collaborate with people from different backgrounds. I am especially grateful to Dr. Shumin Zhai. His HCI classes in Tsinghua

University introduced me to this challenging discipline and he has constantly advised me in various research projects.

I am extremely grateful to my parents and family for their love. I want to thank my parents for giving me the opportunity to pursue my own interests, and encouraging me to study abroad. As the only child of them, I can understand the pain they endure of being separated from their son for years. I must thank my wife, Huijing Yang, who abandoned her career in China and spent the most valuable time of her life with me here. Her encouragement was my energy source which led me to overcome all the obstacles in the Ph.D. pursuit. Finally, I would like to thank my upcoming daughter, Niuniu, who offered me the opportunity to finish this thesis in a joyful mood.

Table of Contents

1 Introduction ...... 1

1.1 Contributions...... 6

1.2 Thesis Outline ...... 8

2 Related Work ...... 10

2.1 Using a Normal-Sized Monitor to Process Desktop Work ...... 10

2.2 Using Dual or Triple Monitors to Process Desktop Work ...... 11

2.3 Large Display Usage ...... 13

2.3.1 Using Large Displays in Desktop Computing Environments ...... 13

2.3.2 Data Visualization...... 17

2.3.3 Virtual Reality Applications ...... 18

2.3.4 Group Work ...... 18

2.3.5 Public displays ...... 20

2.3.6 Specialized Applications ...... 22

2.4 Challenges with Interacting on a Large Display ...... 23

2.4.1 Mice and Keyboards Based Interaction ...... 23

2.4.2 Other Interaction Methods ...... 27

2.5 Managing Desktop Work ...... 30

2.5.1 Document Management Behaviors ...... 30

2.5.2 Advantages of Paper Documents ...... 31

2.5.3 Using Paper Metaphor in Digital Information Management ...... 32

2.6 Summary ...... 33

3 Longitudinal Diary Study of Large Display Usage ...... 34

3.1 Study Design ...... 34

3.2 Results ...... 39

3.2.1 Overall Activities ...... 39

3.2.2 Subjective Opinions ...... 40

3.2.3 Screen Space Partition ...... 43

3.2.4 Window Management...... 44

3.2.5 Seating Location Relative to the Display ...... 49

3.3 Discussion ...... 50

3.3.1 Benefits of Large Displays ...... 50

3.3.2 Challenges of Managing Windows on a Large Display ...... 52

3.4 Conclusion ...... 54

4 Effects of Display Size and Pixel Density on Desktop Work ...... 56

4.1 Experimental Settings ...... 57

4.1.1 Display Conditions ...... 57

4.1.2 Subjects ...... 59

4.1.3 Apparatus ...... 59

4.2 EXPERIMENT 1: Scattered-Information Processing ...... 59

4.2.1 Experiment 1: Task ...... 59

4.2.2 Experiment 1: Design ...... 63

4.2.3 Experiment 1: Data Collection ...... 63

4.2.4 Experiment 1: Results ...... 64

4.2.5 Experiment 1: Discussion ...... 67

4.3 EXPERIMENT 2: Multi-Scale Navigation ...... 68

4.3.1 Experiment 2: Tasks ...... 68

4.3.2 Experiment 2: Design ...... 70

4.3.3 Experiment 2: Data Collection ...... 70

vii

4.3.4 Experiment 2: Results ...... 70

4.3.5 Experiment 2: Discussion ...... 72

4.4 Experiment 3: Single-Window Text Processing ...... 74

4.4.1 Experiment 3: Task ...... 74

4.4.2 Experiment 3: Design ...... 75

4.4.3 Expt 3: Data Collection ...... 75

4.4.4 Expt 3: Results ...... 75

4.4.5 Experiment 3: Discussion ...... 76

4.5 Conclusions ...... 77

5 Effects of Interior Bezels of Tiled-Monitor Large Displays on Visual Search, Tunnel Steering, and Target Selection ...... 79

5.1 Experimental Setup ...... 80

5.1.1 Tiling Configurations ...... 80

5.1.2 Participants ...... 82

5.2 Visual Search Experiment ...... 82

5.2.1 Visual Search: Task ...... 82

5.2.2 Visual Search: Design ...... 83

5.2.3 Visual Search: Results ...... 84

5.2.4 Visual Search: Conclusions ...... 86

5.3 Tunnel Steering Experiment ...... 87

5.3.1 Tunnel Steering: Task ...... 87

5.3.2 Tunnel Steering: Design ...... 88

5.3.3 Tunnel Steering: Results ...... 89

5.3.4 Tunnel Steering: Conclusions ...... 92

5.4 Target selection experiment ...... 92

viii

5.4.1 Target Selection: Task ...... 93

5.4.2 Target Selection: Design ...... 94

5.4.3 Target Selection: Results ...... 94

5.4.4 Target Selection: Conclusions ...... 96

5.5 Discussions on Display Resolution and Size Effects ...... 96

5.5.1 Resolution Effects ...... 96

5.5.2 Display Size Effects ...... 97

5.6 Conclusions ...... 98

6 WallTop: Flexibly Managing Windows on a Large Display ...... 100

6.1 WallTop Interaction Techniques ...... 101

6.1.1 Fringe ...... 101

6.1.2 Jab-to-lift...... 103

6.1.3 Multi- and Single-Window Marking Menus ...... 104

6.2 Implementation ...... 109

6.3 Iterative Usability Tests ...... 110

6.3.1 First Round Test ...... 110

6.3.2 Second Round Test ...... 111

6.4 Discussion ...... 117

6.4.1 Multi-Window Operations ...... 118

6.4.2 Flexibly Arrange Windows ...... 118

6.4.3 Facilitate Moving and Resizing Windows ...... 118

6.4.4 Execute Command In-Place ...... 118

6.5 Conclusions ...... 120

7 Conclusions ...... 122

7.1 Summary ...... 122

7.1.1 Thesis Structure ...... 122

7.1.2 Implications for Display Usage ...... 124

7.2 Limitations ...... 126

7.3 Future Directions ...... 127

References ...... 130

List of Tables

Table 3-1 Hours logged per major activity on the large display ...... 39

Table 6-1 Simple tasks (summary)...... 112

Table 6-2. Compound taskTable 6-3s (summary)...... 114

Table 6-4. Mean completion time and total number of failed trials for simple tasks...... 116

Table 6-5. New functions in WallTop compared with other systems ...... 119

List of Figures

Figure 1-1. Apple Cinema Monitor and Radius320 display ...... 3

Figure 1-2. Using a large high resolution display to process desktop work...... 4

Figure 2-1. A user is working on an experimental DSharp display...... 11

Figure 2-2.A large display consisting of nine 17’’ monitors in a 3 × 3 tiling...... 14

Figure 2-3. Using projector displays as primary workplaces...... 14

Figure 2-4. Kimura...... 15

Figure 2-5. The Prairie...... 16

Figure 2-6. The office of the future...... 17

Figure 3-1. The large display in the study...... 35

Figure 3-2. Example activity log ...... 37

Figure 3-3. Participants’ subjective opinions of the large display by user ...... 41

Figure 3-4. Participants’ subjective opinions of the large display by task...... 42

Figure 3-5. Mouse event distribution ...... 44

Figure 3-6. A screen capture of the large display...... 46

Figure 3-7. Distribution of window management operations per participant...... 48

Figure 3-8. Time spent at different distances to the large display ...... 50

Figure 4-1 Display conditions used in our experiments ...... 56

Figure 4-2. The 2×2 tiled projection screen...... 57

Figure 4-3 The display size and pixel density in each display condition...... 58 xii

Figure 4-4. Applications and resources used in Experiment 1...... 60

Figure 4-5. Simple information-lookup ...... 60

Figure 4-6. Complex information-lookup and processing ...... 61

Figure 4-7. Information comparison...... 61

Figure 4-8. Picture insertion...... 62

Figure 4-9. The apparent sizes of characters in the wiki web-pages...... 62

Figure 4-10. Mean completion time (with std error) in Experiment 1...... 64

Figure 4-11. Mouse-event distribution for Experiment 1...... 65

Figure 4-12. Mean number (with std error) of simultaneously viewable windows ...... 66

Figure 4-13. A screen image in the BH display condition...... 67

Figure 4-14. Experiment 2: Locating marks task...... 69

Figure 4-15. Experiment 2: Tracing a route task...... 69

Figure 4-16. Mean completion time (with std error) in Experiment 2...... 71

Figure 4-17. Mean numbers (with std errors) of navigating operations in Expt 2...... 72

Figure 4-18. The mean window sizes (in pixel) in Experiment 2 ...... 73

Figure 4-19. Expt 3 proofreading task ...... 74

Figure 4-20. The mean number (with std error) of correct edits in Expt 3...... 76

Figure 5-1. Tiled-monitor large displays in personal offices ...... 79

Figure 5-2. The three tiled-monitor large display configurations used in our experiments ...... 81

Figure 5-3. Visual search...... 83

xiii

Figure 5-4. Mean (SD) error rate by symbol-split status...... 85

Figure 5-5. Deflected straight tunnels in (a) [2×2] and (b) [3×3] tiled displays...... 87

Figure 5-6. Steering task ...... 88

Figure 5-7. Mean (SD) steering time by the degree of tiling...... 89

Figure 5-8. Mean cursor speed in each 32-pixel long segment along a tunnel...... 91

Figure 5-9. Target selection task ...... 93

Figure 5-10. Mean cursor speed...... 95

Figure 6-1. the fringe is activated / deactivated...... 101

Figure 6-2. Drawing, clicking and moving windows ...... 102

Figure 6-3. The fringe operations...... 102

Figure 6-4. resizes all of the windows at once...... 102

Figure 6-5. Jab-to-lift ...... 103

Figure 6-6. (a) Multi-, and (b) single-window marking menus...... 104

Figure 6-7. Choosing Spread item on the marking menu...... 105

Figure 6-8. Choosing Splat item on the marking menu ...... 106

Figure 6-9. Choosing Pile item on the marking menu...... 106

Figure 6-10. Choosing Side Right item on the marking menu...... 107

Figure 6-11. Pack (up) and Unpack (left) icons ...... 108

Figure 6-12. Specification of WallTop and usability test setup...... 109

Figure 6-13. Mean () completion time for compound tasks ...... 116

xiv

Figure 6-14. Proportions of used operations under the with new techniques condition ...... 117

Figure 7-1. Office of the Future...... 124

1 Introduction

A desktop computer, which is a personal computer in a form intended for regular use at a single location, is now the main working station for a current information worker. Since its invention in the 1970s, it has been widely used in modern offices. Over the past three decades, the computing capability of such a computer has undergone a drastic increase: the CPU speed has become much faster and the available storage space has become much greater. Along with the rapid development of desktop computers, digital information in the world has been boosted over the past decade. The total amount of digital information reached 281 billion gigabytes in 2007, around 45 GB per person on the earth (Gantz et al., 2008). By the end of 2011, this value will be 10 times the size it was in 2007 (Gantz et al., 2008). The explosively growing “digital universe” leads to an increasing amount of digital information surrounding normal users. Furthermore, the rapid advance of desktop computing technology enables users to process greater amounts of information simultaneously. The daily digital lives of current desktop users are significantly enriched compared to decades ago. On the other hand, the overflowing digital information leads to heavy desktop work (Desktop work is the work processed on a desktop computer.). A recent study (Czerwinski et al., 2004) revealed that current information workers often found themselves in multi-tasking environments which required them to juggle multiple applications processing different types of information on desktop computers.

Intuitively, a large work space is beneficial for processing a large amount of information. Indeed, prior research which investigated office workers’ document management behaviors explains the advantages of a large work space. Bondarenko and Janssen’s (2005) studies showed that to better support information processing, documents should be embedded within meaningful (task-related) contextual information, and the documents should be easily accessible for regrouping as a task progresses. Malone (1983) pointed out that the desk organization was also utilized to remind the user of things to do, not just to help the user find desired information. He (Malone 1983) further identified two basic strategies used for handling paper documents: filing and piling. Filers maintained clean desktops, and did not allow papers to pile up. They systematized their archives (using alphabetical, conceptual, or temporal methods) to support straightforward access to stored data. In contrast, pilers had messy desktops cluttered with paper piles, making few attempts to organize stored information. 1

A large office table, which allows a user to freely lay out multiple documents, has advantages over a small desk in supporting office workers’ document management behaviors. The user has more freedom to file and pile documents on a large table than on a small desk. The filed or piled documents scattered on a large office table provide the contextual information of relevant tasks, and they could serve as reminders of upcoming work. Due to these advantages, office workers usually prefer big tables over small desks in processing a large number of paper documents. For example, a pilot tray table might be sufficient for reading a novel, but a big office desk is usually preferred for sorting bank bills from the past five years.

Although many office workers now process digital information instead of paper documents, the strategies of managing information remain unchanged when they switch from the physical to digital world (Bondarenko & Janssen, 2005). A larger display, which offers users a larger visualization and work space, could benefit users in processing digital desktop work. Since a large amount of information is visualized, it benefits task switching, provides more contextual information for tasks, and reminds users of upcoming work. For example, because multiple windows are visualized simultaneously on a 51” monitor (Ball & North, 2005), context information in different windows is visualized simultaneously together with the focused task. Windows for upcoming tasks reside at the side regions of a display as reminders. The spatial layouts of freely piled and filed windows subtly convey the contextual information of associated tasks. In contrast, users usually keep only one window totally visualized on a typical 17” monitor (Bi & Balakrishnan, 2009). To check contextual information from other windows, they need to frequently switch back and forth among multiple windows. Because of the screen space limit, they are constrained from freely filing and piling windows.

Indeed, previous research reveals overwhelming benefits of working on a big-sized regular desktop monitor1. Simmons (2001) showed that a 21” monitor outperformed a 15”, 17” and 19” in Excel tasks and multitasking. Fridgeman, Lennon and Jackenthal (2003) reported that a 17” (1024 × 768) display led to higher scores in computer-based verbal tests than a 15” (640 × 480) or 17” (640 × 480) monitor. Brujun, Mul, and Oostendorp (1992) showed that a 15” screen led to

1 Regular desktop monitors are displays whose diagonals are shorter than 30”. 2

less learning time than a 12” display in a text reading task. A recent released report revealed that a 30” monitor led to increased productivities not only in professional designs, digital imaging and editing digital videos, but also in office applications such as word processors and spreadsheets, in comparison to 20” and 17” monitors (Pfeiffer Report, 2005).

Research literature shows that desktop monitors between 20” and 30” large are beneficial for desktop work (Fridgeman, Lennon & Jackenthal, 2003; Brujin, Mul & Oostendorp, 1992; Pfeiffer Report, 2005; Simmons, 2001). In addition to it, recent studies revealed that information workers could benefit from using displays much bigger than regular desktop monitors. Ball and North (2005) showed that a tiled display containing nine 17’’ monitors in a 3 × 3 tiling (113 cm wide × 94 cm high) afforded a number of advantages, including improving task switching and viewing large documents; Bishop and Welch (2000) reported that using projector displays (2.4m wide × 1.8m high, resolution 2048 × 768) as primary workplaces improved the productivities for daily work; Czerwinski et al. (2003) showed that a 42” screen outperformed a 15” monitor in various desktop work. In sum, the previous studies unanimously indicate a trend that bigger displays lead to higher performance for desktop work.

Figure 1-1. left: Apple Cinema Monitor with 30” in diagonal and resolution 2560 × 1600, right: Radius320 display with 50” in diagonal and resolution 4800 × 1200. Figures taken from www.apple.com and www.seamlessdisplay.com.

Actual display usage echoes these research results: information workers tend to use bigger and bigger displays for desktop work. Over the past decade, the size of a commonly used desktop monitor has increased from 15” to 25”. Thirty inch screens are also widely seen in many offices at present. In fact, the rapid development of display technology makes large screens increasingly available and affordable: 50” monitors are now commercially available (Figure 1-1 right); many manufactures have released high-resolution LCDs bigger than 100” (e.g., SHARP LCDs, http://www.sharp-world.com/index.html). As this trend develops, normal users will be able to

afford such large desktop displays. For example, users might easily purchase a 50” display at a commodity price (Figure 1-1 right), or even acquire a visualization surface bigger than 100” attached on the wall as their desktop display (Figure 1-2).

Figure 1-2. Using a large high resolution display to process desktop work. Display size: 4m × 1.5m. Resolution: 6144 × 2034.

Constrained by the structure of a personal office, the largest single continuous display that an information worker can work with is at most wall-sized (e.g., 4m wide × 1.5m high). Although research literature indicates the benefits of a big-sized screen, such a wall-size display is much bigger than the regular size. Questions arise as to whether such displays still bring benefits compared to normal-sized monitors? A simple and common concern from a desktop user will be whether it is worthwhile to replace a normal-sized monitor with a large display by paying extra money. (In this thesis, we refer to displays equal to or under 30” as normal-sized monitors, and wall-sized displays around 4 m wide and 1.5 m high as large displays.). A large display can show a large amount of digital information, which might facilitate users processing overflowing digital information. On the other hand, the excessive amount of digital information visualized on a large display might overwhelm users thus hindering their information processing performance. Before we step into a large-display-enhanced desktop environment, it is important to disentangle its pros and cons beforehand.

Besides affecting information processing performance, a large display might substantially affect users’ behaviors. In the physical world, office workers’ activities change drastically when they switch from a pilot tray-table to a large office desk. The latter offers a greater working space thus providing more flexibility for arranging paper documents and making sense of different files. A large display offers information workers a greater amount of screen real estate than a normal- sized display. Will users’ behaviors undergo drastic changes when they replace a normal-sized

monitor with a large display? Given a large visualization surface, how will information workers utilize such big work space to process the excessive digital information?

The answers to the above questions are fundamental and critical: understanding the effects of a large display would lead to effective large display usages, and help software developers design large-display-oriented interfaces and applications.

Another issue pertaining to adopting a large display for desktop work is about managing excessive digital information on a large visualization surface. Digital information in current desktop environment is usually visualized through windows. The increasing amount of digital information on a large display will likely lead to numerous open windows. However, current window management systems, such as WinXP or Mac OS, are designed for normal-sized monitors, which only allow users to operate on a single window at a time. When users switch to work on a large display, questions arise as to will traditional window management systems still suit a large display where a user might deal with numerous windows simultaneously? If not, how should we refine them to fit users’ demands?

In summary, as the amount of digital information has been explosively growing, information workers are demanding bigger and bigger displays to process daily work. Thanks to the quick development of display technology, information workers will be able to acquire big monitors, or even large wall-sized displays in their personal offices. For example, users might purchase digital wall papers and turn an entire wall into a visualization surface in the near future. Such displays could potentially benefit users because of the large visualization surfaces, but also result into various challenges due to the overflowing information. Investigating the effects of such a large display and addressing the challenges of working on it become timely issues. However, previous research has mostly focused on the properties of normal-sized displays (i.e., monitors under or equal to 30”). Issues pertaining to large displays (i.e., around wall-sized) for desktop work are under-researched. This thesis is aimed at shedding light in this area: we systematically investigate the effects of a large display on information workers’ performance and behaviors, and design new interaction techniques to address the challenges of working on it. Although the main focus of this thesis is large wall-sized displays, the effects of big displays (e.g., around 50”) are also initially investigated.

1.1 Contributions

We make the following contributions relating to human factors and interaction designs. There has been no comprehensive qualitative or quantitative study investigating how a large display affects users’ performance and behaviors in desktop work. Our longitudinal diary study in Chapter 3 is the first to systematically investigate such effects. More specifically, it makes following contributions:

1) Distinct Behavior Patterns in Partitioning Screen Space and Managing Windows of Large Display Users.

The longitudinal diary study in Chapter 3 comprehensively compares the usage of a large display with single- and dual-monitor for realistic desktop work. Based on the analysis of users’ detailed activities such as mouse actions and window management operations, the results showed distinct behavior patterns of large display users in partitioning screen real estate and managing windows. Different from single- and dual-monitor users, large display users tended to utilize the center part as the focal region where primary windows were located, while the remaining space was used as the peripheral region where peripheral applications resided. Compared to normal single- and dual-monitor users, large display users invested more efforts to spatially arrange windows in various layouts, and had special interaction and visualization demands of windows in peripheral regions. Also, users on a large display performed more window moving and resizing, but less minimizing and maximizing operations than on a single- or dual-monitor. These distinct behavior patterns led to suggestions for designing large display oriented window management systems.

2) Pros and Cons of Utilizing a Large Display for Desktop Work.

Aside from the unique behavior patterns, the diary study in Chapter 3 also revealed pros and cons of working on a large display. In general, users unanimously preferred working on a large display over single- or dual-monitor. The large display was beneficial in multi-window and rich information tasks; it raised a user’s awareness of peripheral work, and offered him/her immersive experience of desktop work. Despite the overwhelming preferences, users encountered the

challenge of managing overflowing windows on a large display, because current window management systems were mainly designed for normal-sized monitors.

3) Effects of Pixel Density and Physical Size of a Display on Users’ Performance and Behaviors

Following the longitudinal study in Chapter 3, a series of controlled experiments in Chapter 4 disentangle the effects of physical size and pixel density of a display on users’ performance and behaviors. Also, complementary to the qualitative study in Chapter 3, these controlled experiments quantitatively demonstrate the pros and cons of working on a display with big physical size and high resolution.

4) Effects of Internal Bezels of a Tiled-Monitor Display on User’s Perception and Interaction Abilities

Tiled-Monitor large displays are widely used to replace seamless displays due to the low cost and ease of deployment. However, one critical flaw on such displays is the physical bezels surrounding each component monitor. Our study in Chapter 5 is the first to systematically investigate how these bezels will affect users’ basic perception and interaction abilities. The results lead to guidelines about effective tiled-monitor display usages and implications about UI designs on such displays.

5) Interaction Techniques for Managing Windows on a Large Display

As indicated by the diary study in Chapter 3, one big challenge with working on a large display is how to manage numerous windows. Current window management systems, such as window XP or Mac OS are designed for normal-sized desktop screens. They only allow users to operate on a single window at a time, which are too rigid and inappropriate for managing a large number of windows on a large display. Chapter 6 describes a set of new interaction techniques offering information workers high flexibility in managing numerous windows on a large display, including facile methods for selecting, moving and resizing an arbitrary number of windows, and techniques to help users spatially arrange windows. These new techniques are coherently integrated with traditional operations in a large-display window management prototype, called 7

WallTop (A video demo of WallTop is at http://www.dgp.toronto.edu/~xiaojun/WallTop/). Usability tests show that users can quickly and easily learn the new techniques and apply them to increase efficiency in realistic window management tasks.

1.2 Thesis Outline

The remainder of this document is organized as follows:

Chapter 2 presents related work in the fields of ergonomic and HCI as it pertains to working on a large display. This chapter begins with literature investigating effects of a normal-sized monitor on users’ desktop performance and behaviors, followed by a summary of research on multi- monitor usage. An overview of large display usage is presented in the following section, with particular attention paid to scenarios where a large display is used for a single person’s desktop work. The remainder this chapter describes various interaction techniques on a large display, and reviews the prior research investigating office workers’ information management behaviours.

Chapter 3 presents a longitudinal dairy study investigating large display use in a personal desktop computing context by comparing it with single and dual normal-sized monitor use. By analyzing participants’ detailed activities such as mouse actions and window management operations, we discovered distinct behavior patterns of large display users, and indicated the potential benefits as well as shortcomings of working in such environments.

Following this qualitative study, Chapter 4 presents a series of controlled experiments in which pixel density and physical size of a display were accurately controlled. These experiments not only deeply revealed how the physical size, pixel density, and resolution of a display affect desktop work, but also verified some findings from the prior dairy study.

Chapters 3 and 4 focus on a seamless large display, which is created by tiling multiple projectors. Another popular approach of creating a large display is to tile multiple equivalent LCDs, which is usually referred as tiled-monitor large display. Chapter 5 presents a study systematically investigating users’ interaction and perception capabilities when working on tiled-monitor large displays. In particular, we systematically investigated the effects of internal bezels within a tiled- monitor large display on users’ performance and behaviors in the most fundamental perception and interaction tasks: visual search, tunnel steering, and object selection. This study led to 8

implications of UI design on tiled-monitor displays and guidelines of effectively using such displays.

Chapter 6 presents a large display window management system prototype called WallTop. It includes a set of new interaction techniques specifically suited to a large display. Distinct from traditional window management systems such as Win XP or Mac OS, it allows users to operate on multiple windows simultaneously, and offers more flexibility of spatially arranging multiple windows. Formal user evaluations showed that users could quickly and easily learn the new techniques and apply them to boost window management efficiency.

Chapter 7 concludes this thesis.

2 Related Work

In this chapter, we thoroughly review relevant research. Section 2.1 reviews research studying the effects of normal-sized monitors on desktop work. In section 2.2, we discuss research about utilizing multiple monitors for desktop work. Section 2.3 summarizes existing large display usages, with particular attentions to scenarios in which large displays are involved for desktop computing. This is followed by a review of interaction techniques on large displays in section 2.4, and a summary of desktop workers’ information management behaviours in section 2.5.

2.1 Using a Normal-Sized Monitor to Process Desktop Work

A sizeable amount of research has been carried out investigating how normal-sized monitors affect users’ performance and behaviours for desktop work. Simmons (2001) conducted a study comparing user’s performance on 15’’, 17’’, 19’’ and 21’’ monitors in Microsoft Word, Excel spreadsheet, multitasking, web browsing, PowerPoint, and Web encyclopaedia tasks. The results revealed that users performed the Excel task and multitasking in significantly less time on the 21’’ display than on the 15- 17- and 19-inch displays. Statistical analysis also indicated that the 19- and 21-inch displays exhibited preference scores significantly higher than those for the 17- and 15-inch displays. Moreover, in a computer-based verbal test, empirical studies showed that users achieved higher scores on a 17’’ (1024 × 768) monitor than on a 15’’(640 × 480) or 17’’(640 × 480) one, indicating that a larger high-resolution monitor could improve performance (Fridgeman et al., 2003). For a text reading task, a 15’’ screen can generate a more efficient integration process in constructing the semantic representation than a 12” one, leading to less learning time (Brujin et al., 1992). Aside from the performance increase, a bigger monitor also provides ergonomic benefits. Sommerich et al. (1998) revealed that a 19’’ monitor generated less muscle activities than a 13.6’’ one for screen-intensive work. A recent released report revealed that a 30” monitor led to increased productivities not only in professional design, digital imaging and editing digital videos, but also in office applications such as word processors and spreadsheets, in comparison to 20” and 17” monitors (Pfeiffer Report, 2005). Czerwinski et al. (2003) carried out a control experiment to characterize the productivity benefits of using a large monitor for complex, multi-application office work (Figure 2-1). Participants performed multiple-step, cognitively loaded tasks in two conditions: a normal 15’’ monitor, and a 42’’ wide 10

surface, called DSharp, created by using three XGA DLP projectors at 1024 × 768 resolutions onto a curved Plexiglas panel. The result showed a significant performance advantage of using the large display, as well as positive user preference and satisfaction with its use over the smaller one.

Figure 2-1. A user is working on an experimental DSharp display. Figure taken from Czerwinski et al. (2003).

Previous research reveals the overwhelming benefits of a bigger normal-sized monitor over a smaller one. However, a normal-sized monitor only occupies a small portion of a user’s visual filed, roughly less than 10%. When switching to a large wall-sized display that can take upwards 70% of the visual field, the findings discovered for normal-sized monitors might not hold and users’ desktop behaviours might change drastically. Substantial piece of research is desired to investigate the effects of large displays.

2.2 Using Dual or Triple Monitors to Process Desktop Work

Besides using a large display, another approach of obtaining plentiful screen real estate is to use multiple monitors. Current technology allows users to easily and cheaply get additional monitors. Newer video cards support two, three, or even four independent monitors, and many PCs allow users to use multiple video cards simultaneously. Extra monitors provide a user additional screen real estate, offering more visualization surface for daily work.

Grudin (2001) documented the usage of dual monitors, revealing that second monitors were generally used for secondary activities related to principal tasks, for peripheral awareness of information outside the main focus, and for easy access to resources. Hutchings et al. (2004) deployed a tool, called VibeLog, to a group of single, dual and triple monitor users to log window management activities. Analysis of data showed the differences in window management activities between different display configurations. Due to the long distance a user had to traverse to click on the taskbar, dual and triple monitor users tended to access windows directly more and used the taskbar less than single monitor user. As dual and triple monitors provided a greater amount of screen real estate than a single one, there were more visible windows on dual and triple monitor configurations. Moreover, as the larger visualization surface can display more information, users preferred to work with dual or triple monitors.

Dual or triple monitors allow users to visualize more information simultaneously; however, they can also cause some usability problems. For example, a large number of visible windows pose extra burden for windows management. Small windows such as dialogues and toolbars may appear at unexpected positions when invoked (Hutchings & Stasko, 2005). To ease the windows management on dual or triple monitors, Hutchings and Stasko (2007) developed the snip technique. It allowed a user to constrict the view onto any window, thus reducing the space needed to display information, and resulting in fast information access. The Mudibo system alleviated the problem of dialogues and toolbars appearing at unexpected locations by placing the invoked small windows in multiple locations simultaneously (Hutchings & Stasko, 2005). This technique allowed users to easily interact with dialogues/toolbars in any desired location.

Both multi-monitor and a large display offer users plentiful screen real estate. The major difference between them is that multi-monitor introduces physical bezels or even gaps within the visualization surfaces, which cause visual discontinuity and could affect users’ performances and behaviours. Hence, prior research on multi-monitor can at best serve as a very rough guide to single large high-resolution display usage; research explicitly focused on the latter is clearly required, and this thesis takes one step in this direction.

2.3 Large Display Usage

Since a large high-resolution display can show a great amount of information, and be viewed by multiple persons simultaneously, it has been widely used in various scenarios. In this section, we discuss existing large display usages, with special attentions to desktop computing environments.

2.3.1 Using Large Displays in Desktop Computing Environments

Studies Investigating Large Display Properties

To facilitate processing the increasing amount of digital information, some researchers initially investigated using a large display as a desktop monitor.

Andrews, Endert and North (2010) conducted observational studies to investigate how users carried out intelligence analysis exercise on a large display which consisted of a 4 × 2 grid of 30” LCD panels, each with a maximum resolution of 2560 × 1600. Their study results indicated that by creating a virtual workspace with real physical space, the large, high-resolution display offered a number of intriguing possibilities for future sense making tools. Even without any special tolls, the inherently spatial environment already provided support for activities typically done with physical artifacts. The study showed that an analyst using the large display both as a form of rapid access external memory and as an added semantic layer in which meaning was encoded in the spatial relationships between data, documents, display and analyst.

Ball and North (2005) reported an observational study of the use of a large tiled display containing nine 17’’ monitors in a 3 × 3 tiling (Figure 2-2). The entire large display was approximately 37 inches (94 cm) tall, with the total resolution of 3840 × 3072. In their study, five participants used this nine-monitor tiled display for at least three months in a form of time sharing. The behavioural analysis showed that the large-high resolution display afforded a number of advantages, including improving task switching and viewing large documents, increasing users’ collaboration abilities, and enhancing the awareness for secondary tasks. One interesting finding was that the physical bezels surrounding each monitor affected the large display usage. On the one hand, they separated a picture which was positioned across the bezel, resulting in visual discontinuities. On the other hand, they could be utilized to segregate work space. 13

Figure 2-2.A large display consisting of nine 17’’ monitors in a 3 × 3 tiling. Figure taken from Ball and North (2005).

Bishop and Welch (2000) self-reported a one-year experience of using projector displays as primary workplaces (Figure 2-3). Two projectors were mounted overhead to display abutted image on a large, flat display surface. The projectors each displayed 1024 × 768 pixels for a composite display of 2048 × 768 pixels. Numerous advantages were reported throughout the study, including improved social and technical interaction, better ergonomics, and high information content. Aside from the benefits, the large display system introduced some problems, such as generating extra heat and noise, and occasionally compromising privacy. Despite these shortcomings, overall, the authors concluded that they felt better, got more work done on the large display, and never wanted to go back to conventional displays.

Figure 2-3. Using projector displays as primary workplaces. Figure taken from Bishop and Welch (2000).

Andrew et al. (2010), Ball and North (2005), and Weltch and Bishop (2000) qualitatively investigated large display usages in realistic settings. However, the displays in Andrew et al.(2010), and Ball and North (2005) were tiled-monitor displays with physical bezels within the surface, and the display in Weltch and Bishop (2000) was low resolution, only 2048 × 768. None of them truly reflected the usage a contiguous large high-resolution display for desktop work. Our qualitative study in Chapter 3 is the first to study the effects of a contiguous, large high- resolution display on desktop work via a longitudinal study. Also, we are the first to investigate large display users’ behaviour by analyzing the detailed mouse and window operations.

Desktop Computing Systems Involving Large Displays.

In addition to empirical studies, prototypes involving large displays for desktop work have also been designed and implemented.

In Kimura (Figure 2-4), an augmented office environment, large projected displays on nearby walls showed the user’s working contexts, in the forms of a visual collage of images gathered from the activity log (MacIntyre et al., 2001). The user could not only monitor work status and notification via these background peripheral representations, but also resume a work configuration by selecting the corresponding working context.

Figure 2-4. Kimura, an augmented office environment including the focal and peripheral interactive displays. Figure taken from MacIntyre et al. (2001).

The Prairie, a prototype for supporting a knowledge worker in a “virtual enterprise”, was built by utilizing six 29’’, 800 × 600 pixel resolution monitors (Swaminathan and Sato, 1997). The system was aimed to facilitate globally distributed enterprise operation as if all its employees and other resources were located in one place. A large display was chosen over a normal monitor because it was beneficial in displaying collaborative applications, rich information objects and context information. The authors were convinced that it was simply a matter of time before large displays became standards for home and office computers.

Figure 2-5. The Prairie, a prototype for supporting a knowledge worker in a “virtual enterprise”. Figure taken from Swaminathan and Sato (1997).

In “the office of the future” (Figure 2-6), a user could designate every-day real surfaces in a office to be used as spatially immersive display surfaces, and then project high-resolution graphics and texts onto them (Raskar et al.,1998). Also, by using cameras, one could interpret dynamic changes on the surfaces for the purposes of tracking, interaction, or augmented reality applications. With the computer graphics and vision techniques, anything can be a display surface: a wall, a table, and anywhere.

Figure 2-6. The office of the future. Figure taken from Raskar et al. (1998).

Researchers have designed prototypes which explored integrating large displays into office computing environments. Some of them utilized large displays in conjunction with normal-sized desktop monitors (MacIntyre, 2001; Raskar et al.,1998), while others exclusively used a large display (Swaminathan & Sato1997). The thorough understanding of display effects described in this thesis will provide these researchers theoretical bases of designing high-level applications on a large display.

2.3.2 Data Visualization.

Compared to a conventional desktop monitor, a large high-resolution display has the potential to show both a broader overview and more details for a given data set, thus benefiting large data visualization tasks. Yost, Haciahmetoglu and North (2007) showed that a large high-resolution display was indeed beneficial in visual analytic tasks, since it enabled users to use their embodied resources, such as spatial memory, proprioception, and optical flow. Ball, North, and Bowman (2007) revealed that a large display encouraged a user to employ physical navigation methods (e.g., walking, crouching, moving the head), instead of the virtual navigation (zooming, panning, flying), resulting in better performance when navigating large information space. In practice, a large display has been widely used for large dynamic data visualization in meteorology (Semeraro et al., 2004), geology (Johnson et al., 2006), network traffic (Wei et al., 2000) and medical applications (Hibbs et al. 2005). By using a large display, users not only perceive a greater amount of information than on a conventional monitor, but also gain more flexibility: they can either step back to get an overall picture of the data set, or move in to study fine details without changing the visible images.

2.3.3 Virtual Reality Applications

Aside from data visualization tasks, a large display is also beneficial in virtual reality applications. In 3D navigation tasks, a physical larger display could immerse users within the problem space and bias them into using more efficient cognitive strategies (Tan et al., 2004). Therefore, even when the visual angle is identical, a user still performs better in mental rotation tasks on a large projected wall display than on a standard desktop monitor (Tan et al., 2003). In addition, as a large display provides a wide field of view, it narrows the gender gap on spatial performance: women taking a wider field of view can achieve similar virtual environment navigation performance to men (Tan, Czerwinski & Robertson, 2003).

Besides benefiting 3D navigation tasks, large displays also increase the levels of immersion and presence in virtual reality environments. For example, in the CAVE (Cave Automatic Virtual Environment), a room-sized, high-resolution 3D video and audio environment, graphics were projected in stereo onto three walls and the floor, and viewed with active stereo glasses equipped with location sensors (Cruz-Neira et al., 1992). When the user moved within the display boundaries, the correct perspective was displayed in real-time to achieve a fully immersive experience. Because of being surrounded by projected images, the user experienced a greater sense of presence compared to desktop displays. This system could also be used by multiple users at a time, albeit it only tracked the viewpoint of one viewer, and the displayed image was truly corrected only from that viewer’s point of view. In the Varrier system, a large display consisting of 35-panel 20’’ displays was used to free the user from stereo glasses (Sandin et al., 2005). This large display can produce a 2500 X 6000 autostereo imagery over a 120 degree field of view at interactive frame rates without requiring the user to wear any stereo or tracking accessories.

2.3.4 Group Work

One major advantage of a large display over a conventional desktop monitor is that it can be viewed by multiple people simultaneously, thus significantly facilitating multi-user collaboration tasks. Indeed, large displays have been widely used in a variety of scenarios to support group work.

Elrod et al. (1992) built a large, directly interactive display called Liveboard, which allowed users to interact on the surface with a cordless pen. The Tivoli system was built based on it to support an informal workgroup meeting, in which the Liveboard worked as an electronic whiteboard (Pedersen et al., 1993). Pen strokes drawn on the surface were treated as objects and can be easily stored, categorized and annotated. Flatland was another electronic whiteboard system designed for the continuous, long-term office use (Mynatt et al., 1999). It enabled users to flexibly manage the display space and apply semantic behaviours to segments on the display with pen strokes. Moreover, in the interactive Mural system, a large (6’ × 3.5’) high resolution (64 dpi) pen-based display was used to support brainstorming sessions among a small group of people (Guimbretiere, Stone & Winograd, 2001). Combined with the pen-based interaction techniques, this large display allowed users to work with computer based materials, including arbitrary applications and 3D models, with the ease of informal writing, sketching, and space management. Besides the brainstorming, a large display was also used in negotiation tasks among a group of co-located people to enhance the parallel work process (Birnholtz et al., 2007). Another example of using large displays to support group work was the i-Land system, in which vertical large displays were used in conjunction with other interactive devices such as tabletop displays and tablets to facilitate cooperative work among a group of people (Streitz et al.,1999). Other examples included the MERBoard, a touch-sensitive large display providing an immersive and interactive environment for teams to view, annotate and share data (Trimble, Wales & Gossweiler, 2003) , and the Alias Visualization Studio, in which large displays were used to support digital visual communication and collaboration with corporate clients, future customers, employees, and corporate partners (Fitzmaurice et al.,2005).

Aside from supporting collaboration among a group of people, large displays also emerged as community awareness monitors and communication facilitators in hospitals (Bardram, Hansen & Soegaard, 2006,), offices (Greenberg & Rounding, 2001; Fass, Forlizzi & Pausch, 2002), and research settings (Huang & Mynat, 2003). In the awaremedia system, a large display was situated around the hospital to help clinicians coordinate highly cooperative work ( Bardram et al., 2006). In the Notification Collage, a large display served as a real-time collaborative surface onto which distributed and co-located colleagues could post media elements (Greenberg & Rounding, 2001). A large display worked as a shared bulletin board in the MessyBoard system,

which was decorated collaboratively by a small group of users to improve the communication among group members (Fass et al., 2002). Other examples included the Semi-Public displays, which promoted collaboration and foster awareness among small co-located groups by showing lightweight information about group activities (Huang & Mynatt, 2003), and the GroupCast system, in which a large peripheral display was used to create awareness and enhance communication among people in the vicinity (McCarthy, Costa & Liongosari, 2001).

2.3.5 Public displays

In public places, large displays can serve as ambient displays to communicate with users passing by, or as interactive displays to help users access digital information.

Ambient displays

Ambient displays are considered to be a subset of peripheral displays, which communicate with users primarily via the users’ peripheral attentions (Stasko et al., 2007). As large displays can be easily seen by people walking past, they are widely deployed as ambient displays in public places. Because these large displays are often permanently located within an architectural setting, they must not only display information, but also make an attractive addition to the environment. One strategy of keeping this balance is to visualize ambient information with aesthetic adapted from art works. Redstrom, Skog and Hallnäs (2000) first created such ambient displays based on the work of Andy Warhol (“soup clock”) and Bridge Riley (“motion painting”). Other examples included the visualization of email information and world weather conditions using the designs inspired by Piet Mondrian’s work(Skog et al., 2001), and the visualization of activity levels in a café inspired by the wallpaper designs (Skog, 2004). Huang et al. (2008) reported a filed study examining the current use of large ambient information displays in public settings. The study results revealed that glancing and attention at large displays is complex and dependent on many factors. Various factors, including settings, and audiences should be taken into account when designing ambient displays.

In addition to passively presenting ambient information, public large displays can also serve as interactive surfaces. In the Hello Wall system, a wall-sized ambient display coupled with handheld devices supported three zones of interaction according to the distance to the display:

ambient zone, notification zone, and interactive zone (Prante et al., 2003). In ambient zone, the display showed information independent of the presence of any particular person, while in interactive zone, the user could interact with the large display with a hand-held device. Vogel and Balakrishnan (2004) developed an interactive public ambient display which served the dual role of public ambient or personal focused displays depending on the context inferred from various variables, including the attention level to the display, and relationship of available information to an individual currently near the display. The system covered the interactions ranging from the distant implicit public interactions to up-close explicit personal interactions.

Interactive Public Displays

Different from the ambient displays which mainly communicate with users via their peripheral attention, interactive public displays enable users to explicitly or implicitly interact with them, acting as information kiosks or public terminals. They are mainly designed for foreground activities rather than conveying ambient information. Such interactive public displays are usually deployed in offices (e.g. Russell & Gossweiler, 2001, and Churchill et al., 2003), conferences (e.g., McCarthy et al.,2002), occasional meetings (Izadi et al., 2003), parties (Brignall & Rogers, 2003), or even the center of a city (Peltonen et al., 2008).

The BlueBoard was a large plasma display with touch sensing and a badge reader to identify the individuals using the board (Russell & Gossweiler, 2001). Once badged in, a user can access and manage personal web-based content on the BlueBoard. One or more persons may log in at the same time, using it as a collaboration tool between colleagues. The Plasma Posters were Plasma displays with interactive overlays, which were usually deployed in an office environment to augment casual conversations, serendipitous information sharing, and foster an informal sense of co-presence among distributed office colleagues (Churchill et al., 2003). The information on the Plasma Posters was either posted by individuals or automatically sampled from the intranet. The user can actively read, browse, navigate the information and send other people messages via the large displays. Different from the BlueBoard and Plasma Posters, Pro-Active Displays did not require explicit interaction (McCarthy et al., 2002). They reacted implicitly to people in their proximity using RFID rags. For example, in a conference setting, when attendees approached a Pro-Active Display, their name and affiliation were displayed for other attendees to see. In

unfamiliar public places, a large display can serve as a communal multi-user interactive surface. For example, the Dynamo system allowed users to share and exchange information via a large display, promoting lightweight, visible and fluid collaborations (Izadi et al., 2003). Other interactive public display usages included facilitating social activities in parties (Brignall & Rogers, 2003), and city centers (Peltonen et al., 2008).

2.3.6 Specialized Applications

In some specialized application domains such as the automotive design (Buxton et al., 2000), emergency response (TOtal. http://www.city.chiba.jp/fire/english/systeme.html) and traffic control (Electrosonic. http://www.electrosonic.com/command profiles.asp), a large display is widely used to replace a normal monitor.

In automotive designs, a large display not only enables users to view and interact with large- scale representations of vehicles, but also enhances the informal and formal communication among a group of users (Buxton et al., 2000). For example, the traditional drafting table could be replaced by an electronic equivalent called the Active Desk, which is essentially a drafting table with a computer image projected from the rear onto the surface. Compared to using a standard desktop computer and display, this technique allows for direct, 1:1 scale interaction between the user and the digital media. Familiar physical tools such as rulers and drawing templates can also be used instead of virtual ones. In a design studio, a large display can enhance the collaborations across multiple people or car models evaluations, since the large display can render large-scale representations of vehicles and allow multiple persons to view them simultaneously.

Large displays are also widely used in time critical monitoring tasks. For example, emergency response (TOtal. http://www.city.chiba.jp/fire/english/systeme.html) and traffic control centers (Electrosonic. http://www.electrosonic.com/command profiles.asp) have been equipped with digital walls to visualize complex systems. A large display can show a greater amount of information than a normal desktop screen, thus improving working efficiency in monitoring tasks.

2.4 Challenges with Interacting on a Large Display

In this session, we review challenges and techniques for interacting on a large display with special attentions on the mice and keyboards based interaction paradigm.

2.4.1 Mice and Keyboards Based Interaction

Current desktop computing is dominated by mice and keyboards based interaction paradigm. Mice and keyboards are very efficient tools for desktop computing: keyboards are powerful devices for text entry, and mice augmented with accelerating functions perform well for accurate selection. In addition, most of current software is designed to suit mice and keyboards interaction paradigm. Retrofitting the vast number of existing software applications to accommodate other new paradigms is arguably not practically feasible. Thus, we argue that mouse and keyboards will still be the major input devices for large display enhanced desktop computing even though a variety of other input devices are emerging from HCI research community. On the other hand, most of current user interfaces are designed based on the assumption that users access much larger virtual worlds through relative small displays (i.e., normal-sized monitors). They may no longer suit when users switch to work on a large display. This section summarizes known challenges of working on a large display for mice and keyboards based interaction.

2.4.1.1 Windows Management

Given a large visualization surface, desktop users tend to open numerous windows to process increasing amount of digital information. How to effectively manage a large number of windows becomes a big challenge for large display users.

Designs addressing this challenge include WinCuts (Tan, Meyers & Czerwinski, 2004), GroupBar (Smith et al., 2003), and Scalable Fabric (Robertson et al., 2004). The WinCuts (Tan et al., 2004) enabled users to replicate arbitrary regions of existing windows into independent windows. The users can make the information fill as much or as little space as they like. The GroupBar provided task management features by extending the current desktop task bar metaphor. It facilitated task management by letting users drag and drop tiles representing open windows into high-level tasks called groups (Smith et al., 2003). Scalable Fabric managed multi- window tasks on the Windows desktop using a focus-plus-context display to allocate screen real

estate in accordance with users’ attention (Robertson et al., 2004). The user can define a central focus area as well as a peripheral region on a display surface. The focus-area windows behaved like normal desktop windows, while windows in peripheral areas were scaled down.

Since it remained unknown how large display users manage windows on a large display when these techniques were created, none of them fully resolves the window management challenges on large displays. Partially inspired by the previous explorations, the WallTop system in this thesis tackles unaddressed challenges of managing windows on a large display by designing a set of new interaction techniques.

2.4.1.2 Bezel Problems

With current available technologies, a common and economic way of constructing a large display is to tile multiple equivalent LCDs. It occupies less physical space than tiling multiple projectors and eases calibration process. One distinct character of tiled-monitor large display is that it introduces physical bezels surrounding each monitor, which present both opportunities and problems (Ball & North, 2005). The bezel lets users organize their work into different activities partitioned onto different monitors. On the other hand, it results in a visual discontinuity when an image is positioned across bezels, which makes reading text and perceiving image patterns difficult. In addition, when the cursor moves across a bezel, its path often appears deflected because there is no virtual space corresponding to the physical space that the bezel occupies (Robertson et al., 2005).

Snapping, Bumping (Russell, Drews & Sue, 2002), Mouse Ether (Baudisch et al, 2004) and OneSpace (Russell et al., 2002) techniques were developed to alleviate these problems. Snapping let users easily snap windows to bezels, other windows, or any display edges, and bumping technique automatically moved the windows to other displays or nearby empty space to avoid leaving it across a bezel (Russell et al., 2002). The Mouse Ether moved the cursor so that it followed the perceived trajectory it was on the first monitor (Baudisch et al, 2004). The OneSpace adjusted a computer’s geometric model to reflect the actual physical distance between monitors on the user’s desk (Russell et al., 2002). While this approach required hiding image material located behind the bezels, it let users view distortion-free images.

Although it has been aware that bezels on tiled-monitor large displays would affect users’ perception and performance, no previous research has ever systematically investigated these effects. The study in Chapter 5 is the first to comprehensively investigate the effects of internal bezels on users’ basic perception and interaction abilities.

2.4.1.3 Losing the Cursor

As the screen size increases, users tend to accelerate cursor movement to quickly traverse a long distance. The faster the cursors move, however, the more likely users are to lose the track of them. Furthermore, as the screen size increases, it also becomes increasingly difficult to locate a stationary cursor (Robertson et al., 2005).

To alleviate this problem, some new types of cursor are designed. The high-density cursor helped users to keep track of the cursor movement by filling the space between the cursor’s current and previous position with additional fill-in cursor images (Baudisch, Cutrell & Robertson, 2003). Windows XP Operation system provided a locator function to help users find a stationary cursor: an animated circle appeared over the cursor whenever users press the control key. To facilitate cursor locating when it is moving, auto-locator cursor is proposed (Robertson et al., 2005). Whenever the cursor quickly moved a long distance, it automatically invoked an animated location indicator.

2.4.1.4 Invisible Content Monitoring

A large screen can display a large amount of information; however, the large visualization surface might go beyond a user’s visual acuity, which means that some dynamic visual changes may happen without the user being aware of them. Studies have shown that people are rarely able to spot visual changes when they occur during disruption as short as eye saccades (Rensink, 2002), unless explicit notification or history mechanisms are provided (Freeman & Fertig, 1995; Rekimoto, 1999; Renaud, 2000). A variety of approaches have been developed to help users keep track of changes of visualization content. Baudisch et al. (2006) used animation and abstraction to indicate changes in user interface elements. This set of designs assumed high-level system knowledge of the data presented on the screen and the types of dynamic changes that were likely to occur. Bezerianos, Dragicevic and Balakrishnan (2006) proposed an image-based storage, visualization, and implicit interaction paradigm called mnemonic rending to help users 25

understand changes happening out of the user’s sight. Visual cures such as a mask around the targeted window and trails leading to the targeted window were also designed to direct users’ visual attention in windows switching tasks (Hoffmann, Baudisch & Weld, 2008).

2.4.1.5 Distal Access

As the display size increases, it becomes increasingly difficult for users to access distant information. For example, accessing an icon or window at a distance requires moving the cursor a long distance, which is time-consuming and raises cursor tracking problems. This problem is even exacerbated in up-close interaction: some screen real estate is unreachable by physical hands.

Space Distortion. One approach of easing distant object access is to scroll the entire virtual canvas so that remote content moves toward the user for comfortable viewing and manipulation (Bezerianos & Balakrishnan, 2004). A shortcoming of this technique is that it may cause a significant amount of visual disruption thus disorienting users since the display covers most of a user’s visual field.

Alternative views. Another approach is to use alternative views of different display locations as visual and interaction shortcuts. For example, Stoakley, Conway and Pausch (1995) used WIM (World in Miniature) views to access content in immersive 3D virtual environments. Khan et al. (2004) introduced the Frisbee, a widget that acted as an interactive telescope to a remote area on a display. Bezerianos and Balakrishnan (2005) introduced the canvas portal to access remote objects. Alternative view shortcuts bring the distant objects closely to aid content access. However, they segment the visual space and identifying the exact location they point to may be problematic.

Bridging large interaction distances. Bridging large interaction distances may also facilitate remote reaching. Guiard, Blanch and Beaudouin-Lafon (2004) introduced the object pointing, where the cursor skipped empty space, jumping from one selectable target to another. Similarly, the bubble cursor always selected the closer target to the pointer by adjusting the pointer’s size (Grossman & Balakrishnan, 2005). These techniques aid remote target selection in sparsely populated wall displays, but in dense environments intermediate items may exist between the

user and target, so the user needs to physically move closer to the target. Virtually picking up and dropping content was also proposed to facilitate objects movement on a large display (Rekimoto, 1997). Although picking and dropping items are likely more efficient than dragging them across the display, neither might be feasible when content is unreachable. Copying content close to the user’s location is another approach for remote access. Drag-and-pop and push-and- pop brought the remote icons to the user’s location (Baudisch et al., 2003), and the Vacuum, a circular widget, facilitateed the manipulation of distant objects by bringing far away objects into the widget’s center in the form of proxies (Bezerianos & Balakrishnan, 2005) .

Amplifying the user’s input. Other remote access approaches includes amplifying the user’s input, such as the Missle Mouse (Robertson et al., 2005), HybridPointing (Forlines, Vogel & Balakrishnan, 2006), virtual hands extension (Pierce & Pausch, 2002; Poupyrev et al., 1996), and throwing technique (Geißler, 1998). The Missile Mouse enabled a user to “launch a missile” (the cursor) across the screen with a small movement (Robertson et al., 2005). The cursor continued moving in the direction and at the speed of the original motion until the user moved the mouse again, reacquiring the control. The HybridPointing let a user easily switch between absolute and relative pointing with a direct input device such as a pen (Forlines et al., 2006). In the relative mode, users can easily manipulate distant targets, as small movements of the input device are mapped to large movements of the pointer. In Virtual Reality environments, reaching and manipulating distant objects can be easily achieved by virtually growing a user’s arms (Poupyrev et al., 1996). In Geibler (1998), users could send a digital object to other users standing at the opposite side of the display by simply “throwing” it.

2.4.2 Other Interaction Methods

Aside from traditional mice and keyboards based interaction paradigm, numerous interaction methods have been proposed to accommodate unique characteristics of a large display. These techniques fall mainly under two categories: remote and up-close interaction. Remote interaction enables a user to perceive more information than standing close, which is usually preferred when working on non-interactive surfaces (Lund, 1997). In contrast, up-close interaction usually occurs on touch-sensitive displays. It enables a user to manipulate the content or view information in detail.

2.4.2.1 Remote Interaction

Traditional remote interaction devices are keyboards and mice. Aside from them, other devices including isotonic flying mice (Zhai, 1998), hand held isometric devices (MacKenzie & Jusoh, 2001), soaps (Baudisch, Sinclai & Wilson, 2006), mobile phones, and vision based virtual touch pads (Malik, Ranjan & Balakrishnan, 2005). Some of these approaches require the user to rest arms on a desk, which might be more comfortable than direct pointing or touching on the display, but confine the user to a single sitting and viewing position.

Remote pointing, including direct hand pointing (Bolt, 1980; Zhai, 1998; Vogel & Balakrishnan, 2005) and the use of laser pointers (Winograd & Guimbretiere, 1999; Olsen & Nielsen, 2001; Myers et al., 2002), has also been used for large display interaction. For example, Bolt (1980) investigated the combination of direct pointing and voice to disambiguate contexts. Hinckley et al. (1994) and Zhai (1998) studied the issues when using the hand for 6 DOF tasks. In VR environments, direct pointing was used to select objects with the use of a button (Bowman & Hodges, 1997). Furthermore, Vogel and Balakrishnan (2005) proposed direct pointing and selection techniques requiring no buttons or other physical aids. In addition to hands, laser pointers are also used for remote pointing. In interactive Mural, an optically tracked laser pointer was used as a remote input device (Winograd & Guimbreti`ere, 1999). Olsen and Nielsen (2001) designed a series of laser pointer based interaction techniques. Direct pointing is a very natural mechanism for referring to remote content, and provides users more movement flexibilities than traditional mice and keyboards (Kendon, 2004). However, they are inaccurate due to hand jitter (Myers et al., 2002) and fatigue-prone because of the lack of a physical surface to provide hand supports (Hinckley et al., 1994).

Besides remote pointing, researchers explore using mobile devices as input devices interacting with public large displays. Ballagas et al. (2005) implemented Sweep and Point & Shoot techniques which enabled users to use a phone like an optical mouse to interact with a large public display without having to point the camera at the display. Dearman and Truong (2010) presented Bluetone, a framework which allowed users to interact with a large display using dual- tone multi-frequency sounds transmitted from mobile phones paired with the display using the Bluetooth headset profile. Boring et al. (2010) implemented Touch Projector, a system that enabled users to interact with remote screens through a live video image on their mobile devices. 28

As mobile devices are becoming ubiquitous personal computing devices, they are widely used as ad hoc input devices interacting with public large displays. However, due to their limited functions, they are usually used for casual and light weight interactions (e.g., moving objects around), rather than intensive information processing (e.g., editing text-rich documents).

Other remote interaction devices include wands (Cao & Balakrishnan, 2003), cameras (Jiang et al., 2006), and hand-held projectors (Cao & Balakrishnan, 2006). These approaches often introduce uncommon interaction paradigms for casual commonly performed tasks. Moreover, they also suffer from hand jitter since they operate in mid-air and are in general not well suited for detailed and prolonged manipulations.

2.4.2.2 Up-close Interaction

When users are standing up close to a large display, they mainly rely on two hands or a digital pen to interact. Direct hand touch has emerged as an interaction approach since the work by Buxton (Buxton, 1997). Currently, technique advances enable the creation of touch sensitive surfaces ranging from two (Smart Technologies. http://www.smarttech.com/dvit/), to multiple points sensing (Han, 2005). Furthermore, even an entire hand gesture can be interpreted as input on a touch-sensitive large display (Wu & Balakrishnan, 2003).

Grabbing a marker pen and subsequently sketching on a vertical white board is now a common activity in offices. Inspired by it, digital pens emerge as input devices when users are standing up-close to a large display. Liveboard from Xerox was the first step toward a large vertical display integrating pen input (Elrod et al., 1992). Other examples include the Tivoli (Pedersen et al., 1993), FlatLand (Mynatt et al., 1999), and interactive Mural (Guimbreti`ere et al., 2001).

Other uncommon physical devices such as graspable handles (Bae et al., 2004), electronic trackers (Balakrishnan et al., 1999), and bricks are also used as input devices on a large display (Fitzmaurice, Ishii & Buxton, 1995). Typically, these devices are only used for specific interaction purposes. For example, graspable handles (Bae et al., 2004), electronic trackers (Balakrishnan et al., 1999) were used for curve creations, and bricks were used in 2D graphic drawing and 3D modeling (Fitzmaurice et al., 1995).

2.5 Managing Desktop Work

Information management is a core activity of current desktop computer users. As the total amount of information keeps increasing, how to effectively manage it becomes a big challenge. In this session, we review the literature regarding information management.

2.5.1 Document Management Behaviors

A sizable amount of work has been conducted to study how information workers manage documents in both digital and paper domains. Bondarenko and Janssen (2005) reported the results of a two-year ethnographic study of the personal document management of information workers. Their results revealed that document management was strongly related to task management. To better support task management, documents should be embedded within meaningful (task-related) context information, and they should be easily accessible for regrouping as the task goes on. Physical papers supported these needs very well. In contract, digital tools such as emails, digital file folders did not adequately support these two needs. Malone (1983) summarized two strategies regarding naturally and conveniently managing digital information from a series of interviews: 1) desk organization was to remind the users of things to do, not just to help the user find desired information. 2) computer-based systems might help office workers reduce the cognitive difficulty of categorizing information by a) doing automatic classification and b) including untitled “piles” of information arranged by physical location as well as explicitly tilted and logically arranged “files”. These strategies could better support current information workers who found themselves in multitasking environments: tasks were usually interrupted before they were completed and new incoming tasks were often postponed (Czerwinski et al. 2004). Barreau and Nardi (1995) studied the ways users organized and found files on their computers. Their work reaffirmed the importance of reminding: finding and reminding were intimately linked in information organization and they should be considered together. Malone (1983) identified two basic strategies for handling paper documents: filing and piling. Filers maintained clean desktops, and did not allow papers to pile up. They systematized their archives (using alphabetical, conceptual, or temporal methods) to support straightforward access to stored data. In contrast, pilers had messy desktops cluttered with paper piles, making few attempts to organize stored information. Whittaker and Hirschberg (2001) further examined the effects of paper-processing strategies on archive structure. They discovered that different 30

paper-processing strategies (filing and piling) were relatively independent of job type, and filers amassed more information, and accessed it less frequently than pilers. They also argued that filers might engage in premature filing: to clear their workspace, they archived information that later turned out to be of low value. Prior research revealed how users might manage information, including both digital and paper-based documents in a desktop context. This knowledge deepens our understanding of office workers’ information management activities, and also partially explains large display users’ window management behaviour patterns reported in Chapter 3.

2.5.2 Advantages of Paper Documents

In theory, all paper documents nowadays can be handled digitally. However, paper documents are still anywhere in modern offices (Sellen & Harper, 2003). Prior research has demonstrated that paper documents have unique affordances and properties which are advantages over typical digital tools. In this session, we briefly review the literature studying the properties of paper documents, which inspires the designs of interaction techniques in WallTop, a large-display- orientated window management system prototype (Chapter 6).

Sellen and Harper (2003) extensively described the unique affordances of paper. The physical properties of paper (its being thin, light, porous, opaque, and flexible) afforded the human actions of grasping, carrying, folding, writing, and so on. Compared to digital tools (paper as analytic tool), papers better supported authoring, editing, reading, collaborating and annotating. Whittaker and Hirschberg (2001) further discussed the values of paper documents for information workers. Compared to digital information, paper was more available and better supported reminding. Contrary to the instinct, paper information was also valuable for the long- term storage. Despite various digital tools for transferring information, paper was still an important mechanism for delivering information in the modern offices (Sellen & Harper 1997; Whittaker et al., 1995). After examining the effects of paper-processing strategies on archive structure, Whittaker and Hirschberg (2001) suggested making digital data mimic critical properties of paper, such as supporting reminding, and maintaining the context, to effectively manage digital information.

In addition to information management, papers have advantages over the digital counterparts for other tasks. Hara and Sellen (1997) found that reading with paper had major advantages over 31

reading on-line because paper better supported annotations while reading, quick navigation, and flexibility of spatial layouts. Despite the widespread introduction of information technology into health care, medical practitioners continued to use the more traditional paper medical records often alongside the computerized system. The paper records better supported the collaborative work and utilized the conceptions of ‘writers’, ‘readers’, ‘objects’ and ‘records’ (Heath and Luff, 1996). Luff et al., (1992) also discovered that paper documents were being used to perform certain collaborative tasks even though a computer system had been introduced to carry out those tasks. This was not only due to the intrinsic properties of paper or the intrinsic constraints of screens. It was also because paper afforded an interactional flexibility that allowed individuals a range of ways of participating in tasks-in-interaction.

2.5.3 Using Paper Metaphor in Digital Information Management

Inspired by the advantages of paper documents, researchers have explored managing digital information using a paper metaphor. Johnson et al. (1993) built a system called XAX exploring bridging the paper and the digital worlds. The user interface moves beyond the workstation and onto paper itself. Mackay and Pagani (1994) developed the Video Mosaic, a system which allowed users to edit video elements using both paper story boards and on-line editing tools. Paper provided the user with the ability to lay out various temporal sequences over a large spatial area and the ability to quickly sketch, annotate and rearrange the relevant video clip. Newman and Wellner (1992) explored augmenting ordinary paper documents with computing capabilities.

In addition to augmenting physical paper with computing functions, paper has been used as a metaphor for various tasks. Whittaker et al. (1997) supported lightweight interpersonal communications by making communication tasks tangible as “yellow sticky” reminders on a computer desktop. Henderson and Card (1986) introduced the “room” metaphor that allowed users to associate sets of digital documents with different tasks, which generated context when switching tasks. Mander et al. (1992) utilized piling to organize documents on a desktop, and implemented gesture-based piling operations. Agarawala and Balakrishnan (2006) developed BumpTop, a virtual desktop based on physical simulation, which used the piling as the fundamental structure to organize desktop icons. Watanabe et al. (2007) designed bubble clusters for manipulating desktop icons and digital ink; aggregated objects were automatically recognized

as a group so that the user can drag, copy, or delete them effectively. Distinct from the work organizing desktop icons and digital ink, the Walltop focus on managing windows, thus leading to unique visualization design and multi-object operations.

2.6 Summary

This chapter firstly surveys previous research about using normal-sized monitors (Section 2.1), and multiple-monitor (Section 2.2) to process desktop work. Most of the research converges on a conclusion that large screen real estate improves desktop working experience. A large display, which is physical larger and has higher resolution than a normal-sized monitor or multi-monitor, might substantially benefit desktop users. The early-stage explorations described in Section 2.3 initially reveal the effects of tiled-monitor large displays and large low-resolution projected screens. Building on this knowledge, the comprehensive studies in this thesis will provide in- depth understanding of effects of a contiguous large display on desktop work. Section 4 reviews interaction techniques on a large display, with special attention on mice and keyboards based interaction paradigm. Section 5 summaries prior research regarding information management. This knowledge partially explains large display users’ window management behaviour patterns reported in Chapter3, and also inspires us to design new interaction techniques in WallTop (Chapter 6).

3 Longitudinal Diary Study of Large Display Usage

Traditional desktop computing environments can be classified into two categories: single or multiple normal-sized monitors. We start investigating the effects of a large display by comparing it with these two traditional setups in realistic desktop work. Through analyzing users’ detailed interaction activities, we are able to discover large display users’ unique behaviour patterns, summarize its potential pros and cons, and identify the challenges with working on a large display.

More specifically, this chapter presents a longitudinal diary study that investigated users’ behaviors when they switched from standard computing environments (i.e., single- or dual- monitor) to a large high-resolution display for a five day period. By recording participants’ daily activities, window events, and mouse operations, we looked at how users utilized and partitioned screen real estate, and managed windows in different display configurations. Based on the results, guidelines for large display interface design were formulated.

3.1 Study Design

Our goal is to investigate how a person uses a large high-resolution display for daily information processing work. In particular, we aim to compare large display use with traditional personal computing environments (i.e., single- or dual-monitors). Rather than conducting a controlled experiment to examine individual aspects in isolation, we carried out a diary study in a more realistic context, allowing us to explore usage in a broad range of computing activities over a five day period.

Participants

We recruited eight participants by posting on-line advertisements: four who used a single- monitor, and four who used a dual monitor configuration for their daily computing. Two single- monitor users (S1, S2) usually worked on a 17’’ LCD, and the other two (S3, S4) used a 21’’ LCD in their daily work. Two of the dual monitor users (D1, D2) worked on two 18’’ LCDs, one

(D3) used a 21’’ in conjunction with an 18’’ LCD, and one (D4) used two 21’’ LCDs. S4 and D2 were female and the others were male. All participants used computers over 5 hours per day. S3, D2, and D3 were graduate students from Computer Science Department; S1, D1 and D4 were graduate students from Electronic and Computer Engineering Department. According to the pre- experiment questionnaires, their daily computing activities were programming, documents reading, writing, web browsing, and emailing. S2 was a graduate student from Chemical Engineering Department. His major activity during the two weeks of the study was writing up the PhD thesis. S4 was an unemployed person and her major activity was seeking job information on-line. Each participant ensured that he/she performed similar computing tasks during the two weeks of the study. All of the participants were Window XP users and a variety of software was installed on the large display computer to ensure that they can carry out the daily work. The study was conducted in a locked office to ensure the privacy. Each participant received $50 compensation upon the completion of the study. We chose experienced computer users as we wanted to see how their interaction and visualization strategies might change when they moved to a single large high-resolution display; in contrast, inexperienced users would not have a baseline of strategies that they would need to adapt when moving to our large display setup.

Apparatus

Figure 3-1. The large display in the study.

We used a 16’ wide × 6’ high display (Figure 3-1), comprised of 18 projectors in a 3 × 6 tiling. The 18 projectors were carefully calibrated to minimize the seams between them (i.e., the width of each seam < 1mm). None of the participants reported that the seams affected their performances in the experiments. We also carefully adjusted color and brightness across all projectors to minimize the variation between them. Each projector had a resolution of 1024 × 768 pixels, for a total display resolution of 6144 × 2304 pixels. This display is significantly larger and higher resolution than those used in earlier large display studies (Ball and North, 2005; Bishop and Welch, 2000; Czerwinski et al., 2003). As there is no operating system specific to large displays, we ran standard Windows XP on a computer with several multi-headed graphics cards that could drive the multiple projectors concurrently. Various applications were also installed to ensure that participants could perform their daily work.

Design

Participants were categorized into two groups according to their normal computing environments: single-monitor or dual-monitor users. Participants switched to exclusively use the large display as their daily computing environment for five consecutive days, with 5 hours per day. Thus, each group had two working conditions: their normal working environments (i.e., single or dual monitors) as well as the large display condition. The study ran over an eight-week period, with each participant using the large display for one week.

Participants were asked to perform their daily routine work on the large display. Throughout the study, the participant was the only person in the large display room, ensuring a personal office scenario and maintaining the user’s privacy. Since the optimal position and distance of using the large display was unclear, participants could freely adjust the sitting position and distance in the study.

To understand how a user worked on the large display, we employed the following observation methods:

Activity Log

Each participant maintained a daily activity log when working on the large display, in which she wrote down the activities every half an hour. Figure 3-2. shows parts of a daily activity log. The 36

participant briefly described the activities that had occurred in the past half-hour, recorded the approximate distance to the display, and listed the advantages/disadvantages of having performed these activities on the large display. Finally she ranked the large display as “better”, “worse”, or “equal”, in comparison to performing these activities in her normal computing environment.

Daily Interview

To gain further insight of users’ activities and examine the issues not covered by the daily activity log, a follow-up structured interview was carried out at the end of each day. The interview occurred right after the participant finished their daily work on the large display. The whole process usually took one hour, and was taped for later analysis. In the interview, participants were asked to elaborate on the events recorded on the activity log, describe screen real estate usage, and talk about the pros and cons of using a large display. Throughout we asked the users for more detail if the explanations given were unclear. However, in order not to bias the outcome, we paid special attention to ensure that the additional questions were non-suggestive and only clarifying in nature.

Figure 3-2. Example activity log

Recording Windows and Mouse Events

We ran an application called VibeLog (Hutchings et al., 2004) on the large display to record every window event that occurred and detailed information of each running window. It

maintained two kinds of logs: events log, and windows log. The events log had an entry for every window management event that occurred. These activities included closing, activating, moving, resizing, minimizing, and maximizing a window. Each log entry had a timestamp and the title of the window operated. Complementarily, the windows log created a series of entries each minute, one entry for each open window on the system. The entry contained detailed information about the corresponding window, including the spatial coordinates, size, status of the window, and the window’s z-order on the desktop.

We also recorded every mouse event using an application written within our lab called MouseLog. It created an entry for every mouse activity, which contained the spatial coordinates of the mouse cursor, and event type (e.g., left button down/up, right button down/up).

Both VibeLog and MouseLog worked by programmatically hooking into the public window system events made available by Windows XP. They occupied little system resources and did not interfere with normal computer use.

These four recording means are complementary to each other. The daily activity log provides data about a participant’s real-life activities in the workplace, while the follow-up interview extends those results with more qualitative descriptions. However, since the activity log and interview both require that the participants self-report their activities after the fact, they may fail to recall all activities of interest, or their beliefs about the study’s purpose may bias their selection of events to report. Complementarily, the VibeLog and MouseLog objectively record user activity, which can not only corroborate the results drawn from the activity logs and interviews, but also reveal interesting activities that might be missed by the self-report.

Note that before a participant switched to using the large display, VibeLog and MouseLog were deployed in her normal computing environment for five working days to record windows and mouse events. This data served as a baseline. When working on the large display, all four recording methods were employed to gather richer data. All participants reported that they were doing similar work during these two weeks (one week in the normal computing environment, and the other on the large display). We did not do the activity log or daily interview during the five days of standard computing environment usage as we felt it would impose undue demands on our participants who were already committing significant time to this study, while any new 38

insights gained would likely be minimal since standard computing environments have already been well studied.

3.2 Results

Rather than report the results gleaned from each of our four logging methods separately, we instead present a more holistic analysis that extracts the most interesting findings from all our data taken as a whole.

3.2.1 Overall Activities

Table 3-1 shows the categories of activities and the time logged on those activities across all the participants on the large display. The categories that account for most of the hours are web browsing, word processing and reading papers, with each of them constituting more than 20% of the total hours logged. Moreover, all eight participants performed web browsing, word processing, reading papers, and emailing tasks. These four tasks take up nearly 80% of the hours logged.

Table 3-1 Hours logged per major activity on the large display # of participants Activities Hours Percentage reporting activity

Web Browsing 54.5 25.9% 8

Word Processing 48.3 23.0% 8

Reading Paper 42.0 20.0% 8

Emailing 21.8 10.4% 8

Programming 12.4 5.9% 5

Data Analysis (Excel) 10.5 5.0% 3

Preparing presentation 6.5 3.1% 3 slides

Chatting (MSN or 5.0 2.4% 6 GTalk)

3D modeling 5.0 2.4% 1

Graphic Drawing 4.5 2.1% 2

Total 210.5 100%

3.2.2 Subjective Opinions

Figure 3-3 shows each participant’s subjective opinion comparing the large display with their normal computing environments. Perhaps somewhat surprisingly, all participants overwhelmingly preferred the large display: for each participant, “Better” and “Equal” constitute more than 90% of the rankings and more than 50% of the rankings are “Better”. All participants reported hoping to obtain more screen space for their computers. One dual-monitor user (D1) commented:

“When I was working on a 15 inch monitor, I thought a 19’’ LCD would be much better. When I switched to a 19’’ one, I found it was still too small. Now I am using two 18’’ monitors, but I still hope to get more screen space”

In particular, single-monitor user S4 always ranked the large display “Better” or “Equal”. Her activities on the large display included web browsing, word processing, emailing, on-line chatting, and 3D modeling. She expressed a strong preference for working on the large display, commenting:

“The larger one is pretty better. It [The large display] offered me much flexibility of doing daily work. I can choose the optimal amount of screen space according to the task at hand.”

Worse Equal Better 30

15 Hours 10

0 S1 S2 S3 S4 D1 D2 D3 D4

Figure 3-3. Participants’ subjective opinions of the large display compared to the traditional single- or dual-monitor configuration. (S1, S2, S3, and S4 represent four single- monitor participants, while D1, D2, D3, D4 are four dual-monitor users)

Figure 3-3 also demonstrates that single-monitor users have a stronger preference for using the large display than dual-monitor users. Single-monitor participants rated the large display “Better” for 81% of the hours while dual-monitor participants did so only for 61% of the working time. The relatively smaller screen space on a single monitor might make single- monitor users feet like they were benefiting more from the large display than dual-monitor users.

Figure 3-4 breaks down the rankings by task. Both single- and dual-monitor users unanimously preferred the large display to their normal computing environments across all the tasks: for each task, “Better” and “Equal” constituted more than 80% of the rankings. In particular, the large display was always rated “Better” or “Equal” for programming, data analysis (excel), and 3D modeling.

Worse Equal Better 100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0% SM DM SM DM SM DM SM DM SM DM SM DM SM DM SM DM SM DM Word Web Reading Emailing Programming Data Preparing Graphic 3D Processing Browsing Paper Analysis Slides Drawing Modeling

Figure 3-4. Participants’ subjective opinions of the large display compared to the traditional single- or dual-monitor configuration, distributed by task. (SM and DM represent the four single-monitor and four dual-monitor participants, respectively).

Three single-monitor (S1, S2 and S3) and two dual-monitor users (D1 and D3) performed programming tasks on the large display. All reported preferring the large display, because it allowed them to simultaneously view multiple windows, which they felt improved their work flow. For example, D1 coded in JAVA on the large display. He kept a Java NetBeans window, an on-line help document, an electrical dictionary, and a Google search webpage visible concurrently. The Java NetBean was the primary coding window, the on-line help document and the dictionary helped him to check unfamiliar Java functions, and the search page was used to search other help information from the internet. Keeping all these four windows visible eased the access to any of them. In contrast, in his normal dual-monitor setup, he can keep just two of these applications visible, thus requiring frequently interleaving actions.

Spreadsheet data analysis received the highest “Better” ranking (91%). Two dual-monitor (D2 and D4) and one single monitor user (S2) analyzed data using Excel on the large display. All of them reported benefiting significantly from the larger display, because of the huge number of spreadsheet cells that could be kept visible simultaneously.

One single monitor user (S4) used Maya software to build 3D models on the large display. She rated the large display “Better” for more than 80% of her time. The large display can concurrently visualize the several sub-windows she had to use concurrently while working in Maya, thus significantly improving the work flow.

The large display received the highest “Worse” ranking (12%) for web browsing tasks. Participants reported that rendering a large image was slow and there were some usability problems with the web browser (Internet Explorer) as some interface elements did not scale correctly to the large display. For example, activated menus may appear at unexpected positions on the large display.

3.2.3 Screen Space Partition

All the dual-monitor participants reported mentally partitioning the screen real estate into focal and peripheral regions in both the dual-monitor and large-display, while single-monitor participants did so only on the large display. Generally, the focal region is used for primary tasks – writing code, word processing, or drawing a graphic image – to which most interaction activities are devoted over time, while the peripheral region is used for applications that are secondary to the primary tasks, such as email clients, instant messaging clients, and personal “to do” lists.

While the dual-monitor participants partitioned screen space in both the dual-monitor and large- display conditions, they did so differently in these two circumstances.

The distribution of mouse events shows differences in location of both focal and peripheral regions (Figure 3-5). On dual monitors, participants performed more activities in one monitor than in the other: 71% of mouse events occurred within one monitor, and 29% occurred in the other. According to dual-monitor participants’ reports, one monitor was usually used as the focal region, and the other as the peripheral. Due to monitors’ physical bezels, spanning a window across two monitors suffers from visual discontinuities, so the primary activities were usually restricted to one monitor. However, when dual-monitor participants worked on the large display, 81% of the mouse events occurred in the center region of the large display, and 19% occurred in the remaining ”inverted-U” shaped area (Figure 3-5). The center region was used as the focal

region, while the rest of the display was used as the peripheral region. This distribution might be explained by users’ sitting positions. As all the participants sat in front of the horizontal centre of the large display, the center part was the closest region to the users, making it convenient to view and interact with. Consequently, users were more likely to position focus-required work in this part, and peripheral applications requiring less attention slightly further away.

Dual Monitors Large Display 800 1.8

19% of mouse events 600 1.2 71% of mouse 29% of mouse events events 400 81% of mouse events 200 Height (m)

0 1.4 3.5 4.9 Two LCDs 128 X 128 pixels Length (m)

Figure 3-5. Mouse event distribution on dual monitors and the large display, for dual- monitor participants3.3.4 Window Management

3.2.4 Window Management

3.2.4.1 Arranging Windows

Aside from differences in location of focal and peripheral regions, participants managed, interacted with, and displayed windows on the large display in distinct ways than on dual monitors. One major distinction is related to how users arranged their application windows. Due to the relatively limited and divided screen space on the dual monitors, participants usually invested little effort in arranging application windows in this condition. One dual-monitor participant (D2) reported:

“I just throw all the non-primary applications to the secondary monitor, and do not care much about the layout. If I need to use one of them, I maximize it by a simple click.”

However, on a large display, participants tend to expend more effort to optimize the layout of application windows to improve workflow. One participant reported that at the beginning of each day, he spent nearly 5 minutes arranging the applications. He commented:

“It takes a certain amount of time at the beginning of each day, but worthwhile, because this optimal layout help me greatly afterwards”.

In particular, post-interviews revealed that there are two common strategies for arranging windows.

 The first is to position the applications requiring interaction activities, such as MS Word and instant messaging clients, close to the center of the screen, while the applications only passively displaying information, such as a weather forecast window or a calendar, on the side or corner of the screen. Figure 3-6 shows a layout using this strategy captured during one of the participant’s usage sessions. This strategy aims to facilitate the interaction with the applications beside the focal region.

 The second strategy is to arrange windows according to their relevance to the primary task. The more relevant, the closer the application is to the primary task window. For example, when drawing a car, the canvas window was positioned in the center, and surrounded by applications displaying reference materials. Other applications less relevant to the drawing process were placed further away. This strategy is similar to what office worker adopt for managing paper documents on an office table. Paper documents can be defined as three types: “hot”, “warm”, and “cold”. They refer to the documents that are actively used at the current moment, those that were just in use or will be used in the nearest future, and documents that are not used at the moment respectively. Bondarenko and Janssen (2005)’s study revealed that the “hot” and urgent documents were usually moved to the focus of work space (e.g., keyboard and PC screen on an office desk). Similarly, our study showed that large display users tended to place the windows closely related to the primary tasks to the working focus (the center of a large display), and less relevant windows to the sides of the display.

Figure 3-6. A screen capture of the large display from one participant’s session, illustrating how the screen real estate is implicitly partitioned into focal and peripheral regions.

Although large display users tend to invest more effort in arranging windows, it is worthwhile. The visuospatial appearances of these filed/piled windows (sizes, shapes, locations, the way they are grouped, etc.) implicitly contain context information about the task, information that represents the stage the task is at. This context information could help users more effectively process daily work (Bondarenko and Janssen, 2005).

3.2.4.2 Interacting with Windows in the Peripheral Region

Another difference is how users interact with peripheral windows. According to the interviews, users reported that they just slightly turned their heads and bodies to work on applications on the secondary monitor in dual-monitor conditions, such as email and instant messaging clients. These applications were dragged into the primary monitor only when users interacted and focused on them for a long period of time. This behavior occurred because two monitors were usually placed close to each other, requiring only a slight head or body movement to shift attention.

In contrast, when interacting with peripheral windows on a large display, users reported that they often dragged the windows from the peripheral into the focal region, and seldom turned their heads and bodies. This action likely resulted from these peripheral windows usually residing on the sides or corners of the large display, thus requiring plenty of head and body movements to

shift attention. Also, concentrating on distant applications is difficult, which discourages working with them in the peripheral region.

Take replying to emails for example. On dual monitors, the email client usually resided on the secondary monitor. A user (D3) just slightly rotated his head and body to reply to the email whenever a new message arrived. However, when working on the large display, he frequently switched the email client back and forth between the focal and peripheral regions: dragging the email client into the focal region to reply, and then sending the application back to the previous location afterwards. He commented:

“Although dragging the application back and forth between side and center region is time consuming, it is still more comfortable than rotating my head and body to interact with it”.

3.2.4.3 Visualizing Applications in the Peripheral Region

On a large display, since the peripheral region is distant from the seating positions and viewed from more acute angles, pictures and text in this region are usually magnified to ensure clarity and facilitate viewing. As seen in a screen shot (Figure 3-6), the picture within the weather forecast window is magnified by 200%, and the font size for the “to do” list is enlarged to 25 from the usual 10 on a normal desktop screen. A similar enlargement of peripheral content did not occur in the dual-monitor condition.

3.2.4.4 Window Operations

All the window management activities are executed through basic window operations. To gain a deeper insight into unique window management behavior on a large display, we compared basic window operations across different conditions. Figure 3-7 shows the distribution of different window management operations for each person. One common characteristic across all the participants is that moving and resizing operations constitute much higher percentages of total operations on the large display than in their normal computing environments. For single-monitor users, the mean percentage of moving plus resizing operations is 51.5% (std. dev. = 2.4%), on the large display and 16.7% (std. dev. = 6.1%) on the single monitor; for dual-monitor users, it is 58.5% (std. dev = 2.4%) on the large display and 29.5% (std. dev. = 9.7%) on the dual monitors. Whenever a new application was opened, the

following operation was to optimize its position and size. Moreover, frequently switching applications back and forth between the peripheral and focal regions also caused more moving operations on the large display.

Resize Move Maximize Minimize Destroy Activate 100%

80%

60%

40%

20%

0% SM LD SM LD SM LD SM LD S1 S2 S3 S4 y 100%

80%

60%

40%

20%

0% DM LD DM LD DM LD DM LD D1 D2 D3 D4

SM: Single Monitor DM:Dual Monitors LD:Large Display

Figure 3-7. Distribution of window management operations per participant. (S1, S2, S3, and S4 represent four single-monitor participants, while D1, D2, D3 and D4 are four dual- monitor participants).

Another interesting finding is that dual-monitor participants performed more window moving and resizing operations than single-monitor participants in their normal computing environments. However, when using the large display, both single- and dual-monitor participants performed similar percentages of moving and resizing operations, which were much higher than those in their normal environments.

In contrast to moving and resizing, maximizing and minimizing operations constitute a much lower percentage of actions on the large display. Only one (S2) out of eight participants ever maximized windows on the large display. S2 did so to show a Google map. Other participants reported never spanning a window across the entire large display surface. When they needed to visualize windows containing rich information, 60~70% of the entire display surface was typically sufficient. This is a particularly interesting finding in that it indicates an upper bound on window sizes that users are comfortable working with. Minimizing operations were often performed in single- or dual-monitor conditions to save screen space, but it was rarely used on a large display due to the ample available space.

3.2.5 Seating Location Relative to the Display

As the optimal seating location when working with a large display is unclear, participants were allowed to freely adjust their seating during the study. Wheels were mounted on the table and chair to enable easy mobility. All of the participants reported sitting in front of the horizontal centre of the display, because they felt that their visual fields covered most of the screen space from this position. Figure 3-8 shows the durations of all the sitting distances from the display which were self-reported on the activity log. As shown, the distances range from 0.9m to 3.0m, with more than 90% of data between 1.5m to 3.0m. Most relevant is that participants sat within 2.0m to 2.5m of the display nearly 50% of the time.

When participants worked in their normal computing environments, the distances to the screen ranged from 0.75m to 0.85m. All participants reported sitting slightly further when switching to the large display because they wanted to view more of the screen at once. One single- (S2) and one dual-monitor participant (D3) reported that sitting too close (less than 1.0 m) to the large display made them feel like sitting facing upclose to a wall, which was very uncomfortable. However, sitting too far away can also make viewing details on the display difficult. It seems that a distance of 1.5m to 2.5m was the optimal range for most of the participants, a distance at which they could clearly perceive the content on the large display and their visual field covered sufficient screen real estate.

100 95 90

70 64 60

Hours (h) Hours 40 40

20 10.5 10 1.0 0 0.5 1.0 1.5 2.0 2.5 3.0 Distance (m)

Figure 3-8. Time spent at different distances to the large display

3.3 Discussion

3.3.1 Benefits of Large Displays

As shown in Figure 3-3, despite the fact that the Windows XP operating system is designed for a normal desktop screen, participants still overwhelmingly preferred a large display to their normal computing environment (i.e., single- or dual- monitors). Our results indicate that a large display can provide several key benefits:

3.3.1.1 Scattered Information Processing Tasks

A large display benefits scattered information processing tasks by enabling the simultaneous display of multiple windows. Scattered information processing tasks are tasks in which information is scatted across multiple windows. Information workers need to collect, search or compare information across multiple windows. Examples include programming and slide making tasks. In a programming task, although the primary working window is the coding application (e.g., a Java Bean window), the user needs to search for and check information relevant to the coding in other windows (e.g., search for a function name in a Java document window). In a slide making task, the user need to collect materials from different windows such as web pages and picture folders, and edit/organize them within the slide making application (e.g., the Microsoft PowerPoint). As most of the relevant windows are visualized on the large

display, information scattered in different windows are visible simultaneously. Information workers can easily collect, search and compare the information without interleaving windows. In contrast, they need to frequently switch back and forth among multiple windows in single- and dual-monitor conditions because only one or two windows are totally visualized due to the small screen real estate. By reducing the number of window interleaving operations, a large display benefits the scattered-information processing tasks.

Previous research (Czerwinski et al., 2003; Simmons, 2001) has shown that a large display (albeit one that was much smaller and of lower resolution than the display in our study) outperforms a small display for complex, multi-application office work. Complementarily, our study shows that users intentionally optimize window layout to facilitate scattered information processing to improve their workflows.

3.3.1.2 Multi-scale Navigation Tasks

Another obvious benefit is related to applications which contain rich digital information and require users to perform frequent navigation operations in both the lateral and depth directions to search for the desired information. For example, a user pans and zooms in/out on a large map to locate a target; a data analyst scrolls up/down and zooms in/out on a big spread sheet to search for desired data. This type of task is referred as a Multi-scale Navigation task throughout the thesis. As a large amount of information is visualized simultaneously on a large display, users tend to search for desired data via physical navigations (e.g., changing eye positions, rotating heads), as opposed to using virtual navigations (e.g., panning and zooming in/out). Reducing the workload of virtual navigation leads to the better performance on a large display for such tasks. For example, one participant (D2) expressed a strong preference for analyzing data in Excel files on the large display, because it can show all the columns. Unlike working on a large display, he had to frequently scroll left and right, up and down in his usual dual-monitor setup because only partial information was visible.

3.3.1.3 Awareness of Peripheral Applications

Being aware of peripheral applications could improve work productivity (Matthews et al., 2006). Besides offering information workers large work space for primary tasks, a large display also provides plenty of screen real estate for showing peripheral applications. Our study results reveal 51

that the side and top regions on a large display are usually treated as peripheral regions for displaying such applications. For example, as shown in Figure 3-6, the user opened a weather report, “to-do” list, email client, and instant messaging client in the peripheral region. He could maintain awareness of these four peripheral applications while working on the primary task. In contrast, in his usual single-monitor environment, due to the limited screen real estate, only the major application window was fully visible. Peripheral applications were usually obscured.

3.3.1.4 Immersive Experience.

Tan et al. (2003) shows that a large display outperforms a normal monitor in 3D navigation tasks because of the immersive experience generated by the large display. Although a user’s personal desktop work is not typically comprised of 3D navigations tasks, participants reported that a large display engaged them more in their daily work than single- or dual-monitors. When sitting in front of the large display, they felt “surrounded” by the task. This feeling helped them to focus attention on the task, especially when they were performing attentive work such as proofreading, or coding. Additionally, the large display might provide some ergonomic benefits. In the single monitor condition, the user’s head and eyes are restricted to the limited screen space, which might easily cause fatigue if working for a long period of time. However, users felt more relaxed using a large display because they can freely adjust hand and body positions while maintaining a good view of the display.

3.3.2 Challenges of Managing Windows on a Large Display

Despite the overwhelming preferences of working on a large display, challenges arise when users switch from a traditional working environment to a large display. As plentiful digital information and numerous windows are visualized, large display users exhibit distinct patterns in managing windows. However, current window management systems (e.g., WinXp) are designed for a normal-sized monitor, which seems too rigid for a large display. Based on the study results, we suggest enabling the following functions to suit large display users’ unique window management behaviors:

3.3.2.1 Enable Multi-Window Operations

Large display uses tend to spatially arrange multiple windows into various layouts to convey semantic meanings. For example, software developers spread multiple help documents around the code editing window; windows not being used are stacked at the side for later reference. However, current window management systems are not sufficiently flexible to support these multi-window operations. They typically only allow users to operate on a single window at a time. Thus, arranging multiple windows on a large display becomes tedious and time-consuming. To alleviate this problem, a window manager for a large display should offer appropriate multi- window operations.

3.3.2.2 Support Flexibly Arranging Windows

Office workers enjoy greater flexibility in organizing paper documents with two hands in the physical world than with a single cursor in the digital counterpart. They can quickly move documents in and out of the center of a desk for information processing. Relevant files are interleaved for cross-referencing, and documents not being used are roughly stacked to convey subtle information. To mimic these paper organizing behaviors, we suggest that the window management system offer a more natural and flexible approach for arranging windows. For example, a large display oriented window management system should allow users to freely adjust windows z-order or quickly spread, pack windows.

3.3.2.3 Facilitate Moving and Resizing of Windows

Basic window operation analysis shows that users move and resize windows more frequently on a large display than on single or dual monitors – moving and resizing accounts for more than 50% of window operations on a large display while less than 30% on single or dual monitors (Figure 3-7). Unfortunately, many large-display users were not satisfied with current window moving or resizing operations; they complained that the thin title bar or the tiny corner of a window was difficult to point to and drag, especially when windows were located on the right and left sides, far from the user’s focus area. Holding the mouse button and dragging a long distance was tended to be prone to fatigue. A lighter-weight moving and resizing approach might significantly benefit users.

One distinct pattern of large display window management activities is that users frequently switch windows between center and side regions for changing work focus. Easing this switching process could also significantly improve window management efficiency.

3.3.2.4 Allow Executing Commands In-Place

On traditional window management systems such as Windows XP or Mac OS, some window operations are executed by clicking buttons on a window, such as the close and minimize buttons on the title bar. However, accurately clicking a small button distant to a user’s sitting position is error-prone on a large display, as reported by participants in section 3.2.2. Moreover, since these buttons are usually fixed on corners of a window, clicking them may require extra cursor movements To alleviate these problems, the system should allow users to execute commands in- place and avoid high-precision clicking.

3.4 Conclusion

In this chapter, we present a longitudinal diary study comparing usage of a large high-resolution display (4.8m wide × 1.8m high, resolution: 6144 × 2304) to single- and dual-monitor configurations in a desktop work environment. Results indicated users’ unanimous preferences of processing information on a large display and revealed large display users’ distinct behavior patterns in partitioning screen real estate and managing windows. In particular, the following major findings were discovered through the study:

 Users unanimously preferred working on a large display over single- or dual-monitor. A large display could benefit scattered information processing and multi-scale navigation tasks, enhance users’ awareness of peripheral applications, and offer immersive working experiences.

 Large display users exhibited distinct behaviour patterns in partitioning screen real estate on a large display. They tended to utilize the center part as the focal region where primary windows were located. The remaining space was used as the peripheral region, where peripheral applications resided.

 In addition to partitioning screen real estate, large display users also showed unique patterns in managing windows. Compared to normal single- or dual-monitor users, large display users invested more effort to spatially arrange windows in various layouts, and had special interaction and visualization demands of windows in the peripheral region. Also, users on a large display performed more window moving and resizing, but less minimizing and maximizing operations as compared to a single- or dual-monitor. Detailed analysis of window management activities led to four suggestions for designing large display oriented window management systems: 1. Enable Multi-Window Operations 2. Support Flexibly Arranging Windows 3. Facilitate Moving and Resizing Windows 4. Allow Executing Commands In-Place.

4 Effects of Display Size and Pixel Density on Desktop Work A distinct characteristic of a large display is that it possesses a great amount of resolution, which can show enormous digital information simultaneously. The diary study in Chapter 3 initially reveals its benefits on desktop work: participants’ subjective rates and comments showed that a large display is beneficial for scattered information processing and multi-scale navigation tasks. Scattered information processing tasks are the tasks in which information is scatted across multiple windows. Information workers need to collect, search or compare information across multiple windows. Multi-scale navigation tasks contain a large amount of digital information. Users need to perform frequent navigation operations in both the lateral and depth directions to search for the desired information.

In this chapter, we aim to deepen the understanding of resolution effects through rigorous experiments. Since the resolution of a display is determined by physical size and pixel density, we conducted three control experiments (Figure 4-1) to disentangle the effects of these two resolution-related factors on user performance and behavior for normal desktop tasks: (1) scattered-information processing, (2) multi-scale navigation, and (3) single-window text processing. Pixel density Low (58 ppi) High (87 ppi) )  Small (32

(a) (b) Display size )  Big (48

Figure 4-1 Display conditions used in our experiments: (a) small-sized/low pixel-density, (b) small-sized/high pixel-density, (d) big-sized/low pixel-density, (d) big-sized/high pixel- density. 56

The prior dairy study shows that a large display is beneficial for tasks (1) and (2). The control experiments in this chapter verify these findings and further reveal how physical size and pixel density affect users’ performance and behaviors. Task (3) is one of the most common tasks on the large display according to the diary study in Chapter 3. We also include this task in the experiments.

4.1 Experimental Settings

This section outlines the details common to our three controlled-experiments.

4.1.1 Display Conditions To precisely control display size, pixel density, overall display aspect ratio, and pixel aspect ratio in our experiments, we used a remote desktop application running on a tiled projector screen. More specifically, four high-resolution projectors (1280×720 or 1920×1080) were tiled in a 2×2 grid to create a 44.2 wide × 24.8 high display area (Figure 4-2), and connected to a computer with four video-output channels. The four projectors were carefully calibrated to minimize the seams between them (i.e., the width of the seam < 1mm). None of participants reported that the seams affected their performances in experiments.

44.2 44.2   2937 pixels/ 4406 pixels/ 24.8 24.8 50.7 50.7 1280×720 = 58 ppi 1920×1080 = 87 ppi pixels pixels (b) (a) Figure 4-2. The 2×2 tiled projection screen: (a) low-density condition (58 ppi), (b) high- density condition (87 ppi). The only visible application on the four-projector screen was a remote desktop application in the center, which simulate a realistic display (Figure 4-3). Experimental tasks were running on a second computer to which this remote desktop application was connected. To make the “virtual” desktop realistic, we set the background color of the four-projector screen to black, hid the task bar, desktop icons, and constrained cursor movement within the remote desktop application boundary by hooking mouse events. There are two independent variables in our experiment – the display size and the pixel density. Each of them has two levels, resulting in four conditions:

Pixel density Low (58 ppi) High (87 ppi) 27.6 27.6 )    × 1600×900 1836 pixels/ 2400 1350 2754 pixels/

15.6 pixels 15.6 pixels 31.7 31.7 (16:9) = 58 ppi (16:9) = 87 ppi Small (32

(a) (b)

41.5 41.5 Display size )    2400×1350 2754 pixels/ 3600×2025 4130 pixels/

23.3 pixels 23.3 pixels 47.6 47.6

Big (48 = 58 ppi = 87 ppi (16:9) (16:9) (c) (d)

Figure 4-3 The display size and pixel density in each display condition: (a) small-sized/low pixel-density (SL), (b) small-sized/high pixel-density (SH), (c) big-sized/low pixel-density (BL), (d) big-sized/high pixel-density (BH).

 Small-sized/low pixel-density (SL) (Figure 4-3 a): The remote desktop application was 27.6 wide × 15.6 high (31.7 diagonal) with a pixel density of 58 ppi. The resulting screen resolution of the remote desktop application area was 1600×900 pixels.

 Small-sized/high pixel-density (SH) (Figure 4-3 b): The image size of each projector remained the same as that in SL, whereas the projector resolution increased. Thus, the pixel density and the resulting screen resolution of the remote desktop application area increased to 87 ppi and 2400×1350 pixels, respectively.

 Big-sized/low pixel-density (BL) (Figure 4-3 c): The configuration of each projector was the same with that in SL, whereas the size of the remote desktop application was set to 41.5 wide × 23.3 high (47.6 diagonal). The resulting screen resolution of the remote desktop area was 2400×1350 pixels.

 Big-sized/high pixel-density (BH) (Figure 4-3 d): The configuration of the each projector was the same with that in SH, while the size of the remote desktop application set to that in BL. The resulting screen resolution of the remote desktop application area was 3600×2025 pixels.

As a baseline condition, we set the SL around 30 in diagonal with a resolution of 1600×900 because 30 is now a commonly used monitor size and 1600×900 is also a widely used display resolution. The size and pixel density of SH and BL were carefully chosen so that these two conditions had identical resolution (2400×1350) to allow us investigating pixel density and physical display size effects under the same resolution. All the four display conditions have the same screen aspect ratio and pixel aspect ratio on the screen, 16:9 and 1:1, respectively.

4.1.2 Subjects

Sixteen people (11 males, 5 females, age: 18-45) participated in our formal study. All participants were daily computer users and have normal or corrected-to-normal vision. They sat 80~90 cm from the display, which was the preferred sitting distance determined by a pilot study. All the participants firstly performed Experiment 1, followed by Experiments 2 and 3. Each experiment lasted around 40 minutes per participant. Ten minutes break was enforced between experiments.

4.1.3 Apparatus

Four projectors (Epson PowerLite 6100) were connected to a computer with 2.67G Hz CPU, 3Gb RAM, two multi-headed graphic cards (NVIDIA Quadro NVS 290), and Microsoft Windows XP OS. The remote computer to which the remote desktop application was connected to was with 3.00 GHz CPU, and Window XP OS.

4.2 EXPERIMENT 1: Scattered-Information Processing

The goal of this experiment is to investigate how display size and pixel density affects user performance and behavior in tasks where various kinds of information are processed across multiple windows.

4.2.1 Experiment 1: Task

Given application programs and resources (Figure 4-4), participants are asked to complete a set of slides by collecting and comparing information in each display condition.

(a) (b) (c) (d)

Figure 4-4. Applications and resources used in Experiment 1: (a) a slide-making program (Microsoft PowerPoint), (b) five web-browser windows (Microsoft Internet Explorer) in which related wiki web pages are open, (c) a local file-folder with related pictures, (d) a spreadsheet program (Microsoft Excel).

Completing each set of slides consists of the following four types of tasks:  Simple information-lookup (Figure 4-5): Participants replace a question mark on a slide with the correct numerical value from the corresponding wiki web-page.

(a) (b) (c)

Figure 4-5. Simple information-lookup: e.g., to complete the third bulleted-item “Average Depth: ? feet” on the “Lake Huron” slide (a), find the value from the corresponding wiki web-page (b), and replace ? with it (c).  Complex information-lookup and processing (Figure 4-6): Participants collect information from the five wiki web-pages into a table-graph template in the spreadsheet program, and then copy the resulting graph into a slide.

(a) (b) (c)

Figure 4-6. Complex information-lookup and processing: e.g., to complete the “Comparison of Water Volume” slide (a), refer to the five wiki web-pages, and fill the values in the spreadsheet program to create a bar graph (b), and insert and adjust it in the slide (c).  Information comparison (Figure 4-7): Participants choose the correct comparative adjective in a statement on a slide by referring to the two corresponding wiki web-pages.

(a) (b) (c)

Figure 4-7. Information comparison: e.g., to complete the sentence “The Max. Width of Huron is wider/narrower than that of Michigan.” on the “Some Facts” slide (a), refer to the two corresponding wiki web-pages (b), and choose the correct one (wider) (c).  Picture insertion (Figure 4-8): Participants insert a picture into a slide from the given local file- folder.

(a) (b) (c)

Figure 4-8. Picture insertion: e.g., click the “Insert Picture” icon on the “Lake Huron” slide (a), find the corresponding picture (Lake_Huron_2.jpg) in the local file-folder using the file chooser (b), and insert it in the slide (c).

Completing tasks required geographic knowledge. The questions were chosen so that participants were not able to easily guess them without referring to the given wiki web-pages (e.g., the surface area of Lake Ontario). Participants are explicitly instructed to look for the information in the upper-right panel of the wiki web-pages (depicted in orange color in Figure 4-5b, Figure 4-6b, and Figure 4-7b) so as to reduce the effect of their information-search skills on their multi- window processing performance that we want to see. Based on pilot studies, the font size and the zooming factor of Microsoft Internet Explorer 7.0 are set to medium and 125 %, respectively so that the text of the wiki web-pages can be clearly and comfortably read in all display conditions (Figure 4-9).

1cm 1cm (a) (b)

Figure 4-9. The apparent sizes of characters in the wiki web-pages: (a) low pixel-density display conditions (SL and BL), (b) high pixel-density display conditions (SH and BH).

We prepared four different slide sets, whose topics are Great Lakes, States in the United States, Countries in Africa, and Lakes in the UK, respectively. Each slide set consists of 9 slides, including 3 simple information-lookups, 1 complex information-lookup and processing, 2 information comparisons, and 1 picture insertion tasks. The questions in the slides were carefully chosen to ensure a similar difficulty level between the four slide sets.

4.2.2 Experiment 1: Design

A within-subjects design was used. Fully crossing the two independent variables (the display size and the pixel density) generates four display conditions – SL, SH, BL, and BH (Figure 4-3). Each participant is given one of the four slide sets in each display condition, thus completing four different slide sets in total across all display conditions. The presentation order of the four display conditions and the four slide sets are counterbalanced among 16 participants using two Latin Squares.

Prior to the formal experiment, participants finish a 5-slide practice trial to familiarize themselves with the tasks. They were asked to perform the tasks as accurately and quickly as possible.

4.2.3 Experiment 1: Data Collection

We collected three different kinds of data to determine how well participants complete the given tasks in the different display conditions, and to investigate their behavior:

 Completion time and error rate: Participants click the “start” button in a timer application to start the formal experiment, and click the “finished” button after finishing a set of slides. The elapsed time between clicking the “start” and “finished” buttons is the completion time. The error rate is the ratio of the number of incorrect answers relative to all the questions to be filled in.

 Mouse/window events: As in the longitudinal study in Chapter 3, we use the MouseLog and VibeLog programs to record mouse and window events, respectively. MouseLog records every mouse activity including the spatial coordinates of the mouse cursor, and event types (e.g., left-button down/up, right-button down/up, scroll wheel down/up) while VibeLog records

every window event (e.g., window closed/activated/moved/resized/minimized/maximized) and detailed information (e.g., the spatial coordinates, size, status, and depth order of each window on the display).

 Subjective Opinions: After completing the task, participants were asked to rate each condition according to their experiences (How good is the display for the task? 1—very bad, 5—very good).

4.2.4 Experiment 1: Results

Completion Time and Error Rate

The mean error-rate was less than 5 %, indicating that the participants successfully completed the slide-making tasks in all display conditions. The mean completion times were 358 (SL), 305 (SH), 319 (BL), and 289 (BH) seconds. An ANOVA analysis showed significant main effects for the display size (F1,15=5.43, p<.05) and the pixel density (F1,15=7.58, p<.05) on the mean completion time – smaller-sized and lower pixel-density displays required longer time to finish the slide-making tasks than bigger-sized and higher pixel-density ones (Figure 4-10). Pairwise means comparison tests showed a significant difference between every two display conditions

(p<.05). A strong (display size)×(pixel-density) interaction was also observed (F1,15=8.36, p<.05) – with the lower pixel-density, increasing the display size reduced the mean completion time more significantly. Another interesting observation was that when comparing SH and BL with identical resolution, the completion time for SH was shorter than BL.

390

SL 330 BL SH BH 270 Completion time time Completion (s)   Low pixel-density 0 High pixel-density Small Big Size

Figure 4-10. Mean completion time (with std error) in Experiment 1.

Mouse/Window Events

Figure 4-11 shows the aggregated mouse-event distribution for all participants. In SL, mouse events were intensive on the bottom of the screen, where the task bar is located. According to a mouse-event analysis, 26% of mouse events occurred on the task bar in SL, but reduced to 20%, 16 %, and 14 % in SH, BL, and BH, respectively. This was confirmed by a window-event analysis – the mean numbers (standard error) of window switching actions via the task bar are 35.8 (4.6), 26.5 (3.1), 22.5 (2.9), and 18.6 (2.2), in SL, SH, BL, and BH, respectively. On the other hand, in the big-sized display conditions (BL and BH), mouse events were distributed over the entire screen area, with some intensive clusters in the central region. This indicates that users leveraged the large screen real estate that the bigger-sized displays provided to make many windows viewable simultaneously, while devoting more attention to the main application (the slide-making program) located in the central region. This is consistent with the finding of diary study in Chapter 3: large display users tend to use central areas as focus regions.

2.2 x 2.2 cm # of mouse events

0 SL SH BL BH

Figure 4-11. Mouse-event distribution for Experiment 1.

# of windows# SH

SL Low pixel-density

0 High pixel-density Small Big Size

Figure 4-12. Mean number (with std error) of simultaneously viewable windows in Experiment 1.

Figure 4-12 shows the mean number of windows being viewable simultaneously in each display condition. A window was considered as viewable when it was not completely occluded by other window(s). The number of viewable windows was computed every minute and averaged for each display condition per participant. An ANOVA analysis showed significant main effects for the display size (F1,15=9.85, p<.05) and the pixel density (F1,15=10.96, p<.05) on the mean number of simultaneously viewable windows – with more viewable windows on bigger-sized and higher pixel-density screens. Pairwise means comparison tests showed a significant difference between every two display conditions (p<.05), except SH and BL (p=.256). That is, the number of simultaneously viewable windows relates to display resolution (the display size times the pixel density).

Fourteen participants reported that they had kept multiple windows viewable simultaneously in SH, BL, and BH, whereas none of them did so in SL. They commented that arranging multiple windows to be viewable at the same time was beneficial for multi-window office work like our slide-making tasks if enough screen real estate was provided – they could collect and compare information without switching back and forth among several windows. This behavior echoes the finding obtained from the prior diary study (Chapter 3): large display users tend to optimize window layouts to improve workflow. Figure 4-13 is an example screen in BH where a participant intentionally arranged the five wiki-pages side-by-side to ease information collection and comparison.

Figure 4-13. A screen image in the BH display condition.

Subjective Ranks

The mean (Std Dev) subjective scores for SL, SH, BL and BH are 2.4 (0.5), 3.4(0.7), 3.6 (0.8),

4.8 (0.5) respectively. ANOVA shows significant main effects for both display size (F1,15=7.46, p<.05) and pixel density (F1,15=8.22, p<.05) on subjective ranks. This results is consistent with completion time, indicating that big-sized and high pixel-density displays are better than small- sized and low pixel-density displays in scatter information processing tasks.

4.2.5 Experiment 1: Discussion

We carefully designed our slide-making tasks to have a number of common multi-window operations, and include simple and complex information collection and comparison so that they would reflect scattered information process tasks.

As expected, bigger-sized and higher pixel-density displays resulted in better performance of scattered-information processing among multiple windows. As its size and pixel density increased, the display provided bigger screen real estate so that more windows were viewable simultaneously, easing the information collection and comparison process, thus facilitating multiple-window office work. Aside from the main effects of the display size and the pixel density, there was a strong (display size)×(pixel density) interaction – as pixel density was lower, increasing the display size brought more performance gains; increasing the pixel density led to more performance gains on a smaller display than on a bigger one. 67

The SH and BL displays have the identical resolution (2400 x 1350), but different sizes and pixel densities – the former is a small-sized and high pixel-density display, and the latter a big-sized and low pixel-density one. The experiment results showed that the mean completion time in SH is shorter than that in BL, indicating the small-sized and high pixel-density display enabled better performance of scattered-information processing. We reason that the necessary information was positioned close to the main application in the smaller physical area in SH, thus the participants could relatively easily collect and compare information with less head rotation and eye movement.

4.3 EXPERIMENT 2: Multi-Scale Navigation

The objective of this experiment is to investigate the effects of display size and pixel density on dynamic level-of-detail information searching tasks.

4.3.1 Experiment 2: Tasks

Participants perform a set of online map navigation tasks in each display condition:

 Locating marks in a small area (LM-S): Five marks are distributed within a 30 mi2 area (Figure 4-14a). Participants are asked to check and write down the name of the street where each mark is located on a sheet of paper (Figure 4-14b).

 Locating marks in a big area (LM-B): This task is the same as LM-S except that the area within which the five marks are distributed is broader (60 mi2).

 Tracing a route in a small area (TR-S): A route between two positions, A and B, is predefined on a map Figure 4-15a). Participants are asked to check and write down the street names it goes through (Figure 4-15b). The distance between A and B is 7 miles and there are 5 streets along the route.

 Tracing a route in a big area (TR-B): This task is the same as TR-S except that the distance between A and B is longer (14 miles) and the route goes through more streets (10 streets).

All the tasks are performed on Google maps using Microsoft Internet Explorer 7. The text clues to the marks and routes, which are usually displayed on the left-side of the window, are removed

so that participants complete the tasks only relying on virtual navigation – left mouse-button dragging for panning, mouse-wheel scrolling up/down for zooming in/out, and left mouse-button double clicking for zooming in. Similar to Experiment 1, the font size and the zooming factor of the web browser are set to medium and 125 %, respectively, to ensure enough legibility. At the beginning of each task, the application window is placed in the center of the display screen with a size of 500×400 pixels, but participants are allowed to freely adjust its size during the experiment. The length of every street name is under 15 letters to ensure simplicity.

(a) (b)

Figure 4-14. Experiment 2: Locating marks task.

(a) (b)

Figure 4-15. Experiment 2: Tracing a route task.

The map areas of the tasks were determined through a pilot study. They are big enough so that users have to use navigating operations to complete the tasks.

4.3.2 Experiment 2: Design

A within-subjects design is used. For each task (LM-S, LM-B, TR-S, and TR-B), we prepared a set of four maps, for a total of 16 maps across all tasks. Within each task, we ensured that the difficulty level of each of the four maps was roughly similar. Participants performed the tasks in the order LM-S, followed by LM-B, TR-S and TR-B. The presentation order of the four display conditions and the four maps within each task were counterbalanced across the 16 participants using two Latin Squares.

Prior to the formal experiment, participants perform a practice trial containing a LM-S and TR-S task to familiarize themselves with the location and tracing tasks. They were asked to perform the tasks as accurately and quickly as possible.

4.3.3 Experiment 2: Data Collection

As with Experiment 1, we collected three types of data:

 Completion time and error rate: Similar to Experiment 1, participants started and finished each task by pressing the “start” and “finished” buttons in a timer application, respectively. Participants were instructed to press the “finished” button after they wrote down the street names on the answer sheets. We considered the elapsed time between clicking the two buttons as the completion time. The error rate was the ratio of the number of incorrect street names to the total number of street names participants had to identify.

 Mouse/window events: Similar to Experiment 1, we recorded every mouse and window event.

 Subjective ranks: Similar to Experiment 1, we asked participants to rate the four display conditions after the experiment (How good is the display for the task? 1—very bad, 5—very good).

4.3.4 Experiment 2: Results

Completion Time and Error Rate

The mean completion times (s) were 158.3 (SL), 139.3 (SH), 121.4 (BL), and 110.7 (BH) for LM-B task, and 180.7 (SL), 157.5 (SH), 140.4 (BL), and 126.8 (BH) for TR-B (Figure 4-16). For 70

the LM-B and TR-B tasks, an ANOVA analysis showed significant main effects for display size

(F1,15=8.82, p<.05) and pixel density (F1,15=6.03, p<.05) on the completion time – with bigger- sized and higher pixel-density displays the participants completed the tasks faster than with smaller-sized and lower pixel-density ones. Pairwise means comparison tests showed a significant difference between every two display conditions (p<.05). Interestingly, the mean completion time in SH was longer than that in BL for both LM-B and TR-B tasks – this result is contrary to that in Experiment 1. On the other hand, there were no significant main effects for the display size or resolution on the completion time for LM-S and TR-B tasks. The error rates in all the tasks were less than 5% in all display conditions, indicating that the participants completed the tasks without problems.

TR-S TR-B 200 LM-S LM-B 180 160 140 120 100 80 60 40 20 0 SL SH BL BH SL SH BL BH SL SH BL BH SL SH BL BH (a) (b) (c) (d)

Figure 4-16. Mean completion time (with std error) in Experiment 2 multi-scale navigation tasks: (a) locating marks in a small area (LM-S), (b) locating marks in a big area (LM-B), (c) tracing a route in a small area (TR-S), (d) tracing a route in a big area (TR-B).

Mouse/Window Events

Figure 4-17 shows the mean number of panning and zooming operations across all the tasks. An

ANOVA analysis showed significant main effects for display size (F1,15=7.52, p<.05, &

F1,15=5.66, p<.05 ) and pixel density (F1,15=5.75, p<.05, & F1,15=5.02, p<.05) on both panning and zooming operations respectively –participants used less panning and zooming operations on bigger and higher pixel-density displays than on smaller and lower pixel-density ones. The mean number of panning operations in SL was at least 13% higher than that in the other three display conditions; and the number of zooming operations in SL was at least 10% higher than in the other display conditions. 71

70 Panning Zooming 60 50 40 30 20 # of Operations 10 0 SL SH BL BH SL SH BL BH (a) (b)

Figure 4-17. Mean numbers (with std errors) of navigating operations across all tasks in Expt 2: (a) panning, (b) zooming.

As the participants were allowed to change the size of the web browser, they performed the tasks on maps as big as they preferred. Figure 4-18 shows the mean size of the web browser in which the online map application was running. We can see the tendency that the participants tried to maximize the map area as much as possible in all display conditions (at least 84 % of each entire area); this tendency was prominent in SL whose resolution is at least four times lower than that of the other display conditions. All the participants reported that large window area was beneficial in this experiment because they could see more information simultaneously.

Subjective Ranks

The mean (Std Dev) subjective scores for SL, SH, BL and BH are 2.4 (0.5), 3.4(0.6), 3.3 (0.5),

4.8 (0.3) respectively. ANOVA shows significant main effects for both display size (F1,15=7.46, p<.05) and pixel density (F1,15=8.22, p<.05) on subjective ranks. This results show that big-sized and high pixel-density displays are better than small-sized and low pixel-density displays in multi-scale navigation tasks, which concurs with the results about completion time.

4.3.5 Experiment 2: Discussion

Online map navigation is a kind of information searching not only in the lateral space but also in the depth direction. It involves a large amount of information, and requires intensive virtual navigation operations.

3 1600 2400 2400 600

1528 (34.5) 2245 (42.6) 2228 (41.5) 3357 (45.8) 2.7M pixels/ 1.3M pixels/ 1350 900 2.7M pixels/ 6.3M pixels/ 3.2M pixels 2025

1.4M pixels 1350 1189(36.9) 3.2M pixels 845 (12.6) ≈ 84% 7.3M pixels

≈ 90% 1225 (35.4) ≈ 84% 1889 (38.9) ≈ 86%

(a) (b) (d) (c)

Figure 4-18. The mean window sizes (in pixel) in Experiment 2 multi-scale navigation tasks (with std errors in parentheses): (a) SL, (b) SH, (c) BL, (d) BH.

Similar to Experiment 1, bigger-sized and higher pixel-density displays result in better performance in our multi-scale navigation tasks. We reason that simultaneously visualizing large amounts of information reduced the numbers of virtual navigating operations, resulting in the performance improvements. However, when level-of-detail navigation was not important due to the relatively small areas, the performance gains were neutralized. For example, we observed no main effects for display size or pixel density on the completion time in LM-S and TR-S.

A (display size)×(pixel density) interaction was also observed in LM-B and TR-B tasks. Similar to that in Experiment 1, when the display size was smaller, the performance became more susceptible to the pixel density.

Another interesting result is that BL outperformed SH in online map navigation, contrary to the result in Experiment 1. We reason that it might be due to the text legibility difference. As the physical size of a pixel in BL was 1.5 times bigger than in SH, the physical font sizes in BL were accordingly 1.5 times bigger than those in SH. Nine of 16 participants reported that bigger font- sizes made it easier for them to recognize the street name at a glance in BL compared to SH. Although the viewport in BL was larger than that in SH, the characteristics of online map navigation did not require as frequent head rotation and eye movement as for information collection and comparison among multiple windows as in Experiment 1. Thus, the physically larger font sizes might outweigh the weakness of the large display area, leading to the performance improvement.

4.4 Experiment 3: Single-Window Text Processing

The objective of this third experiment is to study how display size and pixel density affect user performance and behavior in (linear) text reading/editing, in a single window.

4.4.1 Experiment 3: Task

Participants were asked to proofread given articles. We use the following rules to introduce grammatical errors into each test article:  Each sentence has at most one error (some sentences might have no error).  Errors were fairly evenly spaced throughout each article  Only two kinds of errors are considered: subject-verb mismatches and inconsistent verb tenses.

These rules are similar to those introduced by Maglio and Campbell (2001), and Tan and Czerwinski (2003) in their reading tasks. Within 300 seconds, participants are asked to find and underline as many errors as they can Figure 4-19). They do not need to actually correct the errors.

Ctrl+ U

Figure 4-19. Expt 3 proofreading task: underline erroneous words using the keyboard shortcut Ctrl+U of Microsoft Word.

We prepared four test articles drawn from the New York Times between January 2008 and January 2009, with the Flesch readability score around 46, representative of reading at the 12th grade level (Flesch, 1948). The test articles are shown double-spaced with the 14-point Times New Roman font. The word processing program (Microsoft Word) is initially placed in the center of the screen with a size of 400×500 pixels, but participants are allowed to freely adjust the application window size during the experiment.

4.4.2 Experiment 3: Design

A within-subjects design is used. Each participant proofreads one article per display condition. The presentation order of the articles and display conditions are counterbalanced across the 16 participants using two Latin Squares.

Prior to the formal experiment, participants perform one practice trial (with a 5th practice article) to familiarize themselves with the task. They were asked to perform the task as accurately and quickly as possible. All the participants were able to find more than 60% errors within 300 seconds.

4.4.3 Expt 3: Data Collection  Correct edits: We count the number of correct edits.  Error rate: The percentage of incorrect underlines among all the edits is recorded.  Mouse events: We record every mouse event.  Subjective ranks: After completing the task, the participant is asked to rate the four display conditions (Is the display good for current task? 1—very bad, 5—very good).

4.4.4 Expt 3: Results

Correct Edits

Figure 4-20 shows the mean number of correct edits for each display condition. ANOVA did not show significant main effects for either the display size (F1,15=2.68, p=.12) or pixel density

(F1,15=2.31, p=.15) on the number of correct edits. The error rate in each condition was less than 5%, indicating that participants seldom incorrectly underlined errors.

25 20

15 10

# of correct edits 0 SL SH BL BH

Figure 4-20. The mean number (with std error) of correct edits in Expt 3.

Mouse Events

The participants performed most mouse events in SL; the mean (standard error) numbers of mouse events in SL, SH, BL, and BH were 15.4 (3.4), 12.0 (3.8), 11.1 (2.9) and 10.0 (2.2), respectively. An ANOVA showed significant main effects for the display size (F1,15=6.25, p<.05) and pixel density (F1,15=5.31, p<.05) on the number of mouse events. Eleven of 16 participants reported that they scrolled the articles more frequently in SL than in the other display conditions due to the smallest amount of information space. However, according to the performance data (Figure 4-20), these extra operations did not jeopardize the proofreading performance.

Subjective Ranks

The mean (Std Dev) subjective scores for SL, SH, BL and BH are 3.6 (0.4), 3.7(0.5), 3.6 (0.6), 3.8 (0.7) respectively. No significant main effect was observed on number of correct edits for either display size (F1,15=1.56, p=0.2308) or pixel density (F1,15=2.01, p=0.1767).

4.4.5 Experiment 3: Discussion

Proofreading was chosen to represent single-window text processing tasks. This task involves only one window and relatively linear information processing. It demands a certain amount of

cognitive load, which is generally required for many single-window tasks. Furthermore, proofreading per se is a very common task for many office workers.

Our results revealed that the display size and the pixel density do not affect proofreading performance. This is consistent with the findings by Miyao et al. (1989), and Zelifle (1998). Due to the small amount of information shown in small and low- resolution displays, users need to perform more document navigating operations (e.g., scrolling, page up/down). However, the linear nature of text reading does not make these extra operations detrimental to the reading performance.

4.5 Conclusions

We conducted three controlled experiments to examine the effects of display size and pixel density on various desktop activities. Our findings are summarized as follows:

(1) Bigger-sized and higher pixel-density displays better support scattered-information processing (multiple windows, information search/collecting/comparison, object inserting/copy- and-pasting) and multi-scale navigation in big areas (huge information space, frequent navigation in both the lateral and depth directions).

(2) Increasing the pixel density is more crucial for smaller displays than for larger ones for scattered-information processing and multi-scale navigation in big areas.

(3) Increasing the display size is more crucial with lower pixel-density than with higher pixel density for scattered-information processing and multi-scaling navigation in big areas.

(4) With an identical resolution, a smaller but higher-density display outperforms a bigger but lower-density one for scattered-information processing.

(5) With an identical resolution, a bigger but lower density display outperforms a smaller but higher-density one for multi-scale navigation in big areas.

(6) The display size and pixel density do not affect the performance of multi-scale navigation in small areas.

(7) The display size and pixel density do not affect the performance of single-window text processing (linear text reading/proofreading, scrolling, page up/down). However, users need to perform more documents navigation operations on small low-resolution displays.

The study results reveal a general trend of how display size and pixel density affect users performance and behaviors. Although the display size in the experimental condition BL and BH (i.e., 1.01 m x 0.6 m) was much smaller than that of a large display in this thesis (i.e., a wall- sized display, 4.5m x 1.5m), study results are generalizable. One major reason is that because a user can freely adjust the window size within the dimensions of a display, he/she can always resize a window to an optimal size for a certain task on a large display. For example, Experiment 2 indicates that a 50” monitor (i.e., the display size in the conditions BL and BH) leads to better performance than a 30” screen (i.e., the display size in the conditions SL and SH) in the multi- scale navigation task. Based on this result, we can predict that a user would also gain better performance for this task on a large display (4.8 m wide x 1.8 m high) than on a 30” monitor: the user can just simply enlarge the application window to 50” on the large display to outperform the 30” monitor. In fact, this prediction concurs with the finding acquired from the longitudinal diary study in Chapter 3: users prefer processing scattered information processing and multi-scale navigation tasks on a large display over single- or dual-monitor.

In addition to disentangling the effects of physical size and pixel density of a display, we also contribute to designing tasks representing information processing work. Slide making, online map navigation, and proofreading tasks represent scattered-information processing, multi-scale navigation, and single-window text editing work, respectively. These are commonly performed tasks in desktop environments. Although these tasks do not reflect every aspect of daily information processing work, they are carefully designed to capture the essence of three common desktop activities. Aside from being used to investigate size and pixel density effects, they can be further adopted as benchmark tasks representing scattered-information processing, multi-scale navigation, and single-window text editing work in other studies.

5 Effects of Interior Bezels of Tiled-Monitor Large Displays on Visual Search, Tunnel Steering, and Target Selection

With currently available technologies, there are two common approaches to construct a large display – tiling multiple projectors or normal monitors. Carefully calibrating and tiling multiple projectors can create a large display surface with either no seams at all or at most very thin seams, such as the large displays in chapters 3 and 4. However, this approach requires relatively large physical space for deploying projectors and excessive effort for calibrating them. An alternative is tiling multiple monitors. A tiled-monitor large display occupies less physical space and eases the calibration process. In addition, it is usually less expensive than a tiled-projector large display. The quick advance of display technology would enable users to afford seamless, large high-resolution displays in future. However, before the seamless large displays become affordable and available to common users, tiled-monitor large displays are widely used as approximations for seamless large displays in many places, such as in conference rooms, public places, data visualization centers, or even personal offices (Ball & North, 2005) (Figure 5-1). Although the main theme of this thesis is seamless large displays, it is necessary to understand how their current approximation—tiled monitor large displays differs from true seamless large displays. This knowledge would lead to effective tiled-monitor display usage and help software developers design suited interfaces on such displays.

(a) (b)

(c) (d) Figure 5-1. Tiled-monitor large displays in personal offices (a and b), data visualization center (c), and public place (d). Figures taken from Swaminathan and Sato (1997), www.cinemassivedisplays.com, and www.tacc.utexas.edu. 79

The distinct difference between a seamless and a tiled-monitor display is the interior bezel, part of the physical bezel surrounding each component monitor. As indicated by previous studies, these interior bezels cause visual discontinuities of displayed images as well as cursor trajectory (Ball & North, 2005; Robertson et al., 2005). On the other hand, since these bezels are relatively small compared to the display surface (e.g., a common 24 LCD – 51 cm wide and 32 cm high – monitor frame is 2 cm wide, which is only 4% of the monitor width), it might be possible that users can successfully compensate for the difficulties caused by interior bezels in realistic tasks.

To understand the difference between seamless and tiled-monitor large displays, we systematically investigate effects of tiled-monitor display (interior) bezels. Three experiments are deliberately designed to better understand bezel effects on user performances and behaviors in visual search, straight tunnel steering, and target selection tasks, respectively. These three tasks are basic visual and interactive tasks on a large display. Straight-tunnel steering and target selection are canonical tasks for GUI interaction (e.g., hierarchical menu navigation and button clicks). Visual search tasks are frequently performed if uses are searching/comparing information. Many desktop workers’ information processing activities could be decomposed into the combinations of these basic tasks. For example, copying and pasting a picture into a PowerPoint slide can be decomposed into searching for the picture (visual search), selecting the picture (target selection), issuing the “copy” command via a mouse right-click menu (straight- tunnel steering and target selection), and pasting the picture in the desired location (target selection). Investigating the bezel effects on visual search, straight-tunnel steering and target selection tasks could deeply reveal how bezels affect various information processing activities which consists of these basic tasks.

5.1 Experimental Setup

In this section, we explain the common details of our three controlled experiments.

5.1.1 Tiling Configurations

One independent variable in all the three experiments is the degree of tiling, which relates to how many interior horizontal and vertical bezels exist on a tiled-monitor large display. In order to keep display size and performance parameters (e.g., brightness, contrast.) constant, thus

separating bezel effects from other factors, we use one projected display, on which virtual bezels are rendered to simulate tiled-monitor displays. We use three different degrees of tiling (1, 2, 3) to generate three tiling configurations as follows:

 [1×1]: There is no bezel on the display surface. This condition serves as baseline in the three experiments (Figure 5-2a).

 [2×2]: The display surface is equally subdivided into four areas by one horizontal and one vertical black bezel. This condition is to simulate a tiled-monitor large display consisting of four 40 monitors (Figure 5-2b).

 [3×3]: The display surface is equally subdivided into nine areas by two horizontal and two vertical black bezels. This condition is to simulate a tiled-monitor large display consisting of nine 26 monitors (Figure 5-2c).

167 cm 81 cm 53 cm 1024 pixels 66 cm

4 cm 101 cm (26) 40 cm 40

61 cm 61 (40) 210 cm (83) 4 cm 127 cm 768 pixels 4 cm 4 cm 180 cm

(a) (b) (c) Figure 5-2. The three tiled-monitor large display configurations used in our experiments: (a) [1×1], (b) [2×2], and (c) [3×3].

For all the three conditions, a single projector with a resolution of 1024×768 pixels (NEC WT610) creates a 167 cm wide ×127 cm high display area. Each interior bezel is 4 cm wide and generated by the computer to simulate plastic bezels of physical tiled-monitor displays, which deflect cursor paths and separate images that are across the monitors. Current plastic frames on LCDs are usually 2 cm wide; hence, 4 cm reflects the width of bezels when multiple LCDs are tiled together. Note that replacing the default plastic frames with thinner frames might reduce the width of bezels. 4-cm width is a conservative estimation; it can be considered an upper bound for bezel width on tiled-monitor displays.

5.1.2 Participants

We recruited twelve participants (seven males and five females) between ages of 18 and 45. All of them are daily computer users and have normal or corrected-to-normal vision. They sat 1.8 m from the display, which is a preferred sitting distance determined by a pilot study, so that they could comfortably view the entire display with slight head rotation, and easily recognize displayed objects. Participants performed the three experiments in turn, with the presentation order of the experiments counter-balanced across the participants.

5.2 Visual Search Experiment

The objective of this experiment is to investigate how interior bezels on tiled-monitor large displays affect users’ visual searching performance and behavior.

Visual search is a commonly performed task on a large display. A larger display can visualize a greater amount of information simultaneously, thus improving visual search performance over a small-sized display (Yost, Haciahmetoglu & North, 2007). On a tiled-monitor large display, however, its interior bezels might distract users. In addition, they may separate objects which are located across interior bezels apart (symbol split), thus impeding users from recognizing them. Given these two factors, we hypothesize that interior bezels are detrimental to user’ visual search performances as follows:

H.1 The presence of interior bezels is detrimental to visual search performance.

H.2 Splitting data across interior bezels is detrimental to visual search performance.

5.2.1 Visual Search: Task

The experiment task is a traditional attentional image-searching task, in which participants are asked to identify whether a target object exists or not within a number of distracters (Wolfe, 1998). Our images use a light gray background, on which the two letters “VI” appear multiple times. In 50% of the images, one “IV” is present (Figure 5-3a). Participants are asked to search for an IV among VI’s as quickly and as accurately as possible. They indicate the presence or absence of the target object using two keys on the keyboard. A beep sound is played if users make a mistake. To prevent participants from “racing through” our trials without regard for 82

accuracy, the testing program freezes for five seconds whenever an error occurs. To specifically investigate symbol split effects, in 50% of the images, at least one VI or IVis located across interior bezels in both the [2×2] and [3×3] conditions (Figure 5-3b).

(a) (b) Figure 5-3. Visual search: (a) task – to identify if an IV exists among VI’s in a given image, (b) the symbol-split condition – symbols might be displayed apart across an interior bezel(s).

5.2.2 Visual Search: Design

We used a within-subject, full-factorial repeated measures design with search time and error rate as our dependent measures. The search time is defined as the time lapse between when a stimulus image is initially presented on the display and when the “Present” or “Absent” key on the keyboard is pressed. The error rate is the percentage of erroneous trials.

The independent variables include the degree of tiling (1, 2, or 3), number of distracters (15 or 30 VI’s), target presence (absence or presence of IV), and symbol-split status (split or non-split).

The presentation order of degree of tiling was fully counter-balanced with 6 combinations across the 12 participants. For each degree of tiling condition, participants searched the same set of 48 images that were presented in a random order. These 48 images consisted of 6 repetitions of 8 conditions, which represented the fully crossed combination of 2 number of distracter, 2 target presence, and 2 symbol-split status conditions. On each of the 48 images, the location of a symbol was randomly generated by the computer. Especially in the symbol-split condition, symbol locations were randomized with a constraint that in both the [2×2] and [3×3] degree of tiling conditions, at least one symbol is located across an interior bezel(s). In the final tested set of 48 images, the mean (SD: Standard Deviation) of the number of symbols across interior bezels was 1.4 (0.4) in the [2×2] × split condition and 1.6 (0.64) in the [3×3] × split condition.

The size of symbols was constant in the whole experiment. It was determined via a pilot study to ensure that users could easily recognize them.

In short, our design was: 12 participants × 3 degrees of tiling (1, 2, 3) × 2 numbers of distractor VI’s (15, 30) × 2 target presences (present, absent) × 2 symbol-split statuses (non-split, split) × 6 repetitions = 1728 total trials.

5.2.3 Visual Search: Results Search Time

The means (SD) of search time for the [1×1], [2×2], and [3×3] conditions were 6125 (3322), 6356 (3301), and 6284 (3394) ms, respectively. An ANOVA showed no significant main effect for the degree of tiling on the search time (F(2, 22)=0.89, p=0.42). The means (SD) of search time for the non-split and split conditions were 6260 (3264) and 6249 (3413) ms, respectively. No significant main effect for symbol-split status on the search time (F(1,11)=0.125, p=0.73) was observed either.

As expected from previous visual search studies, an ANOVA showed a main effect for the number of distracter VI’s on the search time (F(1,11)=54.2, p<0.001), with 30 distracters resulting in longer search time than 15 distracters. The mean (SD) search times were 4970 (2440) and 7540 (3612) ms, with 15 distracters and 30 distracters, respectively.

The target-absent trials took a longer time than the target-present trials (F(1,11)=62.3, p<0.001): the mean (SD) search time for the target absent trials was 7392 (3513), and 5117 (3126) for the target present trials. This is typical in visual search experiments (Forlines & Balakrishnan, 2009).

Error Rate

A repeated-measures ANOVA analysis showed no significant main effect for the degree of tiling on the error rate (F(2,22)=1.35, p=0.28), with the mean (SD) error rates of 5.324% (1.51), 8.33% (3.61), and 8.56% (6.11) for the [1×1], [2×2], and [3×3] conditions, respectively.

An ANOVA showed a significant main effect for symbol-split status on the error rate (F(1,11)=6.45, p<0.05). The mean (SD) error rates for the non-split and split conditions were 5.864% (3.00) and 8.796% (3.33) respectively, indicating that symbol split results in more searching errors (Figure 5-4).

(%) 15

0 Non-split Split Figure 5-4. Mean (SD) error rate by symbol-split status.

Not surprisingly, the number of distracters had a significant main effect on the error rate (F(1,11)=15.6, p<0.05). The mean (SD) error rates for 15 distracters and 30 distracters were 4.63% (1.97) and 10.03% (4.26) respectively, indicating that more distracters cause a higher error rate. Target presence also had a main effect on the error rate (F(1,11)=27.5, p<0.05), with the mean (SD) of 1.39% (0.612) for the target-absence condition and 13.27% (5.7) for the target- present condition; participants often missed targets when they were present, whereas false positives were extremely rare.

Subjective Opinions

At the end of the experiment, a short questionnaire was administered to gather subjective opinions.

 “Did interior bezels hinder or improve search process?” Three participants circled “Improved”. They commented that interior bezels separated the display area into smaller regions so that they could search them one by one, which seemed easier than with no interior bezel. In contrast, four participants reported that bezels hindered their performance because they were distracting and broke the continuity of search. Five participants thought bezels had no effect on the search task. These diverse answers indicated that effects of bezels on visual search might vary on different persons. According to participants’ subjective opinions, bezels effects could be positive, negative, or even neutral in search process. It somehow explains why no significant main effect was observed for bezels on search time.

 “What search strategies did you use?” 11 participants reported that they were searching targets grid by grid in both the [2×2] and [3×3] conditions since the entire display surface was divided into smaller subareas. In the [1×1] condition, various search strategies were applied. Four participants reported that they searched the target by clusters on the display; two participants reported that they searched from left to right and top to bottom on the display; four participants reported that they searched the target circularly. This result indicates that the presence of bezels affects the decision of search strategies; most of participants searched targets grid by grid in bezel-present conditions ([2×2] and [3×3]), and their search strategies varied in no- bezel condition ([1×1]).

5.2.4 Visual Search: Conclusions

We conclude the first experiment with our results with respect to the two hypotheses we formulated.

H.1 The presence of interior bezels is detrimental to visual search performance. This hypothesis was not confirmed. The number of interior bezels did not show any main effect on either search time or error rate.

H.2 Splitting data across interior bezels is detrimental to visual search performance. This hypothesis is confirmed. Splitting symbols across an interior bezel(s) leads to a high rate. We reason that this is because separating symbols apart increases the difficulty of recognizing them.

5.3 Tunnel Steering Experiment

The objective of the second experiment is to investigate how interior bezels affect straight-tunnel steering performance and behavior – straight-tunnel steering is a canonical task for GUI interaction such as hierarchical menu navigation (Accot & Zhai, 1997). As interior bezels deflect tunnels and cursor trajectory (Figure 5-5), we hypothesize that interior bezels are detrimental to steering performances. To be specific, the major hypotheses in this experiment are:

H.3 The presence of interior bezels hinders steering performance.

H.4 As the number of interior bezels increases, steering performance declines.

(a) (b) Figure 5-5. Deflected straight tunnels in (a) [2×2] and (b) [3×3] tiled displays.

5.3.1 Tunnel Steering: Task

Participants perform a straight tunnel steering task (Figure 5-6). At the beginning of each trial, a red start circle appears at one of four home positions (Figure 5-6a). After clicking the circle, a 58-pixel wide, 480-pixel long straight green tunnel appears next to the start circle. The tunnel’s direction is randomized with its start and end lines all residing within the display (Figure 5-6b). The subsequent cursor movement is shown as a red trajectory on the display (Figure 5-6c). When the cursor crossed the start line, a trial begins and the crossing time is recorded. When the cursor crosses the end line, the color of the tunnel turns to yellow, signaling the end of the trial (Figure 5-6d).

Crossing the side borders of the tunnel results in the cancellation of the trial, and an error is recorded. Participants are explicitly asked to perform the task as quickly and accurately as possible. The mapping speed of the cursor is adjusted through a pilot study to ensure that users

can steer through any tunnel without clutching. The “Enhance pointer precision” function in Windows XP is turned on.

The tunnel length is fixed at 480 pixels to ensure that any tunnel crosses at least one interior bezel in the [2×2] condition and at least two in the [3×3] condition. The bezels on the large display act exactly as physical bezels on a tiled-monitor large display: the tunnels are broken as they cross the bezels, but the widths of the tunnels keep constant. Note that this study cannot be considered as a general steering law experiment [1] because we do not systematically vary tunnel length and width. Instead, by varying the number of interior bezels, it allows us to have an insight into interior bezel effects on general steering tasks.

(275,180) (748,180) 480

(275,587) (748,587) 58

(a) (b) (c) (d) Figure 5-6. Steering task: (a) clicking the start circle appearing at one of the four home positions to start a trial, (b) a straight tunnel with a random direction, (c) crossing the start line, (d) crossing the end line (the color of tunnel changes from green to yellow)

5.3.2 Tunnel Steering: Design

We used a within-subject, repeated measures design with steering time, error rate, and cursor speed as our dependent measures. The steering time is defined as the elapsed time from the moment when the cursor crosses the start line until when it crosses the end line. The error rate is the percentage of erroneous trials. The cursor speed is defined as the amount of movement over time. Although the task will result in cursor speed being directly proportional to steering time if the trajectory is perfect, in practice participants will create different trajectories each time they steer through a tunnel. Further, this might be affected by whether or not the tunnel crosses bezel(s). Thus, cursor speed provides a slightly more nuanced measure that considers both speed and trajectory as a whole.

The independent variable is the degree of tiling (1, 2, or 3). The presentation order of these three conditions was fully counter-balanced with 6 combinations among 12 participants. Within each degree of tiling condition, participants first performed 3 trials to familiarize themselves with the task. Practice trials were followed by the experiment sessions, which consist of 6 blocks with each one containing 6 trials. In short, our design was: 12 participants × 3 degrees of tiling (1, 2, 3) × 6 blocks × 6 repetitions = 1296 total trials.

5.3.3 Tunnel Steering: Results Steering time

An ANOVA analysis of the collected data showed a significant main effect for the degree of tiling on the steering time (F(2,22)=5.8 p<0.05). The mean (SD) steering times for the [1×1], [2×2], and [3×3] conditions were 1542 (663), 2025 (869), and 2046 (815) ms, respectively. Pairwise mean comparisons showed significant differences in steering time for [1×1] vs. [2×2] (p<0.01) and [1×1] vs. [3×3] (p<0.01), but not for [2×2] vs. [3×3] (p=0.8) (Figure 5-7). No significant main effect of the number of blocks on the steering time was observed (p=0.233), indicating no main learning effects after the practice session. (ms) 3000

2000

1000

0 [1×1] [2×2] [3×3] Figure 5-7. Mean (SD) steering time by the degree of tiling.

Error rate

The mean (SD) error rates for the [1×1], [2×2], and [3×3] conditions are 7.6% (5.3), 11.5% (4.9), 8.0% (3.9), respectively. No significant main effect of the degree of tiling on the error rate was observed (F(2,22)=3.23, p=0.06).

Cursor Speed

An ANOVA showed a significant main effect for the degree of tiling on the mean cursor speed. (F(2,22)=8.4, p<0.05). Pairwise mean comparisons showed significant differences for any pairs (p<0.005) except for [2×2] vs. [3×3] (p=0.641). The means (SD) of the cursor speed in the [1×1], [2×2], and [3×3] conditions were 0.34 (0.25), 0.26 (0.22), and 0.25 (0.26) pixel/ms, respectively, indicating that the users steered fastest in the no interior-bezel condition.

To further investigate steering behaviors, we divided each 480-pixel long tunnel into 15 32-pixel long segments and calculated the mean speed in each segment. As illustrated by Figure 5-8, the users steered fastest in the [1×1] condition across all the segments.

Interestingly, Figure 5-8 also revealed different cursor speed distributions across the 15 segments. In the [1×1] condition, the cursor speed increased at the beginning of tunnels, kept a relative high value during the middle of operation, and slightly dropped down at the end. Participants commented that since the tunnels were relatively long, they sometimes lost control of the cursor at the end of tunnels thus leading to speed drop. Different from [1×1], strong wavy curves were observed for the [2×2] and [3×3] conditions. The cursor accelerated at the beginning of steering, but then sped up and down multiple times during the remaining steering process – one apparent (extra) deceleration for the [2×2] condition, and two for the [3×3] condition. Note that the average (SD) numbers of interior bezels the tunnels went through were 1.42 (0.49) and 2.88 (0.51), respectively.

(pixel/ms) 0.4

0.2

[1×1] [2×2] [3×3]

0 0 160 320 480 (tunnel length: pixel) Figure 5-8. Mean cursor speed in each 32-pixel long segment along a tunnel.

We reason that this wave-shaped distribution might be attributed to the presence of interior bezels. Participants reported that since a tunnel was separated into several sub-tunnels by interior bezels, they conceptually broke the entire steering task into a combination of multiple sub-tunnel steering tasks. Separating a tunnel into multiple sub-tunnels might break the continuity of cursor movement thus resulting in a wave shape in the cursor speed distribution.

Subjective Opinions

At the end of the experiment, the participants were asked to answer a short questionnaire.

 “Did interior bezels hinder steering performance?” Eight participants answered “Yes”. Some of them explained that the visually broken tunnel and deflected cursor trajectory prevented them from quickly steering through. One of them said, “I saw the tunnel was horizontally offset when it crossed a vertical (interior) bezel, so I moved the cursor to compensate it. However, the tunnel was in fact straight in virtual space. My intentional compensation moved the cursor out of the tunnel.” Four participants commented the bezels did not hinder their steering performance because they felt they could easily adjust mouse movement to meet the changes. In general, most participants indicated that bezels did hinder their steering performance, which is consistent with the Steering Time results – the presence of bezels significantly reduced the tunnel steering speed.

5.3.4 Tunnel Steering: Conclusions

We close this section with the results with respect to the hypotheses of the second experiment.

H.3 The presence of interior bezels hinders steering performance. This hypothesis was confirmed. Results showed a significant main effect for the degree of tiling on steering time, with the conditions having interior bezels leading to longer steering time. There are significant differences in steering time for [1×1] vs. [2×2], and [1×1] vs. [3×3]. Similarly effects were seen for cursor speed. Participants’ subjective opinions also confirmed the negative effects caused by bezels on steering process.

H.4 As the number of interior bezels increases, steering performance declines. This hypothesis was not confirmed. No significant difference in steering time, error rate, or cursor size was observed between the [2×2] and [3×3] conditions. From the cursor speed data, however, we can surmise that a tunnel crossing a much larger number of bezels might well result in lower performance, but our current experiment did not have sufficient number of bezels to test that hypothesis fully.

Besides performance, our data indicates that interior bezels also affect steering behaviors. As a long straight-tunnel is visually broken into multiple sub-tunnels, users tend to treat the steering task as a combination of multiple sub-tunnel steering tasks.

5.4 Target selection experiment

The goal of the third experiment is to investigate how interior bezels affect user performance in a target selection task, which is also canonical for GUI interaction. As target selection is path-free, cursor-trajectory deflection effects should be negligible. Thus, we set our hypotheses as follows:

H.5 The presence of interior bezels does not affect user performance in target selection tasks.

H.6 As the number of interior bezels increases, the performance of target selection tasks remains constant.

5.4.1 Target Selection: Task

We use a traditional 2D selection task consisting of repeated blocks of target selection trials. At the beginning of each block, a red circle appears at one of the four home positions (Figure 5-9a). Selecting the red circle starts a block, during which 10 blue circles appear sequentially. Participants are asked to select them in turn by clicking the mouse left-button while the cursor is in the target (Figure 5-9b). Each target appears when the prior one is selected successfully, and disappears when it is selected correctly.

(275,180) (748,180) 640

(275,587) (748,587)

(a) (b) Figure 5-9. Target selection task: (a) clicking the start circle at one of the four home positions to start a block of 10 trials, (b) clicking a target circle 640 pixels away from the start position (or previous target position).

The radius of each target circle is 12 pixels, and the distance between two successive circles is 640 pixels to ensure a cursor trajectory crosses at least one interior bezel to select a target in the [2×2] condition, and at least two interior bezels in the [3×3] conditions. Note that because our study is not intended to be a general Fitts’ law experiment (Fitts, 1954), we do not systematically vary target width and distance – our purpose is to initially investigate interior bezel effects by varying the number of bezels, not target size and distance per se. The position of each target circle is randomly generated, but not across interior bezels since our main purpose is to investigate if the cursor-trajectory deflection caused by interior bezels affects selection performance and behaviors. Similar to the second experiment, the mapping speed of the cursor is adjusted through a pilot study to ensure that users can select any target without clutching, and the “Enhance pointer precision” function in Windows XP is turned on.

5.4.2 Target Selection: Design

We used a within-subject, repeated measures design with selection time, error rate, and cursor speed as our dependent measures. The selection time is the elapsed time between when a target is presented and when it is selected by clicking the left mouse-button. An error is recorded when participants click outside the target. The error rate is the percentage of erroneous trials. The cursor speed is defined as the amount of movement over time. Unlike in the previous steering task experiment, we do not expect trajectories to significantly differ in this task, but nonetheless it would be useful to look at whether there is any deflection when crossing over bezel(s).

The independent variable is the degree of tiling (1, 2, or 3). The presentation order of these three conditions was fully counter-balanced among 12 participants. Within each degree of tiling condition, participants first performed a practice block consisting of 10 trials. The practice block was followed by the formal experiment, which consists of 6 blocks of 10 trials each. In short, our design was: 12 participants × 3 degrees of tiling (1, 2, 3) × 6 blocks × 10 repetitions = 2160 total trials.

5.4.3 Target Selection: Results Selection Time

The mean (SD) selection times for the [1×1], [2×2], and [3×3] conditions were 1211 (312), 1244 (265), and 1311 (263) ms, respectively. An ANOVA did not show any main effect for the degree of tiling on the selection time (F(2,22)=2.43, p=0.11).

No main effects for number of blocks on selection time was observed (p=0.446), indicating no learning effect in the formal experiment after the practice block.

Error Rate

The mean (SD) error rates for the [1×1], [2×2], and [3×3] conditions are 3.5% (2.42), 4.6% (3.2) and 3.54% (3.14), respectively. No significant main effect for the degree of tiling on error rate was observed (F(2,22)=0.76, p=0.48).

Cursor Speed

The mean (SD) cursor speeds in the [1×1], [2×2], and [3×3] conditions are 0.7648 (1.4), 0.7567 (1.4) and 0.7109 (1.5) pixel/ms, respectively. An ANOVA did not show significant main effects for the degree of tiling on cursor speed (F(2,22)=3.39, p=0.052).

Figure 5-10 shows the cursor speed along normalized target selection time. In each of the three degree of tiling conditions, the cursor speed increased very quickly at the beginning, reached the peak soon, and then gradually dropped down during the rest of the trajectory. There were no other bumps caused by interior bezels as found in the tunnel steering experiment. This result is consistent with our observation – participants first moved the cursor a long distance regardless of intervening bezels, and then slowed it down to accurately select targets.

(pixel/ms) 3

2 [1×1] [2×2] [3×3] 1

0 0 0.5 1.0 (Normalized completion time) Figure 5-10. Mean cursor speed.

Pearson correlation tests showed strong positive correlations for cursor speed among these three conditions (Pearson correlation coefficient > 0.9 for every pair), indicating that the cursor speed distributions in the [1×1], [2×2], and [3×3] conditions were very similar.

Subjective Opinions

At the end of the experiment, we gave the participants a short questionnaire. “Did the bezels hinder selecting targets?” 11 participants answered “No”, indicating that most participants felt that interior-bezel effects on target selection are negligible. It is consistent with

the Selection Time and Error Rate results, which also show that interior bezels have negligible effects on selection performance.

5.4.4 Target Selection: Conclusions

We conclude the third experiment with our results with respect to the hypotheses.

H.5 The presence of interior bezels does not affect user performance in target selection tasks. This hypothesis was confirmed. The results showed no main effect for the degree of tiling on selection time, error rate, or cursor speed.

H.6 As the number of interior bezels increases, the performance of target selection tasks remains constant. This hypothesis was confirmed. The results showed no main effect for the degree of tiling on selection time, error rate, or cursor speed.

Both the cursor speed analysis and subjective opinions also supported that target selection behavior was not affected by interior bezels. Participants performed the tasks similarly across the three conditions.

5.5 Discussions on Display Resolution and Size Effects

5.5.1 Resolution Effects

In our experiments, a single 1024×768 resolution projector was used to simulate all three tiling configurations. Thus, the resulting resolution of each grid in the [2×2] and [3×3] conditions is lower than that on real tiled-monitor displays (e.g., a 3×3 tiled-monitor display composed by nine 1024×768 LCDs). In this section, we discuss the feasibility of applying our findings to real- world settings having higher display resolutions.

Since a pixel is the smallest unit of visualization on a display, we can assume that the maximum display error is at most one pixel. Given the large-display projection area (167 cm wide × 127 cm high), the physical size of each pixel is 0.16 cm wide × 0.16 cm high, thus the maximum error is at most 0.16 cm wide or high. In all the three experiments, the sizes of visual objects are all far beyond 0.16 cm: the rectangular tunnel is 9 cm wide × 77 cm long; the diameter of a circular target is around 4 cm; the VI/IV symbols are displayed in a 4 cm × 4 cm square area. Thus, the

0.16 cm pixel length can cause at most 1.8%, 0.2%, 4%, and 4% error with respect to the tunnel width, tunnel length, circle diameter, and symbol size, respectively. The cursor (arrow-shaped) is displayed in a 2.7 cm high × 1.8 cm wide rectangular area. 0.16 cm will cause at most 5.9% and 8.9% error with respect to cursor height and width. Because our experiments do not require precise operations across objects’ boundary, we argue the current resolution does not substantially affect user performances. Subjective data also support this: all the participants reported that they could easily and comfortably view the tunnel, circular targets, VI/IV symbols and the cursor. In fact, given the 1.8 m sitting distance, 0.16 cm length leads to 0.05 degree visual angle, which is very close to the limit of a normal human’s visual acuity, 1/60 = 0.01667 degree.

Another difference between a 1024×768 projected screen with a higher resolution one is that the latter can show larger amounts of content. However, all the three experiments only involve simple geometric objects and a small number of symbols: the visual search experiment shows 15 or 30 VI/IV symbols; the tunnel steering experiment displays a rectangular tunnel; the target selection experiment shows a circular target. The visualization capability of a 1024×768 projected display is sufficient to clearly illustrate them. If we replace the 1024×768 display with a higher resolution one, we might get higher quality images, but not extra information such as more symbols/objects. Therefore, we argue the difference in visualization capability has minimal effect in the three experiments. The findings from these experiments can be generalized to higher resolution conditions with minimal changes.

5.5.2 Display Size Effects

Although the display in the experiments was 1.7 m wide and 1.3 m high, we argue that conclusions we draw about bezel effects can be easily generalized to other large-sized displays. Internal bezels affect users’ perception/interaction behaviors and abilities because they cause visual discontinuities and cursor trajectory deflections. The size of a display has a minimal impact on the presentations or functions of the bezels. Effects caused by display size on experimental results should be negligible.

5.6 Conclusions

In this chapter, we reported three controlled experiments investigating how interior bezels on tiled-monitor large displays affected user performance and behavior in visual search, straight tunnel steering, and target selection tasks. We summarize our findings as follows:

 Interior bezels are not detrimental to visual search performance; however, splitting objects across interior bezels leads to a higher error rate.

 The presence of interior bezels affects users’ search strategies. As an entire surface is divided into grids by interior bezels, users tend to apply a grid-by-grid search strategy.

 The presence of interior bezels hinders straight-tunnel steering performance.

 The presence of interior bezels also affects steering behaviors – as a tunnel is visually separated into sub-tunnels, users tend to treat the steering task a combination of multiple sub- tunnels steering tasks with a multiple cursor acceleration-deceleration pattern.

 The existence of interior bezels does not affect target selection performance nor user behavior.

Based on the above findings, we can gain some insights that might help with tiled-monitor large display usage and interface designs:

 Tiled-monitor large displays are suitable for visual search tasks. However, if high accuracy is required, objects should not be placed across bezels.

 UI designers should take special care in designing steering-based elements (hierarchical menus, etc.) on a tiled-monitor large display so as not to place them across interior bezels. If it is not avoidable, techniques alleviating bezel effects such as Mouse ether (Baudisch et al., 2004) and OneSpace (Robertson et al., 2004) might be worth considering.

 Selection-based UI elements (e.g., on-screen buttons or icons) can be adopted with no (or at most minor) adjustments on tiled-monitor large displays.

As an initial investigation into internal bezel effects, we did not vary steering/selection task parameters, such as tunnel shape/width/length, and target size/selection distance, as these would have overly complicated these studies. It might be worthwhile in follow-up work to conduct full Fitts’ law (Fitts, 1954) or steering law experiments on tiled-monitor large displays to gain an even deeper understanding of the effect of interior bezels (Accot & Zhai, 1997).

6 WallTop: Flexibly Managing Windows on a Large Display

The studies in Chapters 3 and 4 show various benefits of processing desktop work on a large display. However, since most of current user interfaces are designed for normal-sized monitors, challenges arise when users switch from traditional single- or dual-monitor to a large display. As indicated in Chapters 3 and 4, one challenge is about managing overflowing windows. One distinct behavior pattern of large display users is that they have special demands and require more flexibility when managing windows. However, current window management systems (e.g., WinXp) are designed for normal-sized monitors, which are too rigid for large displays. Based on the study results, we propose four suggestions to suit large display users’ unique window management behaviors:

Enable Multi-Window Operations

Support Flexibly Arranging Windows

Facilitate Moving and Resizing of Windows

Allow Executing Commands In-Place

To meet the aforementioned challenges, we design and implement a set of new interaction techniques to facilitate window management on a large display. These techniques are also carefully designed to ensure high compatibility with traditional window operations which have existed for more than two decades with little changes. Additionally, we deliberately avoid steering-based UI elements to ensure that the new techniques could be easily adapted to a tiled- monitor large display, as suggested by the study in Chapter 5. All these new techniques, together with traditional window management operations are implemented on a large display window management system prototype called WallTop. Two rounds of usability testing show that these new techniques significantly improve the efficiency of managing windows on a large display. A video demo of the WallTop is at www.dgp.toronto.edu/~xiaojun/WallTop/. Details of each interaction techniques and usability tests are reported in the remainder of this chapter.

100

6.1 WallTop Interaction Techniques

6.1.1 Fringe

The fringe is the virtual boundary enclosing selected windows (Figure 6-1). By default, overlapped windows are recognized as a group. The fringe around them fades in when the cursor is approaching the border of the group (within 3cm from any window of the group) (Figure 6-1 a&b), and fades out when the cursor is moving away (7cm away from every window in the group) (Figure 6-1 c). The user can also manually select multiple windows by drawing a rectangular rubber band (Figure 6-2) – windows intersected by the rubber band rectangle are selected. Selecting one window is achieved by directly clicking anywhere inside it (Figure 6-2). In any situation, the fringe is adjusted to exclusively enclose the selected windows. In the current prototype, the fringe is set to 3cm wide to ensure easy selection.

Once multiple windows are selected, users can simultaneously move and resize them: dragging a convex corner on the fringe resizes every window enclosed (Figure 6-4) while dragging any other position on the fringe moves windows simultaneously (Figure 6-2c and Figure 6-3c). Users can freely operate on an arbitrary number of windows with the fringe, no matter whether they are spatially aggregated or not. Since the fringe is much wider than the border of each window as well as its title bar, selecting, moving and resizing windows with the fringe are much easier than with the window border or title bar.

(a) (b) (c)

Figure 6-1. (a) overlapped windows; the fringe is activated when the cursor approaches the border of the windows (b), and deactivated when the cursor moves away (c).

101

(a) (b) (c)

Figure 6-2. (a) Drawing a rubber band on multiple windows, (b) clicking on the fringe of the selected windows, (c) moving them by dragging the fringe

(a) (b) (c)

Figure 6-3. (a) The fringe activates for multiple windows, (b) selecting a window by clicking inside it, (c) moving it by dragging the fringe (or the title bar).

(a) (b) (c)

Figure 6-4. Dragging one of the convex corners of the fringe that encloses selected windows resizes all of them at once.

102

6.1.2 Jab-to-lift

Office workers easily arrange physical paper documents with two hands. They often use the dominant hand to insert documents underneath others while lifting up the corner of those other documents with the non-dominant hand. Inspired by this lifting and inserting behavior, we implemented the jab-to-lift operation as a more flexible supplement to the normal window dragging and dropping activity which places a selected window always on top of the other windows, thus enabling some of the power of the real-world two-handed interaction but with just a single cursor.

Jabbing – performing a scratch motion by quickly and repeatedly moving the selected windows vertically, horizontally, or diagonally (Figure 6-5a) across the boundary of other windows animatedly shifts (folds) the closest corner of the static ones (Figure 6-5b). The selected windows can be inserted beneath folded ones (Figure 6-5b). The folded windows spring back to their original shape when the mouse button is released (Figure 6-5c), or the selected windows are moved away (at least 6cm away from folded windows). Both folding and unfolding are visualized with smooth animation to avoid confusion.

(a) (b) (c)

Figure 6-5. (a) Jabbing a selected window at the boundary of other window to fold its corner, (b) inserting the selected window under the folded window, (c) Unfolding the window when the mouse button is released.

Folding windows have been used to reveal underlying contents (Beaudouin-Lafon, 2001) or to facilitate copy/paste operations (Dragicevic, 2004; Chapuis & Roussel, 2005). We employ this concept in conjunction with the jab gesture that explicitly conveys the user’s intention of interleaving a selected window(s) with other window layers. Traditional window operations only

103

allow users to drag a window over others, which is too rigid to appropriately support arranging windows on a large display. Jab-to-lift offers more flexibility in changing the z-order of windows on a large display using just one cursor with a natural and easily understandable metaphor of peeling back and inserting physical paper documents.

6.1.3 Multi- and Single-Window Marking Menus

Many of the interactions on WallTop are performed via right-click marking menus (Kurtenbach, 1993) which are activated when the user right clicks on the fringe. Dragging the cursor with the right-button down beyond the inner circle highlights a color-coded command wedge (Figure 6-7a, Figure 6-8a, Figure 6-9a, Figure 6-10a). Subsequently releasing the right-button executes the corresponding command. We implemented two types of marking menus on WallTop. The multi- window marking menu (Figure 6-6a) is invoked if the fringe encloses more than one window, otherwise the single-window marking menu (Figure 6-6 b) is activated.

Marking menus are designed to address the fourth design goal in section 3.3.2 – executing commands in-place, and avoiding precise selection. Users can activate marking menus anywhere on the fringe without moving the cursor to a specific small location (e.g., the corners or the title bar of a window(s)). Since menu items are selected by cursor movement direction, precise pointing is not necessary. We only use the fringe area for menu activation because right clicking on other regions of a window might conflict with the traditional window functions. For example, in the Windows XP OS, right clicking on the title bar invokes a window menu and right clicking inside a window invokes an application menu.

Center

Undo Spread Center

Side Left Side Right Side Left Side Right

Splat Pile Close

Close (a) (b)

Figure 6-6. (a) Multi-, and (b) single-window marking menus.

104

We did not employ left-click marking menus to avoid conflicting with moving a window(s) – Left-button clicking and dragging on the fringe moves or resizes selected windows. We considered left-button double-click marking menus as another alternative, but only kept it as an optional setting based on results of the first-round usability test (see First Round Test subsection).

6.1.3.1 Multi-Window Operations

The marking menu on the fringe establishes a framework for performing a variety of multi- window operations. This is used to make typical selection-and-action phrases: the former (the fringe) visually indicates the selected targets and the latter (the marking menu) provides command options. A set of multi-window operations are designed to mimic how office workers organize paper, such as piling, packing, and spreading (Sellen & Harper, 2003).  Spread. Office workers often spread documents out during active reading or comparison reading. The primary document is placed in the center and surrounded by supporting ones (Figure 6-7 c). Large display studies also show such spreading behaviours are common when a task involves multiple windows (e.g., programming or active reading). Invoking Spread command in the marking menu quickly arranges selected windows in a tiled layout (Figure 6-7). Once executed, the top window of the selected windows is placed in the center and surrounded by other windows.

Center Undo Spread 2 2 Side Left Side Right 3 1 Splat Pile 4 3 1 4 Close (a) (b) (c)

Figure 6-7. Choosing Spread item on the marking menu (a) arranges the selected windows (b) in a tiled layout where the window positioned on top is centered (c).  Splat. Splatting is an effective tool for selecting an object from a group of occluding ones (Apple Expose, (Ramos et al., 2006)). Invoking the Splat command in the marking menu temporarily separates windows apart (Figure 6-8). Each separated window is connected to the group center with its ray fan that indicates the splat status. The subsequent mouse left-button 105

click (Figure 6-8c) brings separated windows back to original positions with the clicked one on top. Splat uses a force-directed layout algorithm where each participating window is slid along the ray starting from the group center to its center by the distance proportional to its size (Ramos et al., 2006). This algorithm roughly preserves the spatial relationship of the selected windows.

Center 2 1 Undo Spread 2 Side Left Side Right 1 3 3 Splat Pile 4 4 Close (a) (b) (c) Figure 6-8. Choosing Splat item on the marking menu (a) arranges the selected windows (b) in a radiated configuration from the group center (c).

 Pile. Document piles are commonly seen on office workers’ desks. Piling has been used as an effective way of organizing files and desktop icons (Agarawala & Balakrishnan, 2006; Mander, Salomon & Wong, 1992). Executing the Pile command via the marking menu piles up selected windows with their title bars uncovered to show their names (Figure 6-9). Once piled, leafing through the windows by moving up or down the mouse wheel brings a window from bottom to top or top to bottom, enabling quick window browsing.

Center Undo Spread 2 4 Side Left Side Right 3 1 32 Splat Pile 4 1 Close (a) (b) (c)

Figure 6-9. Choosing Pile item on the marking menu (a) arranges the selected windows (b) in a tidy pile layout (c).

 Center/Side. Users frequently switch windows back and forth between center and side areas of a large display. The former is the focal region where most mouse and keyboard events

106

are triggered while the latter is the peripheral region where windows not being used currently are usually positioned. For example, an email client program is dragged into the focal region from the periphery when the user is to check messages, and is later on repositioned to the peripheral region to make space in the center area for other windows used for the next task to occupy. Large display users reported that dragging windows back and forth between focal and peripheral regions was annoying, mainly due to the fatigue caused by holding the mouse button for a long period of time and traveling a long distance Center and Side Left/Right commands are designed to ease this frequent switching process. Invoking the Center command on the marking menu brings selected windows to the center area of the display, while Side Left and Side Right shift windows to empty space at the left and right areas of a large display, respectively (Figure 6-10). On the marking menu, Side Left and Side Right commands are placed at west and east wedges to be consistent with window moving directions.

Center Left Center Right

Undo Spread 2 2 Side Left Side Right 1 3 1 3 4 Splat Pile 4 Close (a) (b)

Figure 6-10. Choosing Side Right item on the marking menu (a) shifts the selected windows to right. (b) Windows are automatically packed at the side regions.

Scalable Fabric allows users to quickly move windows from center to side by clicking the window’s minimize button, and from side to center by clicking anywhere inside the window (Robertson et al., 2004). On a large display, however, users still click the minimize button to minimize windows and click on windows placed in the peripheral regions to operate on the applications. Thus, Scalable Fabric’s position switching convention can cause operation conflicts. Our use of Center and Side actions with marking menus solves this conflict, and coherently integrates with existing operations while providing users with more flexibility – users can choose on which side (left or right) to place windows.

107

 Undo/Close. Any window arrangement operations executed previously can be reversed by using the Undo command; the Close command closes the selected windows.

 Pack/Unpack. Packing files is another common behavior of office workers. After reading some paper documents, they lazily pack documents into a messy pile. Different from a tidy pile, a messy pile still subtly conveys the initial spatial information of each document, which might ease later retrieving action (Agarawala & Balakrishnan, 2006). We design Pack and Unpack icons on the upper left corner of the fringe for continuously packing and unpacking windows (Figure 6-11a). These two operations are designed as icons because they require subsequent continuous mouse movement to determine operation parameters. It is hard to fit such a combination operation into a marking menu.

2 2 3 1 3 1 4 4

(a) (b) (c)

Figure 6-11. (a) Pack (up) and Unpack (left) icons; clicking Pack icon (b) and dragging in any direction continuously tighten the selected windows (c).

Clicking the Pack icon and dragging in any direction tightens the selected windows (Figure 6-11 b and c). The dragging distance determines how tightly windows are packed. Unpack works in reverse – clicking it and dragging separates windows apart. Automatic Pack and Unpack functions are also enabled to support large display users’ common behaviours of packing documents on the side region and unpacking windows in the focal region. When moved from center to side, windows are automatically packed (Figure 6-10 b), and unpacked when moved from side to center.

108

6.1.3.2 Single-Window Operation

Right click on the fringe enclosing only one window triggers the single-window marking menu, which contains four commonly used commands (Figure 6-6 b). Similar to those on the multi- window marking menu, these options are used to center, side, or close the selected window.

6.2 Implementation

We integrated the new interaction techniques with traditional window operations (Windows XP) into a prototype called WallTop. The current WallTop is a proof-of-concept system (using images of windows instead of real applications) written in C++, OpenGL and GLUT, and running on a 16’ wide x 6’ high large display via the Chromium system (chromium.sourceforge.net). The entire display consists of 18 projectors arranged in a 3×6 configuration, with a total resolution of 6144×2304 pixels (1024×768 pixels each) (Figure 6-12). A video demon of WallTop is at http://www.dgp.toronto.edu/~xiaojun/WallTop/ .

WallTop is designed to facilitate window management on a large display which is big enough to simultaneously visualize multiple windows. We demonstrate the interaction techniques on this wall-size display as a proof-of-concept. They can be easily implemented on large displays of various sizes with at most minor changes.

16

1024×768 pixels 6

Figure 6-12. Specification of WallTop and usability test setup.

109

6.3 Iterative Usability Tests

We conducted two rounds of usability tests of WallTop on a wall-sized display. The first round test was to discover potential usability problems and refine interaction techniques. The second round was to formally evaluate the new interaction techniques.

6.3.1 First Round Test

6.3.1.1 Test Design

Six participants (2 female, 4 male, 18~34 years old), participated in videotaped think-aloud sessions lasting an hour each. After a 20-minute introduction session, participants tried out the new interaction techniques. In particular, they were instructed to compare the right click and left double click for activating marking menus. Participants sat 2 m away from the center of the display (Figure 6-12), which is a normal sitting distance for the large display reported in the longitudinal study.

6.3.1.2 Results and Observations

Subjective data shows that most users prefer right click to invoke marking menus to left-button double click. The average score of right-button click and left-button double click are 4.2/5 and 3.3/5, respectively. Five out of six participants reported a preference for the right-click marking menu. Three users explained that right-click marking menu required only one click action whereas double click required two. One participant said he was familiar with right-click marking menu because of the familiarity with the Autodesk Maya 3D modeling and animation system. The only participant who preferred double click explained that he disliked pressing and holding the right button with the middle finger. Since the majority of the participants preferred the right- click marking menu, we chose it as the default for WallTop. Double-click marking menu is kept as an option that users can enable via a systems setting dialog box.

We initially designed only one icon for Packing and Unpacking and distinguish them by cursor dragging directions. Clicking the icon and dragging down packs the selected windows while dragging up unpacks them. However, we observed usability problems with this approach across all the participants. All six participants reported that they disliked using one icon for both packing and unpacking. Also, five participants said they were confused with using direction to

110

distinguish two opposite functions, even after one hour of practice. To alleviate this problem, we finally designed two separate icons for these two operations.

We also determined several design parameters via this usability test. For example, the fringe is 3cm wide and fades out when the cursor is 7cm away, and folded windows spring back when the cursor is 6cm away. We iteratively found that these values ensure easy selection of windows on their fringe, and prevent them from being deselected unexpectedly in the middle of operations.

6.3.2 Second Round Test

6.3.2.1 Test Design

After refining the techniques according to the first round feedback, we conducted a user study to evaluate WallTop. In particular, we aimed to determine how quickly and correctly users learn our new interaction techniques, and how much these techniques improve the efficiency of managing windows on a large display.

Eight participants (2 female, 6 male, 18~34 years old) who did not participate in the first round test were recruited. Similar to the first round test, they sat 2 m from the center of the display (Figure 6-12). After a 20-minute introduction session, participants performed 29 window management tasks ranging from simple to compound tasks.

Simple Tasks. Tasks labelled ST1~24 were simple ones which required users to roughly reposition or resize windows with one or two specific operations described on the instruction sheets that we provided (Table 6-1). The series of simple tasks were designed to test if users could correctly understand and use the new interaction techniques.  Moving windows (ST1~4): Move single or multiple windows by dragging the fringes  Resizing windows (ST5~7): Resize single or multiple windows by dragging a corner of fringes.  Inserting windows (ST8~10): Change the z-order windows by using the jab-to-lift operation.  Packing/unpacking windows (ST11, 12): Pack or unpack windows using Pack/Unpack icons.  Using marking menus (ST13~24): Perform single- or multi-window operations using marking menus (ST13~20 for multiple windows, ST21~24 for a single window).

111

Table 6-1 Simple tasks (summary). ST Description Perform ST1~4 by dragging the fringes 1 Move A/G/Y/P (Center) simultaneously to Right. 2 Move A (Center) to Right. 3 Move A/Y (Center) simultaneously to Right. 4 Move C/E (Left) simultaneously to Center. Perform ST5~7 by dragging a corner of the fringes 5 Enlarge and then shrink A/G/Y/P (Center) simultaneously. 6 Enlarge and then shrink C/E (Left) simultaneously. 7 Enlarge and then shrink C (Left). Perform ST8~10 by jab-to-lift operation 8 Insert C (Left) under E (Left). 9 Insert C (Left) under A/G/Y/P (Center). 10 Insert A (Center) between G and Y (Center). Perform ST11~12 with Pack/Unpack icons 11 Pack and then unpack A/G/Y/P (Center). 12 Pack and then unpack C/E (Left). Perform ST13~ST20 with multi-window marking menus 13 Spread A/Y/G/P (Center). 14 Undo ST13 15 Pile up A/G/Y/P (Center) and review them with the mouse wheel. 16 Splat A/G/Y/P (Center) and select G as the top window. 17 Side A/G/Y/P (Center) to Left. 18 Side A/G/Y/P (Center) to Right. 19 Center C/E (left). 20 Close A/G/Y/P (Center). Perform ST21~24 with single-window marking menus 21 Center C (Left). 22 Side P (Center) to Right. 23 Side P (Center) to Left. 24 Close E (Left). A: Park-Arches; G: Park-Glacier; Y: Park-Yosemite; P: PowerPoint; C: Calendar; E: Email; Au: CHI-Author; At: CHI-Attending; H: CHI-Home

Compound Tasks. We extracted window management behaviours from the large display user’s realistic activities reported in the longitudinal study to create five compound tasks (Table 6-2). Although we used images of windows instead of real application windows, each of the compound tasks were carefully chosen to emulate a set of common daily window management activities.

112

 Information checking (CT1): Move an application (e.g., personal information program such as a calendar) from side to center, enlarge it to reveal the contents, and then return it back to roughly the original position and size.

 Task switching (CT2): Move windows that have been used for the current task from center to side, pack them, and move other windows that are to be used for the next task from side to center, and spread them.

 Window arrangement (CT3, 4): For focused tasks, find and move relevant windows scattered across the large display next to the main application window, then resize and reposition them to form a spread layout. These tasks were intended to mimic programming and/or active reading activities.

 Tidying (CT5): Clean the workspace by grouping relevant windows together, and move them side.

Here is the description of a compound task written on a piece of paper handed to the participants in the study. e.g., CT2. Move “Park-Arches”, “Park-Yosemite”, “Park-Glacier”, and “PowerPoint-Park” windows from the center region to the left side of the display, and tighten them to save space. Move windows “CHI-Author”, “CHI-Attending”, and “CHI-Home” from the right side to the center region. Surround “CHI-Home” window with “CHI-Attending” and “CHI-Author”.

Different from simple tasks where every operation is specified, users were allowed to perform compound tasks using any available operation. Another difference is that compound tasks were performed under two conditions:  With-condition: Participants can use both new and traditional window interaction techniques (i.e., dragging the title bar to move a window, dragging its border or corner to resize a window, clicking buttons on its upper right corner to close or maximize a window).  Without-condition: Participants are not allowed to use new interaction techniques; they can only use traditional window operations.

113

Table 6-2. Compound taskTable 6-3s (summary). CT Description 1 Information checking Move C (Left) to Center and Move C back to Left and make (Calendar) enlarge it. it small.

A At A At A At C Y Y C Y P H P H P H G Au CG Au G Au E E E

2 Task switching (from Move A/G/Y/P to Left and Move Au/At/H to Center and PowerPoint to CHI) tighten them. enlarge them.

Y A Y A A At G P At G P Au C Y C C At P H H G Au Au H E E E

3 Window arrangement Move A/G to Center next to P. Re-order windows (A/G under (PowerPoint) P).

A Y At Y At Y At C H C H C H Au G A Au G A Au G P P P E E E

4 Task switching and window Move E (and C) to Left. Move G/Y to Center next to P. arrangement

A C A C A C At E At E At H H H G Au G Au Y Au P P G P Y E Y

5 Tidying Move C/E to Side; shrink C and Move/tighten At/Au/H, enlarge E. A/G/Y/P, respectively.

Y Y At Au Y P P H P C G G G A At Au At Au H A H A E E E C C A: Park-Arches; G: Park-Glacier; Y: Park-Yosemite; P: PowerPoint; C: Calendar; E: Email; Au: CHI-Author; At: CHI-Attending; H: CHI-Home

We set these two conditions to test how much new interaction techniques improve the efficiency of managing windows on a large wall display. Four of eight participants performed the compound tasks under the with-condition first and then under the without-conditions. The order was reversed for the other four participants.

For both the simple and compound tasks, participants were provided with sheets of paper on which the detailed description of each task was written. They were instructed not to start each task until they completely understood it, and to start each task by clicking the Start button and end by clicking the Finish button (both buttons were located at the bottom of the screen).

114

6.3.2.2 Dependent Variables

Simple Tasks. We focused on two dependent variables for each simple task.

 Completion time: the time lapse between the user clicking the Start and Finish buttons.

 Number of failed trials: If windows are repositioned or resized differently from the description of each task, it is a failed trial and the participant has to redo it. An experimenter monitors the experiment and judges each trial to be successful or failed immediately after the Finish button is clicked. Furthermore, simplified tasks must be performed with operations specified on the script, otherwise it is failed. Take task 19 for example. It is a failed trial if a participant moves the window “Calendar” to the center region by dragging the fringes. The system logs every operation and signals if wrong commands are used.

Compound Tasks. Similar to simple tasks, we record both completion time and number of failed trials for each compound task. The only difference is that the logged operations are not used to judge a trial to be successful or failed, because participants are free to use any available operations for completing compound tasks.

6.3.2.3 Results

Simple Tasks. Table 6-4 shows the mean completion time and total number of failed trials for each task. The total number of failed trials sums failed trials of all the participants for each task. The mean completion time of 24 tasks is a reasonably quick 7.9s (standard deviation, =1.3), implying that users executed these operations in a timely manner. The total number of failed trials is zero for 18 out of 24 tasks, indicating that users could learn most of the interaction techniques without any trouble. For ST2, 4, 9, 13, and 20, this dependent variable is less than or equal to 2, showing that most users correctly performed these tasks the first time. ST10 was found to be the most difficult task that asks users to insert one window between two overlapped windows. Four users encountered difficulties when performing it for the first time, but they succeeded it in the second trial. In general, all the participants were able to learn our new interaction techniques quickly and easily. The post-study questionnaire confirms that the new techniques were easy to learn (4.5/5), that users were able to accomplish the tasks (4.4/5), and that users liked the new techniques (4.6/5). 115

Table 6-4. Mean completion time and total number of failed trials for simple tasks. ST (Simple 1 2 3 4 5 6 7 8 9 10 11 12 Task) Completion 7.1 6.5 6.3 7.2 6.1 7.4 6.8 7.6 8.1 11.4 7.7 10.7 time (s) Standard 1.4 1.3 1.5 1.8 2.3 2.8 3.9 3.2 1.9 4.5 2.2 2.0 deviation () Number of 0 1 0 1 0 0 0 0 1 4 0 0 failed trials ST (Simple 13 14 15 16 17 18 19 20 21 22 23 24 Total Task) Completion 8.2 7.2 7.8 9.3 8.1 8.2 9.1 9.5 7.9 6.9 7.3 7.6 7.9 time (s) Standard 2.4 1.2 3.8 4.2 2.3 3.2 3.7 4.7 3.5 3.3 3.5 3.2 1.3 deviation () Number of 2 0 0 0 0 0 0 1 0 0 0 0 10 failed trials

Compound Tasks. Figure 6-13 shows the mean time for performing CT1~5. The participants completed each task much faster with new interaction techniques. The average mean completion times are 59.54s (=16.4) and 77.86s (=21.4) under the with- and without-condition, respectively. A dependent t-Test shows a significant difference in completion time for every task between the two conditions (p<0.01 for all the tasks). Only two failed trials were observed across all participants and tasks – one for CT1 under the with-condition and the other for CT5 under the without- condition. Participants who committed the failed trials explained that they misunderstood the descriptions of the tasks.

(s) 100

50 With new techniques

0 Without new techniques CT1 CT2 CT3 CT4 CT5 (Traditional ones only) Total

Figure 6-13. Mean () completion time for compound tasks

116

To answer why participants performed given compound tasks much faster with new interaction techniques, we analyzed the logged operations. The average numbers of operations for each task are 8.7 (=3.5) and 14.3 (=5.6), with and without the new techniques, respectively. It indicates that the participants used fewer operations to complete tasks when they were allowed to use the new window operation techniques. Furthermore, the proportions of used operations under the with-condition (Figure 6-14) shows that more than 85% of used operations were the new interaction techniques for each task despite the fact that participants were free to use both the new techniques and traditional operations. In particular, the fringe usage accounted for nearly 50% in average. Thus, we can say that the performance gain in the tests mainly comes from using our new interaction techniques, and that applying them would likely significantly improve window management efficiency on a large display since these compound tasks represent realistic window management activities.

Window move/resize w/ fringe Jab-to-lift Marking menu Pack/unpack Traditional window operations CT1 CT2 CT3 CT4 CT5 Total

Figure 6-14. Proportions of used operations under the with new techniques condition

Five participants reported that the most helpful technique was the fringe because it helped save time and effort enabling them to move or resize multiple windows simultaneously. Four noted that Center and Side Left/Right commands were very helpful. One explained, “I use the menu to move windows because I am just too lazy to move the cursor a long distance.” In the post-study questionnaire, all participants agreed as to the usefulness of WallTop: do the additional window operations make window management better or worse? (4.6/5) (1: much worse, 5: much better).

6.4 Discussion

These new interaction techniques are designed meet the challenges of managing overflow windows on a large display discussed in session 3.6.2. Informed by the usability results, we now reflect on how the new interaction techniques meet our objectives. 117

6.4.1 Multi-Window Operations

The fringe helps users select multiple windows easily, and the multi-window marking menu offers a variety of operations including quickly centering/siding windows and arranging them in different layouts. The results of the simple task test show that users can quickly and easily learn these operations, and those of the compound task test show that using these techniques can significantly shorten the completion time of realistic window management activities.

6.4.2 Flexibly Arrange Windows

The jab-to-lift technique adds flexibility in arranging windows. It allows users to insert a window(s) under others, complementing traditional window dragging and dropping only on top of others. The usability tests showed that users could quickly learn this operation after a short period of practice. One participant reported that the jab-to-lift seemed a bit slower than traditional ways, but she still liked it because it was more natural and flexible, with a dose of fun.

The multi-window operations via marking menus also allow users to quickly spread, pile up, pack and unpack multiple windows. Together with the Fring, they offer greater flexibility of managing windows than traditional window management operations.

6.4.3 Facilitate Moving and Resizing Windows

Because the fringe is wider than traditional window borders or title bars, moving and resizing windows become much easier from a basic Fitts’ law (Fitts, 1954) perspective. The second usability test, given compound tasks involving many moving and resizing operations, confirmed this with the fringe usage accounting for 50% of the number of all the operations done, and that the participants performed the tasks much faster with the fringe than without it. Furthermore, the Center, Side Left/Right operations on marking menus meet the demands of quickly switching windows between center and side regions.

6.4.4 Execute Command In-Place

We achieved this goal by using marking menus. Users can invoke the menus anywhere on the fringe instead of on a fixed corner. In addition, the marking menus utilize cursor moving direction instead of precise clicking for command execution, avoiding the error-prone clicking

118

operations on a large display. One exception is the Pack and Unpack icons, which are located on the upper left corner of the fringe. Because they require subsequent continuous mouse movement, it is hard to fit them into a marking menu. But, to ease clicking them, these icons are bigger than traditional Minimize and Close buttons. One participant suggested placing Pack/Unpack icons on every convex corner of the fringe. However, we did not incorporate this suggestion as we felt it would result in visual clutter.

In addition to meeting the challenges of managing windows, the new techniques are also designed to ensure high compatibility with traditional window operations. In the WallTop system, none of the new interaction techniques conflicts with the existing window operations. All the traditional operations are also maintained in our prototype and can be used in conjunction with new ones. In addition, we note that some of the new interaction techniques are judiciously designed to leverage the user’s familiarity with existing interaction conventions, such as dragging the fringe to move windows, and dragging its corner to resize them.

WallTop includes a set of new interaction techniques addressing challenges neglected in previous research. Table 6-5 summarizes the novelty of WallTop in comparison with other window management systems.

Table 6-5. New functions in WallTop compared with other systems

Multi-Window Operations Flexibly Arrange Windows

 Enable Multi-Window Resizing  Job-to-lift adjusts z- WallTop and Moving via Fringe order of a window.  Enable multi-window spread,  Multi-window marking pile, pack, unpack menus enable several layouts.

Scalable Fabric Enable multi-window grouping/ No function for adjusting z- (Robertson et al., ungrouping order. 2004)

119

Metisse (Chapuis & Not Enabled Windows can be rotated, and Roussel, 2005) peeled back.

Facilitate Moving and Resizing of Execute Commands In-Place windows

 The fringe eases moving and WallTop resizing. Marking Menus on Fringe  Center/Side commands on enable executing commands marking menus allow quick switch In-Place. between sides and center

Scalable Fabric Minimize/Maximize buttons enable Not Enabled (Robertson et al., quick switch between sides and center 2004)

Metisse (Chapuis & Traditional Operations Not Enabled Roussel, 2005)

6.5 Conclusions

To meet the challenges of managing numerous windows on a large display, we design a set of new window operations. These techniques include the fringe, jab-to-lift, and various window arranging functions such as Spread, Pile, Center, and Side. All these techniques were coherently integrated with traditional operations in our proof-of-concept window management prototype, WallTop. The usability tests showed that these techniques are easy to learn and can significantly improve the efficiency of managing windows on a large display.

Aside from studying users’ behaviors and designing appropriate interaction techniques, we also made contributions in proposing benchmarks for evaluating window management systems. In the 120

second round user test (section 6.3.2), we designed a set of window management tasks ranging from basic operations on a single window to complex arrangement tasks involving multiple windows. These tasks are extracted from the longitudinal diary study to reflect the commonly performed window management tasks for desktop work. They can also serve as benchmarks to evaluate the efficiencies of other window management systems.

121

7 Conclusions

This dissertation systematically investigates the effects of a large display on information workers’ performance and behaviors in desktop work. Guided by the UI design implications drawn from the studies, we design a set of interaction techniques to address the challenges of working on a large display. This work exemplified a commonly used human computer interaction research methodology: understanding human interaction/perception abilities through theoretical analysis and empirical experiments, and using such understanding to guide interaction technique design. In this final chapter, we provide a summary of our work, summarize its limitations, and discuss the opportunities for future work.

7.1 Summary

7.1.1 Thesis Structure

We conducted the research following a progressive plan. Following the aforementioned research methodology, we first conducted a longitudinal diary study (Chapter 3) to investigate the effects of a large display in realistic settings, and to compare it with traditional computing environments (i.e., single- and dual-monitor settings). We then carried out controlled experiments (Chapter 4) to validate the findings acquired from the prior longitudinal diary study, and to deepen the understanding of large display effects. Tiled-monitor large displays are presently widely used as approximations of seamless large displays in many places. To understand the difference between tiled-monitor and seamless large displays, we conducted a series of experiments to investigate effects of interior bezels on tiled-monitor large displays (Chapter 5). Based on the understanding of users’ abilities and limitations reported in Chapter 3, 4 and 5, we designed and implemented a set of new interaction techniques to address the challenges of working on a large display. These techniques were also formally evaluated to ensure high usability in practice (Chapter 6). Detailed content of each chapter is presented as follows.

In chapter 1, we describe the background and discuss the motivation of investigating large display effects and designing large display-oriented interaction techniques. To better process an increasing amount of digital information, information workers tend to work on a big visualization surface. In the meantime, the rapid development of display technologies has 122

enabled a user to easily obtain a large display for daily work: a 30” monitor can be purchased at a moderate price; one of the biggest LCDs on the market is 100”. It is possible that users could easily obtain a large, or even wall-sized display as their primary working space for daily desktop work in the near future. This dissertation is to address two research questions under this background: how will a large wall-sized display affect users’ performances and behaviors in desktop work? How should we design appropriate interaction techniques to suit such a large display?

In chapter 2, we thoroughly review relevant literature, including using a big-sized monitor or multiple monitors to process desktop work, current large display usages in various scenarios, interaction techniques on a large display, and desktop workers’ information management behaviors. Prior research reveals various advantages of a large display and converges on a conclusion that office workers could potentially benefit from a large visualization surface. However, there has been little research systematically investigating the effects of a large display or designing large display oriented interfaces for desktop work. This dissertation is aimed to shed some light in these areas.

In chapter 3, we report a longitudinal diary study comparing the usage of a large display (4.8 m wide x 1.8 m high) to single- and dual-monitor configurations in a desktop work environment. Four single- and four dual-monitor users switched to work on a large display for a one-week period each person. Results show the users’ unanimous preference of working on a large display and reveal distinct behavior patterns in partitioning screen real estate and managing windows on a large display. The analysis also leads to implications for designing window management operations specifically suited a large display.

To further investigate how a large high-resolution display affects users’ performance and behaviors in desktop work, we report three control experiments in chapter 4. By precisely controlling display physical size and pixel density, we detangle effects of these two resolution- related factors. The results not only validate findings from the prior diary study in chapter 3, but also provide more in-depth understanding of display physical size and pixel density effects.

Tiled-monitor large displays are now widely used as approximations of seamless large displays in many places. To understand the difference between such displays and true seamless large 123

displays, we report three control experiments which systematically investigated the effects of interior bezels within tiled-monitor large displays on users’ basic perception and interaction abilities in chapter 5. The study results reveal bezel effects on visual search, tunnel-steering and object selection tasks, and also suggest guidelines for designing tiled-monitor suited UI elements.

Guided by the study results in chapters 3, 4, and 5, we designed, implemented, and evaluated a large display oriented window management system prototype—WallTop, which is elaborated on in chapter 6. This system prototype is designed to address a significant challenge of working on a large display: managing overflowing windows. The WallTop enables users to operate on multiple windows simultaneously, and also provides a set of novel interaction techniques to suit large display users’ behaviors. Formal usability tests show that WallTop significantly improves window management workflow on a large display over traditional window management systems such as Window XP.

7.1.2 Implications for Display Usage

As display technologies advance rapidly, future offices may be equipped with large displays in various shapes and forms (Figure 7-1): the tables and walls in an office become displays, which office workers use as primary work spaces. This dissertation systematically investigates the effects of a large display for desktop work. Lessons learned from the research would guide future office workers to choose appropriate displays.

Figure 7-1. Office of the Future. Figures taken from http://www.officelabs.com.

In general, our studies show that a large display offers great benefits for daily desktop work. We suggest that office workers, especially those who frequently process scattered-information and 124

perform large scale navigation tasks, switch from normal-sized monitors to large displays. A large display can visualize a great amount of digital information simultaneously, which facilitates the information collection and comparison. It also reduces the number of navigation operations when processing large data sets. Besides the benefits for specific tasks, a large display benefits general digital information management. As a large display allows users to freely file and pile windows, and to arrange them into various meaningful layouts, it provides contextual information for different tasks, eases the regrouping of windows as a task goes on, and reminds users of upcoming jobs. It allows users to flexibly manage digital information with few space constraints.

Our studies indicate that the performance gain obtained from using a large display depends on the tasks. We suggest that desktop workers choose displays according to the types of their typical working tasks. Switching from normal-sized monitors to large displays benefits scattered- information processing and large scale navigation tasks. However, performance remains unchanged for single-window text processing tasks as display size and pixel density increase. Furthermore, different display size and pixel density combinations affect users’ performance. With the identical resolution, a high pixel-density and small-sized display outperforms a low pixel-density and big-sized monitor for scattered-information processing, while the opposite is observed for large scale information navigation tasks.

Our studies reveal not only the overwhelming benefits of using large displays but also the challenges of working on them. Since current UIs are mainly designed for normal-sized monitors, simply switching to large displays causes usability problems such as managing windows: as users tend to operate on multiple windows simultaneously, current window management systems such as window XP and Mac OS do not fit users’ behavior patterns on a large display because they only allow users to operate on a single window at any time. Using the window management techniques designed in WallTop significantly improves the window management workflow on a large display. We expect these technologies will be integrated into upcoming operation systems to benefit future office workers.

Besides large flat displays, another option for processing future desktop work is to use curved displays. Precious research (Shupp et al., 2009) reveals that flat and curved displays have their

125

own pros and cons. A curved display outperforms a flat display in geographic search and comparison tasks because a curved display changes the type of physical navigation, from translational (e.g., walk) to rotational (e.g., head rotation), which improves the performance. However, a flat display is preferred over a curved display in insight tasks because 1) a flat display is good at providing overall information, 2) users are familiar with flat forms, and 3) the flat forms better support searching by grids. Also, images are distorted when they are displayed on a curved surface, which might hamper users’ ability to recognize objects. Due to the structures of modern offices, large flat displays usually fit the offices well, but curved displays might have installation issues. Therefore, we suggest that users choose appropriate displays according to the types of tasks they work on, the structures, and sizes of the offices.

7.2 Limitations

In this session, we outline limitations associated with work presented in this dissertation.

To form a large high resolution display in studies in chapters 3 and 4, we tiled multiple projectors and did careful calibration to minimize seams between them (i.e., the width of the seam < 1mm). All the participants reported that the seams were negligible and had minor impacts on their behaviors. However, using multiple projectors still caused some technical problems such as slight uneven brightness across projected screens. This uneven brightness might negatively affect large display users’ performance in long-term usage. However, participants’ comments revealed that the effects on studies presented in this thesis were negligible. All the participants in experiments presented in chapter 4 commented that uneven brightness was too slight to be noticed. In the diary study in chapter 3, seven out of eight participants reported that the uneven brightness had very minor effects on their performance or behaviors throughout the entire studies. One participant commented that the uneven brightness was negligible most of the time except that it caused some eye-fatigue on the last day of usage.

All the participants in the longitudinal dairy study reported in chapter 3 were experienced computer users, with each of them using computer over 5 hours per day. As such, it might be difficult to generalize the results to inexperienced persons. However, we chose experienced computer users as we wanted to see how their interaction and visualization strategies might change when they moved to a single large high-resolution display; in contrast, inexperienced 126

users would not have baselines of strategies that they would need to adapt when moving to the large display setup.

To investigate display size and pixel density effects on desktop work, we used slide making, online map navigation, and proofreading to represent scattered-information processing, multi- scale navigation, and single-window text editing, respectively (chapter 4). While we carefully designed these tasks to capture the essences of common desktop activities, they obviously do not reflect every aspect of daily information processing work. However, we believe the results shed some light as to the relative trade-offs between size and pixel density for a reasonable proportion of usage scenarios, and is one step towards gaining a deeper understanding of how these display properties might affect user performance.

As an initial investigation into internal bezel effects in chapter 5, we did not vary steering/selection task parameters, such as tunnel shape/width/length, and target size/selection distance, as these would have overly complicated these studies. It might be worthwhile in the follow-up work to conduct full Fitts’ law (Fitts, 1954) or steering law experiments on tiled- monitor large displays to gain an even deeper understanding of the effect of interior bezels (Accot & Zhai, 1997).

The number of participants which were used should also be discussed. All of our studies used 8- 16 participants. While this is fairly typical for studies in the Human-Computer Interaction literature, it is a relatively low number of participants. The decision to use a lower number of participants was mostly due to available resources. However, we feel that this limitation is minor, since our analysis of these studies did produce statistically significant results.

7.3 Future Directions

Based on the study presented in this dissertation, we believe there are opportunities to perform additional studies, create additional techniques, and explore new display usages.

Multi-Channel Input on a Large Display

This dissertation is focused on the traditional mouse and keyboard based interaction paradigm, because these two devices are the de facto the standard input devices for desktop work and most

127

of current software interfaces are designed them. However, challenges arise when users interact on a large display with these two devices. For example, the input bandwidth provided by a single cursor might be too narrow to process a rich amount of digital information, and frequently moving the cursor over a long distance on the large display may cause hand fatigue. As various new input methods are being explored, such as multi-touch surfaces and eye-tracking devices, incorporating them into a desktop environment might alleviate the emerging interaction challenges. For example, bringing multi-touch surfaces into desktop work might ease the window management tasks. Manipulating windows with multiple fingers on a multi-touch surface improves the input bandwidth over a singer cursor-based interaction paradigm. By tracking users’ eye movements, we could roughly acquire the current focused region on a large display. This information can be used to quickly shift cursor positions, or change the display visualization style (e.g., highlight the focused region). The mouse and keyboard have existed for decades and will continue to be used as major input devices for desktop work. However, other extra input channels could serve as useful supplements. Integrating these extra input channels with the traditional mouse and keyboard-based interaction paradigm could be an interesting research direction to pursue.

Multi-Device Desktop Environment

As computing technology is becoming ubiquitous, many information workers own multiple computing devices with different display sizes and computing capabilities, such as laptops, desktops, and PDAs. While this dissertation is focused on exclusively using a large display for desktop work, it would be interesting to explore using a large display to support computing activities across multiple computing devices. For example, a large display in a personal office could serve as a visualization hub connecting multiple computing devices for a single person. Office workers can use a large display to share digital information among multiple computing devices, and to synchronize work across different devices. As a large display can visualize a large amount of digital information, it might ease the information share and synchronization process.

The research methodology in this dissertation could be adopted to carry out such research. For example, we could firstly carry out empirical studies to observe how users process digital

128

information across multiple devices. Based on the study results, we could then design appropriate software infrastructure and interaction techniques to better support cross-device computing activities.

Using a Large Display at Home.

Large displays are now widely used in various scenarios. This dissertation is focused on a personal office scenario: using a large display to process desktop work. A possible future research direction is to explore using a large display at home. As opposed to a personal office where a large display is used mostly for daily office work, a large display at home is multi- functional: it serves as an entertainment platform when people need relaxing; it is also the primary working environment when users work at home. In addition, as there are usually multiple family members at home, a large display should be able to accommodate multiple users. As large displays are becoming increasingly available and affordable, it would be interesting to research how to use them to support home activities.

129

References

1. Accot, J. and Zhai, S. (1997) Beyond Fitts' law: models for trajectory-based HCI tasks. ACM CHI Conference on Human Factors in Computing Systems, 295-302.

2. Agarawala, A. and Balakrishnan, R. (2006) Keepin' it real: pushing the desktop metaphor with physics, piles and the pen. ACM CHI Conference on Human Factors in Computing Systems, 1283-1292.

3. Andrews, C., Endert, A. and North, c. (2010). Space to think: large high-resolution displays for sensemaking. ACM CHI Conference on Human Factors in Computing Systems, 55-64.

4. Bae, S., Kobayash, T., Kijima, R., and Kim, W. (2004). Tangible NURBS-curve manipulation techniques using graspable handles on a large display. ACM UIST Symposium on User Interface Software and Technology, 81-90.

5. Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G. and Buxton, W. (1999) Digital tape drawing. ACM UIST Symposium on User Interface Software and Technology, 161–169.

6. Ball, R. and North, C. (2005) An analysis of user behavior on high-resolution tiled displays. IFIP International Conference on Human-Computer Interaction (INTERACT), 350–364.

7. Ball, R., North, C. and Bowman, D.A. (2007) Move to improve: promoting physical navigation to increase user performance with large displays. ACM CHI Conference on Human Factors in Computing Systems, 191–200.

8. Ballagas, R., Rohs, M. and Sheridan, G. J. (2005). Sweep and point and shoot: phonecam-based interactions for large public displays. ACM CHI Conference on Human Factors in Computing Systems (Extend Abstract), 1200-1203.

130

9. Bardram, J.E., Hansen, T.R. and Soegaard, M.. (2006) Awaremedia: a shared interactive display supporting social, temporal, and spatial awareness in surgery. ACM Conference on Computer Supported Cooperative Work, 109–118.

10. Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. (2003) Drag-and-pop and drag-and-pick: Techniques for accessing remote screen content on touch- and pen-operated systems. IFIP International Conference on Human-Computer Interaction (INTERACT), 57–64.

11. Baudisch, P., Cutrell, E., and Robertson, G. (2003) High-density cursor: A visualization technique that helps users keep track of fast-moving mouse cursors. IFIP International Conference on Human-Computer Interaction (INTERACT), 236–243.

12. Baudisch, P., Cutrell, E., Hinckley, K., and Gruen, R. (2004) Mouse ether: accelerating the acquisition of targets across multi-monitor displays. ACM CHI Conference on Human Factors in Computing Systems, 1379–1382.

13. Baudisch, P., Sinclair, M., and Wilson, A. (2006) Soap: a pointing device that works in midair. ACM UIST Symposium on User Interface Software and Technology, 43–46.

14. Baudisch, P., Tan, D., Collomb, M., Robbins, D., Hinckley, K., Agrawala, M., Zhao, S. and Ramos, R. (2006) Phosphor: explaining transitions in the user interface using afterglow effects. ACM UIST Symposium on User Interface Software and Technology, 169–178.

15. Beaudouin-Lafon, M. (2001) Novel interaction techniques for overlapping windows. ACM UIST Symposium on User Interface Software and Technology, 153-154.

16. Bezerianos, A. and Balakrishnan, R. (2005) The vacuum: facilitating the manipulation of distant objects. ACM CHI Conference on Human Factors in Computing Systems, 361– 370.

17. Bezerianos, A. and Balakrishnan, R. (2004) Interaction and visualization techniques for very large scale high resolution displays. DGP-TR-2004- 002, University of Toronto.

131

18. Bezerianos, A. and Balakrishnan, R. (2005) Canvas portals: View and space management on large displays. Special issue of IEEE Computer Graphics and Applications, 25(4): 34–43.

19. Bezerianos, A., Dragicevic, P. and Balakrishnan, R. (2006) Mnemonic rendering: an image-based approach for exposing hidden changes in dynamic displays. ACM UIST Symposium on User Interface Software and Technology, 159–168.

20. Bi, X. and Balakrishnan, R. (2009) Comparing usage of a large high-resolution display to single or dual desktop displays for daily work. ACM CHI Conference on Human Factors in Computing Systems, 1005-1014.

21. Birnholtz, J. P., Grossman, T., Mak, C. and Balakrishnan, R. (2007) An exploratory study of input configuration and group process in a negotiation task using a large display. ACM CHI Conference on Human Factors in Computing Systems, 91-100.

22. Bishop, G., and Welch, G. (2000). Working in the office of "real soon now". IEEE Computer Graphics and Applications, 20(4 ). 76-78

23. Bolt, R. A. put-that-there: Voice and gesture at the graphics interface (1980). SIGGRAPH ’80: Proceedings of the 7th annual conference on Computer graphics and interactive techniques, 262–270.

24. Bondarenko, O. and Janssen, R. (2005) Documents at Hand: Learning from Paper to Improve Digital Technologies. ACM CHI Conference on Human Factors in Computing Systems, 121-130.

25. Bowman D.A. and Hodges L.F. (1997) An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. Proceedings of the 1997 symposium on Interactive 3D graphics, 35–38.

26. Flesch, R. (1948). A New Readability Yardstick. Jour-nal of Applied Psychology, 32, 221-233.

132

27. Fridgeman, B., Lennon, M. L., and Jackenthal, A. (2003) Effect of Screen Size, Screen Resolution, and Display Rate on Computer-Based Test Performance. Applied Measurement in Education, Volume 16, Issue 3, 191-205

28. Barreau, D. and Nard, B. (1995) Finding and reminding: file organization from the desktop. SIGCHI Bull. 27, 3, 39-43.

29. Boring, S., Baur, D., Butz, A., Gustafson, S. and Baudisch, P. (2010) Touch projector: mobile interaction through video. ACM CHI Conference on Human Factors in Computing Systems, 2287-2296.

30. Brignall, H. and Rogers, R. (2003) Enticing people to interact with large public displays in public space. IFIP International Conference on Human-Computer Interaction (INTERACT), 17–24.

31. Brujin, D. D., Mul, S. D., and Oostendorp, H. V. (1992) The influence of screen size and text layout on the study of text. Behaviour & Information Technology, vol 11, No.2, 71- 78

32. Buxton, W. (1997) Living in augmented reality: Ubiquitous media and reactive environments. In K. Finn, A. Sellen, and S. Wilber, editors, Video Mediated Communication, pages 363–384. Hillsdale, N.J.: Erlbaum.

33. Buxton, W., Fitzmaurice, G., Balakrishnan, R., and Gordon Kurtenbach. (2000) Large displays in automotive design. IEEE Computer Graphics and Applications, 20 (4):68–75.

34. Cao, X. and Balakrishnan, R. (2006) Interacting with dynamically defined information spaces using a handheld projector and a pen. ACM UIST Symposium on User Interface Software and Technology, 225–234.

35. Cao, X. and Balakrishnan, R. (2003) Visionwand: Interaction techniques for large displays using a passive wand tracked in 3d. ACM UIST Symposium on User Interface Software and Technology, 193-202.

133

36. Chapuis, O. and Roussel, N. (2005) Metisse is not a 3D desktop! ACM UIST Symposium on User Interface Software and Technology, 13-22.

37. Churchill, E., Nelson, L., Denoue, L. and Girgensohn, A. (2003) The plasma poster network: Posting multimedia content in public places. IFIP International Conference on Human-Computer Interaction (INTERACT), 599-606.

38. Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V., and Hart, J. C. (1992). The CAVE: audio visual experience automatic virtual environment. Commun. ACM 35, 6 (Jun. 1992), 64-72.

39. Czerwinski, M., Horvitz, E., and Wilhite, S. (2004) A diary study of task switching and interruptions. ACM CHI Conference on Human Factors in Computing Systems. 175-182.

40. Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G. and Starkweather, G. Toward characterizing the productivity benefits of very large displays. IFIP International Conference on Human-Computer Interaction (INTERACT), 9–16

41. Dearman, D. and Truong, K.N. (2009) BlueTone: A Framework for Interacting with Public Displays Using Dual-Tone Multi-Frequency through Bluetooth. International Conference on Ubiquitous Computing, 97-100.

42. Dragicevic, P. (2004) Combining crossing-based and paper-based interaction paradigms for dragging and dropping between overlapping windows. ACM UIST Symposium on User Interface Software and Technology, 193-196

43. Electrosonic. http://www.electrosonic.com/command profiles.asp

44. Elrod, S., Bruce, R., Gold, R., Goldberg, D., Halasz, F., Janssen, W., Lee, D., McCall, K., Pedersen, d., Pier, K., Tang, J., and Welch, G. (1992) Liveboard: A large interactive display supporting group meetings. ACM CHI Conference on Human Factors in Computing Systems, 599–607.

134

45. Fass, F., Forlizzi, J., and Pausch, R. (2002) Messydesk and messyboard: two designs inspired by the goal of improving human memory. DIS ’02: Proceedings of the conference on Designing interactive systems, 303–311

46. Fitzmaurice, G., Khan, A., Kurtenbach, G., and Binks, G. (2005) Cinematic meeting facilities using large displays. IEEE Computer Graphics and Applications, 25(4):17–21.

47. Fitzmaurice, G., Ishii, H., and Buxton, W. (1995) Bricks: laying the foundations for graspable user interfaces. ACM CHI Conference on Human Factors in Computing Systems, 442-449

48. Fitts, P., (1954) The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 381-391.

49. Forlines, C., and Balakrishnan, R. (2009) Improving visual search with image segmentation. ACM CHI Conference on Human Factors in Computing Systems, 1093- 1102.

50. Forlines, C., Vogel, D., and Balakrishnan, R. (2006) Hybridpointing: fluid switching between absolute and relative pointing with a direct input device. ACM UIST Symposium on User Interface Software and Technology, 211–220

51. Freeman, E. and Fertig, S. (1995) Lifestreams: Organizing your electronic life. In AAAI Fall Symposium: AI Applications in Knowledge Navigation and Retrieval, 38-44.

52. Gantz, J. F., Chute, C., Manfrediz, A., Minton, S., Reinsel, D., Schlichting, W., and Toncheva, A. (2008). The Diverse and Exploding Digital Universe: An Updated Forecast of Worldwide Information Growth Through 2011. An IDC White Paper

53. Geißler, J. (1998) Shuffle, throw or take it! working efficiently with an interactive wall. ACM CHI Conference on Human Factors in Computing Systems (Extended Abstracts), 265-266.

135

54. Greenberg, S. and Rounding, M. (2001) The notification collage: posting information to public and personal displays. ACM CHI Conference on Human Factors in Computing Systems, 514–521.

55. Grossman, T. and Balakrishnan, R. (2005) The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor’s activation area. ACM CHI Conference on Human Factors in Computing Systems, 281–290.

56. Grudin, J. (2001) Partitioning digital worlds: focal and peripheral awareness in multiple monitor use. ACM CHI Conference on Human Factors in Computing Systems, 458–465.

57. Guiard, Y., Blanch, R. and Beaudouin-Lafon, M. (2004) Object pointing: a complement to bitmap pointing in guis. Graphics Interface, 9–16.

58. Guimbreti`ere, F., Stone, M., and Winograd, T. (2001) Fluid interaction with high- resolution wall-size displays. ACM UIST Symposium on User Interface Software and Technology, 21–30.

59. Han, J. (2005) Low-cost multi-touch sensing through frustrated total internal reflection. ACM UIST Symposium on User Interface Software and Technology, 115–118.

60. Hara, K. and Sellen, A. (1997) A comparison of reading paper and on-line documents. ACM CHI Conference on Human Factors in Computing Systems, 335-342.

61. Heath, C. and Luff, P. (1996) Documents and professional practice: “bad” organisational reasons for “good” clinical records. ACM conference on Computer supported cooperative work 354-363.

62. Henderson, A. Jr. and Card, S. (1986) Rooms: the use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface. ACM Trans. Graph. 5, 3, 211-243.

63. Hibbs, M. A., Dirksen, N. C., Li, K. and Troyanskaya, O. G. (2005) Visualization methods for statistical analysis of microarray clusters. BMC Bioinformatics, 6(115).

136

64. Hinckley, K., Pausch, R., Goble, J. C., and Kassell, N. F. (1994) A survey of design issues in spatial input. ACM UIST Symposium on User Interface Software and Technology, 213–222.

65. Hoffmann, R., Baudisch, P., and Weld, D. (2008) Evaluating visual cues for window switching on large screens. ACM CHI Conference on Human Factors in Computing Systems, 929-938.

66. Huang, E.M., Koster, A. and Borchers, J. (2008) Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at Public Displays?. International Conference on Pervasive Computing, 228-243.

67. Huang, E.M., and Mynatt, E.D. (2003) Semi-public displays for small, co-located groups. ACM CHI Conference on Human Factors in Computing Systems, 49–56.

68. Hutchings, D., Smith, G., Meyers, B., Czerwinski, M., and Robertson, G. (2004). Display space usage and window management operation comparisons between single monitor and multiple monitor users. Advanced Visual Interfaces, 32-39.

69. Hutchings, D. and Stasko, J., (2005) Mudibo: multiple dialog boxes for multiple monitors. ACM CHI Extended Abstracts, 1471-1474

70. Hutchings, D. and Stasko, J. (2007) Quantifying the performance effect of window snipping in multiple-monitor environments. IFIP International Conference on Human- Computer Interaction (INTERACT), 461-474.

71. Hyperwall http://www.nas.nasa.gov/Groups/VisTech/hyperwall/

72. Izadi, S., Brignull, H., Rodden, T., Rogers, Y. and Underwood, M. (2003) Dynamo: a public interactive surface supporting the cooperative sharing and exchange of media. ACM UIST Symposium on User Interface Software and Technology, 159–168

73. Jiang, H., Ofek, E., Moraveji, N. and Shi, Y. (2006) Direct pointer: direct manipulation for large-display interaction using handheld cameras. ACM CHI Conference on Human Factors in Computing Systems,1107–1110. 137

74. Johnson, W., Jellinek, H., Klotz, L. Rao, R., and Card, S. (1993) Bridging the paper and electronic worlds: the paper user interface. ACM CHI Conference on Human Factors in Computing Systems, 507-512.

75. Johnson, A., Leigh, J., Morin, P. and Van Keken, P. (2006) Geowall: Stereoscopic visualization for geo science research and education. IEEE Computer Graphics and Applications, 26 (6):10–14

76. Kendon, A. (2004) Gesture: visible action as utterance. Cambridge University Press.

77. Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N. and Kurtenbach, G. (2004) A remote control interface for large displays. ACM UIST Symposium on User Interface Software and Technology, 127–136

78. Kurtenbach, G. (1993) The Design and Evaluation of Marking Menus. Ph.D. Thesis, University of Toronto.

79. Luff, P., Heath, C. and Greatbatch, D. (1992) Tasks-in-interaction: paper and screen based documentation in collaborative activity. ACM conference on Computer-supported cooperative work, 163-170.

80. Lund, A. M. (1997) The influence of video image size and resolution on viewing- distance preferences. Society of Motion Picture and Television Engineers Journal, 102:406–415.

81. MacIntyre, B., Mynatt, E. D., Voida, S., Hansen, K. M., Tullio, J. and Corso, G. M. Support for multitasking and background awareness using interactive peripheral displays. ACM UIST Symposium on User Interface Software and Technology, 41–50

82. Mackay, W. and Pagani, D. (1994) Video mosaic: laying out time in a physical space. In ACM international conference on Multimedia (MULTIMEDIA '94), 165–172

83. MacKenzie, I. S. and Jusoh, S. (2001) An evaluation of two input devices for remote pointing. IFIP Working Conf. on Engineering for HCI - EHCI 235–249. S

138

84. Maglio, P. P. and Campbell, C. S. (2000) Tradeoffs in displaying peripheral information. ACM CHI Conference on Human Factors in Computing Systems, 241–248.

85. Malik, S., Ranjan, A. and Balakrishnan, R. (2005) Interacting with large displays from a distance with vision-tracked multi-finger gestural input. ACM UIST Symposium on User Interface Software and Technology 43–52.

86. Malone, T. (1983) How do people organize their desks? : Implications for the design of office information systems. ACM Trans. Inf. Syst. 1, 1 99–112.

87. Mander, R., Salomon, G. and Wong, Y. Y. (1992) A “pile” metaphor for supporting casual organization of information. ACM CHI Conference on Human Factors in Computing Systems, 627-634.

88. Matthews, T., Czerwinski, M., Robertson, G. and Tan, D. (2006) Clipping lists and change borders: improving multitasking efficiency with peripheral information design. ACM CHI Conference on Human Factors in Computing Systems, 989–998.

89. McCarthy, J. F., Costa, T. J. and Liongosari, E. S. (2001) UniCast, OutCast & GroupCast: Three Steps Toward Ubiquitous, Peripheral Displays. Ubicomp 332-345.

90. McCarthy, J. F., Nguyen, D., Rashid, A., and Soroczak, S. (2002) Proactive Displays and The Experience UbiComp Project. SIGGROUP Bull. 23, 3 (December 2002), 38-41

91. Miyao, M., Hacisalhzade, S., Allen, J. and Start, L. (1989) Effects of VDT resolution on visual fatigue and readability: an eye movement approach. Ergonomics 32, 603-614.

92. Myers, B. A., Bhatnagar, R., Nichols, J., Peck, C. H., Kong, D., Miller, R. and Long, A. C. (2002) Interacting at a distance: measuring the performance of laser pointers and other devices. ACM CHI Conference on Human Factors in Computing Systems, 33–40.

93. Mynatt, E., Igarashi, T., Edwards, W. and LaMarca, A. (1999) Flatland: New dimensions in office whiteboards. ACM CHI Conference on Human Factors in Computing Systems, 346–353.

139

94. Newman, W. and Wellner, P. (1992) A desk supporting computer-based interaction with paper documents. ACM CHI Conference on Human Factors in Computing Systems, 587– 592.

95. Olsen, D.R. Jr., and Nielsen, T. (2001) Laser pointer interaction. ACM CHI Conference on Human Factors in Computing Systems, 17–22.

96. Pedersen, E. R., McCall, K., Moran, T. P. and Halasz, F. G. (1993) Tivoli: an electronic whiteboard for informal workgroup meetings. ACM CHI Conference on Human Factors in Computing Systems, 391–398.

97. Peltonen, P., Kurvinen, E., Salovaara, A., Jacucci,G., Ilmonen, T., Evans, J., Oulasvirta, A. and Saarikko, P. (2008) It's Mine, Don't Touch!: interactions at a large multi-touch display in a city centre. ACM CHI Conference on Human Factors in Computing Systems, 1285-1294

98. Pfeiffer Report, (2005) The 30-inch Apple Cinema HD Display Productivity Benchmark

99. Pierce, J. S., and Pausch, R. (2002) Comparing voodoo dolls and homer: exploring the importance of feedback in virtual environments. ACM CHI Conference on Human Factors in Computing Systems, 105–112.

100.Poupyrev, I., Billinghurst, M., Weghorst, S. and Ichikawa, T. (1996) The go-go interaction technique: non-linear mapping for direct manipulation in vr. ACM UIST Symposium on User Interface Software and Technology, 79–80.

101.Prante, T., Rocker, C., Streitz, N., Stenzel, R., Magerkurth, C., Alphen, D.v. and Plewe, D. A. (2003) Hello Wall – Beyond Ambient Displays. Video Track and Adjunct International Conference on Ubiquitous Computing, 277-278

102.Ramos, G., Robertson, G., Czerwinski, M., Tan, D., Baudisch, P., Hinckley, K., and Agrawala, M. (2006) Tumble! Splat! helping users access and manipulate occluded content in 2D drawings. Advanced Visual Interface, 428-435.

140

103.Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L. and Fuchs, H. (1998) The office of the future: a unified approach to image-based modeling and spatially immersive. ACM SIGGRAPH. 179-188.

104.Redström, J., Skog, T., and Hallnäs, L. (2000) Informative art: using amplified artworks as information displays. Proceedings of DARE 2000 on Designing Augmented Reality Environments, (Elsinore, Denmark). 103-114.

105.Rekimoto, J. (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. ACM UIST Symposium on User Interface Software and Technology, 31–39.

106.Rekimoto, J. (1999) Time-machine computing: a time-centric approach for the information environment. ACM UIST Symposium on User Interface Software and Technology, 45–54.

107.Renaud, K. (2000) Expediting rapid recovery from interruptions by providing a visualization of application activity. OZCHI 2000, 348–355.

108.Rensink, R.A., (2002) Change detection. Annual Review of Psychology, 53:245–277

109.Robertson, G., Horvitz, W., Czerwinski, M., Baudisch, P., Hutchings, D.R., Meyers, B., Robbins, d., and Smith G., (2004) Scalable fabric: flexible task management. Advanced Visual Interfaces, 85–89.

110.Robertson, G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., and Tan, D. (2005) The large-display user experience. IEEE Computer Graphics and Applications, 25(4): 44–51.

111.Russell, D.M., and Gossweiler, R. (2001) On the design of personal & communal large information scale appliances. International Conference on Ubiquitous Computing, 354– 361.

141

112.Russell, D.M., Drews, C. and Sue, A. (2002) Social aspects of using large public interactive displays for collaboration. International Conference on Ubiquitous Computing, 229–236.

113.Sandin, D.J., Margolis, T., Ge, J., Girado, J., Peterka, T. and DeFanti, T. A. (2005) The VarrierTM autostereoscopic virtual reality display. ACM Trans. Graph. 24, 3, 894-903

114.Sellen, A. and Harper, R. (1997) Paper as an analytic resource for the design of new technologies. ACM CHI Conference on Human Factors in Computing Systems 319-326.

115.Sellen, A., and Harper, R. (2003) The Myth of the Paperless Office. MIT Press.

116. Semeraro, D., Wilhelmson, B., Bramer, D., Leigh, J., Porter, D., and Ahrens, J. (2004) Collaboration, analysis, and visualization of the future. Meteorological Data 20th International Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology.

117.SHARP LCDs, http://www.sharp-world.com/index.html

118.Shupp, L., Andrews, C., Dickey-Kurdziolek, M., Yost, B. and North, C. (2009) "Shaping the Display of the Future: The Effects of Display Size and Curvature on User Performance and Insights", Human–Computer Interaction, vol. 24, issue 1, 230-272.

119.Simmons, T. (2001) “What's the optimum computer display size?” Ergonomics in Design Vol. Fall 19 – 25.

120.Skog, T., Holmquist, L.E., Redstrom, J. and Hallnas, L. (2001) Informative Art. In SIGGRAPH 2001 Conference Abstracts and Applications (Emerging Technoloies exhibitions).

121.Skog, T. (2004) Activity Wallpaper: Ambient Visualization of Activity Information. DIS (Poster)

122.SmartTechnologies. Smart technologies inc. digital vision touch technology http://www.smarttech.com/dvit/.

142

123.Smith, G., Baudisch, P., Robertson, G., Czerwinski, M., Meyers, B., Robbins, D., Andrews, D. (2003) GroupBar: The TaskBar evolved. OZCHI, 34-43

124.Sommerich, C.M., Joines, S.M.B. and Psihogios, J.P. (1998) Effect of VDT viewing angle on user biomechanics, comfort, and preference. Human Factors and Ergonomics Society 42nd Annual Meeting, 861-865.

125.Stasko, J., Miller,T., Plaue, C., Pousman, Z. and Ullan, O. (2004) Personalized Peripheral Information Awareness through Information Art. International Conference on Ubiquitous Computing, 18-35

126.Stoakley, R., Conway, M.J. and Pausch, R. (1995) Virtual reality on a wim: interactive worlds in miniature. ACM CHI Conference on Human Factors in Computing Systems, 265–272.

127.Streitz, N., Geißler, J., Holmer, T., Konomi, S., M¨uller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P. and Steinmetz R. (1999) i-land: an interactive landscape for creativity and innovation. ACM CHI Conference on Human Factors in Computing Systems, 120–127.

128.Swaminathan K. and Sato, S. (1997) Interaction design for large displays. IFIP International Conference on Human-Computer Interaction (INTERACT) 4(1): 15–24,

129.Tan, D. and Czerwinski, M. (2003) Effects of visual separation and physical discontinuities when distribut-ing information across multiple displays, OZCHI 2003, 184-191.

130.Tan, D., Czerwinski, M. and Robertson, G. (2003). Women go with the (optical) flow. ACM CHI Conference on Human Factors in Computing Systems 209-215.

131.Tan, D., Gergle, D., Scupelli, P. and Pausch, R. (2004) Physically large displays improve path integration in 3d virtual navigation tasks. ACM CHI Conference on Human Factors in Computing Systems, 439–446.

143

132.Tan, D., Meyers, B. and Czerwinski, M. (2004) Wincuts: manipulating arbitrary window regions for more effective use of screen space. ACM CHI Conference on Human Factors in Computing Systems, 1525–1528.

133.Tan, D., Gergle, D., Scupelli, P. and Pausch, R. With similar visual angles, larger displays improve spatial performance. ACM CHI Conference on Human Factors in Computing Systems, 217–224.

134.TOtal. http://www.city.chiba.jp/fire/english/systeme.html

135.Trimble, J., Wales, R. and Gossweiler, R. (2003) Nasas merboard: An interactive collaborative workplace platform. Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies, pages 18–44.

136.Vogel, D. and Balakrishnan, R. (2004) Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. ACM UIST Symposium on User Interface Software and Technology, 137–146.

137.Vogel, D., and Balakrishnan, R. (2005) Distant freehand pointing and clicking on very large, high resolution displays. ACM UIST Symposium on User Interface Software and Technology, 33–42.

138.Wei, B., Silva, C., Koutsofios, E., Krishnan, S. and S. North. (2000) Visualization research with large displays. IEEE Computer Graphics and Applications, 20(4):50–54.

139.Whittaker, S. and Hirschberg, J. (2001) The character, value, and management of personal paper archives. ACM Trans. Comput.-Hum. Interact. 8, 2, 150-170.

140. Whittaker, S., Swanson, J., Kucan, J., and Sidner, C. (1997) TeleNotes: Managing lightweight interactions in the desktop. ACM Transactions on Computer Human Interaction 4, 137–168.

141.Winograd, T. and Guimbreti`ere, F. (1999) Visual instruments for an interactive mural. ACM CHI Conference on Human Factors in Computing Systems (Extended Abstract), 234–235. 144

142.Wolfe, J. (1998). Visual search. In Pashler, H. (Ed.), Attention, Psychology Press.

143.Wu, M. and Balakrishnan, R. (2003) Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. ACM UIST Symposium on User Interface Software and Technology, 193–202.

144.Yost, B., Haciahmetoglu, Y. and North, C. (2007) Beyond visual acuity: the perceptual scalability of information visualizations for large displays. ACM CHI Conference on Human Factors in Computing Systems, 101-111.

145.Ziefle, M. (1998) Effects of display resolution on vis-ual performance. Human Factors 40 (4), 554-568

146.Zhai, S. (1998) User performance in relation to 3d input device design. SIGGRAPH Comput. Graph., 32(4):50–54.

145