The Design of Table-Centric Interactive Spaces by Daniel J. Wigdor a Thesis Submitted in Partial Conformity with the Requirement

The Design of Table-Centric Interactive Spaces

Daniel J. Wigdor

A thesis submitted in partial conformity with the requirements for the degree of Doctor of Philosophy

Graduate Department of Computer Science

University of Toronto

The Design of Table-Centric Interactive Spaces, by Daniel J. Wigdor

A thesis submitted in partial conformity with the requirements for the degree of Doctor of Philosophy

Graduate Department of Computer Science, University of Toronto

Direct-touch tabletops offer compelling uses as direct, multi-touch, multi-user displays for face to face collaborative work. As task complexity and group size increase, the addition to the tabletop of multiple, vertical displays allows for more information content, while reducing the need to multiplex the tabletop display area. We dub such systems table-centric interactive spaces .

Although compelling, these spaces offer unique challenges. In particular, the displays in such spaces are seen by the users at angles not typically found in combination in other environments. First, the viewing imagery shown on a horizontal display by seated participants means that that imagery is distorted, receding away from the users’ eyes. Second, the sharing of information by users sitting around a horizontal display necessitates that on-screen content be oriented at non- optimal angles for some subset of those users. Third, positioning vertical displays around the table means that some subset of the seated users will be looking at those displays at odd angles.

In this thesis, we investigate the challenges associated with these viewing angles. We begin with a examination of related work, including tabletop technology and interaction techniques. Next, we report the results of controlled experiments measuring performance of reading, graphical perception, and ancillary display control under the angles we identified. Next, we present a set of design issues encountered in our work with table-centric spaces. We then review a series of interaction techniques built to address those issues. These techniques are evaluated through the construction and validation of an application scenario.

Through these examinations, we hope to provide designers with insights as to how to enable users to take full advantage of ancillary displays, while maintaining the advantages and affordances of collocated table-centric work.

ii Acknowledgements

A great number of people contributed significantly to my research over the last 4 years. The direction of Professor Ravin Balakrishnan provided invaluable guidance in helping me to graduate from young student to an independent researcher. Professors Mark Chignell and Ron Baecker provided a great deal of support in transitioning my work from a body of individual publications to a single, unified research thread. Professor Rob Jacob of Tufts University provided the challenge to recognize the greater context of the work, and freed me to expand my vision. Thank you.

Research for my thesis was conducted at two labs: the Dynamic Graphics Project (DGP) lab at the University of Toronto (www.dgp.toronto.edu), and Mitsubishi Electric Research Labs (www.merl.com) in Cambridge, MA. Both of these labs are fabulous research environments, filled with keen minds with a thirst for ideas and knowledge. Many members of these institutions provided invaluable help and support. Two collaborators in particular have had a great influence on my research: Clifton Forlines, whose words make up more than a small part of this thesis from our many co-authored papers, taught me the value and importance of presentation and of thoroughly examining all details of a problem, as well as a great deal about how to find my voice in a choir. Tovi Grossman helped me to understand the importance of highly critical thinking, distilling ideas down to their basic research questions, and the value of the social side of the research community. I owe them both a great debt.

At MERL, John Barnwell was my friend and frequent guide on late night hardware building marathons. Our hacking efforts were aided by frequent support (both technical and personal) from Paul Dietz, Darren Leigh, Jonathan Westhues, and Bill (Crash) Yerazunis. Alan Esenther, Kathy Ryall, Sam Shipman, Chris Wren, Yuri Ivanov, and Paul Beardsley were highly valued collaborators on various projects. Just about every other member of MERL provided invaluable insights into some otherwise unseen avenue; my time at MERL was a wonderful and treasured experience.

iii In the DGP in Toronto, my research was aided at various points by Jonathan Deber, Michael McGuffin, and John Hancock, and by general discussion with the Interaction Research Group.

I owe a great debt to my supportive family: in Canada, this is Robin, Irene (who were both a great help with the text of the thesis), Adam, and Noel Wigdor. In Boston, this has been Amy MacLearn, Jill Hirschen, and various other characters I encountered while living at 86 Kirkland Street. I love you all.

Although she was not officially my supervisor, the majority of this work was conducted under the direct supervision of Dr. Chia Shen at MERL, and then at the Initiative in Innovative Computing at Harvard University. She served as my de facto advisor in the later years of my PhD, lending me her vision in areas of research, career development, professional conduct, and all manner of personal affairs. In appreciation of her maternal guidance, unfailing support, and loving friendship, I dedicate this thesis work to her.

iv Copyright Notices and Disclaimers

Several sections of this document have previously appeared in publications; in all cases, permission from the publisher has been granted for these works to appear in this thesis. Where appropriate, each chapter is marked with references to the relevant publication(s), each of which is listed below, along with the appropriate copyright notice or disclaimer for the given work.

Springer

© 2005 Springer No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Printed in the Netherlands. With kind permission of Springer Science and Business Media.

Chapter 3

Wigdor, D., Balakrishnan, R. (2005). Empirical investigation into the effect of orientation on text readability in tabletop displays. Proceedings of the 9 th European Conference on Computer-Supported Cooperative Work (ECSCW 2005) , 205-224 Information Processing Society of Japan

The copyright of this material is retained by the Information Processing Society of Japan (IPSJ). This material is published in this thesis with the agreement of the author(s) and the IPSJ. Please be complied with Copyright Law of Japan and the Code of Ethics of IPSJ if any users wish to reproduce, make derivative work, distribute or make available to the public any part or whole thereof. All Rights Reserved, Copyright © Information Processing Society of Japan. Comments are welcome. Mail to address [email protected], please.

Chapter 6

Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R., (2006). Table-Centric Interactive Spaces for Real-Time Collaboration: Solutions, Evaluation, and Application Scenarios. Proceedings of the 2006 conference on Collaborative Technologies (CollabTech 2006), 9-15.

v Association for Computing Machinery

ACM COPYRIGHT NOTICE. Copyright © 2005, 2006, 2007 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or [email protected]. © 2005, 2006, 2007 ACM, Inc. Included here by permission.

Chapter 4

Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R. (2007). Perception of Elementary Graphical Elements in Tabletop and Multi-Surface Environments. Proceedings of the 2007 SIGCHI conference on human factors in computing systems (CHI 2007), 473-482.

Chapter 5

Wigdor, D., Shen, C., Forlines, C., and Balakrishnan, R. (2006). Effects of display position and control space orientation on user preference and performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2006), 309-318.

Chapter 6

Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R., (2006). Table-Centric Interactive Spaces for Real-Time Collaboration. Proceedings of the 2006 conference on Advanced Visual Interfaces (AVI 2006), 103-107. ARDA and NGA

This study was partially supported by the Advanced Research and Development Activity (ARDA) and the National Geospatial-intelligence Agency (NGA) under Contract Number HM1582-05-C- 0028. The views, opinions, and findings contained in this report are those of the author(s) and should not be construed as an official Department of Defence position, policy, or decision, unless so designated by other official documentation.

Chapter 5

vi Video

A video (2:51) demonstrating the interaction techniques described in Chapter 6 can be found online. Given the impermanence of URL’s, it has been placed at multiple locations in the hopes that at least one will continue to be available:

Dynamic Graphics Project at the University of Toronto

http://www.dgp.toronto.edu/~dwigdor/nb/videos/wigdor_cscw_2006.wmv

University of Toronto

http://individual.utoronto.ca/dwigdor/dwigdor_TableCentricSpaces.wmv

Mitsubishi Electric Research Labs

http://www.diamondspace.merl.com/videos/2006_wigdor_WIM.wmv

Wigdor.com

http://www.wigdor.com/daniel/vid/wigdor_cscw_2006.wmv

YouTube

http://www.youtube.com/watch?v=HgBx_9F_Zw8

This video was published previously as part of the video proceedings of CSCW 2006 (Wigdor et al. 2006e).

vii Table of Contents

Acknowledgements ...... iii Copyright Notices and Disclaimers ...... v Video ...... vii List of Tables ...... xii List of Figures ...... xiii Chapter 1: Introduction ...... 1 1.1 Problem Space ...... 5 1.1.1 The Trouble with Angles ...... 5 1.1.1.1 Angle of Table Relative to the Eye ...... 5 1.1.1.2 Angle of Table Content Relative to the User ...... 6 1.1.1.3 Angle of Vertical Displays Relative to the User ...... 6 1.2 Contributions ...... 7 1.2.1 Human Performance Measurement ...... 7 1.2.2 Interaction Design...... 8 1.3 Thesis Roadmap ...... 8 1.3.1 Chapter 2: Background Literature ...... 9 1.3.2 Chapter 3: Reading ...... 9 1.3.3 Chapter 4: Perception of Elementary Graphical Elements ...... 9 1.3.4 Chapter 5: Input and Control ...... 10 1.3.5 Chapter 6: Interaction Techniques ...... 10 1.3.6 Chapter 7: Conclusions ...... 11 Chapter 2: Background Literature ...... 12 2.1 Direct-Touch Tabletops ...... 12 2.2 Hardware Implementations ...... 13 2.2.1 Vision Based Systems ...... 13 2.2.2 Capacitive Systems ...... 15 2.2.3 Frustrated Total Internal Reflection ...... 16 2.2.4 Object Tracking Technologies ...... 16 2.3 Extensions to the Direct-Touch Tabletop ...... 18 2.3.1 User-Differentiated Views ...... 18 2.3.2 Inverted and Two-Sided Touch Tables ...... 18 2.3.3 3D Tabletops ...... 19 2.4 Behavioural, Perceptual, and Physical Issues ...... 19 2.4.1 Applications and Long-Term Use of Tabletops ...... 19 2.4.2 Group Behaviours ...... 21 2.4.3 Orientation ...... 24 2.4.4 Reach ...... 25 2.5 Interaction Design ...... 26 2.5.1 Collocated Groupware GUI Design ...... 26 2.5.2 Tangibles on the Tabletop ...... 37 2.5.3 Gestural Interaction ...... 39 2.5.4 Mixed-Presence Groupware ...... 40 2.6 Interactive Spaces ...... 42 2.7 Multi-Terminal Roomware ...... 42 2.8 Multi-Surface Systems ...... 43 2.8.1 Pointing and Space Management ...... 43 2.8.2 Multi Surface Implementations ...... 44 2.8.3 Table-Centric Multi-Surface Environments ...... 45 viii Chapter 3: Reading ...... 48 3.1.1 Angle: Vertical Display to the User ...... 48 3.1.2 Angle: Table to the Eye ...... 49 3.1.3 Angle: Table Content to the User ...... 49 3.2 Terminology ...... 50 3.3 Related Work ...... 51 3.3.1 Orientation in Collaborative Groupware Research ...... 51 3.3.2 Human Ability to Read Text at Various Orientations ...... 52 3.4 Research Goals ...... 54 3.5 Experiment 1 ...... 55 3.5.1 Goals and Hypotheses ...... 55 3.5.2 Apparatus ...... 56 3.5.3 Participants ...... 57 3.5.4 Procedure ...... 57 3.5.5 Design ...... 58 3.5.5.1 Alternative Experimental Designs ...... 59 3.5.6 Results ...... 60 3.5.7 Discussion ...... 63 3.6 Experiment 2 ...... 64 3.6.1 Goals and Hypotheses ...... 64 3.6.2 Apparatus ...... 64 3.6.3 Participants ...... 64 3.6.4 Procedure ...... 65 3.6.5 Design ...... 67 3.6.6 Results ...... 67 3.6.7 Discussion ...... 68 3.7 Implications for design ...... 69 3.8 Avenues for Future Research ...... 70 Chapter 4: Perception of Elementary Graphical Elements ...... 72 4.1 Related Work ...... 72 4.1.1 Basic Visual Elements ...... 73 4.1.2 3D and Perception Under an Angle ...... 74 4.1.3 Magnitude Perception ...... 75 4.1.4 Distance Estimates and Virtual Environments ...... 76 4.2 Research Goals ...... 77 4.2.1 Terminology ...... 78 4.2.2 Angle: Vertical Display to the User ...... 79 4.2.3 Angle: Table to the Eye ...... 80 4.2.4 Angle: Table Content to the User ...... 81 4.3 Experiment 1: Single Display ...... 81 4.3.1 Goals and Hypotheses ...... 81 4.3.1.1 Visual Variables ...... 82 4.3.2 Apparatus ...... 83 4.3.3 Participants ...... 85 4.3.4 Procedure ...... 85 4.3.5 Design ...... 86 4.3.6 Results ...... 87 4.3.7 Discussion ...... 91 4.4 Experiment 2: Multi Display ...... 91 4.4.1 Goals and Hypotheses ...... 91 ix 4.4.2 Apparatus ...... 92 4.4.3 Participants ...... 93 4.4.4 Procedure ...... 93 4.4.5 Design ...... 93 4.4.6 Results ...... 94 4.4.7 Discussion ...... 95 4.5 Implications for Design ...... 95 4.5.1 Tabletop Visualization ...... 95 4.5.2 Mixed-Display Visualization ...... 98 4.6 Avenues for Future Research ...... 98 Chapter 5: Input and Control ...... 100 5.1.1 Angle of Table Relative to the Eye and of Table Content Relative to the User ...... 100 5.1.2 Angle of Vertical Displays Relative to the User ...... 101 5.1.3 Terminology ...... 102 5.1.3.1 Display Space ...... 102 5.1.3.2 Control Space ...... 103 5.1.3.3 Example ...... 103 5.2 Related work ...... 104 5.2.1 Collaborative Systems ...... 104 5.2.2 Effects of Control Space Orientation on Performance ...... 105 5.3 Research Goals ...... 106 5.4 Experiment 1: Adjustable Control Spaces ...... 106 5.4.1 Goals and Hypotheses ...... 106 5.4.2 Apparatus ...... 107 5.4.3 Participants ...... 108 5.4.4 Procedure ...... 108 5.4.5 Design ...... 110 5.4.6 Results ...... 111 5.4.6.1 Preferred Display Space Position ...... 111 5.4.6.2 Preferred Control Space Orientation per Display Position ...... 111 5.4.6.3 Effect of Display Space Position on Performance ...... 113 5.4.6.4 Relationship of User Preference to Performance ...... 114 5.4.7 Discussion ...... 114 5.5 Experiment 2: Fixed Control Spaces ...... 116 5.5.1 Goals and Hypotheses ...... 116 5.5.2 Apparatus ...... 116 5.5.3 Participants ...... 117 5.5.4 Task and Procedure ...... 117 5.5.5 Design ...... 117 5.5.6 Results ...... 117 5.5.7 Discussion ...... 119 5.6 Implications for Design ...... 120 5.7 Avenues for Future Research ...... 124 Chapter 6: Interaction Techniques ...... 126 6.1 Design Issues ...... 126 6.1.1 Non-Linear Alignment of Vertical Displays ...... 126 6.1.2 Awareness of Input-Output Mapping ...... 127 6.1.3 Contextual Association ...... 127 6.1.4 Direct-Touch Input Paradigm ...... 127 6.1.5 Multiple Control Orientations ...... 128 x 6.1.6 Multiple Granularities of Remote Interaction ...... 128 6.1.7 Support for Non-Interfering Concurrent Tasks ...... 128 6.2 Interaction Design ...... 128 6.2.1 Visual Connectivity between Displays ...... 128 6.2.2 Enhanced Contextual Associations ...... 129 6.2.3 World in Miniature: WIM ...... 131 6.2.4 Meta World in Miniature ...... 133 6.2.5 Multiple Control Granularities ...... 134 6.2.6 Workspace Pointer ...... 136 6.2.7 Mechanism for Moving (‘Teleporting’) Objects Between Displays ...... 137 6.3 Example Usage Scenario ...... 138 6.3.1 Description ...... 138 6.3.2 User Feedback ...... 141 6.4 Interaction Technique Discoverability ...... 142 6.4.1 Design and Procedure ...... 143 6.4.2 Participants ...... 145 6.4.3 Results and Discussion ...... 145 6.5 Implications for Design ...... 147 6.6 Avenues for Future Research ...... 147 Chapter 7: Conclusions ...... 149 7.1 Summary ...... 149 7.2 Additional Display Parameters ...... 152 7.2.1 Very Large Displays ...... 152 7.2.2 Table Size ...... 153 7.2.3 Additional Tables...... 153 7.2.4 Multiple Table-Centric Spaces ...... 154 7.2.5 3D Displays ...... 154 7.3 Avenues for Future Research ...... 154 7.3.1 Additional Physical Parameters ...... 154 7.3.2 Task Considerations ...... 155 7.3.3 Additional Interaction Mechanisms ...... 155 7.3.3.1 Non-Tabletop Input ...... 155 7.3.3.2 Alternative Table / Wall Couplings ...... 156 7.3.4 Extending from One to Many Users ...... 156 7.3.5 Application Building ...... 156 7.3.5.1 Sports Strategy ...... 157 7.3.5.2 Surveillance ...... 157 7.3.5.3 Research and Exploration Spaces ...... 157 7.4 Conclusions ...... 157 Bibliography ...... 159

xi List of Tables

Table 1-1. Classification of multi-display collocated environments augmented with one or more shared large- scale displays. The columns indicate the devices used to manipulate the large display(s) are, rows indicate the nature of the primary work area for the collaboration ...... 3 Table 1-2. Crossing the angular problems with primary interaction tasks provides an overview of the problems this thesis seeks to address. Individual cells identify whether this has been examined in previous work, a chapter in this thesis, or is not yet explored...... 8 Table 3-1. Summary of mean ( µ) and variance ( σ), of reading times, and percentage deviation from mean reading time for un-rotated text of the same type ...... 61 Table 4-1. error (mean, standard deviation) by up/down distance for each visual variable type, displayed on a tabletop...... 90 Table 4-2. error (mean, standard deviation) by up/down distance for each visual variable type, displayed on a tabletop...... 90 Table 4-3. error (mean, and standard deviation) for both display conditions for each visual variable ...... 94 Table 5-1. Preferred display space position for each participant ...... 111 Table 5-2. Mean control space orientation ( µ) and variance ( σ) across all participants for each display-space position ...... 112 Table 5-3. Average task completion time in msec for all participants for each display position, as well as how much slower (percentage) participants were to complete the task at that display position relative to the overall fastest one (NE) ...... 113 Table 5-4. Mean time for each control space orientation at each display space position, expressed as a percentage of the value of the optimal cell (0º,NE: 0.786 seconds) ...... 119 Table 5-5. Optimal control space orientation (relative to user 1) and performance penalty at that orientation (as a percentage of optimal, NE/0 º pairing) for a display positioned at each position relative to user 1 (rows) for the given user position(s) around a table combination, determined using data from Table 5-4...... 121 Table 5-6. Optimal control orientation and its penalty for that orientation for each user for a display at the given position. The display position is expressed relative to user 1, while the control orientation is expressed relative to the individual user ...... 122 Table 5-7. Worst-case performance penalty for each user / display position configuration. The display position is expressed relative to user 1 ...... 123 Table 6-1. Each feature tested in the study, and whether each user discovered that feature ...... 145

xii List of Figures

Figure 1-1. Left: Traditional meeting spaces have often had a table-centred focus. Right: more advanced spaces have added information displays on the surrounding walls (photo from Catford and Jenner 2001) ...... 1 Figure 1-2. Planning and control rooms often have digital terminals between users, or are oriented towards a large, shared display ...... 1 Figure 1-3. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Right: the same map, as it appears to the user of a tabletop display ...... 5 Figure 1-4. Left: the same map of Cambridge as above. Right: the tabletop as seen by another user seated at the same table ...... 6 Figure 1-5. Left: A vertical display as seen when angled optimally for a user. Right: the same vertical display as seen by another user at the same table ...... 6 Figure 2-1. Vision based systems make use of cameras to detect the presence of objects on the table. Figure from Wellner (1993) ...... 14 Figure 2-2. The ThinSight system. Left: IR emitters and detectors are laid out in an array beneath the LCD. Centre: the image detected by the device. Right: the ThinSight can be embedded within a thin LCD, such as a laptop. Images from Hodges et al. (2007)...... 14 Figure 2-3. A diagram of the DiamondTouch, which performs a capacitive coupling for each user. Figure from marketing material ...... 15 Figure 2-4. Left: Using Frustrated Total Internal Reflection to detect touch points. Right: the image as seen by the camera, before processing. Images from Han (2005) ...... 16 Figure 2-5. Left: the TViews sensor configuration. Right: the mobile ‘pucks’ are the only objects tracked by the system. Images from Mazalek et al. (2006) ...... 17 Figure 2-6. Left: the ActiveDesk used magnetically tracked bricks to perform two-handed manipulations. Right: the ActiveDesk in action. Images from Fitzmaurice et al. (1995) ...... 17 Figure 2-7. The Lumisight Table presents a different view to each user. Figure from Matsushita et al. (2004) ...... 18 Figure 2-8. Khandelwal’s “Teaching Table” provided pre-kindergarten students with a tangible interface. It was intended to act as a teaching tool for fundamental math skills ...... 20 Figure 2-9. Visual field with no target present (left), another with a gun (centre), and the same field with all items but the gun removed (right). From Forlines et al. (2006a) ...... 23 Figure 2-10. Groups performed a coordinated search of a maze, with each participant’s unique view presented either on a Lumisight table (left) or a vertical display (right) ...... 24 Figure 2-11. Left: Animators make use of a turntable to rotate their drawing surface without rotating their work area or handling the cell directly. Right: an artist’s finished product (e), and their pencil strokes relative to the table surface (f). From Fitzmaurice et al. (1999) ...... 25 Figure 2-12. Left: Vernier et al.’s circular tabletop interface. Right: a user gives input to the system using a stylus. Images from Vernier et al. (2002) ...... 27 Figure 2-13. Left: objects in the current flow around the space defined by that current. Right: a display bubble shows a portion of the interface, and avoids overlapping with a mobile phone. Images from Hinrichs et al. (2006) and Cotting and Gross (2006) ...... 28 Figure 2-14. The opportunistic browsing system built by Brujin and Spence Images from Brujin and Spence (2001), on the LIME table (Philips.com) ...... 28

xiii Figure 2-15. Two widgets for single-point high precision touch screen interaction. Left: Cross-keys, Right: Precision-Handle. Images from Albinsson and Zhai (2003) ...... 29 Figure 2-16. Left: The cursor is displayed between the middle finger and thumb. Right: the mouse-click is simulated when the index finger is touched to the display ...... 29 Figure 2-17. Left: tracking state is provided by pressing less of the finger to the display – increasing the finger area on the display performs a ‘click’. Right: the non-dominant hand is used to adjust various controls for the dominant hand’s manipulation of the cursor ...... 30 Figure 2-18. Selection of distant objects using the laser pointer- like interaction of the Tractor Beam . Image from Parker et al. (2005b) ...... 30 Figure 2-19. DTLens. Left: user defines zoom area with fingertips Right: zoom widget with additional controls. Figure from Forlines and Shen (2005) ...... 32 Figure 2-20. Left: individual CoR 2Ds can be used to perform commands. Right: grouped CoR 2Ds provide a mechanism for commands to be closer to a user ...... 36 Figure 2-21. The Video Draw system. Left: schematic. Right: system in use. Images from Tang and Minneman (1991)...... 40 Figure 2-22. The Clearboard system. Image from Ishii and Kobyashi (1993) ...... 41 Figure 2-23. The Collab installation at Xerox PARC ...... 42 Figure 2-24. Left: a multi-surface environment. Right: the same environment in ARIS. Image from Biehl, 2004 ...... 43 Figure 2-25. The Roomware system, including Comm Chairs and Dyna Wall. Images from Streitz et al. (1999) ...... 44 Figure 2-26. Left: the laptop is recognized, and the user has moved an image out on to the table. Right: the user has moved the image to the vertical display ...... 45 Figure 2-27. Left: the iRoom topology. Right: various iRooms built from their system architecture. Images from Johanson et al. (2002b)...... 46 Figure 2-28. Table-based control of ancillary displays. Left: world in miniature metaphor. Right: camera metaphor. Images from Wigdor et al. (2006b) and Forlines et al. (2006b) ...... 47 Figure 3-1. Text rotations. (a) No rotation applied. (b) Positive roll. (c) Positive pitch. (d) Positive yaw. Figure from Grossman et al. 2007 ...... 48 Figure 3-2. Degrees of roll rotation as referred to throughout this chapter. In all cases, angles are measured relative to the edge of the table at which the participant was seated ...... 50 Figure 3-3. Top-down diagrammatic view of the experimental apparatus ...... 56 Figure 3-4. Left: orienting crosshair primes the participant as to the location of text. Centre: after 0.65 seconds, the crosshair disappears and is replaced by the rotated text. Right: as the subject begins to type, the text disappears from the screen. The black arrow is for illustration only and was not present in the actual experiment ...... 57 Figure 3-5. Box plots of time, in seconds, required to read a single word at each position and orientation. Outliers (>1.5 * IQR) removed ...... 62 Figure 3-6. Box plots of time, in seconds, required to read a 5-6 word phrase at each position and orientation. Outliers (>1.5 * IQR) removed ...... 62 Figure 3-7. Box plots of time, in seconds, required to read a 6-digit number at each position and orientation. Outliers (>1.5 * IQR) removed ...... 63 Figure 3-8. Serial search task with all targets oriented at 0º ...... 65 Figure 3-9. Serial search task with targets oriented at -45º, 0º, and 45º ...... 65

xiv Figure 3-10. Serial search task with targets at all 8 orientations ...... 66 Figure 3-11. Left: Screen displaying search target to the participant. Right: the search field presented once the subject presses the “space” key, and timing starts for the search ...... 66 Figure 3-12. Box plot of time required for the serial search under each of the three target / distracter orientation conditions: 0: all oriented towards the user, 1: all oriented at -45 o, 0 o, or 45 o, 2: all oriented at one of the 8-compass positions. Outliers (>1.5 * IQR) removed ...... 68 Figure 4-1. Cleveland and McGill’s elementary perceptual tasks. All visual representations of quantitative information require decoding using one or more of these ...... 73 Figure 4-2. The relative position of imagery on-screen is measured by its left/right distance (left) and its up/down distance (right) ...... 78 Figure 4-3. The orientation of on-screen imagery – as demonstrated here by two on-screen lines – is either upright (left) or lateral (right) ...... 79 Figure 4-4. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Right: the same map, as it appears to the user of a tabletop display (repeat of Figure 1-3) ...... 80 Figure 4-5. Left: two horizontal lines of equal size as they appear to a user on a vertical display. Right: the same scene, as it appears to the user of a tabletop display ...... 80 Figure 4-6. Left: a map of Cambridge, as seen a tabletop by a user seated at the table. Right: the tabletop as seen by another user seated at the same table. (repeat of Figure 1-4) ...... 81 Figure 4-7. Left: two vertical lines of the same length as seen by a user seated at a tabletop. Right: lines of the same length presented laterally have less distortion ...... 81 Figure 4-8. The visual variables in our experiments; a subset Cleveland and McGill (1985) ...... 83 Figure 4-9. The apparatus as employed in our experiment. The user was seated in front of the display (left), which was precisely oriented at 4-different angles ...... 84 Figure 4-10. (Left). Experimental display in Cleveland and McGill (1985). (Right). Display used in the present experiment. Modulus locations (M), and stimulus locations (S) ...... 85 Figure 4-11. Examples of on-screen stimuli as presented to the participant. From left to right: position (upright), line length, and angle ...... 86 Figure 4-12. Mean size of error by up/down distance between objects for each display angle across all visual variable types...... 89 Figure 4-13. Error size by up/down distance between compared objects for each visual variable type on a tabletop display ...... 90 Figure 4-14. Display conditions for our second experiment. (Left) Vertical+vertical. (Right) Vertical+tabletop. The 3D position of the stimuli is fixed between conditions ...... 92 Figure 4-15. Values encoded in position (left) are less accurately perceived on a tabletop than those encoded in length (right) ...... 96 Figure 4-16. A traditional box plot (left), and an alternative (right): this box plot design replaces judgements of position with length for reading the interquartile range by connecting the ranges to the edges of graph with lines whose length will be judge ...... 97 Figure 5-1. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Centre: the same map, as it appears to the user of a tabletop display. Right: the same map, as it appears to another user of the same tabletop display ...... 100 Figure 5-2. Display space position: location of the screen relative to the position of the user and table where input is made. The “X” marks the centre of the chair where the user is seated, the rectangle above it is the table on which input is made ...... 102

xv Figure 5-3. Control space orientation: the rotation of the control space about the axis perpendicular to the table. Left: labels for the various orientations. Right: e.g., with a 135 c orientation, to move up on the display (top) the user must move back and to the left ...... 103 Figure 5-4. Example of user orienting his control space to roughly 135º with a display at S ...... 104 Figure 5-5. Control space and stylus input. Left: Position and orientation of the control space is controlled by a device held in the non-dominant hand. Right: Diagrammatic illustration of the relationship between the device (red) and control space: green rectangle represents the control area, the arrow indicates the “up” vector for the control space (arrow is shown here for illustration only, and was not displayed during the experiment) ...... 107 Figure 5-6. Display and input table. Left: plasma display on cart used to vary the display’s position. Right: table and chair used in the study. The 8 possible display space positions, equidistant to the centre of the chair, were marked on the floor with tape and the cart placed accordingly ...... 108 Figure 5-7. The docking task used in experiment 1: Participant touches the pen to the table, 2: crosses the blue object to select it, 3: drags blue object to the red dock, which then turns light-blue 4: when released the red dock moves to a new location ...... 109 Figure 5-8. The direction of each line indicates the mean control space orientation for a participant for a display space position. Each user’s particular orientation is rendered in the same colour in each display position ...... 112 Figure 5-9. The experimental apparatus for the second experiment. The physical template controlled only the position, and not the orientation (right), of the input space. The green-blue gradient was used to show the ‘up’ vector of the input area to the user...... 116 Figure 5-10. Mean task completion time at a given control space orientation encoded as the length of the line in that direction (longer the line the slower the performance). Display space position indicated by the position of the perpendicular line. Overlaid in red on each is the bounds of preferred orientations (longer lines) from experiment 1 ...... 118 Figure 5-11. An actual path (in black) followed by a participant using a spiral-strategy to dock the blue square onto the red square under a transformed control – display mapping. Three distinct spirals (S1-S3) are visible ...... 120 Figure 5-12. Left: the 4 canonical positions for users seated at a table. Middle: the 8 canonical control orientations for input on the table. Right: the 8 canonical positions for a vertical display around the table (partial repeat of Figure 5-3 for convenience) ...... 120 Figure 6-1. Schematic diagram of our system with screenshots overlaid: the matching colours and shapes of the repeating pattern on the vertical displays and associated proxies on the table allows precognitive connections ...... 129 Figure 6-2. Cords connecting related objects are routed between displays via the proxy objects Note the Cord connecting the website and its URL appears to ‘pass through’ the proxy...... 130 Figure 6-3. Bottom: the contents of the tabletop: three small WIM’s are shown on the edge closest to the ancillary displays they represent. The user has opened and expanded a larger WIM view of the green ancillary display. Top: the right ancillary display ...... 133 Figure 6-4. Left to Right: the content of a WIM as it zooms out to show a photograph of the room. The user can then zoom the WIM back in by touching a display in the photograph ...... 134 Figure 6-5. Top: screenshot of ancillary display, which remains static during a WIM zoom. Bottom: three stages of a zoom of the world in miniature (partial screenshot of table)...... 135 Figure 6-6. Each user’s point of contact on one or more WIMs is shown on ancillary displays, uniquely identified by color ...... 137 Figure 6-7. An object is moved (‘teleported’) from the tabletop to an ancillary display by dragging it onto a WIM. The orientation of the object is corrected once it is placed on the vertical display ...... 137

xvi Figure 6-8. The emergency management scenario environment: an interactive table augmented with two large displays: the video wall (left) and deployment wall (right). Figure 6-9, Figure 6-10, and Figure 6-11 show detailed views of each of the displays ...... 138 Figure 6-9. Real-time surveillance video is displayed on the video wall . The video feeds are augmented with geospatial information to aid with field situation assessment ...... 139 Figure 6-10. An application to allow the monitoring and deployment of special police forces is displayed on the “deployment wall” and controlled from the table ...... 140 Figure 6-11. The contents of the interactive touch-table, including police-forces unit information, special bulletins, and control areas for the other surfaces which make-up the system...... 140 Figure 6-12. The demonstration of our system to senior members of the RTCC team ...... 141 Figure 6-13. The New York Police Department’s Real Time Crime Center in New York City, USA features several workstations facing a shared 10’ x 26’ tiled display ...... 142

xvii Chapter 1: Introduction

There are many tasks and scenarios requiring members of a group to work collaboratively in a co-located setting. Traditional meeting spaces, such as the one shown in Figure 1-1 (left), have often centred around a conference or meeting table. In more complex or dedicated spaces, information displays can be added to the walls, such as the ones used to help plan the RAF fight in the Battle of Britain, Figure 1-1 (right).

With the advent of digital technology, many meeting spaces have lost their table-centred focus, in favour of various digital devices such as computer terminals and large displays. Many dedicated planning, control, and decision support rooms are now equipped with displays that inhibit direct face to face discussion, or which change the display area from a shared public space, such as those seen in Figure 1-1, to a private individual space, or spaces which require users to sit side by side, such as those shown in Figure 1-2.

Figure 1-2. Planning and control rooms often have digital terminals between users, or are oriented towards a large, shared display

1 Recent strides in technology have seen the advent of commercially available digital tabletops. As a result these spaces may once again become table-centred environments, without sacrificing the advantages of digital devices.

The focus of the present work has been the development of a dedicated meeting area, dubbed a table-centric interactive space . In such a space, users are seated around a digital touch table, collaboratively performing some shared task. Such an arrangement facilitates easy face to face interaction and collaboration, leveraging the advantages of traditional meeting tables. A table-centric focus comes at a cost, however: moving from one display per user to a shared display configuration necessarily reduces the number of pixels and display space available to any one user. To alleviate this, table-centric interactive spaces include one or more large vertical displays, visible to anyone seated at the table. These vertical displays can be used to display information feeds, provide additional interaction space, or facilitate different points of view of a shared data set.

Although advantageous, the introduction of these displays could present a danger to the face to face interaction facilitated by the touch table. They can draw user focus, especially if interacting with these displays requires a physical transition to a different location or input device in order to control them. To ensure that these displays enhance the interaction rather than distract from it, we have focused on providing mechanisms allowing users to interact with these displays directly from their seat at the tabletop, without changing physical location or input device. By maintaining the table-centricity of our space, we attempt to maintain all of the advantages of tabletop groupware, while simultaneously leveraging the advantages of additional, shared displays.

Although some of the earlier research we will explore examines environments in which tables and walls were both included, there are key differences between our problem space and that prior work. In particular, we can classify the previous work as varying along two dimensions: 1. whether the primary interaction area is a personal device or a shared table, and 2. whether the shared large-screen display(s) are controlled directly, or via the primary interaction area. From this classification (Table 1-1), it is apparent that our vision of a table-centric interactive space has yet to be explored.

2 Control-Mechanism of Large Display(s) On Large Display Off Large Display

#2 #1 Project Nick (Begeman 1986) Personal device i-Land (Streitz 1999) iRoom (Johanson 2002a) Primary i-Land (Streitz 1999) Interaction #3 Area i-Land (Streitz 1999) #4 Shared table MultiSpace The present work (Everitt 2006) Table 1-1. Classification of multi-display collocated environments augmented with one or more shared large-scale displays. The columns indicate the devices used to manipulate the large display(s) are, rows indicate the nature of the primary work area for the collaboration

Cell #1: systems devoted primarily to interaction with personal devices, with direct control of large displays. These systems require physical movement by the user to transition between devices. This movement might be as limited as turning the body, or as elaborate as requiring that the user get up and walk between displays. In the present work, we examine systems that do not require this movement. Further, while personal devices provide interesting affordances and advantages, we have focused our work on the design of systems which make use of a shared direct-touch tabletop as their primary interaction area. This has been done in order to leverage the advantages of a tabletop.

Cell #2: systems devoted primarily to interaction with personal devices, using those devices to control the large displays. These systems eliminate the need for a physical transition: the personal devices, such as terminals, provide full control over the large displays. The focus on personal devices is distinct from the present work, which instead focuses on a single, shared touch table as the primary interaction area.

Cell #3: systems devoted primarily to interaction on a touch table, with direct control of vertical displays. These systems also seek to leverage the advantages of direct-touch tables. Like cell # 1, however, these systems require physical transitions from the primary interaction area to the large displays. This transition can be as simple as changing input devices, or as elaborate as walking over to give direct-touch input to the vertical displays. In either, case, the table-centricity of the space is reduced.

3 The present work is dedicated to providing design recommendations for systems characterized by cell #4: systems which make use of direct-touch tables as their primary interaction area, and which provide mechanisms to control large display(s) from the table. In addition to its novelty as an unexplored area, we find this to be a compelling pairing of properties. The advantages of tabletop interaction are leveraged because the primary interaction area is the table, rather than a personal device. Users are able to easily transition between controlling the table and those displays because the primary interaction area, the table, is used to control the displays. These spaces maintain their table-centricity as control is exercised directly from the table. Because the primary interaction and control device is the tabletop, we will sometimes refer to the vertical displays making up the balance of the environment as ancillary displays .

The i-Land project, presented by Streitz et al. (1999), focused on multiple participants switching back-and-forth between collaborative and individual work, moving among and using the different displays in a distributed manner. To that end, they explored interaction mechanisms which would fall into all of cells 1-3. Our vision for a system occupying cell #4 is distinctly different from that of Streitz et al. We envision a control room with a single touch table providing a shared physical and virtual space for multiple collaborators. From that table, users are able to exert full control over large, vertical display(s). Although direct interaction with the ancillary displays might provide more flexibility, it is important that the full-range of actions that can be performed on these displays be supported from the table. This is desirable for several reasons. • all virtual elements are within physical reach of all participants • participants can remain comfortably seated • a consistent input and interaction paradigm is maintained throughout the system • advantages of table-centred interaction are leveraged Given the advantages and opportunities in our design space, and recent developments in interactive tabletop input technology, our goal is to explore the scenario where all interaction occurs solely on the tabletop. This allows multiple users to simultaneously interact - directly from the tabletop using multi point direct-touch input - with the full content of multiple surrounding displays.

4 1.1 Problem Space

Although the distinction between Streitz et al.’s work and our own may seem subtle, the goal of supporting interaction with multiple displays directly from the tabletop bears a high cost: designers must support the need to read, perceive graphical data, and exert control of all surfaces, all while users remain seated at the table. As a result, we do not have the luxury of assuming that users will simply walk between displays to get a better look, or utilize direct-touch to give input to remote displays. In this thesis, we will examine two key issues of paramount concern to designers: how remaining seated at the table affects the performance of fundamental tasks, and the design of interaction techniques to support such an environment.

1.1.1 The Trouble with Angles

A significant factor introduced in a table-centric interactive space is the various non- optimal angles between the user and the display and input devices. We are particularly concerned with three angles created by the unusual display configuration of these spaces.

1.1.1.1 Angle of Table Relative to the Eye

When creating software for a traditional desktop system, designers typically make the reasonable assumption that the user will be viewing a vertical display, oriented directly towards them. In a table-centric interactive space, this assumption no longer holds true. When information is displayed on a tabletop, it will be viewed on a plane receding from the user, providing a distorted retinal image. Figure 1-3 illustrates this distortion.

Figure 1-3. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Right: the same map, as it appears to the user of a tabletop display

5 The angle of the table relative to the eye is dependent on a number of factors. One such factor is the angle at which the table has been installed. There are instances where users have installed interactive touch tables at angles above the horizontal (Wigdor et al. 2007b). In this thesis, however, we concentrate on spaces utilizing a horizontal table. This is for two reasons: first, a horizontal table would be required for multiple users to sit around it comfortably, and so it is the most likely candidate for a table-centric space. Second, a horizontal table presents the largest likely angle of the table to the eye – thus, studying the horizontal condition will provide the best range of design solutions.

1.1.1.2 Angle of Table Content Relative to the User

Since multiple users may be seated around the table, content is displayed at varying angles to each collaborating participant. Content facing one user is necessarily shown at a non-optimal angle for another user. Figure 1-4 illustrates this.

Figure 1-4. Left: the same map of Cambridge as above. Right: the tabletop as seen by another user seated at the same table

1.1.1.3 Angle of Vertical Displays Relative to the User

Finally, vertical displays may be erected at any position or angle around the tabletop. It is reasonable to assume that, in environments where such displays are intended for use by those sitting at the table, the displays will be oriented to face towards those users. Even so, as in Figure 1-5, vertical displays might still be non-optimally oriented for some users.

Figure 1-5. Left: A vertical display as seen when angled optimally for a user. Right: the same vertical display as seen by another user at the same table 6 This angular issue is less of a problem than the others described above, because users can largely correct for this by turning their heads. Despite this, the magnitude of these three angles introduces a novel human-factors element to interaction in a table-centric space.

1.2 Contributions

The primary goal of this thesis is to provide designers with a set of tools which they can employ to better design a table-centric interactive space. These tools will come in two forms. First, a set of studies that measure human performance with systems that include the angles we have described. The results of these studies will serve as first steps towards the design of input mechanisms and information displays for table-centric spaces. The second set of tools is a series of interaction techniques, built in an attempt to address a set of design issues we have identified. These interaction techniques are intended to facilitate a table-centric space by allowing participants to gain an easy understanding of the display topology, and to have complete control of all surfaces while remaining seated at the table.

1.2.1 Human Performance Measurement

There are many methods to allow us to understand how the angles we have described might affect users in these spaces. In an effort to uncover a baseline, we will concentrate on their affects on low-level, fundamental tasks.

There exists a myriad of mechanisms for defining task types. In the first part of this thesis, we will focus on low level tasks users perform when working with an interactive system. It is our belief that, by examining these, we can provide a basis for the design of general-use spaces, rather than spaces intended for a particular type of high-level task. In particular, we will examine three tasks: reading, perceiving information graphics, and giving input to the system. When we cross these with the angular distortion types identified above, we define the scope of work needing to be addressed in order to design a table-centric interactive space. By combining the work of the current thesis with work done previously in the domains of psychology (Cunningham 1989), information visualisation (Cleveland and McGill, 1984, 1985), and human computer interaction (Grossman et al. 2007), we are able to provide a thorough examination of the issues of angular distortion and table-centric spaces. The research space is shown in Table 1-2.

7 Task Type

Angle Type Reading Graphical Perception Input

[Chapter 3] Table to eye [Chapter 4] - Grossman et al. (2007)

[Chapter 3] [Chapter 4] Table content to user Koriat and Norman Cleveland and McGill Cunningham (1989) (1984, 1985) (1984, 1985)

Vertical display to user Grossman et al. (2007) [Chapter 4] [Chapter 5]

Table 1-2. Crossing the angular problems with primary interaction tasks provides an overview of the problems this thesis seeks to address. Individual cells identify whether this has been examined in previous work, a chapter in this thesis, or is not yet explored.

While several of these cells have previously been explored, in many instances it was appropriate to revisit this work. In addition, one cell has not been explicitly studied either by the previous work, or by the work described in this thesis. We will address this in detail in the relevant chapters.

In addition to attending to low-level tasks, designers must also consider the design of interaction techniques to support table-centric interactive spaces.

1.2.2 Interaction Design

There exist several challenges for the effective design of interaction techniques supporting a table-centric space. These include: providing users with awareness of the mapping between table input and the effect on the vertical display; providing mechanisms to associate content between the displays; maintaining a direct-touch input paradigm; providing multiple granularities of remote interaction; and providing support for users to engage in non-interfering concurrent tasks. We will present interaction designs intended to overcome and address all of these issues.

1.3 Thesis Roadmap

We will begin with the study of fundamental capabilities of users performing low-level tasks in table-centric spaces. We will make our way through each of the cells defined in Table 1-2, using a task-centric approach: we examine each of our three key tasks

8 (reading, graphical perception, and input), and, for each, examine the affects of the three types of angular distortion (table to eye, table content to user, and vertical display to user). From each of these examinations will flow two key contributions: guidelines for the design of table-centric spaces, and open research questions requiring further study. Following the presentation of these low-level issues and design recommendations, we will then present and evaluate a series of interaction techniques designed for use in these spaces. We will now briefly examine the content of each remaining chapter of this thesis.

1.3.1 Chapter 2: Background Literature

In Chapter 2, we review the current state of tabletop and related research, including tabletop sensing and display technologies and extensions to the basic tabletop. We also examine work by various tabletop researchers related to human behavioural, perceptual, and physical issues, including group behaviours, and physical limitations in design. We then examine various multi-display environments, and their relevance to table-centric interactive spaces. In addition to this overview of tabletop and multi-display related research, each of the remaining chapters examines works from other areas of research which relate closely to the content of that chapter.

1.3.2 Chapter 3: Reading

As we have discussed, table-centric spaces require the reading of text presented at non- optimal angles. Experimental results in the field of psychology (Koriat and Norman 1984, 1985) exaggerate the reading-speed penalty associated with this reading for the sake of studying internal mechanisms (Wigdor and Balakrishnan 2005). In this chapter, we present the results of a study specifically intended to measure the impact of non-optimal orientation on reading speed. By providing performance results, designers will be better equipped to determine the trade-offs between the various solutions to oriented reading and deciding at what angles such solutions become necessary.

1.3.3 Chapter 4: Perception of Elementary Graphical Elements

Because tabletops require the horizontal presentation of information, the graphical content of their information displays is distorted when it reaches the eye. Although principles such as shape constancy suggest that there exist compensatory mechanisms

9 which allow us to overcome these effects (Pizlo 1994), there did not previously exist a precise measure of the efficacy of these mechanisms in decoding information graphics. In this chapter, we present the results of controlled experiments, intended to precisely measure the accuracy of perception of graphical elements on a tabletop, as well as those compared across a table and a vertical display. The result is a set of measures of the users’ varying acuity at perceiving different methods of graphically encoding numerical quantities. Using these results, designers may be better equipped for selecting among the various visual encodings of quantitative data.

1.3.4 Chapter 5: Input and Control

Table-centric interactive spaces require the user to give input to vertical displays while seated at the table. This requires an offset of input to output, potentially breaking the direct-touch input paradigm. Although indirect interfaces are not uncommon, that the vertical displays in table-centred environments are often placed at odd angles to the users causes problems in the mapping of input and display spaces. Such non-optimal mappings have previously been studied by psychologists (Cunningham 1989, Cunningham and Welch 1994, Dobbelsteen et al. 2004). However, their focus on internal mental mechanisms, rather than the actual metrics produced in their experiments, limited their relevance to practical applications. In this thesis, we present the results of controlled experiments measuring a user’s ability to control ancillary displays while seated at a table. The result is a detailed understanding of users’ abilities to perform motor functions on displays of varying position, using controls at various orientations. This allows us make recommendations to designers about the physical arrangement of vertical displays, as well as the about the location and orientation of control mechanisms for those displays.

1.3.5 Chapter 6: Interaction Techniques

Having examined the three low-level tasks and their performance within an interactive space, we will turn our attention to producing effective interaction techniques for those environments. In this chapter we present techniques which seek to address each of the design challenges described in section 1.2.2 . In order to maintain a table-centric space, our techniques require input from only the table, while allowing for the control over content on ancillary displays, as well as moving content between those displays and the 10 table. We present an example application, validated with a potential user population, and also explore the results of a user study examining the ease of learning of these techniques.

1.3.6 Chapter 7: Conclusions

In the final chapter, we will summarise the results of the previous chapters and examine open issues for the design of table-centric environments.

We now review tabletop and related research. Although there are clearly areas of human factors, psychology, and other research related to the problems of angle we have described, we will reserve reviews of those works for the relevant chapters.

11 Chapter 2: Background Literature

We begin by examining many of the technologies which have been applied to build tabletop systems, and also discuss a number of extensions to the traditional definition of a direct-touch tabletop. For these technologies and extensions, we will examine how each attempts to build on previous efforts, and review the pros and cons of the approaches in building towards a tabletop capable of all of the sensing envisioned by researchers.

Having examined the technological baselines, we will then examine efforts which have been made to discover fundamental human behavioural, perceptual, and physical issues to be addressed by applications on these systems. This will include an examination of group dynamics around a table, perception of content under rotation, and physical limitations for tabletop interaction. This process will attempt to uncover underlying value in the direct-touch tabletop paradigm and to better inform the design of interaction techniques in support of that paradigm.

We will then turn our attention to interaction design. We will conduct an examination of tabletop research, beginning with early motivational works and moving to the state of the art in interaction design. Next, we briefly review non-tabletop efforts in collaborative groupware, before beginning our review of researchers’ efforts to extend the tabletop with multiple devices, displays, and technologies. Finally, we conclude with possible lines of further research which would increase our understanding of tabletop groupware, and in particular its extension to multi-surface, table-centred collaborative spaces.

2.1 Direct-Touch Tabletops

An interactive tabletop is a touch-sensitive surface, large enough to allow for simultaneous input by multiple users, upon which is overlaid a digital display. The tabletop becomes direct-touch when the hardware or application designer matches the location of input to the graphical of system response on the display. In recent years, the advent of new tabletop technologies has spurred a flurry of research activity. We will begin our exploration of tabletop interaction with a review of technologies that have been employed to develop these systems.

12 2.2 Hardware Implementations

The ideal direct-touch table has yet to be implemented. Based on the various properties of existing technologies, and those envisioned in research, the ideal would be comprised of:

1. A large horizontal surface, all of which is capable of displaying digital information (Shen et al. 2003a) 2. Ability to detect the location and shape of area of contact to the table of touches of body parts (Forlines and Shen 2005), styli (Wu and Balakrishnan 2006), and other objects (Fitzmaurice 1995) 3. Ability to detect the precise location and orientation of the user, stylus, or other object while above, and not in contact with, the display surface (Subramanian et al. 2006) 4. Detecting the moment and location that an object moves from being above to being in contact with the table (Grossman et al. 2004) 5. Identification of these users, styli, and other objects on or above the display surface (Dietz and Leigh 2001) There are several underlying technologies which have been deployed in an effort to produce a direct-touch tabletop. None of these technologies is capable of the entirety of our ideal touch table, but each provides advantages and disadvantages over the other implementations. Here, we will review work which introduces new approaches for the construction of interactive touch-tables. Many of the devices presented here have been employed by researchers to explore compelling interaction and system designs. The efforts of those researchers will be presented later in this discussion.

2.2.1 Vision Based Systems

Vision based systems employ cameras to detect the location of objects, users’ hands, fingers, or styli. Several vision based systems have been presented. The first appearance in the literature of the use of vision based systems for direct-touch on a tabletop is the Digital Desk (aka ‘Marcel’) (Newman and Wellner 1992, Wellner 1993). In their system, cameras mounted above the desk were used to track position of a pen, held by the user, as well as paper documents (Figure 2-1). Interactions were generally conducted ‘through’ the paper: pressing drawn buttons, entering data in to columns, etc. This approach allows for the blurring of the lines between the physical and the virtual elements of the system.

Figure 2-1. Vision based systems make use of cameras to detect the presence of objects on the table. Figure from Wellner (1993)

A plethora of other vision based systems have also relied on the installation of cameras, including Underkoffler and Ishii (1998), Rekimoto and Saitoh (1999), Kato et al. (2000), Koike et al. (2001, 2004), and Berard (2003). Vision based systems are subject to occlusion, and detection of identity of the objects is difficult. Because of the necessary physical installation, traditional systems can be difficult to set-up and relocate.

Figure 2-2. The ThinSight system. Left: IR emitters and detectors are laid out in an array beneath the LCD. Centre: the image detected by the device. Right: the ThinSight can be embedded within a thin LCD, such as a laptop. Images from Hodges et al. (2007)

A research system by Wilson (2005) addressed this by integrating a digital projector and camera into a single unit, placed on a table surface. Although the prototype was quite bulky, the vision of integration into a single, portable unit is desirable. A modern example of a commercially available, vision based touch table system is the DViT from Smart Technologies, which has cameras embedded in a bezel installed around a flat-panel display (Smart 2006). By placing the cameras on the periphery, the need for overhead mounting is avoided, and the cameras are better positioned to detect precise vertical distance of objects from the display. The DViT is capable of tracking only two simultaneous touch locations – far fewer than would be required for implementation of many of the tabletop systems developed by researchers. More recently the ThinSight

14 system (Hodges et al. 2007) embedded infrared emitters and detectors in an array beneath an LCD. While an optical system, it avoids many of the disadvantages described above.

In addition to these touch technologies, researchers have recently begun to explore the use of stereo cameras for the detection of hands and objects in three dimensions. Such systems include Wilson’s use of Depth Sensing Video Cameras (Wilson 2007) and Izadi et al’s C-Slate system (2007).

2.2.2 Capacitive Systems

Capacitive touch-systems use changes in electrical potentials to detect the position of touches. Capacitive systems alleviate some problems inherent in vision based systems, since they integrate all sensing technology into the surface of the touch device, are not subject to interference through the occlusion of a sensor, and have the potential to detect a much larger number of contact points. Two capacitive touch tables have been presented in recent research: the SmartSkin by Rekimoto (2002) at Sony, and the Diamond Touch at MERL (Figure 2-3, Dietz & Leigh, 2001). Each of these implementations has advantages and disadvantages. The Diamond Touch provides higher resolution, and can identify which user is making each touch, but cannot detect objects on or above the display (Dietz and Leigh 2001). However, the x and y coordinates of touch points are decoupled, so that multiple touch points result in a series of x and y values not tied to one another, creating ambiguity. The SmartSkin uses an antennae mesh to overcome the ambiguity of touch points, but offers lower resolution data, and offers no user identification.

Figure 2-3. A diagram of the DiamondTouch, which performs a capacitive coupling for each user. Figure from marketing material

15 2.2.3 Frustrated Total Internal Reflection

Frustrated total internal reflection (FTIR) is an optical technique in which lights are positioned along the edges of a sheet of glass. When an object comes in contact with the display, the light is reflected downward and detected by cameras positioned below the surface of the table. Figure 2-4 illustrates a system developed by Han (2005).

Figure 2-4. Left: Using Frustrated Total Internal Reflection to detect touch points. Right: the image as seen by the camera, before processing. Images from Han (2005)

Like vision based systems, FTIR-based systems require the use of cameras offset from the display device. Additionally, the technique is incapable of detecting objects not in contact with the touch surface, and is incapable of user identification.

2.2.4 Object Tracking Technologies

Several tabletop projects have relied on two types of technologies which are able to track only special objects, rather than providing true direct-touch functionality. Because these systems are unable to detect hand position independent of those objects, they are of limited applicability to many of the interaction techniques explored by tabletop researchers. Despite this limitation, however, many of the techniques developed for these systems offer compelling avenues for study.

Kakehi et al. describe an enhancement to their Lumisight table, in which objects transparent to IR light, save for an opaque pattern unique to each object, are placed on the surface. A camera beneath the table is used to track a unique opaque pattern on each object (Kakehi et al. 2006). They point out that their system would function correctly on any transparent display, but advocate specifically for its use on the Lumisight table. This is because the tangible objects can have different patterns projected onto them, depending on the user’s point of view, creating multi-purpose tangibles. The ThinSight system allows for similar tracking, but with sensors embedded in the LCD, eliminating the need

16 for an external camera (Hodges et al. 2007). The ThinSight includes no mechanism for projecting on to the tangibles.

Acoustic systems emit ultrasonic waves which are then received by objects, which can in turn determine their location on the table based on properties of this sound wave. Two systems have made use of acoustic trackers: Shen et al. (2003b) used a commercially available Mimio system (www.mimio.com). Mazalek et al. (2006) later described a system in which large pucks on the table surface can be tracked for positional and rotational information.

Figure 2-5. Left: the TViews sensor configuration. Right: the mobile ‘pucks’ are the only objects tracked by the system. Images from Mazalek et al. (2006)

The Flock of Birds from Ascension Inc (http://www.ascension-tech.com/) provides 3D magnetic-based tracking of small objects. Fitzmaurice et al. (1995) developed a series of interactions for their ActiveDesk , which used these tracked objects, which they dubbed bricks , to perform tabletop interactions. Like acoustic based systems, magnetic tracking recognizes only specific trackable objects, and cannot be used for direct-touch interaction on a tabletop.

Figure 2-6. Left: the ActiveDesk used magnetically tracked bricks to perform two-handed manipulations. Right: the ActiveDesk in action. Images from Fitzmaurice et al. (1995)

17 2.3 Extensions to the Direct-Touch Tabletop

Each of the systems we have seen so far implements some of the functionality of the ideal touch table described earlier. In addition to these implementations, there have been a small number of extensions to the touch table paradigm.

2.3.1 User-Differentiated Views

One of the fundamental characteristics of touch table interaction is that each user seated around the table is able to see and work with all of the elements on the table. Another vision is to present designers with the option to differentiate the view for each user. This differentiation can be used for a multitude of purposes, from correcting for orientation of text, to providing some visual privacy for data display on the shared surface. Technology to enable this includes the Lumisight table, introduced by Matsushita et al. (2004), which uses material with particular refractive properties to present users seated in different positions with images displayed by different projectors, as illustrated by Figure 2-7.

Figure 2-7. The Lumisight Table presents a different view to each user. Figure from Matsushita et al. (2004)

2.3.2 Inverted and Two-Sided Touch Tables

Another extension to the traditional tabletop is the advent of inverted and two-sided touch tables. First presented by Wigdor et al., these tables are built by either turning a traditional touch table upside down (in the case of an inverted table), or by stacking a traditional touch table on top of an inverted table (in the case of a two-sided table). Advantages of these designs include the elimination of occlusion while touching, natural mapping of input mode to touch-surface, more accurate pointing, and allowing bimanual input to a single point (Wigdor et al. 2006d).

18 2.3.3 3D Tabletops

A natural extension of tabletops is a system that would allow the projection of 3D imagery in the space above the table. Grossman and Wigdor presented a taxonomy of 3D for the tabletop (Grossman and Wigdor 2007). Approaches for implementing 3D on a tabletop include projecting images in stereo which must then be viewed with special glasses (Krüger and Frohlich 1994, Krüger et al. 1995), using head-tracking and peer- through glasses to add a 3D image (Benko et al. 2005), or the use of true volumetric displays (Grossman et al. 2004). Each of these approaches has potential benefits and disadvantages. For techniques in which imagery is projected on the table, because the faux position of the 3D objects may not actually be between the table surface and the viewer, it is impossible to display objects ‘above’ the table. Next, for both that and peer- through glasses technologies, the user must wear goggles or headgear, and the user’s point of view must be tracked, or the 3D imagery may not be correct for the viewpoint. Finally, a volumetric display, while alleviating the need for head gear or position tracking, requires that the display be encased in a dome or other shape, making direct- touch interaction with the 3D objects impossible (Grossman et al. 2004).

2.4 Behavioural, Perceptual, and Physical Issues

In this section, we discuss various research efforts focusing on human behaviour, perception, and physical characteristics which may be of use to designers.

2.4.1 Applications and Long-Term Use of Tabletops

Projects have recently explored tabletop systems which have been deployed to users, and in some cases present data about their long-term use of over a period of weeks or months. Various aspects of their findings fit into the particular elements discussed in the following section, but their distinction as longitudinal studies bears individual examination.

In the first, Ryall and her colleagues report on observations of the use by many different users of located in the lobby of their lab. They report various findings about pointing and group interaction. Particularly compelling are their observations of the lack of spontaneous gestures on the table, the importance of a mapping between finger touch area and a single coordinate of the input space (Ryall et al. 2006b).

19 Khandelwal explored the use of a tabletop system by pre-kindergarten students. The author describes the design and implementation of the “Teaching Table”, a touch table augmented with physical objects. The table can track tagged objects placed on its surface, accurately identifying their type and location while providing a coincident visual display and audio feedback. The aim was to demonstrate how a table can work as part of a classroom environment, and to explore how it can enhance learning experiences of children in an interactive and playful way by involving them in physical activities. Focus group feedback and usability studies were employed to parameterize and optimize the design, but an actual deployment is not described in the publication (Khandelwal, 2006).

Figure 2-8. Khandelwal’s “Teaching Table” provided pre-kindergarten students with a tangible interface. It was intended to act as a teaching tool for fundamental math skills

In her MSc thesis, Mazalek explored the use of tangibles on the tabletop to create interactive, point-of-view narratives for her users. The system is intended to provide a story-telling experience which responds to inputs from the user to drive elements of the narrative (Mazalek, 1999). In another work, Mazalek and her colleagues installed a TViews table in the home of a user over a one-month period. The table is used through interaction via physical objects, rather than direct-touch, as described earlier (Mazalek et al. 2006). Their observations and conclusions include a plethora of information about the table’s technical performance, interaction with its applications and objects, and its assimilation into the home environment. Their study reveals the importance of overcoming what they term the ‘novelty factor’, which they point out, may operate as confound in many studies of tabletop use (Mazalek et al. 2007).

20 In yet another study, Wigdor et al. presented the study of a regular tabletop user over a 13 month period. This user employed a tabletop as the primary interaction method with his desktop computer, using DT Mouse software (Esenther and Ryall 2006) to translate multi-touch tabletop input to mouse events for everyday applications. Their study included an examination of touch location behaviour, the linguistic content of e-mail composed from the table, and an interview with the user (Wigdor et al. 2007b).

2.4.2 Group Behaviours

Because tabletop applications are generally intended for group interaction, research on the behaviour of individuals while in groups around a table can inform the design of these systems. Two particular aspects of group dynamics have been explored: territoriality and the effects of group and table size on the performance of tasks around a tabletop.

Scott et al. (2004) set out to explore user behaviour while working with shared objects around a traditional table. They observed participants conducting cooperative and competitive group tasks while seated around tables of three different sizes. The key result from their study is that the participants tended to divide their spaces into three areas: group , storage , and personal territories. Objects located in a group territory tend to be free to be used by any participant seated around the table. The researchers found that this area is located in the centre of the table, within easy reach of all participants. Objects collected from this space and moved to one of the other spaces take on a set of social rules that govern their use which can be shed by returning the object to the public space.

While working on objects collected from the public space, participants tend to move these objects immediately in front of them, in order to manipulate them more comfortably. Interestingly, even when objects in the space in front of users are not actually in use, other users will tend to regard them as belonging to that user. Others will not reach for objects they perceive to be within the private space of another user.

Storage territories are areas reserved for task-essential artefacts not currently in use. These territories tended, in their study, to be on mobile platforms which could be passed around to other people performing a task requiring the resources contained in the storage bin. 21 Rogers et al. point out that one of the promising features of tabletop interaction is that input is egalitarian: users sit around a table, so no seat is preferred, and input is enabled for all users. Accordingly, many assumptions of user behaviours applied in the development of other collocated groupware may not apply to tabletops. One issue they examine is decision making. There is extensive research in other disciplines surrounding this issue, but many groupware systems have been successful while ignoring it, because they supported input from only one user at a time. Rogers et al. (2004) made some initial exploration of this issue by asking groups of users to assign digital photographs to each month of the year, creating a photo calendar. They recorded interactions, and made observations about how participants used the system to aid in forming group decisions. They noted several behaviours, including the use of the system to ask other users questions, to offer instructions, to make suggestions, to request confirmation of choices, to offer opportunities, and to encourage contribution from other group members.

Ryall et al. (2004) asked groups of varying size to collect words on virtual tiles scattered around a table into a pre-determined poem. They observed the participants and recorded the location of each input and time to completion. Participant performed these tasks in three different group sizes (2, 3, or 4) and around two different tables (88cm diagonal or 107cm diagonal). Groups were also given different numbers of copies of the target poem printed on paper: either one shared copy, or one copy for each participant.

The researchers identified three of their findings as key results. First, table size had no effect on speed, while group size was inversely correlated with completion time. Interestingly, there was no interaction between these variables, contradicting their expectation that larger tables would penalize smaller more than larger groups. Second, task distribution are affected by group size: smaller groups (pairs of 2) tended to work together throughout the task, while larger groups vary their working style over the course of the task, tending towards parallel action at the beginning and working increasingly cohesively as they searched for the last few words. They note that social interaction varies with respect to table size. Finally, the researchers observed behaviours consistent with those in Scott et al. (2004), that participants tended to not reach into personal space as defined by it proximity to another participant.

22 In a related work, Forlines et al. (2006a) conducted an analysis of visual search performance with groups of varying sizes. The task was simulated baggage screening, in which participants were required to identify contraband items from a field of objects. The researchers predicted that larger groups would more quickly and more accurately detect the presence of items, and more quickly reject fields without an item present.

Figure 2-9. Visual field with no target present (left), another with a gun (centre), and the same field with all items but the gun removed (right). From Forlines et al. (2006a)

This work also sought to examine the benefits of a tabletop for the baggage screening task. The authors hypothesized that task performance time and accuracy would be better for groups standing around a tabletop than those looking at a vertical display, since each participant would have a different viewing angle to a target object, and would be able to identify it more quickly when it was closer to its canonical view (this hypothesis was suggested by past work in cognitive psychology). Their results were surprising on a number of fronts. First, although groups were more accurate and faster in both target present and target absent trials, the overall improvement in accuracy was far less than might be expected of 4 individuals working individually. Additionally, they found no effect for display configuration: viewing the scene on a horizontal surface gave no advantage or disadvantage over vertical orientation of the display.

Matsuda et al. set out to examine the use of the Lumisight table, seen previously in Matsushita et al. (2004), for performance of tasks with asymmetric distribution of information. In their study, they had groups perform a maze-search task. Each member of the group could see only the area of the maze which surrounded his on-screen avatar. In order to capture their moving target, the group members needed to share information with one another quickly. Two display conditions were used; as illustrated in Figure 2-10: participants were seated either around either the Lumisight table, or vertical displays.

Figure 2-10. Groups performed a coordinated search of a maze, with each participant’s unique view presented either on a Lumisight table (left) or a vertical display (right)

The researchers found that performance was better on the Lumisight table. They hypothesized that this was due to participants’ being able to see one another in, and were therefore able to better communicate. They attribute the benefit of this to ‘closeness’, after they found a correlation between performance and participants’ desire to perform the task with the same group again. Unfortunately, they make no attempt to determine causation: it seems equally likely that a respondent would be more inclined to perform the task with the same group if that group had done well the first time. No matter the cause, this result points to the benefits of a shared display over individual displays.

2.4.3 Orientation

The work with visual search revealed that the different point of view afforded to each in a team of searchers did not result in increased speed. This divergence of viewpoint, however, can have effects on individual and group performance of other tasks.

An issue with orientation is how it is used as a language to imbue meaning for the user and other members of a group. Kruger et al. (2003, 2004) observed that spatial positioning and orientation of items on a tabletop was used to establish and maintain personal and group spaces. They reported that that items placed in group space were generally oriented similarly, usually such that most group members could view them easily. Once an object was removed from the group territory and moved to a user’s personal territory, it was oriented for optimal viewing by that user.

24 Fitzmaurice et al. (Fitzmaurice et al. 1999) observed that, while drawing, artists often rotate the page to non-rectilinearly aligned orientations. In particular, animators were seen to make use of an animator’s turntable , as seen in Figure 2-11. They performed a study in which they asked professional and student artists to draw, using a pencil, on paper placed on one of 1. the turntable, 2. a digital tablet on a table, or 3. a digital tablet which they were invited to place in their lap. In all three conditions, they report similar results: that the artists routinely reposition and rotate the canvas to work on it at different angles. In particular, they observed that professional artists tended to reorient and reposition their canvas more than the students, and theorize that artists may optimize for speed. Figure 2-11 illustrates the extent of the reorientation in the most extreme case. Based on these observations, the authors advocate that drawing tools include the ability to dynamically reorient the canvas while keeping user interface elements towards the user.

Figure 2-11. Left: Animators make use of a turntable to rotate their drawing surface without rotating their work area or handling the cell directly. Right: an artist’s finished product (e), and their pencil strokes relative to the table surface (f). From Fitzmaurice et al. (1999)

2.4.4 Reach

Toney and Thomas (2006) formally address the issue of reach, and point to the Kinetosphere , or dynamic reach envelope, as defined by anthropometricists (Hedge, 2002). They point out that there are, for the user, effectively three regions of the tabletop as defined by reach: within comfortable reach, outside of comfortable but still within reach, and out of reach. They suggest that this may be a method for more formally defining the territoriality regions described by Scott et al. (2004), and that better understanding the Kinetosphere will lead to better interface design for tabletops.

25 2.5 Interaction Design

Having now reviewed various touch table technologies, and human behavioural, perceptual, and physical issues in using those technologies, we turn our attention to the ongoing research effort of developing effective interaction techniques. In this section, we conduct a review of this literature, beginning with efforts focusing on the development of a GUI for the tabletop. Next we will review various projects which have made use of tangible objects on the table, followed by efforts in gestural interaction. Finally, we will review projects designed to support mixed-presence groupware.

2.5.1 Collocated Groupware GUI Design

The primary focus of interaction designers on tabletops has been the development of collocated collaborative groupware. A large number of projects have taken on many different aspects of the GUI: from low-level issues, such as precise pointing and selection, to overarching look and feel issues. Here, we provide a review of these projects.

Although much of tabletop research has focused on particular aspects of interaction, a number of research projects have provided visions for the overall look and feel of potential direct-touch table systems. Although these projects include details beyond look and feel, much of their value in research comes from establishing more completely thought-out UI’s. There are a number of projects which might have been included here, but are instead positioned as foundational works in other sections for their innovations in specific aspects of tabletop UI.

The first such project was the Digital Desk . As described previously, the system worked by projecting digital objects on to the surface of a desk, as well as onto documents. Controls were provided in the form of images printed on note cards that could be placed on the surface of the desk and tapped. These controls were context aware, in that a button placed next to an on-screen feature would have its effect executed on that feature (for example, a ‘save’ button would be placed on the document to be saved). In so doing, it provided the first tangible UI for tabletops, as well as the first projected direct-touch tabletop (Newman and Wellner 1992, Wellner 1993).

26 Vernier et al. (2002) were the first to describe a circular tabletop. Although the actual physical table in the project was square, the interaction area was circular. The shape of the table, though rarely studied in the field, is an interesting problem for UI designers. With a circular table, for example, it is a reasonable assumption that items rotated such that they are oriented along the tangent to the closest point on the edge will face the user. Vernier et al.’s table was later used to implement a compelling tabletop application (Shen et al. 2004, 2006). The circular table is illustrated in Figure 2-12.

Figure 2-12. Left: Vernier et al.’s circular tabletop interface. Right: a user gives input to the system using a stylus. Images from Vernier et al. (2002)

Nishimoto et al. (Nishimoto et al. 2006) present a tabletop system, dubbed pHotOluck which uses dinner plates as virtual displays. Although the content is actually projected from the ceiling, the system projects imagery on to the plates at a table setting. Although the system is capable only of receiving and displaying digital photographs, the use of physical artefacts as display objects is compelling. Hinrichs et al. presented interface currents (Hinrichs et al. 2005, 2006). Currents provide a mechanism to keep non-active objects continually in motion. As seen in Figure 2-13 left & centre, currents can be established around the perimeter of the table, causing objects to flow through the working spaces of all users, or in arbitrarily defined storage areas. Although this approach addresses only the issue of storage, its application clearly defines the look and feel of a system which employs it. Cotting and Gross (2006) introduced display bubbles (Figure 2-13). Display bubbles allow the user to use a laser pointer to define arbitrary shapes in which to display interactive content on a tabletop. The bubbles reshape and move themselves to avoid overlapping with physical objects on the table. Interactions are performed via the laser pointer. 27

Figure 2-13. Left: objects in the current flow around the space defined by that current. Right: a display bubble shows a portion of the interface, and avoids overlapping with a mobile phone. Images from Hinrichs et al. (2006) and Cotting and Gross (2006)

A final, very different approach to tabletop interaction is that of opportunistic browsing . Brujin and Spence (2001) implemented a system on the LIME tabletop from Philips (http://www.design.philips.com/about/design/section-13506/) where the tabletop is an interactive coffee table, and not the primary focus of any kind of interaction. Instead, the table randomly displays news or other information. If the information catches the user’s eye, he can get more information by dragging the object to a search mechanism. The results of the search then become the content of the information display. Figure 2-14.

Figure 2-14. The opportunistic browsing system built by Brujin and Spence Images from Brujin and Spence (2001), on the LIME table (Philips.com)

One of the more compelling properties of an interactive tabletop is that input and display spaces are overlaid, thus providing a natural platform for direct-touch input. In fact, as was reported by Potter et al., in systems with such a confluence of display and input devices, users react negatively when input is not interpreted in a direct-touch way (Potter et al. 1988). A limitation of this, however, is that highly accurate selection is made difficult, since the selecting finger occludes the target.

28 Three research efforts have presented techniques to provide precise selection on a tabletop. The first dealt with single-point touchscreens: Albinsson and Zhai (2003) present a pair of widgets intended to allow for more precise pointing on a touch table, without breaking the direct-touch input paradigm. Figure 2-15 illustrates the two widgets.

Figure 2-15. Two widgets for single-point high precision touch screen interaction. Left: Cross-keys, Right: Precision-Handle. Images from Albinsson and Zhai (2003)

The cross-keys widget requires coarse positioning of the widget, followed by finer- grained movements in touching one of the four direction buttons, similar to selection with a keypad. The precision-handle provides a graphical mechanism to maintain direct-touch input, while offsetting the input and modulating C/D gain to allow accurate selection.

The remaining two selection projects assume the presence of tabletop technology capable of detecting multiple points of input. The first such project is Esenther’s and Ryall’s gestural techniques to simulate the functionality of a two button mouse with scroll wheel. In order to facilitate more precise selection, the selection point can be represented by a cursor positioned between two fingers touched to the display. When two fingers are touched to the screen, the pointer is in tracking mode. As shown in Figure 2-16, some element of the direct-touch paradigm is maintained by attaching the selection action to touching of the index finger to the on-screen cursor (Esenther and Ryall 2006).

Figure 2-16. Left: The cursor is displayed between the middle finger and thumb. Right: the mouse-click is simulated when the index finger is touched to the display

29 In a parallel effort, Benko et al. provide a heavier-weight mechanism to support pixel- accurate pointing. In their system, the non-dominant hand is used to zoom the user interface, change the C/D gain of the cursor and finger, and snap the cursor to targets. In their system, the direct-touch paradigm is all but abandoned (Benko et al. 2006).

Figure 2-17. Left: tracking state is provided by pressing less of the finger to the display – increasing the finger area on the display performs a ‘click’. Right: the non-dominant hand is used to adjust various controls for the dominant hand’s manipulation of the cursor

Each of these efforts goes to great lengths to maintain finger-based input to the table. Many advantages can come, however, by moving selection to a hand-held stylus. In a series of papers, Parker et al. describe a system, the Tractor Beam , which allows users to select tabletop object using a stylus. To select an object, the user can tap it with the tip of the stylus, or she can point at it with the stylus, as shown in Figure 2-18 (Parker et al. 2005a). In a subsequent publication, they added ‘snap to’ selection, so that when the pointer comes within a threshold distance of a selectable object, it snaps to that object (Parker et al. 2005b, 2006).

Figure 2-18. Selection of distant objects using the laser pointer- like interaction of the Tractor Beam . Image from Parker et al. (2005b)

30 Since display angle and object orientation are such an issue for interactive tabletops, a great deal of research has attempted to provide mechanisms for translation and rotation of these on-screen objects. Hancock et al. (2006) provide a summary of mechanisms described in the literature. Wu and Balakrishnan (2003) provide the most basic of mechanisms to address this issue on direct-touch tabletops. They provided in their system a method to numerically specify the orientation of objects by touching controls for rotation angle. Another mechanism is to provide widgets, attached to the object, which can be dragged to adjust position and orientation. This approach was used by several projects: Streitz et al. (1999), Tandler et al. (2001), and Shen et al. (2004).

As Kruger et al. (2003) observed, translation and rotation of objects are usually tied, since objects are reoriented as they are moved from one territory to another. A promising approach is to provide mechanisms to allow the user to simultaneously move and rotate objects. Another approach is to tie position and orientation, so that objects placed at a particular location always have a particular orientation, usually towards the closest outside edge of the table (Vernier et al. 2002, Shen et al. 2004, Forlines et al. 2005). Although simple, such a technique means that not all orientation/positions are possible. As was observed by Kruger et al., objects in a public territory are generally placed at some ‘group’ orientation (Kruger et al. 2003). This group orientation and position may not be included in the pairings that this technique may offer.

An alternative design is to simulate integrated rotation and translation that occurs with physical objects on a table surface. One such model, proposed by a pair of researchers, works as follows: the user selects an object by placing his finger on it. When the user drags his finger, the larger, ‘heavier’ portion of the object trails behind the finger. Such a technique has been described by both Beaudouin-Lafon (2001) and Kruger et al. (2005). A final design is described by Hancock et al (2006) and Han (2005): to translate an object, the user places two fingers on it, and then drags the object along the surface of the table. The object translates and rotates so that the points touched when the user put his hand down remain below the fingers throughout the drag. If the user’s fingers do not remain at a fixed distance, then the object must be scaled, or one of the points ignored.

31 An issue that arises with collocated groupware is the need of a mechanism for users to assign one another permissions for various objects. Ringel et al. (2004) describe a series of mechanisms that allow for users to easily share documents. To change their permissions, objects can be resized (larger objects are owned by the group), reoriented (objects facing a user belong to them), passed (touching the object at the same time as its owner conveys ownership) or moved (objects in front of a user belong to that user). As the authors point out, two of these techniques are based on ethnographic observation done elsewhere: Kruger et al. (2003) observed that users reorient objects and Scott et al. (2004) observed they move them in order to change their “ownership”. In a subsequent work, Morris et al. presented a work which described a series of multi-user coordination policies defining system behaviour when multiple users attempted to work with objects simultaneously. These are rank (higher ranked user has control), replicate , (both users get a copy) and tear (the object is broken in half) (Morris et al. 2004b, 2006e).

Since the tabletop is a shared display, zooming the interface disrupts the work of other users. Forlines and Shen (2005) presented DTLens to attempt to mitigate this effect. In their system, users can zoom an area of the display by touching two points on the surface, and separating their fingers. The fingertips are taken to define the corners of a rectangular zoom window, whose size and magnification are adjusted by the drag. As Figure 2-19 illustrates, additional controls are presented on the zoom window, such as the ability to pin down the window, and adjust zoom. If not pinned-down, the window will snap- closed, giving an elastic, stretched-canvas feel to the zoom.

Figure 2-19. DTLens. Left: user defines zoom area with fingertips Right: zoom widget with additional controls. Figure from Forlines and Shen (2005)

32 As we have seen, the issue of orientation is of particular interest to UI designers for the tabletop, since the viewing angle for each user may be significantly different. We also have seen that the Lumisight table is able to provide a different field of view for each participant, provided that each participant is offset 90 o from one another (Matsushita et al. 2004). In its first presentation, the Lumisight table was demonstrated as a possible technique for correcting text orientation, since labels could be oriented differently for each user. In a follow-up work, Yoshino et al. propose a system architecture to allow for a multi-lingual display, so that labels and other on-screen text is presented to a user in her native language (Yoshino et al. 2006).

Two approaches have been proposed to leverage different aspects of space to modify input. In the first, by Everitt et al. (2005), the tabletop surface is divided into multiple modal spaces . Within each modal spaces, the same posture or gesture performs a different function. For example, drawing the finger across an object in the ‘cutting’ space will divide the object, but moving the object to the ‘drawing’ space will result in drawing a line across it.

In a subsequent work, Subramanian et al. (2006) describe multi-layer interaction. In their paradigm, the vertical space above the tabletop is divided into multiple layers. When the user holds his stylus inside of one of these layers, the on-table visual content updates to show this layer, and input is sent to that layer. Their work includes a mechanism to avoid accidentally changing layers while performing actions on a given layer. They described four techniques to perform a selection without the benefit of a physical object against which to tap: pointing with the stylus and pushing a button with the other hand, pen-air- tap (rapid up/down movement), pin-through (crossing a plane below a layer), and crossing within a layer.

In situations where a tabletop is used by multiple users, audio feedback from the system may be ambiguous. Thus, providing a mechanism for audio feedback is non-trivial, as is the use of audio as a medium for richer information. Two projects explored this issue.

Hancock et al. (2005) examined in detail the problem of audio as a feedback mechanism for multi-user tabletop applications. They first examined both positive (confirming an

33 action) and negative (pointing-out errors) feedback, and found that the presence of positive feedback hindered personal awareness but enhanced group awareness. Negative feedback, however, was found to enhance both personal and group awareness of errors. In a follow-up experiment, they examined the individual and simultaneous use of two mechanisms for disambiguating audio feedback: localised and coded feedback. Localised feedback was provided by speakers placed near each individual. Coded feedback varies the sounds depending on which participant they are intended for. They found that, when coded or localised sound (or both) were employed, there was an increased awareness of the user’s own errors, and better focus on individual work. However, awareness of the errors of other members of the group suffered as a result.

In another examination of the impacts of types of audio feedback, Morris et al. (2004a) asked groups to assign music tracks to scenes from a popular film. The researchers provided a tabletop interface which included music playback. As a within-subject variable, groups were provided with a speaker to play music, or individual ear buds for each participant. They found that, although participants reported nearly identical enjoyment, ability to complete the task, and satisfaction with the end product, a number of behavioural differences were exhibited. Specifically, there was less dominance by a particular group member in the headphone condition, that groups using ear buds were more likely to reconsider and replace song selections already made, and that the amount of communication between group members was actually higher in the headphone condition. It is worth noting that the researchers report that they believe the headphone condition provided two seemingly contradictory results: that more actions were happening in parallel, and thus the task was more thoroughly completed, and that there was a great deal more communication between group members.

One obvious question presented to designers is whether to provide a single, shared control palette, or to provide a copy of needed controls for each user. Morris et al. (2006b) explore this issue in detail, noting design considerations such as ease of reading, size of footprint, comfort for reach, user preference, and impact on collaborative practices. They found that users overwhelmingly preferred individual control palettes for the task they were performing. They also found that, for a task which can be done in

34 parallel but would benefit from collaboration, shared controls led to slightly more collaboration, in the number of users contributing to each unit of work product. They also found, from analysis of video shot of the trials, that shared controls led to more time spent in conversation while performing their task. No performance metrics were included in the study. In another work, Ryall et al. (2006a) explored the use of user identification as a parameter to the function of shared controls. The parameter was used to modify targets of actions (‘undo’ undoes the last action of the user who pressed it), to evoke different responses (personalized audio captions on photos), to exhibit different behaviour for different users (scroll bars are continuous for one user and discrete for another), or to enforce individual permissions (only particular users may delete a photo).

Identifying users allows the system to respond differently to their input. Three techniques have been described in the literature: object association, capacitive coupling, and disoriented gesture recognition. In the first two approaches, pairing user with identifiable tracked objects (Agrawala 1997) and using different receivers of coupling for each user (Dietz and Leigh 2001), specialized hardware is required. Users are required to sit on a pan, hold a particular stylus or object, or wear gloves, which is are used to make input to the system. Another approach, which requires only that users be positioned at different angles to the table, is to attempt to evaluate characteristics of the input to determine the user. In a work which combines temporal and spatial information, Mohamed et al. (2006) claim to be able to detect the location of user around the table, looking only at characteristics of their gestural input. They report accuracy in excess of 98%.

In a series of works, Tse et al. describe their introduction of combined speech and gestures for control of a tabletop. First, they modified to a pair of popular computer games, enabling users to specify action type with speech and locations for those actions with gestures (Tse et al. 2006a). In a subsequent work, their system is extended to allow for programming by demonstration and multiple users (Tse et al. 2006b, 2006c).

Because tabletops are larger than traditional displays, associated objects may be displayed at different angles, and because visual style of the tabletop has not yet been determined, there is a need to provide mechanisms for the visuals for the association of

35 objects on screen. Shen et al. (2005) present two graphically similar but conceptually dissimilar mechanisms, dubbed individual and grouped CoR 2Ds (‘cords’). Grouped cords provide a method for visually connecting a menu with the object upon which its actions will take effect. The group can be dragged by the handle, and oriented and positioned for more convenient use without disturbing the associated object. Individual cords are intended to represent objects and their properties or functionalities, and individual cords from different objects can be brought together to set properties or execute functions. For example, a presentation file may be an individual cord from its source folder, as is the “Current Poster” a corded property of a display window. By dropping the presentation on to that cord, the property is changed, and the presentation loaded as the current poster.

Figure 2-20. Left: individual CoR 2Ds can be used to perform commands. Right: grouped CoR 2Ds provide a mechanism for commands to be closer to a user

As we have seen, a number of projects have attempted to examine human behaviour around a traditional table, in an attempt to inform on the design of interactive systems. We now examine a series of projects based on such work.

In work examined earlier, we saw that Scott et al. discovered three territories: personal, group, and storage. The authors describe a pair of interface ideas which leverage these observations. These include reorienting objects based on their territorial location, and reducing the size of objects that are moved out of personal (working) territory (Scott et al. 2004). In a follow-up, Scott et al. (2005) describe a system to support in software the mobile storage territories they observed on real tables. Their technique, dubbed storage bins , is very similar to a technique developed previously by Wu and Balakrishnan (2003). Both techniques allow for users to arbitrarily define bins in which on-screen objects can be placed, and then moved as a group, or collapsed, similar to minimizing a window in a WIMP system. Scott et al’s innovation seems to be that, in their project, the dimensions and shape of the storage areas can be arbitrarily redefined.

36 A pair of papers examined the development of interfaces to support searching of digital media by teams of users. The issue of collaborative searching is an interesting application for tabletops, since the task allows users to work in parallel and in support of one another. The tabletop system must be designed to support both of these working styles and support fluid switches between them. Note that this is distinct visual search task, examined by Forlines et al (2006a), since participants are searching for information not necessarily present on the screen, and because there is some element of interaction.

In the first work to address this issue, Smeaton et al. (2006) presented a system intended to facilitate multiple users simultaneously searching and browsing a video interface. The system employed the Diamond Spin toolkit (Shen et al. 2004). The authors did not present any new interaction techniques, but do discuss in detail each of their decisions about the various parameters of the toolkit and their applications to their task.

In another paper, Morris, et al. describe their TeamSearch application, also implemented using the Diamond Spin toolkit (Morris et al. 2006a). Unlike Smeaton et al.’s system, TeamSearch requires that the media be marked with meta information, and is intended only to search photos. TeamSearch provides ‘tokens’ representing values of various meta properties. The users filter images by placing these tokens on to widgets. Two policies for user coordination were examined: collective (where the filter is built by Boolean and between each user’s token placements) and parallel (where each user’s token placements are independent, which were joined by or between users, and and within each user).

2.5.2 Tangibles on the Tabletop

Tangible objects provide affordances for touch and manipulation absent from software interfaces. Tabletops afford the use of tangibles in coordination with on-screen elements, since the horizontal surface allows objects to remain in place. In order to be as inclusive as possible, we have broadly defined tangibles to be any physical, non-digital object used to interact with a digital system. The first digital tabletop application, the Digital Desk, included elements of tangibles: paper on the desk surface was used to frame interaction. This included a physical form for the calculation of an expense report, and paper labels used to represent functions (Newman and Wellner 1992, Wellner 1993).

37 In a very different type of interaction Fitzmaurice et al. used trackable bricks to provide bimanual and tangible interaction with the Active Desk . A brick, placed on an object, becomes a physical proxy for that object, allowing simultaneous rotation and translation, similar to some of the techniques described by Hancock et al. (2006). Attaching two bricks to the same object allows it to be scaled or deformed with six degrees of freedom (Fitzmaurice et al. 1995). In a later, similar use of tangibles, Ullmer and Ishii presented the metaDESK . This tabletop system provided two tangible objects representing buildings which were moved to translate, rotate, and scale the map, such that the two buildings were always in the correct positions on the map. Additionally, two virtual lenses were included in the system: the passive lens, which provided a tangible method for defining a software lens on the desk surface, and the active lens, a flat panel display mounted on an arm and used to browse 3D space around the campus. (Ullmer and Ishii 1997).

Two subsequent works from the MIT Media Lab provide further refinement of tangible interfaces on a tabletop: first, the Sensetable , provided a more scalable and responsive mechanism for tracking objects on its surface (Patten et al. 2001). Recently, another project at the MIT Media Lab which leveraged the Sensetable technology was presented by Yoon et al. (2004). Their system, meant as a mechanism for spontaneous social interaction, provided an ‘emergent game’ in a coffee table: when the first participant placed her cup on the table, a fish would be attracted to the ripples. When a second user placed his cup on the table, the fish would begin to oscillate between the two cups. This system was meant to allow each pair of users to discover and play a game of Fish Pong .

A later project, dubbed the Actuated Workbench , allowed computer control over the location of physical objects on the surface of the tabletop. By embedding an array of electromagnets below the surface and varying the power to those magnets, magnetic objects could be made to slide across the surface of the table (Pangaro et al. 2002). Rogers et al. (2006) sought to examine the added value of using tangible objects on the tabletop. They conducted a study, in which participants were asked to plan the layout of a garden, using either digital only or digital and physical objects on a tabletop. They concluded the users are more likely to thoroughly explore an environment and examine multiple options when physical objects are included in the interaction.

38 2.5.3 Gestural Interaction

Because the tabletop input surface is generally situated at a comfortable height and immediately in front of each user, it is a compelling platform for gestural interaction. An important limitation for the potential for the use of tabletops for gestural interaction is that their input and display spaces are overlaid. Care must be taken either to use these in a direct-touch manner (Potter et al. 1988), or to provide sufficient mechanisms to allow the user to break their direct-touch assumptions. Defining gestural interaction is challenging, since it might be as narrowly construed as a series of hand postures, or as broadly interpreted as any kind of input where the system response to one frame of input relies on a previous frame. Here, we will sidestep this debate, and examine several works which provided significant steps in gestural interaction on tabletops.

Cutler et al. defined a set of two-handed operations for the responsive workbench. The gestures are performed using gloves, a stylus, or tools. They provide a series of unimanual, bimanual symmetric, and bimanual asymmetric interactions. These operations provide manipulations in both 2D and 3D, and allow the user to perform actions such as rotation, translation, scale, define a cutting plane, and to measure temperature (Cutler et al. 1997). In a related work, Grossman et al. defined a set of postures and gestures for interaction with volumetric displays. Like Cutler, they supported both unimanual and bimanual gestures to interact with a 3D space. A defining difference between the two works is that the 3D space in the volumetric display is contained in a hemispheric dome, requiring designers to rethink many of the gestures (Grossman et al. 2004).

For gestural interaction on the surface of a tabletop, three research efforts are of particular interest. Wu and Balakrishnan defined a set of gestures to be performed on the tabletop, in the context of a room-planning application. Their gestures included mechanisms to bring up menus, make selections, rotate, scale, and group objects, as well as to define lenses for the display of meta and private information (Wu and Balakrishnan 2003). In a subsequent work, Wu et al. examined mechanisms to ease the use of gestures on the tabletop. Their system included two key features: gesture relaxation, where a complex posture used to initiate a gesture need not be maintained throughout the gesture, and a mechanism to temporarily produce copies of objects to perform gestures on. This second

39 feature was included in order to overcome two limitations of direct-touch input: 1. that input to the same virtual space cannot be done with two hands simultaneously; and 2. that hands occlude objects they are manipulating (Wu et al. 2006). In another innovation for gestures on the tabletop, Morris et al. explored what they termed cooperative gestures: gestures requiring the actions of multiple participants in order to complete them (Morris et al. 2006c).

2.5.4 Mixed-Presence Groupware

As we have seen, tabletop systems provide a promising avenue for in-place meeting and decision spaces. A natural extension of this paradigm is the extension of meeting spaces across multiple sites. Several projects in tabletop research have explored this idea.

The first mixed-presence groupware application on a direct-touch tabletop was the Video Draw collaborative drawing application. In that application, multiple participants, one on each table, used styluses to draw on a shared canvas. The user input areas were overlaid with each other in video, so that each user saw both the other user’s drawings and his arms making the drawings. Figure 2-21 illustrates (Tang and Minneman 1991).

Figure 2-21. The Video Draw system. Left: schematic. Right: system in use. Images from Tang and Minneman (1991)

An interesting subsequent work is the Clearboard . Like Video Draw, the Clearboard presented a drawing area, with the collaborator artwork overlaid. Instead of showing disembodied arms, however, the other user would appear as a mirrored image below the table. The strokes of both participants appeared on the board, producing a shared drawing. Figure 2-22 illustrates.

Figure 2-22. The Clearboard system. Image from Ishii and Kobyashi (1993)

A series of modern mixed-presence tabletop systems have attempted to extend the state of the art to include remote collaboration to a more general set of applications. In an exploratory work, Perron and Laborie (2006) built a pair of tables meant to allow two remote teams of designers to collaborate. Their design included video conferencing on a separate monitor, as well as interface artefacts to support cross-site collaboration. The researchers particularly emphasise the need for their and other systems to support the sharing of all manner of digital and physical documents and objects, facilitated in their case via a boom-mounted camera.

The ViCAT system (Chen et al. 2006) provides vertical screen video conferencing, atop a shared video editing application on the tabletop. Manipulations to the application on the table were mirrored on all sites. In a related work, Hutterer et al. (2006) presented a toolkit for the introduction of a remote-devices layer for tabletop applications. Their layer seamlessly integrates input from multiple locations, allowing perfect mirroring of input across each location. In another effort, Wesugi and Miwa (2006) present a Lazy Susan communication system. In their system, all remote collaborators are seated at a table, with a platform in front of them. When a user applies rotational force to the platform, that rotational force is conveyed to the remote sites, causing the same force to be applied to every participant’s platform. The contents of these platforms are shared via vertical screens and cameras. The goal of the project was to explore how haptics could be used to enhance the feeling of presence.

41 2.6 Interactive Spaces

Also relevant to our table-centric interactive spaces are collaborative systems that make use of multiple displays and devices to form larger interactive spaces. We will examine two form factors: multiple computer-terminal systems, multi-surface systems.

2.7 Multi-Terminal Roomware

Multi-terminal Roomware systems allow each user her own terminal, each of which is connected together to form a larger interactive space. Although many technologies and commercial software tools support the building of spontaneous networks, our focus here is on installed spaces, specifically designed for interaction.

The foundational work in multi-terminal Roomware systems is the Collab system. This system featured multiple terminals which could interact on shared documents, and a large display wall. Users could work together or at multiple sites (Stefik et al. 1987). The system is described in more detail in a follow-up work, which studied the design and use of Cognoter, a tool for collaboratively planning presentations and papers. Cognoter includes tools to support distributed brainstorming, and users working together to evolve a coherent structure to a talk. In addition to the system description, they make several observations about user behaviour when using the tool, such as the iteration of unstructured to structured brainstorming notes in a shared window (Stefik et al. 1987b).

Figure 2-23. The Collab installation at Xerox PARC

42 2.8 Multi-Surface Systems

Multi surface environments feature multiple displays for one or more users (Coutaz et al. 2003). Here, we examine first generally applicable pointing and space management issues. Next, we perform a review of multi-surface projects. Finally, examine in more detail multiple table-centred multi-surface environments.

2.8.1 Pointing and Space Management

In a multiple-display system, navigating a pointer across various, non-aligned surfaces creates a mismatch between the continuous 2D motor space and the 3D display space. Several efforts have been made to attempt to address this issue. In the first, Biehl et al. described a mechanism for display management, which they dubbed ARIS . Notionally, the ARIS system displays a flattened display environment, representing every display in the virtual space. Objects on the displays are represented iconically, and manipulations to these iconic representations, such as moving on and between screens, are conveyed to the object (Biehl and Bailey 2004, 2005). Figure 2-24 illustrates.

Figure 2-24. Left: a multi-surface environment. Right: the same environment in ARIS. Image from Biehl, 2004

In another effort, Baudisch et al. introduce Mouse Ether , which attempts to unify multiple coplanar displays into a larger motor space, allowing the mouse-controlled cursor to move in space on the plane but outside of a particular display (Baudisch et al. 2004). Subsequently, Nacenta et al. presented the Perspective Cursor , which attempts to map the 2D motor space of the mouse to the image plane of the viewer. Wherever the use moves the cursor in the multi-display arrangement, the shape, size, and motor control are adjusted to create the same physical image of the cursor as viewed by a single user (Nacenta et al. 2006). In another piece of work, Wigdor et al. presented a study of how

43 the position of ancillary displays and the control orientation for those displays on the table interact to affect performance at pointing and manipulation. From this, they were able to provide design recommendations for display placement and orientation of their controls (Wigdor et al. 2006c). This work is described in detail in Chapter 5.

2.8.2 Multi Surface Implementations

Multi surface projects make use of multiple displays, sometimes including both vertical and tabletop systems.

In a series of publications, Streitz et al. have described digital furniture designed to support spontaneous collaboration. Their systems include tabletop ( InteracTable ), vertical displays ( Dyna Wall ), and chairs ( CommChairs ) with built-in displays. They provide mechanisms for users to dynamically interconnect laptops and various furniture components in order to construct ad hoc spaces to support collaboration (Streitz et al. 1999, 2002, Prante et al. 2004).

Figure 2-25. The Roomware system, including Comm Chairs and Dyna Wall. Images from Streitz et al. (1999)

In a subsequent effort, Everitt et al. provided mechanisms for interaction between vertical displays, a table, and portable devices. Users working on any of these surfaces could define clips of information to be moved to and assembled on the interactive tabletop. Objects were moved by dropping them onto portals representing each of the connected devices (Everitt et al. 2006).

44 2.8.3 Table-Centric Multi-Surface Environments

Although both of the multi-surface spaces we have seen include interactive tabletops, the interactive table has not been the centre of interaction, but rather simply a device included in the interactive landscape. We now turn our attention to systems designed to position users at an interactive table which is the focus of interactions.

A pair of projects have focused on enabling interactive tables for walk-up-and use functionality, allowing connections with and support for mobile devices. The first, by Rekimoto and Saitoh, allow users to move objects from their laptop computers onto a table surface. Objects on the surface continue to be associated with the home computer with graphical sticks. These objects can be relocated on the table surface, placed on a vertical display, or shared with another laptop user (Rekimoto and Saitoh 1999).

Figure 2-26. Left: the laptop is recognized, and the user has moved an image out on to the table. Right: the user has moved the image to the vertical display

In a subsequent project, Shen et al’s UbiTable was also intended to provide a mechanism for spontaneous, walk-up-and-use functionality for easy sharing of data, such as photos or notes (Shen et al. 2003a). The system supported the sharing of these objects by moving them to the table, and then transferring them to other users using some of the fluid sharing techniques described previously (Ringel et al. 2004).

As we have seen, a table can provide a natural focal point for interaction. In a number of projects, these tables are augmented with the ability to connect with other devices, including vertical displays. What we examine now are systems which have tables as their focal point of interaction, but which are supported by additional devices and surfaces completely controllable from the table, without the need to physically relocate to another surface.

45 The iRoom project at Stanford university has as its stated goal the building of seamless interactive spaces (Johanson et al. 2002a). To enable this, the group has built the Point Right system, which can enable a single system pointer and keyboard cursor to be seamlessly controlled by any device connected to the system, enabling direct and indirect input from any supported device and display (Johanson et al. 2002b). Although one of the goals of the system is to allow input directly to ancillary displays, it also enables complete control over the environment while remaining seated at a meeting table. A pair of follow-up projects have made use of the iRoom infrastructure. These include the Multibrowser project, which allowed web content to be moved across multiple displays (Johanson et al. 2001), and a system to support meetings of architects for building design (Fischer et al. 2002). Figure 2-27 illustrates the iRoom system.

Figure 2-27. Left: the iRoom topology. Right: various iRooms built from their system architecture. Images from Johanson et al. (2002b)

The Diamond Space from MERL represents their vision for table-centric interactive spaces. In a pair of publications, they offer two similar, yet distinctly different, ideas for control over vertical displays from a table. The first, Wigdor et al. 2006a and 2006b, contain descriptions of the work of this thesis, presented in Chapter 6. They present the use of a world in miniature metaphor, with miniature views of the vertical display being presented on the table (Wigdor et al. 2006a, 2006b). In the second vision, Forlines et al. present a system which uses vertical displays to present views from cameras, the position and orientation of which are controlled in 3-space from the table (Forlines et al. 2006b).

46 Both projects show mechanisms to extend the digital table to multiple surfaces, and provide control from the table. The world in miniature approach is more easily generalized than the camera-manipulation metaphor, but provides less fluid control.

Figure 2-28. Table-based control of ancillary displays. Left: world in miniature metaphor. Right: camera metaphor. Images from Wigdor et al. (2006b) and Forlines et al. (2006b)

As we have seen, the area of interactive tabletops has recently seen a great deal of interest. Prior to most of this work being done, multi-display interactive spaces also were the subject of a great deal of research. Although recent significant efforts have examined multi-surface systems, which include both horizontal and vertical displays, the design space of table-centric interactive spaces has not yet been thoroughly explored. It is our intention to explore the unique properties of such spaces, in order to enable their design and increased adoption.

In each of the following three chapters, we will introduce issues of importance to designers of table-centric interactive spaces. In each, we will begin with an introduction to the issue, followed by an examination of work related to that particular issue, generally from research areas outside of direct-touch tabletops and interactive spaces. We will then present experiments intended to provide insights for designers, and conclude with an examination of the results, followed by design recommendations and open research questions. We begin this exploration with the issue of reading of text in interactive spaces.

47 Chapter 3: Reading

Reading text is an essential task in any interactive system. As we have seen, table-centric interactive spaces necessitate that display content be presented at unusual angles. The three angles of concern are that of the vertical display(s) to the user, the table to the eye, and of table content to the user. We will begin by exploring the issue of these angles, and how they might affect reading. In order to help us to place this work among others relating to the affects of orientation of text on readings speed, we will adopt the terms: roll , pitch , and yaw to define the axes of rotation, as shown in Figure 3-1:

Figure 3-1. Text rotations. (a) No rotation applied. (b) Positive roll. (c) Positive pitch. (d) Positive yaw. Figure from Grossman et al. 2007

Roll is a rotation of the text about the axis normal to the plane upon which it is inscribed, pitch as rotation about the axis along which the text is written, and yaw as rotation about the axis orthogonal to the other two. Previous work has suggested that reading text subjected to these orientations will cause a decrease in reading speed. In this chapter, we examine now which of these rotations occurs in a table-centric space, and then present the results of a pair of experiments measuring the magnitude of this decrease in such a space.

3.1.1 Angle: Vertical Display to the User

The angle of the vertical display to the user is significant because each user’s viewing angle to any given vertical display may not be direct. Even under the assumption that ancillary displays meant to aid collaborators seated around the table will be pointed towards that table, users may still not be facing directly towards a display from which they wish to read text. Using Grossman et al.’s terminology, this text might be subject to either positive or negative yaw . As we have previously explored, however, the issue of yaw is not significant in reading in a table-centric environment, since it is almost entirely eliminated if a user turns her head to face the particular vertical display. Because of this,

48 and because reading of text subject to yaw has already been thoroughly explored by Grossman at al. (2007), the issue of the angle of vertical display to the user is not further explored in this chapter.

3.1.2 Angle: Table to the Eye

The angle of the table to the eye is introduced because the tabletop display recedes away from the user, rather than in the user’s viewing plane. Using the terminology in Figure 3-1, unless the user is looking directly down onto the text (by moving her head directly above the table), that text will be subject to some amount of negative pitch . In Grossman et al. (2007), the researchers reported that small amounts of pitch had little effect on reading speed. Their studies, however, were limited to examining pitch in isolation, without considering it in combination with roll or yaw. Thus, we will examine the issue of table angle to eye as part of the experiments of this chapter.

3.1.3 Angle: Table Content to the User

When multiple users are sitting around a tabletop, objects displayed on screen can never be optimally oriented for all: an object facing one user will be at a non-optimal orientation for another. Using the terminology from Figure 3-1, text shown optimally for one user will be shown at a positive or negative roll . When creating content for display on a tabletop, software designers have control of only the roll of that content.

Research has been conducted into how participants in table-centred activities make use of the orientation of artefacts (Tang 1991, Kruger et al. 2003). They have found that users prefer a straight-on orientation for reading text, and orient objects towards themselves, others, or inline with shared artefacts to ease reading in different circumstances. They have discovered that in a collaborative setting, a straight-on planar orientation toward the reader is not always desired or exercised. In fact, orientation is employed as a tool to aid in interaction with other users. Despite this, some designers of collaborative systems have, opted to attempt to orient text towards the reader, as seen in Bruijn and Spence (2001), Rekimoto and Saitoh (1999), Shen et al. (2003b, 2004), and Streitz et al. (1999). Thus, there is a tension between a desire to allow for the use of orientation as an aid to collaboration, and the apparent need for textual objects to face the user to aid in reading.

49 Although users seem to prefer a “normal” orientation of text (Kruger et al. 2003), and although studies in the psychology literature by Tinker (1972) and Koriat and Norman (1984, 1985) indicate that readability is compromised when text is otherwise oriented, it is clear that there exist circumstances under which non-straight-on orientation of text is desirable (Kruger et al. 2003, Tang 1991). Despite Tang and Kruger’s insights, they do not provide an investigation into the parameters for determining when a solution to the text-roll problem should be applied in the context of tabletop groupware. Is roll angle so critical for text readability that design elements must be sacrificed in order to minimize it? Or, should designers simply leave it to the users to orient their own on-screen objects? Without empirical data quantifying the extents to which readability is compromised in less-preferred orientations, it is difficult for designers to make informed choices when confronted with these tradeoffs. The work described in this chapter provides such empirical data via two experiments that examine the effect of text orientation on the performance of tasks common to the tabletop. Based on the results, we hope to provide insights for designers into when the issue of text orientation takes precedence, and when it can be safely ignored. Our work also contributes as replication and extension of experimental work by re-examining and extending the studies of readability of text orientation by Tinker (1972) and Koriat and Norman (1985) to tasks relevant to a new domain, and where users’ head and body movements are unconstrained.

3.2 Terminology

Throughout this chapter, we will use the degree convention for orientation of text on the tabletop, shown in Figure 3-2.

Figure 3-2. Degrees of roll rotation as referred to throughout this chapter. In all cases, angles are measured relative to the edge of the table at which the participant was seated

50 3.3 Related Work

Of relevance to the present work are two areas of research. First, an examination of how the issue of reading orientation has been addressed by researchers of tabletop collaborative groupware will provide context for the present research. Second, research in the field of cognitive psychology concerned with reading, especially of text at non- horizontal orientations, is reviewed.

3.3.1 Orientation in Collaborative Groupware Research

In section 2.4.3 , we examined the issue of orientation and tabletop research. Here, we provide a more detailed re-examination of that same work.

An examination of the role of artefact orientation in collaborative settings was been presented by Tang (1991) and Kruger et al. (2003). The lessons they present are clear: orientation has a role to play in collaboration, and any simple, one-sized-fits-all algorithmic solution to the issue of orientation of screen artefacts will deprive participants of some of the richness of interaction afforded in paper-based tasks. What they also provide is a clear demonstration of the intuitive notion that users regard text presented in a “normal” orientation as more readable. They both agree that, similarly to what was reported by Fitzmaurice et al. (1999), users do not always orient artefacts in the same right-angled orientation found in most modern GUI. For example, orientation for readability is often used as a mark of ownership, and re-orientation towards a collaborator is a strategy for passing over or signifying a desire to share. Further to this point, Morris et al. (2006b), found that reading comfort was a contributor to the user preference of replicated, individual controls over shared group controls for a tagging task in which controls were used with high frequency, and were labelled with terms many of the users were unfamiliar with.

From this research, we make two important observations: first, that right-way orientation of artefacts is an important theme in tabletop collaboration, and second, that users regard the orientation of task objects as important cues for ownership and collaborative behaviour.

51 Despite the tradeoffs inherent in the above observations, systems continue to be designed that attempt to “solve” the issue of text orientation. For example, by taking advantage of advanced display technologies, in both Agrawala et al. (1997) and Matsushita et al. (2004), users view a common scene orientation, but textual object labels are oriented toward each participant. Additionally, many systems attempt to dynamically and automatically re-orient objects; Kruger et al. (2003) and Hancock et al. (2006) present a thorough review of these algorithmic approaches, to which we refer the reader for more information. The reasoning behind these approaches seems sound: if text is easier to read at a particular orientation, an ideal approach would always present that orientation to any user attempting to read its content. It is their implicit contention that right-way up reading is so important that the group-dynamic affordances described in previous research should be sacrificed. But, how real is the tension between readability and system flexibility?

In an attempt to answer this, we now examine the research in text orientation that has been conducted in the field of psychology. This area of research has not previously been examined in this thesis.

3.3.2 Human Ability to Read Text at Various Orientations

Research into the effect of graphical considerations on reading performance began with Huey (1898). Although the issue of orientation was not discussed, the effect of vertical vs. horizontal alignment of words was examined, as well as the effect of partially occluding portions of characters. The issue of orientation and its effect on reading was first explored in detail by Tinker (1972). He conducted two experiments: in the first, subjects performed the Chapman Cook Speed of Reading Test at various orientations (Chapman 1923). The Chapman-Cook test involves the presentation of a paragraph, where a single word “spoils the meaning” of the text. The subject is required to identify this word, either by crossing it out, or speaking it aloud. The speed of reading is measured by how quickly this is accomplished. In this experiment, Tinker (1972) found that the reading of text rotated at 45 o in either direction was, on average, 52% slower than reading normally oriented text, and text rotated at 90 o in either direction was 205% slower on average. In the second experiment, participants performed the Luckiesh-Moss Visibility Meter to determine the physical visibility of rotated text (Luckiesh 1944). Tinker (1972)

52 discovered that the speed of reading was affected much more dramatically by orientation than was visibility, and thus concluded that visibility was not the only factor that contributed to the decreased speed of reading at non-standard orientations.

Koriat and Norman (1984) used a text-reading task to evaluate the relative merits of two theories of how mental rotation is performed. Although this was informative to the present study, it was their later work (1985) which examined the issue of readability of rotated text in more detail. Specifically, they examined the effect of rotating text on the identification of strings as words or non-words. They found that performance of their task in the range of -60 o to +60 o from the horizontal was not significantly variable, but that a performance cliff was reached once rotation exceeded 60 o in either direction. Once this cliff was reached, word/non-word identification speed decreased by more than 120% (Koriat and Norman 1985).

The work of both groups seems to confirm the intuitive notion that reading orientation should be a key concern to designers of tabletop systems. However, their work is not directly applicable to tabletop collaborative groupware research. First, rather than allow participants free and uncontrolled movement during the experiments, the position and orientation of their heads was constrained. Second, in both sets of experiments, readability was determined by how quickly non-conforming strings were identified. In the Tinker (1972) study, this was done at the semantic level, as participants were required to find the word in paragraphs that spoiled their meaning. In the Koriat and Norman (1985) experiment, this was done at the syntactic level, as the study consisted of the presentation to the participant a series of strings of characters, which subjects were required to identify as either words or non-words. Though this experimental design enabled them to answer their research questions, we note that the identification of non- conforming or gibberish strings is not directly applicable to real user interface scenarios. In most applications, textual artefacts consist of either common words or domain terms that might be expected by the user. This assumption might aid in the reading of text at varying orientations, and so should be considered when evaluating user performance of reading at varying orientations.

53 In our studies, we attempted to provide a less controlled environment where user head movements are not constrained and measured the performance of reading non-gibberish text at various orientations. It was our hypothesis that, given this environment, the effect of orientation on task performance would be less dramatic. If this were the case, we believe that the tension between orientation as a tool for collaboration and the apparent need to use the “right” orientation for text readability can be relaxed, and systems could be designed that heed the observations of Tang (1991) and Kruger et al. (2003). The final piece of related work is that by Grossman et al. (2007). Their work was conducted as a follow-up to the work described in this chapter, which was first published in 2005 (Wigdor and Balakrishnan, 2005). In that follow-up, the results of the first experiment below were extended, in order to examine the effects of both pitch and yaw on reading, in addition to the effect of roll described in this chapter. Further, they incorporated their performance data for pitch and yaw with our results for roll, in order to create an algorithm to determine the orientation of text in a 3D display. That work did not include any examination of the effects of roll on reading, but relied instead on the results contained in this chapter, as published previously as Wigdor and Balakrishnan (2005).

3.4 Research Goals

In order to begin to examine the issue of orientation on the reading of text, we designed a pair of experiments. Through these experiments, we wished to establish a baseline for quantifying the effects of presenting text at various orientations and pitches (angle to eye) on reading speed. Ultimately, we wish to provide a tool for designers, so that they might quantify the trade-off we described earlier: providing an uncontrolled, physical-table like experience of allowing users to orient objects arbitrarily, while maintaining a minimum level of readability of text associated with those objects.

We wanted to conduct an experiment that would measure performance in an actual collaborative tabletop environment, in contrast with the artificially constrained environments used by Tinker (1972) and Koriat and Norman (1985). To this end, we present the text to the user on a tabletop, and allow participants free movement of their body and head, to allow for any tendency toward personal re-orientation.

54 3.5 Experiment 1

3.5.1 Goals and Hypotheses

In our first experiment, we seek to quantify the effect of both roll and pitch on three types of text: a single word, a short phrase, and a 6-digit number. To this end, we varied the presentation of text by controlling its roll, as well as its pitch. Because text was presented only on a tabletop, we controlled its pitch by placing the text at varying distances from the participant: text presented further away would have a greater negative pitch, while text presented closer would have a smaller negative pitch. Although a greater range of pitch measurements would have been possible by presenting the text on a display other than a tabletop, this has already been explored by Grossman et al. (2007). The goal in the present experiment with respect to pitch, therefore, is to measure the effects of the levels of pitch that are controllable by a designer of tabletop applications: that pitch which is introduced by the placement of text at different positions on the tabletop. Because of the relatively small difference in pitch, and based on the results in Grossman et al.’s work, we hypothesized that the location of text would not have a significant effect on speed of reading.

In order to ensure that participants could easily comprehend the words and phrases, single word stimuli were only simple, 5-6 letter words, while the phrases presented were coherent and meaningful. It was our hypothesis that the effect of orientation on reading of a single word would be less dramatic than what Koriat and Norman (1985) observed. Because our set consisted of only common words, participants would be able to trust their immediate identification of a rotated word, rather than graphically examining each character, as was required in the Koriat and Norman (1985) experiment. Further, we believed that the performance in reading longer phrases would be better than what Tinker (1972) reported. Because our phrase set consisted only of short, logical phrases, participants would be able to rely on the context of the surrounding text to aid in identification of less recognisable words. Lastly, we believed that the reading of numbers would be most affected by orientation, since no particular grouping of the numbers could be assumed. Thus, we expected that our results in this situation would be similar to that previously reported.

55 3.5.2 Apparatus

Text was presented to the user using a ceiling-mounted digital projector, and was projected onto a large DiamondTouch tabletop, introduced by Dietz and Leigh (2001), at which the participant was seated. Although the DiamondTouch is intended for touch- input, we used it only as a display screen since object manipulation was not required in our experiment. While we could have just as easily projected the image onto any tabletop surface, we chose the DiamondTouch since it is one of the common platforms for tabletop research and has a diffuse surface that provides for high-quality projection, and because its size is a commonly studied form for tabletop research, as described by Ryall et al. (2004). The text was presented in a sans-serif font, and rotated in software to be presented to the participant. There was no significant degradation in the quality of the rendering at the various orientations.

Text entry was facilitated by a standard QWERTY keyboard placed directly in front of the participant, who was seated at a chair placed directly in front of and centred on the longer side of the table. The system was driven by a windows-based Intel Pentium 3.0GHz system, equipped with an NVIDIA GeForce FX Go5700. Figure 3-3 illustrates the experimental environment.

Figure 3-3. Top-down diagrammatic view of the experimental apparatus

56 3.5.3 Participants

Fifteen participants, recruited from undergraduate classes and graduate students, volunteered for the experiment. 12 were male, 3 were female. 13 spoke English natively, while the remaining 2 reported to be excellent readers. All self-reported as excellent typists. Participants received no compensation for their participation.

3.5.4 Procedure

Users were repeatedly presented with a string that they were asked to read, memorize, and type-in to the system. We wished to measure how long they would spend reading the text before they were sufficiently confident that they would remember it long enough to enter it.

The string types consisted of 5-6 letter words, 6-digit numbers, and 6-7 word phrases. The words and numbers were randomly selected and generated, while the phrases were selected from the corpus developed by MacKenzie and Soukoreff (2003). The location of each string on the screen was primed with the display of a red cross for 0.65 seconds before the appearance of the text. When the user began to type, the text would disappear from the screen, but would return whenever “escape” was pressed. Figure 3-4 illustrates the procedure:

Figure 3-4. Left: orienting crosshair primes the participant as to the location of text. Centre: after 0.65 seconds, the crosshair disappears and is replaced by the rotated text. Right: as the subject begins to type, the text disappears from the screen. The red arrow is for illustration only and was not present in the actual experiment

Timing began when the string was first displayed, and stopped when the user began to enter input. In order to prevent participants from “racing through the experiment”, they were required to enter perfect input, and were told to correct their entry if it was incorrect

57 when they pressed “enter”. Trials where such correction occurred were not included in our timing analysis. Participants were given initial instructions as follows:

The experiment will require you to read some text on screen. Each time text is presented, you should memorize it (as quickly as you can), and then start to type it into the computer. When you begin to type, the text will disappear from the screen.

At any time, you can view the text again by pressing “escape”.

You will be required to enter the text absolutely correctly. If you make a mistake, the system will tell you, and ask you to correct your entry. Press enter to begin the experiment. Before each priming/string was presented, they were given on-screen instruction:

You will now be presented with a red cross, which will shortly be replaced with text. Read the text, keeping in mind that you will need to remember the text long enough to type it into the computer after it disappears. When you are ready, type the text in to the computer. If you make a mistake the system will alert you. Press "space" to begin. To ensure that participants understood what was required, they were allowed a practice dataset consisting of several short phrases. They were instructed to enter as many as was required to become familiar with the apparatus. Participants were directed to rest as required between strings, but to continue as quickly as possible once a stimulus was presented on-screen.

3.5.5 Design

We wished to measure the effects of both roll and pitch on reading speed of various types of text. Strings were presented in three datasets of 96 elements each: one set of 5-6 letter words, one set of 6 digit numbers, and one set of 6-7 word phrases. The order of presentation of the datasets was counterbalanced between-participants using a Latin- square (Montgomery, 2001). Pitch was controlled by presenting the strings at two different distances from the user – at the edge closest to them and at the edge furthest from them – thus providing the widest range of pitch values a user would encounter when using a table such as the one in our experiment. Although variable due to its sensitivity to participants’ height, the relative pitch of the text varied between lower and upper quadrant orientations by up to 30 o, a spread which Grossman et al.’s findings suggest could lead to significant differences.

58 Roll was controlled by making each presentation at one of the 8 roll values shown in Figure 3-2, which includes straight up, and each position around the compass at 45 degree increments. Within each set the position and orientation of a string were randomized, but controlled such that each position and orientation at that position was presented an equal number of times. Roll values were measured relative to the side of the table at which the participant was seated. Because head and chair positions were not constrained beyond the need to reach the keyboard to enter text, the exact orientation of the presented strings relative to the participants’ eyes was not measured. Assuming a comfortable typing distance, as was generally observed, the angle of the 0-degree oriented-text relative to the centre of the participant’s nose was approximately 12 o for the upper-quadrant cases, and 17 o for the lower-quadrant cases, well within the range of orientations shown to have little effect on reading speed in the works discussed previously. In summary, the design was as follows: 3 datasets (single word, number, short phrase) x 4 on-screen positions (each corner of the tabletop) x 8 orientations (starting at 0 o, in 45 o increments) x 3 strings at each position/orientation x 15 participants = 4320 strings entered in total.

3.5.5.1 Alternative Experimental Designs

Two alternative experimental designs were considered, but rejected in favour of the design we have presented. We will briefly present each of the alternatives:

The Stroop test is one of the most famous in psychology. As first demonstrated by Stroop (1935), it was found that participants were slower to identify the colour of a string if the text was an interfering colour name than if it was not the name of a colour. It is believed that the participants were reading the text more quickly than they were identifying the colour, and that the string was interfering with the identification of the colour. Our first experimental design consisted of a modified Stroop test, where the strings would be presented in various orientations. We believed that the Stroop effect would continue to assert itself, and so demonstrate that the participant was able to quickly read the text. We rejected this design for two reasons: first, we wished to measure performance of reading

59 of various types of strings. Second, we realized that, even if the effect continued to assert itself, we would be demonstrating only that reading speed was above the colour- identification threshold. While informative, this would fail to measure with sufficient fidelity the effect of orientation on reading performance.

Alternatively, a read aloud design would have consisted of the presentation of text at various orientations, the user reading the string aloud, and the measurement of the total time required to read the text. This design would have mimicked the design presented by Huey (1898). We chose not to employ this design for two reasons: first, limitations in speech recognition technology would limit our reporting accuracy. Second, differences between silent and spoken reading speed would limit the efficacy of the experiment in demonstrating reading performance at varying orientation.

3.5.6 Results

We discarded from the timing data all trials where the participant entered erroneous data that was then subsequently corrected. We utilized a repeated-measures ANOVA to measure effects. Orientation had a significant effect on error rate ( F7,98 = 2.321, p =

.0.031). The type of stimuli did have an effect on errors (F1,14 = 13.644, p =0.002 – a lower-bound result is reported because type was non-spherical with respect to error, using the Greenhouse-Geiser method), with mean error rates of 15%, 8%, and 4% for single word, short phrase, and number treatment respectively. The location of presentation of the stimuli among the 4 on-screen position had a borderline-significant effect on performance time ( F1,14 = 3.657, p = .077 – the lower-bound results were used because the affect of position was non-spherical, employing the Greenhouse-Geiser method). Although borderline-significant, the size of the difference between the means was small. This implies that the small differences in angle between the user’s eye and the table (pitch) for different positions has only a minor effect on reading speed.

As expected, orientation had a statistically significant effect on speed of reading (F7,98 <

0.001), single word ( F7,98 = 11.435, p < .0001), short phrase ( F7,98 = 23.838, p < .0001), and numbers ( F7,98 = 5.476, p < .0001). Pair wise means comparisons of entry time were conducted to determine which orientations were significantly different from one another.

60 In the single word and short phrase conditions, time to entry of stimuli presented at orientations of -135 o and 135 o were not significantly different from one another, but were significantly different from those at -90 o and 90 o. In the number condition, time to entry of stimuli presented at all orientations beyond -45o and 45 o were not significantly different from one another.

As we hypothesized on page 55, the effect of orientation on reading speed was far less dramatic than had been previously reported. Contrary to our hypothesis, however, the effect of rotation on 6-digit numbers was the least dramatic of the three conditions. Table 3-1 summarizes the mean rates for each orientation and condition.

Single Word Short Phrase 6-Digit Number Mean Std. o Mean Std. % off Mean Std. % off % off 0 o o (secs) Dev (secs) Dev 0 (secs) Dev 0 -135 o 1.19 0.67 64.70 3.82 1.52 107.13 2.85 1.06 17.48 -90 o 0.92 0.40 26.60 2.66 1.07 44.25 2.85 1.63 17.19 -45 o 0.78 0.60 7.98 2.07 1.02 12.62 2.36 1.17 -2.71 0o 0.72 0.22 - 1.84 0.86 - 2.43 1.57 - 45 o 0.77 0.22 5.93 1.97 0.70 7.19 2.39 1.19 -1.65 90 o 0.91 0.37 25.78 3.09 1.30 67.71 2.78 1.21 14.56 135 o 1.35 1.00 86.42 3.90 1.97 112.82 3.01 1.24 24.26 180 o 1.11 0.57 53.67 3.69 1.89 100.27 3.03 1.16 24.87 Table 3-1. Summary of mean ( µ) and variance ( σ), of reading times, and percentage deviation from mean reading time for un-rotated text of the same type

Below, find results for required to read the stimuli under each condition, broken down by screen-quadrant. Our results correspond with those of Koriat and Norman (1985) in that, for the word and phrase conditions, the worst mean performance for reading stimuli in the upper quadrants of the screen was 135 o in the upper-left quadrant, and -135 o in the upper- right quadrant. Since the participant was positioned at roughly the centre of the table, these orientations represent text written bottom-to-top, and away from the participant.

Figure 3-5. Box plots of time, in seconds, required to read a single word at each position and orientation. Outliers (>1.5 * IQR) removed

Figure 3-6. Box plots of time, in seconds, required to read a 5-6 word phrase at each position and orientation. Outliers (>1.5 * IQR) removed

Figure 3-7. Box plots of time, in seconds, required to read a 6-digit number at each position and orientation. Outliers (>1.5 * IQR) removed

3.5.7 Discussion

These results confirm our primary hypothesis: that although significant, the effects of orientation on reading speed are not as large as previous work reported. Where Tinker’s (1972) test found a penalty of over 200% for reading of a paragraph once text is oriented at 90 o, we found penalties of only 26%, 54%, and 17% for short words, phrases, and numbers respectively. Although our results do not refute the notion of a performance cliff at 60 o as was found by Koriat and Norman (1985), what is clear is that if such a cliff exists, it is not nearly as large an effect as they reported when performing our tasks.

We attribute the differences in our results to two key experimental differences: the experimental condition and the task. Unlike the previous work, our participants were free to move their bodies and orient their heads to aid in reading as they would be able to do in real use of tabletop groupware applications. Furthermore, our task required the reading and understanding of words and phrases that were consistently and reliably “real”, so participants could trust their first-glance understanding of the text, rather than scrutinize as was required in the Tinker (1972) and Koriat and Norman (1985) experiments.

63 We were surprised by the finding that 6-digit numbers suffered less from rotation-speed effect than did words and phrases. In our post-experimental interviews, several subjects reported that, when reading numbers, they would begin to type almost right away, and rely on their visual memory of the stimulus as they continued to enter the remaining digits, rather than carefully scanning each digit. We were unable to find references to this behaviour in previous work, which suggests an area of future research.

Having found these baseline results for text orientation, we turned our attention to new but related issues.

3.6 Experiment 2

3.6.1 Goals and Hypotheses

A common task in many applications is the serial search: examining many on-screen elements in search of a desired target. Given the results of the previous experiment, we wished to examine the effect of orientation of target and distracters on efficiency in conducting a serial search. We hypothesized that although the amount of rotation of the target and distracters would have a significant effect on performance, the degree of this effect would be less than previous experimental results might suggest. We also hypothesized that a field with more greatly rotated targets and distracters might provide more visual cues, and thus aid in learning. We believed that, with repeated searches of the same field, this learning would mean that search times for the all-orientations condition would be reduced more than those of the no-orientations condition.

3.6.2 Apparatus

The apparatus used was identical to that in Experiment 1.

3.6.3 Participants

Nine participants, recruited from undergraduate and graduate courses, volunteered for the experiment. 7 were male, 2 were female. 6 spoke English natively, while the remaining 3 reported to be excellent readers. Participants received no compensation.

64 3.6.4 Procedure

Participants were presented with a search field of randomly positioned 5-6 letter words, each suffixed with a colon and an Arabic numeral between 1 and 3 (eg: “silly : 2”). For each trial, the position, rotation, and text of each of the words remained the same, but the suffixing numeral was randomly changed. For each trial, participants were told to search the field for a given word (different for each trial), and indicate when they had found it by entering its suffixing numeral. Three datasets were used, which varied only in the degree of orientation of the constituent strings. In the first dataset, each of the strings was presented “right side up”. In the second, strings were drawn at randomly assigned rotations of -45 o, 0 o, and 45 o. In the third, strings were randomly assigned rotations of all 8 multiples of 45 o. The following Figures demonstrate the three treatments:

Figure 3-8. Serial search task with all targets oriented at 0º

Figure 3-9. Serial search task with targets oriented at -45º, 0º, and 45º

Figure 3-10. Serial search task with targets at all 8 orientations

Initial instructions were given to the participant as follows:

This experiment is broken into three parts. For each part, you will, 72 times, be presented with a field of strings, each ending with a ":" followed by a number. Each of the 72 times the field will be the same, except that the numbers at the ends of the strings may change. Each time, you will be asked to find a particular string, and indicate to the computer which number it ends with. If you give an incorrect reply, the system will ask you to try again.

Please relax between presentations, but once you have started a trial, find the string and give the input as quickly as you can.

You will be given the opportunity to rest between each presentation. Please do not take long breaks while you are completing a block. You may rest as long as you like between blocks. The string to find was presented first, and participants then pressed a keyboard button to display the field and begin the timer. Figure 3-11 demonstrates this sequence.

Figure 3-11. Left: Screen displaying search target to the participant. Right: the search field presented once the subject presses the “space” key, and timing starts for the search

66 Timing began when the field was first displayed, and stopped when the subject entered a numeral. Participants were required to enter perfect input, and were told to correct their entry if it was incorrect. Trials with erroneous entries that were subsequently corrected were not included in the analysis.

To ensure that participants understood what was required, they were allowed a practice dataset consisting of several typical searches. They were instructed to conduct as many as was required to become familiar with the apparatus.

3.6.5 Design

Searches were conducted for each of the three datasets. For each dataset, an identical field of 24 5 to 6 letter words was repeatedly presented. Participants were asked to search for each string in the field on three different occasions within each dataset, resulting in 3x24 = 72 searches per dataset.

The order of presentation of the datasets was counter-balanced between participants using a Latin-squares design. The assignment of words to treatment, the order of searches, and the makeup (position and orientation) of the field was randomized between participants. In summary, the design was as follows:

3 datasets (no rotation, small rotation, complete rotation) x 24 strings per dataset (position and orientation randomized) x 3 searches per string (search order randomized) x 9 participants = 1944 searches conducted in total.

3.6.6 Results

As before, trials with erroneous input were discarded from the analysis. There was no significant effect for treatment on error rate, with each treatment yielding an error rate of 0.01 across all participants, indicating that orientation did not mislead the subjects. Repeated measures ANOVA found that the orientation of the target and distracter words did not have a statistically significant effect on the search time in the some-rotations condition (F 1,8 = 0.836, p = 0.387 – the lower-bound result was used because orientation was non-spherical, using the Greenhouse-Geiser method), while it was significant in the

67 all-rotations condition ( F7,56 = 2.883, p = 0.012), though pair wise means comparisons revealed that the search time for the zero and some rotation treatments were not significantly different from one another. As we hypothesized, the difference in performance time between the all-rotations treatment and the others was less dramatic than might have been expected: the mean performance times for the search tasks were 3.3, 3.4, and 3.9 seconds for the no-rotation, small rotation, and complete rotation tasks respectively. Figure 3-12 is the box plot for search time for each of the three search tasks.

Figure 3-12. Box plot of time required for the serial search under each of the three target / distracter orientation conditions: 0: all oriented towards the user, 1: all oriented at -45 o, 0 o, or 45 o, 2: all oriented at one of the 8-compass positions. Outliers (>1.5 * IQR) removed

Learning of the field did take place: we broke the results in to seven blocks, and found that block number had a significant effect on the time required to conduct it ( F6,48 = 4.133, p = 0.02). Contrary to our hypothesis, however, there was no interaction effect between search number and dataset ( F1,8 = 1.419, p = 0.268 – the lower bound result is reported because type was non-spherical, using the Greenhouse-Geiser method), indicating that the degree of orientation did not affect the learning of the field.

3.6.7 Discussion

In the second experiment, our primary hypothesis from page 64, that orientation would have a significant but minor effect on serial search, was confirmed. We found that the average search time to find a target word among 23 distracting words suffered only a (statistically insignificant) 3% increase between the zero and some-rotation conditions, and only an 18% increase between zero and all-rotations conditions. 68 The effect of orientation on task performance is even smaller in the serial search task than it is on the reading task. If we consider the results of single-word reading from the first experiment, and weight the difference in speed of reading from the zero-rotation condition by the proportion of strings in the search field that were at that orientation, we find that a speed difference of 5% in the -45 o to 45 o condition (1/3 x 0% (0 o) + 1/3 x 7.98% (-45 o) + 1/3 x 5.93% (45 o) = 4.6%), and a speed difference of 34% (1/8 x 64.7% (- 135 o) + 1/8 x 26.6% (-90 o) + … 1/8 x 53.7% (180 o) = 33.89%) in the all-rotations condition, would be expected, versus 3% and 18% determined empirically. Thus, serial search of a field of short words is affected less than is the reading of an individual word rotated by the same amount. This finding might be of significance to those conducting general research in the serial search task.

Although the hypothesized learning did take place as participants continued to make 72 searches within the same field, there was no difference in the learning effect due to text orientation. This was contrary to our secondary hypothesis, which was that when strings were presented in varying orientations, learning of the locations of those strings would be enhanced over a field of uniform rotation. With only three repetitive searches of the same item within a given field and all done within a short timeframe, our experiment was able to evaluate only immediate learning and not long-term recall. It is possible that varying orientations may aid in longer term recall of items, and is worthy of further research.

3.7 Implications for design

The results of the first experiment are promising for tabletop designers: the reading speed penalties associated with oriented text are significantly lower than was suggested in the previous research. Of course, this data cannot be used in isolation for system design: user preference will continue to play a role. Building on Kruger’s examination of orientation on a physical tabletop, Morris et al.’s exploration of centralised versus replicated controls provided empirical evidence that users have strong preferences for text oriented towards them (Morris et al. 2006b). The results of the present work, therefore, cannot be interpreted in isolation to determine when textual information needs to be oriented towards each user, leveraging one of the techniques described earlier.

69 What the results do tell designers, however, with high precision, is exactly when, from a performance standpoint, it might become necessary to provide individually rotated copies of textual information. Although useful to the design of any multi-surface environment, this information might become especially important in the performance of time-critical tasks where the small differences in performance can have an important consequence to the success of the task. If, for example, two users are playing a competitive game while seated at the table, one user might be significantly disadvantaged because of the orientation of the text on the display. Designers such a game could utilize the results of this work to determine when such an advantage would be gained, and change either game rules or the interface to remove this advantage.

3.8 Avenues for Future Research

In addition to the design insights described above, our experiments reveal several avenues for further research.

The most apparent avenue is the opportunity to further explore the issue of reading performance, given a multitude of other variables. First, whether graphical rendering issues, such as font selection or aliasing introduced under rotation, has any effect. In particular, an examination of fonts that are more robust to orientation would be a compelling area of research. Second, the experiments we have performed provide data on only particular types of text: short words, phrases, and numbers. Other types of text might provide for different types of performance – proper nouns or other less common words might prove to be more difficult to read under orientation, as might paragraphs of text.

This work has already been extended to other dimensions: Grossman et al. (2007) extended our first experiment to volumetric displays, presenting text rotated around the two remaining spatial dimensions. Combining their results with those in this paper, they then created an algorithm to determine when duplicate copies of labels would become necessary depending on two parameters: the spatial orientation of the object being labelled, and the spatial position of the users attempting to read those labels. Future researchers might consider adapting their algorithm for use on the tabletop. A first step

70 would be to set their ‘pitch’ and ‘yaw’ parameters to ‘0’, but this simple approach might not necessarily achieve optimal results.

Also open to further research is informing the other side of the trade-off we have discussed: user preference. How strongly do users prefer text oriented at ‘optimal’ orientations; how far from these optimal orientations can the text diverge; and is this preference is sufficiently strong to overcome the limitations imposed by the potential solutions we have described. A first step down this path has already been published: Morris et al. examined user preference for form factor, including tabletop and traditional display, in an active reading task (Morris et al. 2007). Their work did not include the issue of orientation, but future researchers might consider it as a framework for examination of preference.

Next, although our second experiment did not find our hypothesized benefits for visual search, the results do suggest an area requiring further exploration: for visual search tasks on the tabletop, the orientation of the elements on the table will have a significant effect on search times. How, then, can interfaces be designed in order to reduce these effects when multiple users are seated around the table? Forlines et al. have already begun to explore this issue, and have found promising results for teams conducting visual searches while working around a table (Forlines et al. 2006a).

Having examined the issue of reading in table-centric environments, we turn our attention now to the perception of graphical elements in such environments.

71 Chapter 4: Perception of Elementary Graphical Elements

Information visualization relies on encoding numerical values as graphical features. In order for data to be accurately decoded by readers, it is essential that they are able to precisely perceive the relative magnitudes of those features. As we have seen, information displayed in a table-centric space is often viewed under three types of rotation due to non-optimal viewing conditions: the angle of the table relative to the eye, the angle of the table content relative to the user, and the angle of the vertical displays relative to the user. Viewing graphical imagery at these angles results in a distortion of the retinal image.

Humans clearly have some ability to compensate for these visual distortions. We are, for example, able to recognize objects under rotation (Shepard and Metzler 1971), and perception of shapes drawn on a plane remain constant under rotation (Pizlo 1994). It is unclear, however, how much of an impact the second type of distortion – due to variance in viewing angle of the table relative to the eye – may have on the accuracy of perception of elementary graphical elements employed in information visualisations, as described by Bertin (1977) and Cleveland and McGill (1984, 1985). In this chapter, we seek to examine and quantify those effects. We conducted a pair of experiments designed to measure them, and to test our hypothesis that the viewing angles in a table-centric environment will induce distortions in the perception of information graphics. Because there is a great deal of domain terminology which comes from the previous efforts, we will begin by reviewing related work, and defer a more detailed description of our experiments.

4.1 Related Work

Research from three fields is relevant to our work. First, we examine work of theorists in the basic elements of information visualization. Next, we survey research on perception of objects under rotation. Finally, we consider the design of experiments involving magnitude perception.

72 4.1.1 Basic Visual Elements

The basic elements of information visualization have been seminally and distinctly defined by two works, initially, by Bertin (1977), and then by Cleveland and McGill (1984, 1985). Each of these authors believed that every visualization of information required that the data be encoded using one or more of the visual variables (Bertin) or elementary perceptual tasks (Cleveland & McGill) which they described. Although each defined their own list of these, it is Cleveland and McGill’s list (Figure 4-1) which is most relevant to our work. This is because they authors recognized that some of the broader categories of Bertin’s list contained elements to which we have varying acuity, and so split those categories along perceptual lines. The effectiveness of their categorization was demonstrated by their experiments of perceptual ability. Through them they were able to order their list in terms of how accurately a viewer can decode quantities represented using each visual variable (Cleveland and McGill, 1984).

An example of the splitting of Bertin’s variables along perceptual lines is the dimensional breakdown of Bertin’s size variable. Cleveland and McGill split Bertin’s size between three elementary perceptual tasks: length , area , and volume. This was done because, as was found in a study by Baird (1970), each is perceived with varying accuracy. Since we are concerned with users’ perceptual ability in evaluating visual variables, we will use Cleveland and McGill’s definitions, although our findings should be equally applicable to those seeking to apply Bertin’s work to tabletop and multi-surface environments, and more generally to any designer of information visualizations.

Figure 4-1. Cleveland and McGill’s elementary perceptual tasks. All visual representations of quantitative information require decoding using one or more of these

73 It should be noted that each of Bertin’s, and Cleveland and McGill’s works offers a great deal more than the definition of these variables. They provide guidance for visualization designers wishing to leverage principles of perception, and describe other stages of perception of information visualization. Readers might wish to examine follow-up works by Green (1996) and Carpendale (2003), and applications of their results by Beattie and Jones (2002) and Nowell (1997).

4.1.2 3D and Perception Under an Angle

The work of psychophysicists in the areas of perception of objects at an angle, and of 3D objects, is vast. Providing a complete review of work on human perceptual mechanisms is beyond the scope of this chapter. Some of the work from this area, however, is of particular interest.

It is likely that in order to perform magnitude perception of visual elements presented on a surface angled away from the user, such as a tabletop, some amount of mental rotation must be performed. Research on rotation of objects around the axis perpendicular to the display surface has shown that there is a linear relationship between the angle of rotation and the time required to perform it (Corballis and McLaren (1984), Corballis et al. (1985), and Shepard and Metzler (1971). Also of interest is that all of gender, level of spatial ability, and visual field of presentation seemed to affect the speed of rotation, although they do not seem to affect accuracy (Voyer and Brydon, 1990). Our desire to measure accuracy, rather than time, meant that we did not perform any screening for these in recruiting our participants.

Finally, psychophysicists have examined the issue of the effect of viewing angle on the perception of elementary graphical objects. In particular, the work of Schneider et al. (1978) examined the effect of viewing angle on the perception of length. The angles of interest in their work, however, were between the line and the center of the retina. We wished to examine in-situ perceptual accuracy, and so our experimental participants were free to move their eyes. Thus, Schneider et al.’s results are not directly relevant.

74 4.1.3 Magnitude Perception

Models for the perception of relative magnitudes of a variety of physical phenomena provide an understanding of the internal perceptual processes in the brain (for example, Fechner (1860) and Stevens (1957). While valuable, the details of these models are beyond the scope of the present work, and we refer the reader to Gescheider (1988) for a thorough overview. Of more direct relevance to us is the ongoing debate as to how best to measure a participant’s perceived magnitude of a given stimulus. Much of this work is similar in technique to the experiments performed by Cleveland and McGill (1984, 1985): a modulus object is shown to a participant who is then asked to evaluate the relative magnitude of subsequent objects as a fraction or ratio to that modulus. This technique, however, measures the reported relative magnitude, meaning that the accuracy of the collected data depends on participants correctly converting their perception into a numerical quantity. There is ongoing debate of how best to collect the actual perceived magnitude, including applying models to results to avoid relying on participant reports (for examples, see Ellermeier and Faulhammer (2000), Narens (1996), and Zimmer (2005). In the present work, we avoid this controversy, since our goal is not to directly measure perceived magnitude, but rather the change in perception between multiple viewing angles. We require only that this change be present in whatever result we collect, and not that the reported results precisely match the actual perception. With this in mind, in designing our experiments, we have not attempted to model the underlying perception, as was done by Narrens (1996), but have instead relied on the reported results, as was done by Cleveland and McGill (1984, 1985).

We model our experimental design on that employed by Cleveland and McGill, with one important exception. In their work, they report that as the distance between modulus and stimulus objects increases, the error of the participants’ estimate of magnitude increases (Cleveland and McGill, 1984, 1985). What may be a flaw in their experimental design, however, is that distance is correlated with the order of magnitude comparison against a given modulus. More recent research, reviewed by Gescheider (1988), has shown that subsequent comparisons against a single modulus are not generally independent, and so error tends to increase over time. As a result, it may be the case that the effect for

75 distance reported by Cleveland and McGill may actually be an effect of the ordering of comparisons. In our experiment, we have taken steps, described in the Procedure section, to eliminate this confound. Also relevant is the independence of magnitude estimations. Repeated studies have shown that changes in one sufficiently different property do not affect the perception of a second visual property of that object, as reviewed by Green (1991). This is especially relevant to the present study, where we will be comparing magnitude perceptions across various viewing angles of a display. This past work suggests that the brightness and colour representations of on-screen objects, which might change with the viewing angle, will not affect the perception of the magnitude of those objects.

4.1.4 Distance Estimates and Virtual Environments

Although there is a multitude of particular magnitude estimations which have been explored by researchers, one in particular is especially relevant to the present work: distance estimation in virtual environments. This has been investigated in both human factors and psychology.

When navigating virtual environments, the scene shown to the user is as might be captured by some theoretical ‘camera’, under a projection of the 3D scene on to the 2D display. A slew of researchers have investigated what angle that camera should have with respect to the scene’s ground. Angling the camera orthogonally, as in a map, results in all scenery being collapsed, giving few cues as to the heights of virtual objects. This view, however, provides the least amount of distortion of the relative distances of objects in any direction along the ground plane. At the other extreme, placing the camera on the ground plane itself, provides the most true representation of the relative heights of objects in the scene, while depriving the viewer of many common depth cues.

This issue has been explored thoroughly by Kim et al. (1987, 1991), Yeh and Silverstein (1992), St. John et al. (2001), and by Adam (2002). Their efforts have focused on the position and angle of the virtual camera, as well as rendering elements and cues about the 3D space. While these efforts are similar to the present research, in that they are concerned with the perception of a space while modulating the viewing angle, they are

76 distinct from the present work because they deal with a virtual viewpoint, with the user generally viewing the scene with a traditional viewpoint on the display device. The position of this virtual viewpoint affects the rendering of on-screen imagery: viewing an object from a camera at 45 degrees versus 30 degrees results in a difference of how the object is actually rendered on the screen.

In our work, we focus on lower-level issues: how does the angle of an actual, physical display affect the perception of graphical elements that are rendered precisely the same . As a result, the applicability of the previous findings to the present work are limited.

4.2 Research Goals

Our ultimate goal is to provide guidelines for overcoming the visual distortion(s) introduced by working in a tabletop and multi-surface environment. As we have described, the effects of these distortions depend largely on the particulars of the graphical representation and the task being performed. The goal of the present work, therefore, is to provide a baseline for the effects of distortion on basic graphical elements, as a beginning to an understanding of how best to overcome those effects.

We have extended the work of Cleveland and McGill, by conducting two experiments evaluating the performance of their elementary perceptual tasks in these environments. In the first, we sought to determine which elementary perceptual tasks are appropriate for use in a single display environment. We believed that, when the display is tilted close to or on the horizontal, the distortion introduced by the perspective change between two graphical elements displayed at different up/down distances from the viewer may impair users’ abilities to perform an accurate estimate of their relative magnitudes. The second experiment extends the goals of the first by examining which basic graphical elements work best for comparisons across two displays: one oriented vertically, and the other oriented horizontally as a tabletop. We believed that since the difference in angle of the visual variables to the user is even greater between vertical and tabletop displays, so too would the level of impairment of perception.

Tufte (1983) claimed that visualizations of quantitative information must afford two levels of use. First, a quick glance should provide an immediate, if coarse, understanding 77 of a data set. Second, finer detail should be available upon closer scrutiny. It should be noted that, even when encoding information in such a way that it requires the execution of what we have found to be a less preferred elementary task, it is possible to provide additional visual information or cues that would allow the data to be decoded. This could be accomplished as simply as by providing numeric values written beside each visual encoding. The goal of the present work, therefore, is not to claim that it is impossible for a reader to perceive values encoded with these variables. Rather, it is to provide guidance for designers in selecting the variables which best facilitate the first of Tufte’s requirements: a quick glance will provide coarse understanding and comparison of relative values.

4.2.1 Terminology

Before discussing the details of our work, it is helpful to establish a set of terms for the various presentations of objects and imagery that we will use in our experiments. When discussing the orientation of the physical display, we will use the terms vertical for an upright display, and tabletop for a display laid flat. The display’s orientation angle will range from 0° (tabletop) to 90° (vertical). The position of on-screen imagery will be defined by where it appears on the left/right (on-screen x-axis) and up/down (on-screen y- axis) axes. Figure 4-2 illustrates these dimensions:

Figure 4-2. The relative position of imagery on-screen is measured by its left/right distance (left) and its up/down distance (right)

78 The orientation of on-screen imagery will be defined as upright when aligned with the up/down axis, and lateral when aligned with the left/right axis. Figure 4-3 illustrates the two on-screen orientations:

Figure 4-3. The orientation of on-screen imagery – as demonstrated here by two on-screen lines – is either upright (left) or lateral (right)

Up/down and left/right distances between objects on an angled display are distinct: increases in up/down distance could increase the relative visual distortion between two objects depending on orientation of the display, while changes in left/right distance may not. As we have described, imagery viewed in a table-centric interactive space is subject to distortions imposed by three different viewing angles. In order to better understand the experiments in this chapter, we now frame them in the context of which of the three inherent distortion angles each of them sought to explore.

4.2.2 Angle: Vertical Display to the User

As was previously described in Chapter 3, the angle of the vertical displays to the user is typically not a problem when viewing content, because the angle is marginalised when the user simply turns her head towards the display. Of concern, however, is whether the comparison of visualisations across the table and the vertical display will be subject to distortion, because of the differing viewing angles to the user. In the second experiment, we examined the issue of inter-display comparison, by asking participants to perform such comparisons across two vertical displays and across a tabletop and vertical display.

79 4.2.3 Angle: Table to the Eye

The angle of the table to the eye is the primary concern of the first of our experiments. As was described previously, display content is distorted when displayed to a user on a tabletop display, because that display recedes away from the viewer, as illustrated in Figure 4-4.

Figure 4-4. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Right: the same map, as it appears to the user of a tabletop display (repeat of Figure 1-3)

Shape constancy is available to the user to help them to understand these distortions. In our first experiment, we will investigate and quantify the accuracy of this ability to process information graphics, such as shown in Figure 4-5:

Figure 4-5. Left: two horizontal lines of equal size as they appear to a user on a vertical display. Right: the same scene, as it appears to the user of a tabletop display

80 4.2.4 Angle: Table Content to the User

The angle of content to the user is not, in isolation, of concern: previous research by Cleveland and McGill (1984, 1985) presented elementary graphical elements at random angles, and found no effect for the orientation of those angles on the accuracy of magnitude judgements. As we see in Figure 4-6, however, the angle of the content to the user does not occur in isolation – imagery on the table is simultaneously subjected to this distortion and to that imposed by the angle of the table to the eye, described above.

Figure 4-6. Left: a map of Cambridge, as seen a tabletop by a user seated at the table. Right: the tabletop as seen by another user seated at the same table. (repeat of Figure 1-4)

As a result, the angle of the table content to the user may well have an effect. Horizontal lines, for example, are subject to less distortion than vertical lines as we see in Figure 4-7:

Figure 4-7. Left: two vertical lines of the same length as seen by a user seated at a tabletop. Right: lines of the same length presented laterally have less distortion

The angle of content to the user may have a significant effect on perception. In our experiments, we distinguish between information graphics presented upright and laterally, so that the effects of the angle of the table content to the user can be measured.

4.3 Experiment 1: Single Display

4.3.1 Goals and Hypotheses

In this experiment, we wished to examine whether compensatory processes within the brain would allow for uniform perception of visual variables across a tabletop surface. In order to evaluate this, we asked several participants to make magnitude comparisons of pairs of values of various visual variables on a display oriented at four different angles, and measured the accuracy of their estimates. Our hypotheses are as follows:

81 H1: As the display is tilted, the accuracy of relative magnitude judgements decreases H2: The up/down distance between objects is positively correlated with the increase in error in magnitude judgements due to display angle. H3: Different visual variable types have differing increases in the error in judgements. H4: Lateral presentations of objects experience less error in magnitude judgements due to display angle than upright presentations. H5: When the up/down on-screen positions of the modulus and stimulus objects are the same, judgement accuracy will be consistent across display angles. The distinction between H2 and H5 is important: our belief is that, for comparisons where the amount of perspective distortion is consistent between the modulus and stimulus, there will be no increase in error as the display is angled, since the distortion will affect the two equally.

Additionally, as was previously stated, we hypothesize that the effect of distance on error reported by Cleveland and McGill may have been due to an experimental confound. As described in the Procedure sections below, we have taken steps to eliminate this confound. Subsequently, we have an additional hypothesis:

H6: There will be no effect for left/right distance on the accuracy of magnitude perception. This contradicts the conclusions of Cleveland and McGill, but is based on recent results in magnitude perception reviewed by Gescheider (1988).

4.3.1.1 Visual Variables

The visual variables in our experiment (Figure 4-8) are a subset of those examined by Cleveland and McGill (1985), which differ from those in their earlier work (1984) (Figure 4-1). Specifically, between these two publications, curvature was removed, and direction was replaced with slope . To maintain consistency, we based our list on that in their more recent publication (1985). We asked participants to report their perceived magnitude of each of length , angle , position , slope , and area . Unlike Cleveland and McGill, who presented their visual variables at random angles, we chose to control this

82 variable, and presented each of length , angle , and position in two orientations, one laterally and the other upright. As explained previously, this was done because we hypothesized that lateral presentations would suffer less visual distortion when the display is laid flat. Thus, this allowed us to test the effect of the angle of display content to the user independently from the angle of the eye to the table.

Like Cleveland and McGill, we chose not to test the visual variable colour saturation . This was done for two reasons. First, the presentation of colour to the retina is not distorted by varying viewing angle, and is unlikely to be perceived differently on angled displays. Second, display technology is such that accurately reproducing colours across viewing angles is difficult. We also chose not to examine position, same scale , since this would not lend itself to up/down position modulation, and because their results suggest that the differences between it and position , different scale are not substantial.

Length (upright & lateral) Angle (upright & lateral)

Position of dot from bottom (upright ) Slope Area or left-side (lateral) of box Figure 4-8. The visual variables in our experiments; a subset Cleveland and McGill (1985)

4.3.2 Apparatus

Participants were seated at a table, which was 66cm off the floor, on which was positioned a flat panel 43cm x 27cm NEC MultiSync LCD 2070WNX display with 1680x1050 pixel resolution. Each visual variable occupied a maximum on-screen size of 10cm 2. The display was in the “portrait” orientation relative to the user, so that it was taller than it was wide, and was attached to a mount which could reposition the display at 90° (vertical), 60°, 30°, or 0° (tabletop). To minimize the impact of display orientation, contrast and brightness were adjusted, and all imagery was displayed in black on a white 83 background. When repositioned, the display was moved so that the top of the display, when angled as a tabletop, would be in the same location as the bottom of the display when vertical. This meant that the maximum distance that an object could appear on- screen from the user’s eye was approximately the same in these two conditions, so as to eliminate any confound between distance and viewing angle. Figure 4-9 illustrates the physical experimental apparatus.

Figure 4-9. The apparatus as employed in our experiment. The user was seated in front of the display (left), which was precisely oriented at 4-different angles

Participants were instructed to position their chair comfortably, and to keep it in the same position throughout the experiment. As was done in previous experiments by both Cleveland and McGill (1984, 1985) for magnitude perception work, and by Wigdor and Balakrishnan for tabletop work (2005), the relative position and angle of the user’s head was not controlled, to allow them to perform whatever compensatory positioning of their head they felt was necessary, thus better approximating real-world conditions.

The apparatus was placed in a dark room, with the only light sources being the displays and a small light used to illuminate the keyboard, which was positioned carefully to ensure that it caused no glare on the display.

84 4.3.3 Participants

Twelve participants (7 men and 5 women) between the ages of 22 and 47 were recruited from the local community by posting on an Internet bulletin board. Participants had completed at least one year of an undergraduate degree. Because the research was conducted at a corporate lab, (and unlike in the previous experiment), participants were given $30 in compensation for their time. Payment was entirely independent of performance, and so we believe it did not at all affect the results of the experiment.

4.3.4 Procedure

Our experiment was similar to Cleveland and McGill’s, which presented a group of four numeric values encoded using a single visual variable type (1985). In their study, participants were asked to compare each of three of the objects with a fourth, largest, modulus object, by entering each of their magnitude as a percentage of the modulus’. As we have described, and which was reviewed by Gescheider (1988), multiple comparisons against a single modulus are not independent. Researchers, such as Galanter (2004) and Green and Luce (1974), have suggested that each magnitude comparison can be made independent by forcing the participant to revisit the modulus, or by showing a new modulus for each comparison. Although we mimicked all other aspects of Cleveland and McGill’s experiment, we modified it such that a new, randomly determined modulus was presented for each comparison. We used the relative placement of objects from their design, but repeated their pattern three times on the display, so that there were 3 modulus and 9 stimulus positions. This was done to increase up/down distances between moduli and stimuli for larger visual distortion. Figure 4-10 illustrates the locations:

Figure 4-10. (Left). Experimental display in Cleveland and McGill (1985). (Right). Display used in the present experiment. Modulus locations (M), and stimulus locations (S)

85 Participants were briefed on the task, and shown each of the 8 types of visual variables used in the study. To ensure understanding, they were asked to practice interactively with the experimenter: after reporting their response, they were told the correct response for each trial. During the actual experiment, no accuracy information was given.

As was done by Cleveland and McGill (1985), participants were presented with a series of pairs of visual variables, and asked to report the magnitude of the stimulus as a percentage of the modulus. Figure 4-11 illustrates the screen as seen by the participant:

Figure 4-11. Examples of on-screen stimuli as presented to the participant. From left to right: position (upright), line length, and angle

Also like Cleveland and McGill (1985), the amount of time participants were allowed to view the stimulus was not limited, ensuring that user responses would be as accurate as possible. When the participant began to type, the on-screen objects were hidden, replaced with their typed text. They could recall the objects for comparison by pressing escape; these trials were marked as erroneous. Each session lasted about 2 hours.

4.3.5 Design

Each participant was presented with only 4 of the 8 visual variables included in our experiment. This was done because a full experiment with all 8 variable types would have extended the session beyond a reasonable length. The pairings of variables were balanced so that, if a participant were to judge a particular variable, he would judge both the upright and lateral orientations of that variable. The remaining pairings of the visual variables were fully counter-balanced between participants. 86 Three on-screen modulus positions were employed, along with 9 on-screen stimulus positions. As was done by Cleveland and McGill, the relative position of these pairings was measured from the centre of the stimulus. For each visual variable and for each display angle, the participant completed three magnitude estimation trials: a total of 81 comparisons. These 81 comparisons were done in a block, so that the participant was not asked to jump back and forth between variable types. At the beginning of each block, on- screen instructions informed the participant of which visual variable would be presented, and they were required to complete 3 practice trials before beginning the block.

For each of the 4 display angles (90° vertical, 60°, 30°, and 0° tabletop), the participant completed one of these blocks of 81 trials for each of the four visual variables. The ordering of display angles, and the order of visual-variable types within each position, was controlled with a Latin-square. The experimental design can be summarized as: 12 participants x 4 display angles x 4 visual variables (per participant) x 3 modulus positions x 9 stimulus positions x 3 magnitude estimates = 15,552 total comparisons

4.3.6 Results

Our dependent variable was error , calculated as in Cleveland and McGill (1984, 1985) as: error = | judged percent – true percent |

In all cases, practice trials, and trials with a completion time or error of more than two standard deviations from the mean were excluded (131 trials). A computed independent variable was up/down distance , the distance between the modulus and stimulus objects along the vector perpendicular to the edge of the display closest to the user (always one of 0cm, 14cm, 28cm). This measure is independent of another computed variable: the left/right distance between those objects.

87 As was expressed in H1, we anticipated that as the display was tilted away from the user, and therefore as the visual distortion of the on-screen objects increased, we would find a corresponding increase in error. Looking only at the effect of display angle on error, we did see this effect: for each of 90°, 60°, 30°, and 0°, the mean for error across all visual variables was 8.9%, 8.3%, 10.5%, and 12.0%, F 1,5=277.303, p < .0001 (lower-bound test indicated due to lack of sphericity, using the Greenhouse-Geiser method). The levels for each of 30° and 0° were significantly different from each other, and from 60° and 90°, which were not significantly different from one another. Thus, we confirm H1.

Although this overall difference is statistically significant, such a small increase in error between display angles for a first-stage analysis of a particular visualization is likely to be acceptable. However, as we analyze the source of this error, we see that the decrease in accuracy is actually much more significant in several cases. In particular, we are able to confirm H2 – that up/down distance is positively correlated with the significance of this increase in error. As we see in Figure 4-12, for the 30° and 0° display angles, the amount of error increased with up/down distance , while this error remained insignificantly changed for 60° and 90°. This suggests that the perspective distortion introduced by the display being angled has caused this difference (F 1,5 =92.765, p < 0.0001 – lower-bound test reported due to lack of sphericity, using the Greenhouse-Geiser method).

In the 0°, or tabletop condition, the error increases from 8.69% for up/down aligned objects, to 12.61% for objects separated by approximately 14cm up/down, to 15.77% for objects separated by 28cm up/down.

88 16 14 12 10 8 6 90 degrees 4 60 degrees 30 degrees 2 0 degrees Error : l Judged - Actual | 0 0cm 14cm 28cm Up/Down Distance Figure 4-12. Mean size of error by up/down distance between objects for each display angle across all visual variable types.

We are also able to confirm H3: that some visual variables are more subject to this error than others. Post-hoc pairwise tests found that, in all cases, up/down distance did not significantly affect error for the 90° and 60° angles. For the remaining angles, however, all visual variables had significant differences in error between different levels of up/down distance , though to varying degrees. As we see in Figure 4-13, each of the visual variables was affected in how accurately it was perceived by the increase in perspective distortion. In Table 4-1, we see the precise mean error sizes for each. We can partially confirm H4, that variables presented laterally would suffer less when presented on a tabletop: post hoc pairwise means testing showed that, of the three variables presented both upright and laterally, two showed a significant difference between upright and lateral presentation. As seen in Table 4-1, length was not significantly different, while perception of both angle and position was significantly better at lateral than at upright presentations. We are also able to confirm H5, that in trials in which the modulus and stimulus are at the same up/down position, error will be consistent across display angles. For all visual variables, the level of error was not significantly different between any display angles for those trials where up/down distance between the modulus and stimulus objects was 0.

Finally, we are also able to confirm H6. For 7 of the 8 visual variables, we found that the left/right distance between the stimulus and modulus did not have a significant effect on error. The only exception to this was slope : for slope judgements, as left/right distance increased, so too did error . Post hoc testing revealed that each of these levels of error for

89 slope was significantly different from one another (mean = 13.42, 14.08, 15.29) for each of the three left/right distances). That this effect is not present for the other variables directly contradicts the work of Cleveland and McGill (1984, 1985).

25 Length (U) Length (L) 20 Angle (U) 15 Angle (L) Position (U) 10 Position (L) Slope 5 Area Error: | Judged - Actual | 0 0cm 14cm 28cm Up/Down Distance

Figure 4-13. Error size by up/down distance between compared objects for each visual variable type on a tabletop display

Up/Down Length Length Angle Angle Position Position Distance (Upright) (Lateral) (Upright) (Lateral) (Upright) (Lateral) Slope Area 6.68 7.34 8.93 8.68, 6.75, 7.71, 11.75, 11.64, 0cm (6.28) (6.22) (7.76) (6.95) (6.70) (9.12) (9.94) (10.47) 8.11 7.72 14.94 12.24, 12.90, 11.76, 17.88, 15.57, 14cm (5.38) (5.40) (10.07) (9.84) (6.58) (12.73) (10.69) (9.97) 9.60 9.24 20.10, 14.02, 17.92, 12.53, 25.60, 17.31, 28cm (6.52) (5.75) (9.96) (10.06) (9.29) (7.88) (12.12) (9.62) Table 4-1. error (mean, standard deviation) by up/down distance for each visual variable type, displayed on a tabletop.

Left/Right Length Length Angle Angle Position Position Distance (Upright) (Lateral) (Upright) (Lateral) (Upright) (Lateral) Slope Area 8.51 8.17 14.54 12.01 12.38 11.03 18.68 14.82 0cm (6.73) (5.80) (8.651) (9.02) (7.25) (10.30) (10.42) (10.05) 8.01 8.03 14.82 11.31 12.54 10.73 18.43 15.06 12cm (6.038) (6.03) (9.861) (9.12) (7.64) (11.89) (10.33) (10.43) 8.23 8.18 14.67 11.62 12.73 10.24 19.07 14.64 20cm (6.379) (7.10) (9.71) (9.28) (9.18) (10.54) (12.29) (9.32) Table 4-2. error (mean, standard deviation) by up/down distance for each visual variable type, displayed on a tabletop

90 4.3.7 Discussion

One of the more interesting results from this experiment is a contradiction with Cleveland and McGill’s (1984, 1985), in that we found that larger left/right distance between compared objects did not increase error. As we have discussed, we attribute this finding to our improved experimental design, which was based on findings in magnitude perception largely not available to Cleveland and McGill. As previously described, we suspect that the previous finding was due to an experimental confound.

More generally, our results suggest that certain types of information visualizations may need to be modified in order to be displayed on a tabletop, since, as we have seen, comparisons of some basic elements of those visualizations may be distorted when working on the table. In particular, we found that both of the angles of concern we were investigating, the angle of the table to the eye, as well as the angle of the display content to the user, had a significant effect on the accuracy of perception of elementary graphical elements.

A natural extension to the tabletop is to add vertical displays on which to show these visualizations, since comparisons of visual variables on the same, vertical surface would not be subject to the distortions we have found for tabletops. Before a multi-surface environment, however, we wished to determine first how well users of such systems will be able to compare visual variables between tabletop and vertical surfaces.

4.4 Experiment 2: Multi Display

4.4.1 Goals and Hypotheses

We extend our exploration from the single display study in Experiment 1 to begin to explore the issue of multi-display environments. We repeated a similar procedure from the first experiment, this time placing the modulus and stimulus objects on different displays kept immediately adjacent to one another. The display with the modulus object remained vertical, while the stimulus display was positioned either vertically (i.e., vertical+vertical condition) or as a tabletop (i.e., vertical+tabletop condition). Since this arrangement would result in an even greater difference in apparent angle between the two

91 objects, we believed we would find a corresponding increase in the amount of error in the reported relative magnitudes. Our hypotheses were: H1: There is an increase in error when comparing visual variable magnitudes across vertical and tabletop displays (i.e., vertical+tabletop) versus comparing on displays of a single orientation (i.e., vertical+vertical). H2: The error increase when comparing between displays is unevenly distributed across visual variable types. H3: The size of the error on the vertical+tabletop condition is larger than in the up/down distance = 28cm condition on the tabletop in the previous experiment, since the angular distortion is greater.

4.4.2 Apparatus

The software and visual variable rendering were nearly identical to the first experiment. We changed the physical apparatus so that the modulus was shown on one display, while the stimulus variable was rendered on an adjacent display. Both were rendered on identical Dell 2000FP displays, with the same settings for brightness and contrast. As in the first experiment, the apparatus was installed in a darkened room with a small light for keyboard illumination.

Unlike in the first experiment, the left/right and up/down on-screen positions of the displayed objects remained constant. The up/down positions of the two variables were aligned, and the left/right position was such that they were presented adjacent to one another. The display with the stimulus was placed either vertical or as a tabletop. The display was positioned such that the axis of rotation was the up/down centre of the visual variables, effectively fixing the physical position of the centre the variables. Figure 4-14 illustrates display angle and the on-screen positions.

Figure 4-14. Display conditions for our second experiment. (Left) Vertical+vertical. (Right) Vertical+tabletop. The 3D position of the stimuli is fixed between conditions 92 4.4.3 Participants

8 male undergraduate students, aged 19 to 26, from the local university community participated in the experiment. All had normal or corrected-to-normal vision, and were given $20 upon completion of their participation.

4.4.4 Procedure

As in the first experiment, participants were briefed on the experimental task, and shown each of the 8 visual variables used in the study. To ensure proper understanding, they were asked to practice interactively with the experimenter, and, after reporting their response, were told the correct response for each trial. During the actual experiment, no accuracy information was provided to the participant.

Each participant was presented with a series of pairs of visual variables, and asked repeatedly to report the magnitude of the stimulus as a percentage of the magnitude of the modulus. When the participant began to type, the on-screen objects would be hidden, and replaced with their typed text. We recorded the value of their response.

4.4.5 Design

Each participant was presented with both display conditions, with the ordering of the two balanced between participants. For each display condition, they were asked to make comparisons of all 8 types of visual variables, in blocks of 31 comparisons. The ordering of visual variable presentation was balanced with a Latin-square between display condition for the same participant and between participants. The study design can be summarized as:

8 participants x 2 display conditions x 8 visual variables x 31 magnitude estimates = 3,968 total comparisons

93 4.4.6 Results

As in the previous experiment, our dependent variable was error , as in the previous experiment, as: error = | judged percent – true percent | In all cases, practice trials, trials in which the participant called for the objects after they began to type, and trials with a time or error of more than two standard deviations from the mean were excluded. We analyzed the results using a repeated measures ANOVA.

There was a significant effect on error for type * angle on (F 1,7 = 28.201, p = 0.001 – type * angle was non-spherical, and so the more stringent lower-bound test was used, using the Greenhouse-Geiser method). As in the first experiment, we are able to confirm H1, that there is an increase in error between values of a visual variable at the same display angle vs. comparing on displays of differing angles.

Post hoc tests showed significant differences for each of the visual variables across the two display conditions. As is apparent in Table 4-3, and as expected based on the results of the first experiment, we can also confirm H2, that the increase in error when comparing across displays (both in the vertical+vertical and vertical+tabletop) is unevenly distributed across visual variables.

Interestingly, we are unable to confirm H3, that the increase in error for the vertical+tabletop condition would be larger than in the first experiment’s tabletop condition. Although the results for the vertical+vertical condition are nearly identical to that of the first experiment, the error size is, in many cases, actually lower for the vertical+tabletop condition in the second experiment than it is in the first experiment’s tabletop condition. In only one case, when the position variable is presented upright, is the error higher in the second study than it is in the first.

Display Length Length Angle Angle Position Position Condition (Upright) (Lateral) (Upright) (Lateral) (Upright) (Lateral) Slope Area

Vertical+ 5.06 5.98 8.01 6.61 6.55 4.58 7.67 7.84 Vertical (6.15) (6.02) (8.89) (7.73 ) (5.43) (5.45) (8.73) (10.32)

Vertical+ 5.99 5.37 8.65 7.92 11.03 6.20 12.10 8.68 Tabletop (7.89) (7.13) (10.05) (10.06) (11.35) (12.26) (14.68) (17.43) Table 4-3. error (mean, and std. Dev.) for both display conditions for each visual variable 94 4.4.7 Discussion

What is probably the most surprising result of the second experiment is that, for the tabletop condition, there is no increase in error over the results of the first experiment, despite similar results for the vertical condition. As with any result in these types of experiments, accurately attributing this finding to an underlying cause is challenging. A possible explanation is that participants were able to utilize the additional rotational cue of the display bezel to better perform the mental rotation necessary to make the comparison. A follow-up study could examine this and other potential cues in detail, perhaps providing designers with means of overcoming the effects we uncovered, and improving the reading of information graphics for both mixed-display and tabletop systems.

Also candidates for future study are extending this work to multiple vertical display orientations, very large displays, or volumetric displays, all of which may cause differing perspective distortion under certain circumstances. An important addition to this work will be the effects of rotation of objects around the axis perpendicular to the display surface, since this effect is also present in collaborative tabletop groupware. Ultimately, the development of a model for predicting the effects of all of these variables on perception accuracy would be of the most use to designers.

4.5 Implications for Design

From the results of these two studies, we are able to make several recommendations for the design of information visualizations in tabletop and multi-display environments.

4.5.1 Tabletop Visualization

For the design of graphical information visualizations for tabletop displays, we draw the following key results from our first experiment:

1. Perception of relative values of visual elements is less accurate on a tabletop when those presentations are not at the same up/down distance from the user. 2. The larger the up/down separation between elements on a tabletop, the less accurate comparisons between those elements will be.

95 3. Some visual variables are more accurately compared than others. Based on our results, the ordering from most to least accurate type is: length (l), length (u), position (l), angle (l), area , position (u), angle (u), slope . 4. Of the visual variables which are more robust for tabletop display, two (position , angle ) require that they be presented laterally relative to the viewer in order to maintain their robustness to reorientation. 5. If variables must be perceived both upright and laterally, position is more accurate than angle . 6. Larger left/right distances between compared elements do not yield an increase in errors, with the exception of the slope visual variable. From these results, we can make several design recommendations. The first, second and sixth results above suggest that items to be compared should be carefully positioned at the same up/down distance from the user, in order to equalize the amount of perspective distortion. With multiple users, it may be necessary to repeat information displays, or position them within the private area (as described by Scott et al. (2005) of the particular user for whom they are intended. These approaches would also conform with the fourth result, by avoiding undesirable orientation of the visual variables that are rotationally sensitive.

From the third result, we see that information visualizations that encode using length, distance, and angle are significantly less subject to error than those that make use of slope or area. From the fourth result, however, we see that, for information visualizations that encode two dimensions of information, position is not an appropriate variable, since upright and lateral positions would effectively be subjected to different scales (because of their degrees of distortion). This is seen in the graphs in Figure 4-15.

Figure 4-15. Values encoded in position (left) are less accurately perceived on a tabletop than those encoded in length (right)

96 For visualizations which attempt to use multiple visual variables to simultaneously encode multiple categories of data, it is especially important to present objects for comparison at the same up/down distance. An example of such a graph is a box plot, which use both position and length to encode information. When comparing multiple box plots on a tabletop, those displayed farther away from the user can be compared for their lengths, but not for the up/down position, to those positioned closer on the table. Thus, the size of inter-quartile ranges can be compared across multiple plots, but the actual bounds of that range will be inaccurately compared. Figure 4-16 illustrates a possible alternative design, which replaces the use of position with length judgments, creating a visualization more suitable for a tabletop display.

Figure 4-16. A traditional box plot (left), and an alternative (right): this box plot design replaces judgements of position with length for reading the interquartile range by connecting the ranges to the edges of graph with lines whose length will be judge

Finally, our findings for the accuracy of comparisons of slope and area values suggest strongly that they should not be employed on the tabletop. This is consistent with previous recommendations made by Cleveland and McGill, who suggest that every line graph should be accompanied with a second graph encoding the slope values of the first (Cleveland and McGill, 1985). Our findings match theirs, except that this second graph would best be designed to use length rather than position judgments, such as a bar graph.

The examples shown here are just a few of the implications of the application of our results to visualization design for the tabletop. Those seeking to design tabletop systems might be guided by these examples, as well as by seeking to apply our results to their own visualization designs.

97 4.5.2 Mixed-Display Visualization

In our second experiment, we found that errors of comparison of visual variables across displays were not as severe as those made directly across large up/down distances on a tabletop. We also found that one of the visual variables, position , is even less robust to differing orientations between displays than it is to large perspective distortions on the same display. Designers of systems which mix tabletop and vertical displays will need to apply the tabletop design results outlined above when designing for their tabletop surface alone. When designing software that is meant to be used on multiple surfaces simultaneously, the following findings of our second study may be useful:

1. Information visualizations should not be compared across display orientations. 2. If comparisons across display orientations are unavoidable, the first three in this ordered list by accuracy are far better than the rest: length (l) , length (u) , position (l) , angle (l) , area , angle (u) , position (u) , slope . 3. If a second variable is required to be perceived both upright and laterally, angle is more accurate than position , unlike on a tabletop. The key difference between designing information visualizations for tabletop and multi- surface environments is that, in the latter, position is even less accurately perceived than angle for upright presentations. Given this, in multi-surface environments where the variables will be seen both upright and laterally, angle should be used before position for encoding information.

4.6 Avenues for Future Research

In both experiments, because we did not wish to create artificial limits on performance, participants were allowed as much time as they wished to perform the magnitude estimates. Examining a potential viewing time vs. accuracy trade-off is an avenue for further exploration.

Many of the results reported in the two experiments are arguably small differences in perception accuracy. An interesting future examination is whether these small differences in perception of visual variables can be overcome by providing additional contextual information in information visualizations that employ them. For example, by displaying a

98 background grid behind a histogram, judgements of location may become de facto judgements of length , and therefore more robust to display on a tabletop or across display orientations. Such techniques should be explored in future research. The intention of this chapter is to highlight an issue that needs to be overcome, not to attempt to close doors for future visualization designs.

One possible application of these results would be to calculate an inverse distortion that could be applied to a given visualisation before it is presented to the user. As we have described, Nacenta et al. (2006) presented a system in which the visual presentation of the cursor is modified so that it always presents the same retinal image when viewed on the various screens in a multi-display environment. In addition, a research project which showed some success with a similar idea was presented by Harrison at al. (2007), who distorted the presentation of progress bars to achieve a shorter perceived time. An implementation which combines elements of these two ideas might indeed lead to more accurate perception of information graphics.

What is perhaps the most important issue for further consideration is how well these results translate to real-world viewing. While the results of these psychophysical tasks has previously been applied to the design of visualisations (Cleveland and McGill, 1984, 1985), it is not a certainty that the results of the present work will also translate. The addition of more graphics might, perhaps, provide additional visual cues for mental rotation, and so help to reduce the effects we uncovered. Whether this occurs, and what cues might best be suited to this, remains the subject of future work.

Having now examined two of our three fundamental tasks, reading and perception of graphics, we turn our attention now to the performance of input in a table-centric interactive space.

99 Chapter 5: Input and Control

The issue of angle has a significant impact on the two issues we have examined so far: reading and perception of graphics. Our final low-level task is the issue of control , examining whether the use of the table as the primary input space for both itself and the vertical ancillary displays comes with any kind of performance penalty. In most computer systems, input devices are typically offset from the consequences of those inputs. In the case of desktop computers, most users appear to easily handle the transformation of mouse movements on a horizontal surface to cursor movements on a vertical display. However, research by Stratton (1897a & 1897b), Cunningham (1989), Cunningham and Welch 1994, and Krakauer (2000) has shown that performance of motor tasks can incur significant penalties under more dramatically offset input/output spaces, such as rotated mappings of up to 180 o. While these penalties are reduced with practice, they are typically not completely eliminated. With this in mind, we now examine the angles inherent in a table-centric interactive space, and how they might impact this offset.

5.1.1 Angle of Table Relative to the Eye and of Table Content Relative to the User

For the performance of the previous two elementary tasks, reading and graphical perception, the issues of the angle of the table relative to the eye (Figure 5-1 left and centre), and the angle of the table content relative to the user (Figure 5-1 centre and right) have had varying but significant consequences.

Figure 5-1. Left: a map of Cambridge, MA as it appears to a user on a vertical display. Centre: the same map, as it appears to the user of a tabletop display. Right: the same map, as it appears to another user of the same tabletop display

100 For the issue of input, however, these are all but eliminated: in table-centric interactive systems, input to objects on the tabletop is done through the table itself, and almost always in a direct-touch way. As a result, the angles of the table relative to the eye, and of the content relative to the user, have no impact on any offset in the mapping between input and display. Because of this, we do not further examine this issue in this thesis. The same is not true of the vertical displays, since input to these surfaces is made through the table, even when the consequences of that input are realised on the ancillary displays. As a result, the angle (and thus position) of the vertical displays relative to the user is strictly correlated in the offset between the input and the display. The balance of this chapter investigates this issue in detail.

5.1.2 Angle of Vertical Displays Relative to the User

It is important that designers of environments employing shared vertical displays consider the penalties associated with an offset between input and display. For example, if a system designer wished to build a table-centric environment augmented with a single vertical display, where should that display be positioned to allow for optimal use by each participant seated around the table? Furthermore, given that it is impossible for such a display to be located directly in front of all participants, what is the appropriate input/output mapping? Although informative, it is difficult to apply the results of the previous research to these new multi-user scenarios because the experimental setups have typically positioned the display directly in front of the user, resulting in only a simple translational offset (as in the desktop computing scenario) plus the experimentally manipulated rotation of the displayed image. Although it is intuitive that the traditional mapping of moving the arm forward in order to move the cursor upward on the vertical display is ideal when facing toward the display, it is not at all clear what happens when the display is moved such that users are no longer facing it directly. Is the ideal mapping body centric, such that forward movement should continue to move the cursor upward; or display centric, such that movement toward the display should move the cursor upward; or, something else entirely? The previous research suggests that selecting the wrong mapping can have a dramatic effect on performance, resulting in penalties in time and accuracy of well over 100% (Cunningham 1989). In this chapter, we present two studies

101 that investigate the effects of display space location and control space orientation on interaction performance. In the first, we varied the position of the display and gave participants dynamic control over the position and orientation of the input space while performing a docking task. This enabled us to determine how users naturally orient their input spaces when using a display in varying locations. In the second study, we forced participants to use a particular orientation of the input space, allowing us to determine performance in more fixed configurations. In combination, these experimental designs cover a broad range of possible display and control space location/orientation scenarios, and the results can help designers of collocated collaborative environments make informed choices as to the placement of shared displays and their input devices.

5.1.3 Terminology

To facilitate discussion of these issues, we first define several terms.

5.1.3.1 Display Space

The virtual display where the user sees the results of her input manipulations is defined as the display space. We define the display space to be a two-dimensional vertically- oriented plane, located at various positions around the table at which the user is seated. We assume that the display faces the user’s body such that the centre of the display is the point that is closest to the user. The position of the display space is a measure of the angle between the user and the vertical displays: Figure 5-2 shows the labels we assign to the different display space positions. For example, a display facing the user but located behind her is at the “S” (south) position; a screen to her left is in the “W” (west) position.

Figure 5-2. Display space position: location of the screen relative to the position of the user and table where input is made. The “X” marks the centre of the chair where the user is seated, the rectangle above it is the table on which input is made

102 5.1.3.2 Control Space

The area used by the user to provide input to the system is defined as the control space . In this chapter, control space is a two-dimensional input plane with a particular position and orientation relative to the table on which it is located. We assume that the control- space is on a horizontal plane at right angles to the display space, similar to the situation in a standard desktop computer where movements of a mouse on horizontal plane are mapped to a vertical screen. The position of the control space on the table can be varied.

Control space orientation refers to the rotational transformation of the control space about the axis perpendicular to the table surface (Figure 5-3). Note that control space orientation is relative to the top of the table, and not to the display space position. To distinguish between the two, we use compass directions (North, South, etc) for display space position, and angles (0 o, 45 o, etc) for control space orientation.

Figure 5-3. Control space orientation: the rotation of the control space about the axis perpendicular to the table. Left: labels for the various orientations. Right: e.g., with a 135 c orientation, to move up on the display (top) the user must move back and to the left

5.1.3.3 Example

On a standard desktop computer setup, the computer monitor has a display space position of “N”, and the control space is the area on which the mouse operates, which is typically a mouse pad with a control space orientation of approximately 0 o. The control space orientation can be dynamically changed by rotating the mouse. Figure 5-4 shows another example, which motivates our exploration.

103

Figure 5-4. Example of user orienting his control space to roughly 135º with a display at S 5.2 Related work

We consider two relevant areas of prior work: collaborative systems where the positions of ancillary vertical display do not allow the familiar N display position, and psychology work which has explored how rotationally transformed input-output mappings impact motor-control.

5.2.1 Collaborative Systems

A study of a related issue was undertaken by Su and Bailey (2005) who sought to determine the optimal relative distance and angle of two vertical displays when performing a docking task requiring the movement of the docking object from one display to another. In conjunction with this work, our results could be used to inform designers as to the optimal position of multiple displays, in addition to providing the optimal control space orientation. In some of this previous work by Streitz et al. (2002), users could move to the ancillary vertical displays in order to interact with them, while others, such as Johanson et al. (2002b), advocated an approach where all interaction occurs while users remained seated. Nacenta et al. (2005) compared techniques for manipulating virtual objects between a tablet and ancillary displays and found that the most effective was a radar (or ‘world in miniature’) approach, where a portion of the tablet acts as a proxy of the primary display. Although they provide a thorough comparison of known techniques, this study only considered the situation where the display is placed at the N position, and the control space orientation is always at 0 °. The problem is exacerbated when multiple users are seated around a table and it is physically

104 difficult for everyone to be optimally oriented to one or even multiple surrounding shared display(s) (e.g., Bentley et al. (1992), Heath and Luff (1992), Nardi et al. (1993), Covi et al. (1998), Teasley et al. (2000), Hindermarsh and Pilnick (2002), Mark (2002), and Tollinger et al. (2004). Thus, their work provides no guidance in the more general situation where ancillary displays may be positioned anywhere around the table.

5.2.2 Effects of Control Space Orientation on Performance

Psychologists and cognitive scientists have long studied the effect of distorting the orientation of control space relative to the display space. The earliest work, conducted with optical prisms mounted on eyeglasses, was conducted by Hemholtz (1866) and Stratton (1897a & 1897b) in the 19 th century. Both found that inverting the visual input to the eyes resulted in severe disruptions to motor activity, but that these disruptions were reduced over time.

Cunningham (1989) sought to determine the structure of the mapping between the visual and motor systems. Participants sat at a table with a digital tablet, and performed a pointing task on a display oriented vertically and positioned at N (directly in front of the user). Performance was measured under control space orientations of 0 °, 45 °, 90 °, 135 °, and 180 °. Participants were instructed to make straight-line movements between targets in the pointing task, and the effect of the control space orientation was measured as the deviation from straight. A control space orientation of 90 ° was found to be the most difficult, while 45 ° and 180 ° orientations had the lowest rate of spatial error relative to 0 °, and the relative difficulty of the 135 o° orientation varied between participants. In subsequent work, Cunningham and Welch (1994) examined how people learned multiple rotated control spaces. Unlike the previous study (Cunningham 1989), participants did not do whole blocks of tasks at a particular orientation, but instead switched back and forth between different orientations. They also measured the effect of different types of cues used to prime participants to the orientation used in the trial. They found that with practice these cues could significantly reduce the interference effects of switching between orientations.

105 5.3 Research Goals

Although there is research that extends the work of Cunningham and colleagues, none has examined the issue of how control space orientation impacts performance under different display space positions (Kagerer et al. (1997), Dobbelsteen et al. (2004), Seidler (2004). User preference for control orientation and display position has also not been investigated. Both these issues are of significant importance not only to the design of collaborative spaces but also to our basic understanding of human capabilities when faced with transformed input-output mappings. We seek to explore these issues via two experiments, and specifically attempt to answer the following questions:

1. Which display space position do users prefer when given a choice? 2. Given a particular display space position, what control space orientation do users prefer? 3. Given that in real environments it may not be possible to position displays and orient control spaces to every user’s preference, what is the penalty on performance if either or both of these preferences are not met? 5.4 Experiment 1: Adjustable Control Spaces

5.4.1 Goals and Hypotheses

In this first experiment, we sought to answer our first two research questions: what are users’ preferences with respect to display space position and control space orientation? We also partially explored our third question by asking participants to perform a task with the display space positioned at each of the eight possible locations (Figure 5-2) while allowing them to orient their control space as they wished. The impact of a fixed control space is explored in the second experiment.

Based on the results of previous experiments in the literature and our experiences with collocated table-centric multi-display environments, we formed several hypotheses:

H1: Participants would most prefer the N display position. H2: Participants would least prefer the S display position. H3: Display space position would have a significant impact on the selected control space orientation. 106 H4: Participants would generally orient their input space such that the traditional mapping of forward/up would be maintained (0 ° for N, 45 ° for NW, 90 ° for W, etc). H5: Display position would have a significant impact on performance. H6: Performance would be best at display positions most preferred by the participants.

5.4.2 Apparatus

The participant sat in a chair in front of a DiamondTouch multi-point input surface (Dietz and Leigh 2001). Although the DiamondTouch is capable of acting as a touch-input device, we did not make use of this feature; instead, as was done by Cunningham (1989), input was made using a stylus, effectively turning the DiamondTouch into a large input tablet. Since the intent in this experiment was to allow participants to orient the control space as they preferred, we built a simple plastic template designed to be manipulated by the non-dominant hand and tracked on the DiamondTouch. The position and orientation of the control space was mapped to the boundaries of this template, allowing participants to easily reposition and reorient the control space by moving the template appropriately. A ceiling mounted projector displayed a green rectangle on the DiamondTouch to provide the user with a visual indication of the extents of the control space. The control space was 17x13 cm while the DiamondTouch surface was 67x51 cm, thus allowing the participant to manipulate the control space over a reasonable area. The stylus was held in the dominant hand and its input was only effective when used within the control space controlled by the non-dominant hand. Figure 5-5 illustrates this apparatus.

Figure 5-5. Control space and stylus input. Left: Position and orientation of the control space is controlled by a device held in the non-dominant hand. Right: Diagrammatic illustration of the relationship between the device (red) and control space: green rectangle represents the control area, the arrow indicates the “up” vector for the control space (arrow is shown here for illustration only, and was not displayed during the experiment) 107 For the display space, we used a large plasma screen (display area approximately 75 x 56 cm) positioned atop a wheeled cart. The display was placed at eight different positions throughout the experiment, each of which was marked with tape on the floor, and positioned 140 cm from the centre of the chair upon which the user sat to perform the study. Figure 5-6 illustrates.

Figure 5-6. Display and input table. Left: plasma display on cart used to vary the display’s position. Right: table and chair used in the study. The 8 possible display space positions, equidistant to the centre of the chair, were marked on the floor with tape and the cart placed accordingly

The software used for the experiment was written in Java, and was executed on a 3.2GHz Pentium PC running Microsoft Windows, which was disconnected from all network traffic. Both the plasma display and overheard projector were driven by a graphics card running at a resolution of 1024 x 768 pixels with a 60Hz refresh rate.

5.4.3 Participants

8 participants (6 male and 2 female) between the ages of 19 and 28 were recruited from a local university community and our lab, and paid $20 for their time, irrespective of performance. All were undergraduate or graduate students. All were right-handed, had completed at least one year of a Bachelors degree, and had little to no experience working with stylus, tablet, or tabletop input.

5.4.4 Procedure

There are three canonical tasks in a GUI: selection (moving a cursor to an object), docking (selection + drag of an object to a location), and path-following (e.g., steering through levels of a hierarchical menu, or drawing a curve with the pointer). We chose a

108 docking task, since it encompasses the simpler selection task and also evaluates movement trajectories while giving participants freedom to move in the wrong direction and then make corrections to their path. This task also varies from Cunningham’s work, where first a selection task in Cunningham (1989), and then a path-following task Cunningham (1994), were used. The stylus, held in the participant’s dominant hand, controlled the absolute on-screen cursor position. Selections were made by crossing the stylus over the desired object, and releasing by lifting the pen from the surface of the table. Docking tasks were grouped into several pre-computed “walks”, which would begin with a blue square positioned at the centre of the screen. Participants would select this square and then drag it to the position of a larger red square “dock” which would change colour to a light blue to indicate success. Participants would then lift the stylus from the surface of the table to complete a successful dock. The red square would then move to a new location. The blue square remains in the same position so that it does not have to be reselected. The blue square is then dragged again to the red square’s new position, and this process continues for four different locations of the red square. Thus, four docking tasks are accomplished in each such “walk”. Figure 5-7 illustrates.

Figure 5-7. The docking task used in experiment 1: Participant touches the pen to the table, 2: crosses the blue object to select it, 3: drags blue object to the red dock, which then turns light-blue 4: when released the red dock moves to a new location

Before beginning the experiment, the procedure and apparatus were explained, and participants were allowed to practice the task until they felt comfortable with it, which usually occurred within 30 practice trials. We recorded the times taken to make the initial selection and perform the docking task successfully, control space orientation throughout the experiment, and the number of errors. An error occurred when the blue square was released outside the red square. To prevent participants from “racing through the experiment”, they had to successfully complete each docking task, even if errors occurred, before the red square would move to a new position. 109 5.4.5 Design

Each participant performed 40 docking tasks for each of the 8 display positions. To counterbalance for learning effects, each participant began with a different display space position and then worked through the remaining 7 display positions in anti-clockwise order (i.e., participant #1 began with the display positioned at N, then NW, then W, etc; participant #2 began with the display positioned at NW, then W, etc).. Although we considered randomizing the order of the display orientation sequence, previous work by Cunningham (1989) and Cunningham and Welch (1994) indicates that radical changes in orientation can result in significant temporary performance degradation before participants adapt to the new orientation. By using a sequential presentation, the orientation changed gradually, thus averting this temporary spike, and reducing the time required to adapt to the new orientation. This was important as our focus was to measure true, adapted performance at each orientation, rather than any transitional effects. Further, by starting each participant at a different orientation, each orientation appears in a different presentation slot per participant (i.e., P1 sees Order 1 first, Order 2 second, etc. P2 sees Order 2 first, Order 3 second, etc) resulting in a between-subjects counterbalanced presentation order. The direction of movement for each docking task was randomized but controlled such that movements in all 8 compass directions were performed an equal number of times by each participant at each display position. In summary, the design was as follows:

8 participants x 8 display positions x 40 docking tasks = 2560 total dockings

110 5.4.6 Results

5.4.6.1 Preferred Display Space Position

We administered a post-experiment questionnaire to answer our first question: what are the most and least preferred display space positions? Table 5-1 summarizes the results.

Most Least Participant Preferred Preferred 1 N S 2 NE S 3 NE & NW S 4 NW S 5 NE S 6 NE SW 7 NE & NW SE & S 8 N SW Table 5-1. Preferred display space position for each participant

That most participants least preferred the S display space position confirms Hypothesis H2. This is unsurprising since S represents the greatest offset between hand and display positions, thus requiring the least comfortable posture. Participants cited body comfort as the primary reason for selecting S as their least preferred display space position. Much more surprising is that participants predominantly (75%) selected a display space location offset 45 o from a traditional N position (we attribute NE / NW asymmetry to right- handedness of all participants). Although all participants were asked to provide an explanation for their selection, none was able to articulate her preference to her own satisfaction: a typical response, stated by participant 3, was “it just feels better”. Based on these results, we reject hypothesis H1, but note that this may well vary by input device, and suggest that designers consider the ergonomics of their input device before applying this particular result.

5.4.6.2 Preferred Control Space Orientation per Display Position

Our second research question, what is the preferred control-space orientation for a given display space position, was answered by allowing the participants to dynamically reorient their input space throughout the experiment and recording the results. Table 5-2 summarizes the average control space orientation each participant used across the entire

experiment for each display position. 111 Control Display Orientation Location (µ) σ S 27.2º 45.8º SE -35.1º 21.3º E -26.0º 19.9º NE -13.3º 15.4º N -4.7º 14.0º NW 12.6º 18.1º W 22.5º 34.1º SW 36.9º 32.7º Table 5-2. Mean control space orientation ( µ) and variance ( σ) across all participants for each display-space position

Display position had a significant effect on the control space orientation selected by the user (F 7,49 = 9.049, p < .0001), thus confirming Hypothesis H3. Further, the high variance between users’ selected orientations, seen especially at the more extreme screen positions, this suggests that significant individual differences between users may play a role in their preferred orientation. These differences are to be expected, since users were given no instruction as to how to make their orientation selection. Figure 5-8 shows the mean control space orientation used by each user across the experiment per display space position.

Figure 5-8. The direction of each line indicates the mean control space orientation for a participant for a display space position. Each user’s particular orientation is rendered in the same colour in each display position

Predominantly, participants chose their orientation at the beginning of a block of trials at a given display-space position, and rarely changed their control space orientation during a block. We measured both inter-trial reorientation (changed in orientation between trials) and intra-trial reorientation (changes in orientation during a trial). On average, for each participant, when the first trial of each block was excluded, instances of inter and intra- trial reorientation in excess of 1 o did not exceed 6 trials.

112 Figure 5-8 illustrates the trend of orienting the control space depending on the position of the display. With the exception of participant 1who kept the control space at an orientation of 0 º for the entirety of the experiment, participants did not strictly maintain the traditional forward/up control space orientation, regardless of whether we consider forward/up to be away from the body or towards the display. We therefore reject Hypothesis H4.

5.4.6.3 Effect of Display Space Position on Performance

We first measured performance as the time required to perform the entirety of the task from selection of the blue square until it is successfully docked with the red square. Analysis of variance showed a significant main effect for display space position on task completion time (F 7,49 = 2.164, p = 0.05), confirming hypothesis H5. Pair wise means comparisons across all participants revealed that performance at screen positions N, NW, E, and W were not significantly different from one another, but were from the rest, as was the case for S and SW; SE and NE were significantly different from one another and from the rest. However, the magnitude of the performance difference between positions was not very large, as shown in Table 5-3.

Display Position

S SE E NE N NW W SW

Time 1255 1176 1112 1024 1088 1113 1110 1227

Efficiency 23% 15% 9% - 6% 9% 8% 20% Table 5-3. Average task completion time in msec for all participants for each display position, as well as how much slower (percentage) participants were to complete the task at that display position relative to the overall fastest one (NE)

We also examined the path traversed by the participants during the docking task. Unlike Cunningham (1989), we did not instruct participants to attempt to move in a straight line when performing the task. Thus the resulting paths in our experiment are more reflective of how users might perform such tasks in a real application, thus increasing the ecological validity of our results. Motivated by previous research on input coordination by both Zhai and Milgram (1998) and Masliah and Milgram (2000), we computed the ratio of the total length of the actual path to the ideal minimum-length straight line path.

113 This metric provides an indication of the additional distance participants travelled by deviating from an ideal path. We recognize that this metric only considers path length and not the complexity of a path as might be, for example, measured by the number of turns. However, given that path complexity metrics are not the focus of our research, we chose to rely on the established path length ratio metric. There was a significant main effect for display space position on this path ratio (F 7, 49 = 8.01, p < .0001). There was also a significant correlation (R 2=.72) between this ratio and performance time, which is expected as larger rotations imply a longer path which require more time to complete. In combination, this further supports Hypothesis H5.

Errors were measured in two ways: a trial was deemed to have been erroneous if the participant released the blue square before placing it in the red dock, or if the blue square entered the red dock’s area and exited again before being released. There was no significant effect for display space position or participant on either error metric.

5.4.6.4 Relationship of User Preference to Performance

Our third research question: what is the effect on performance of not meeting user preference with respect to display space placement and control space orientation can be partly addressed by the results of this experiment. Although the participants were able to adjust the control space orientation, they had to perform dockings with the display positioned at each of the 8 locations, and thus this experiment provides data as to what happens when display position is not at a user preferred location.

Preference did not correlate with optimal performance: 4 of 8 participants performed fastest with their preferred display placement, and 2 of 8 participants had the lowest performance at their least preferred display location. We thus reject hypothesis H6.

5.4.7 Discussion

The lack of correlation between preferred and best performing control space orientation in experiment 1 suggests that either participants are not able to correctly assess their performance, or, more likely, that they consider factors other than performance when determining preference. In particular, the absence of inter-trial reorientation suggests

114 physical comfort may be more important than performance, since it is likely that the initial orientation of the control space was made to optimise comfort. That preference is more closely tied to physical comfort than to performance is a likely explanation for the rejection of Hypothesis H6: that performance would be best at those display positions most preferred by the participants. Also interesting was that when asked for their least preferred display position, only 2 of 8 participants chose the position where their performance was worst. The rejection of hypothesis H1, that participants would most prefer the traditional N display space position, provides further evidence that participants were optimising for comfort. Accordingly, our finding that users least preferred the S display position is not surprising, since it requires the most effort to turn the body to enable them to see it. This trade-off between performance and comfort should be considered when designing multi-display environments.

Although significant individual differences were present, some general trends were visible with regards to how display space position influenced the choice of control space orientation (see Figure 5-8): for all of the eastern display space positions (NE, E, SE), participants chose to orient their control space between 0 o and -90 o, or, generally, to the east. For all of the west display space positions, participants chose to orient their control space between -17.1 o and 90 o, or, generally, to the west. We suspect that the asymmetry between these two ranges may be because our participants were all right handed.

We did find a significant effect for display space position on performance, there was on average a maximum 23% penalty when users were able to adjust their control space orientation as in experiment 1. As Figure 5-10 illustrates, there is a performance trend when participants are not able to adjust their control-space orientation. For those display spaces that are in front of them (NW, N, NE), a 45 o offset from a straight-on control- orientation produced the best results. For the remaining positions, a 90 o offset towards 0 o is optimal. These results will be of use to designers of systems where physical constraints limit the users’ ability to reorient their control space, such as in operating theatres, as in Nardi (1993). There it is suggested that if the monitor of a closed-circuit video feed is used by a surgeon to view the movement of her tools, and that monitor is placed directly in front of her, the video image should be rotated in to create a 45 o control orientation.

115 5.5 Experiment 2: Fixed Control Spaces

5.5.1 Goals and Hypotheses

In the first experiment, our third research question: what is the effect on performance of not meeting user preference with respect to display space placement and control space orientation, was only partially explored. That is we allowed users to manipulate the control space to their preferred orientation. In this second experiment, we further explore this question, this time using a fixed control space orientation that users could not alter. Thus, this experiment considers the situation that is common in real environments where both display position and control orientation are fixed and users have to work within the given parameters. We formulated the following hypotheses:

H7: Inability to adjust control space orientation will have a significant effect on performance: we will find a significant main effect for control space orientation on task performance time. H8: Performance at a given control space orientation will vary between display space positions.

5.5.2 Apparatus

The apparatus for this experiment was the same as in experiment 1, except that the physical template’s orientation no longer affected the orientation of the control space. To compensate for the gap in feedback created by the removal of this pairing, we added a visualisation to the rendered control space: a gradient from green at the bottom (toward 180 o) to blue at the top (toward 0 o) of the space (creating a ground/sky effect). To provide the same positioning flexibility as in experiment 1, the template continued to control the position of the control space.

Figure 5-9. The experimental apparatus for the second experiment. The physical template controlled only the position, and not the orientation (right), of the input space. The green- blue gradient was used to show the ‘up’ vector of the input area to the user

116 5.5.3 Participants

8 participants (4 male and 4 female), undergraduate and graduate students and different from those in experiment 1, between the ages of 19 and 25 were recruited from the community, and paid $20, irrespective of performance. All were right-handed, and had little to no experience working with stylus, tablet, or tabletop input.

5.5.4 Task and Procedure

The task and procedure were virtually identical to those in experiment 1, except that participants were presented with a particular control space orientation, rather than being allowed to dynamically reorient the space.

5.5.5 Design

The task was performed for the 8 display space positions (Figure 5-2) and the 8 control space orientations (Figure 5-3). To reduce the time required to participate, the display control conditions were not fully crossed: each participant performed the task at 4 control orientations for each of 4 different positions. A Latin-square design was used to avoid ordering effects and to ensure that each display space position and control space orientation pairing occurred an equal number of times in the experiment. Because of the learning and interference effects observed by Cunningham (1989) and Cunningham and Welch (1994), we increased the number of docking tasks in each block from the 40 used in the previous experiment to 80. In summary, the design of the experiment was as follows:

8 participants x 4 display positions x 4 control orientations x 80 docking tasks = 10,240 total dockings

5.5.6 Results

We performed a repeated-measures ANOVA to determine significance. There was a significant effect for block number on overall performance time within each display position / control orientation pairing pairs on task performance time (F 15,105 = 2.217, p =

117 0.01). This suggests that, as discussed by Cunningham and Welch (1994), the transformed spatial mappings of control to display space were interfering with one another. We found that after the first 50 trials per condition, the order effect ceased to be statistically significant, indicating that with sufficient practice the prior spatial mappings ceased to interfere with the one currently being used. Accordingly, in the remaining analyses we consider only the last 30 trials per condition, treating the first 50 trials as practice.

There was a significant main effect for control space orientation on task performance time (F 7,7 =10.05, p = 0.003), confirming Hypothesis H7. There was also a significant interaction between control space orientation and display space position on task performance time (F 7,7 =7.31, p < .005), indicating that the effects of control orientation differ depending on display space position. This confirms Hypothesis H8. Also interesting was that the shortest times were seen to roughly correspond to the preferred range of control space orientations that users chose when given the ability to manipulate the control space in experiment 1. Figure 5-10 illustrates these effects, while Table 5-4 provides numerical results for the entire experiment.

Figure 5-10. Mean task completion time at a given control space orientation encoded as the length of the line in that direction (longer the line the slower the performance). Display space position indicated by the position of the perpendicular line. Overlaid in red on each is the bounds of preferred orientations (longer lines) from experiment 1 118 Control Space Display Space Position Orientation S SE E NE N NW W SW 180 130% 273% 108% 129% 176% 178% 183% 180% 135 50% 313% 119% 256% 227% 333% 83% 124% 90 26% 157% 413% 291% 32% 39% 106% 57% 45 53% 32% 62% 75% 23% 38% 41% 24% 0 163% 46% 18% 0% 75% 36% 17% 123% -45 174% 15% 80% 61% 55% 279% 495% 352% -90 128% 108% 117% 221% 305% 353% 118% 121% -135 132% 164% 228% 261% 132% 230% 140% 261% Table 5-4. Mean time for each control space orientation at each display space position, expressed as a percentage of the value of the optimal cell (0º,NE: 0.786 seconds)

5.5.7 Discussion

Given the small sample sizes, comparing between experiments 1 and 2 requires an assumption that the populations did not vary significantly. Given that they came from the same sample pool, we can make comparisons with some confidence. Interestingly, the correlation between actual path to optimal path ratio and task completion time was significantly lower (R 2 = 0.23) in comparison to the results of experiment 1. One possible explanation is that several users adopted what we have dubbed the spiral strategy to moving under a transformed spatial mapping: rather than attempt a seemingly optimal straight-line movement, they instead chose to move in circular motions. Because the control space was offset rotationally, a circular motion can be more easily anticipated than a straight line – moving in a clockwise circle in the input space produces a clockwise motion in the display space, no matter the control space orientation. While still moving, users can probe the input/output mapping, making their circle steadily larger until its radius is equal to the distance between the source of the drag and the target, or by briefly breaking out of the spiral when the correct direction is found. Figure 5-11 illustrates this approach, where we see three distinct anticlockwise spirals as the pointer approaches the red square dock. Note that the blue square was moved very close to the dock near the beginning of spiral S2, but the participant elected to continue the spiral pattern. Although this spiral path clearly deviates from the optimal straight line path, participants who employed it reported that they felt it was faster than trying to learn the more difficult transformed mappings.

119

Figure 5-11. An actual path (in black) followed by a participant using a spiral-strategy to dock the blue square onto the red square under a transformed control – display mapping. Three distinct spirals (S1-S3) are visible 5.6 Implications for Design

In table-centric environments, users control ancillary displays placed around the room from a single, shared input device: the table. For such environments, the data from our second experiment can shed some light on optimal display placement. For example, for a square table with four participants, there are four typical seating positions to be considered, as illustrated in Figure 5-12. Of the 24 possible permutations of user seating positions, 4 are of interest: (1,2), (1,3), (1,2,3), and (1,2,3,4), since all others are repeated cases of these.

Figure 5-12. Left: the 4 canonical positions for users seated at a table. Middle: the 8 canonical control orientations for input on the table. Right: the 8 canonical positions for a vertical display around the table (partial repeat of Figure 5-3 for convenience)

120 By using our results from experiment 2, we are able to compute the optimal position for a display in such an environment, for each possible seating arrangement of users. If we assume that all users will be making use of a shared table to control the vertical displays, and that the table will have a fixed control orientation shared by all users, we can determine both where the display should be located, as well as what the optimal control orientation would be for such a display. That determination is summarized in Table 5-5.

Display User Configuration Position 1 & 2 1 & 3 1, 2, 3 1, 2, 3, 4 S 45º (53%) 90º (32%) 45º (53%) 180º (183%) SE 45º (32%) 45º (38%) 45º (38%) 180º (273%) E 45º (62%) 0º (18%) 45º (62%) 180º (183%) NE 0º (46%) 45º (75%) 45º (75%) 180º (273%) N 45º (62%) 90º (32%) 45º (62%) 180º (183%) NW 0º (36%) 45º (38%) 0º (46%) 180º (273%) W 45º (41%) 0º (18%) 45º (62%) 180º (183%) SW 45º (38%) 45º (75%) 45º (75%) 180º (273%) Table 5-5. Optimal control space orientation (relative to user 1) and performance penalty at that orientation (as a percentage of optimal, NE/0 º pairing) for a display positioned at each position relative to user 1 (rows) for the given user position(s) around a table combination, determined using data from Table 5-4.

Table 5-5 can be used in a number different ways by designers of spaces utilizing controls with fixed orientations. First, if the number of seats around a table is known, the designer can designate their position (the designer will note, for example, that seating two participants across from one another will yield better performance for most vertical display conditions). Second, if the number of and placement of the seats around the table is fixed, the table can be used to decide on the placement of vertical displays (or the angle of the vertical display relative to the table). For example, if users are to be seated across from one another and at a third side of the table (positions 1, 2, 3), the optimal placement of the display would be at the SE position. Finally, if the positions of the users, the table, and the vertical displays is fixed, the table can be used to provide the optimal orientation of table-based control to give input to such a display. For example, with users seated at positions 1, 2, and 3, and the display at SE, the controls should be oriented at 45 o for optimal performance.

121 If technology allows, an alternative scheme would see each individual user provided with a personally oriented control space. This could be enabled by either employing user- aware tabletop technologies (such as proposed by Dietz and Leigh, 2001), or dedicating only a small portion of the central table’s input area to serving as each user’s control space for each ancillary display. Under such a scheme, designers would no longer be constrained to finding the control orientation with the minimal overall penalty for all users, but could rather employ the ideal orientation for each individual user. Free of the constraints imposed in the construction of Figure 5-5, we can re-examine the results shown in Table 5-4, in order to find the optimal control orientation for each user and display space configuration, shown in Table 5-6. The table’s columns indicate the optimal orientation for each user, for any configuration of users.

Display User Position Position 1 2 3 4 S 90º / 26% 0º / 17% 45º / 23% 0º / 18% SE -45º / 15% 45º / 24% 0º / 36% 0º / 0% E 0º / 18% 90º / 26% 0º / 17% 45º / 23% NE 0º / 0% -45º / 15% 45º / 24% 0º / 36% N 45º / 23% 0º / 18% 90º / 26% 0º / 17% NW 0º / 36% 0º / 0% -45º / 15% 45º / 24% W 0º / 17% 45º / 23% 0º / 18% 90º / 26% SW 45º / 24% 0º / 36% 0º / 0% -45º / 15% Table 5-6. Optimal control orientation and its penalty for that orientation for each user for a display at the given position. The display position is expressed relative to user 1, while the control orientation is expressed relative to the individual user

For a designer, Figure 5-6 provides the optimal control orientation for any given user controlling a vertical display in a table-centric interactive space. This table can be used to optimally orient the controls presented to any one user in a table-centric space. For example, in a space with users at position 1 and 2 and a display at to the west, the designer would provide control oriented at 0 o for user 1, and at 45 o for user 2.

To further aid designers, we can utilize the data in Table 5-6 to compute the maximum penalty for each display position and user configuration. Similar to Table 5-5, Table 5-7 indicates the worst-case performance penalty for each display position and user seating

122 combination – like Table 5-5, the designer can use it to select the optimal vertical display position for any given set of user positions around the table. The difference between these two tables is that Table 5-5 should be employed by designers utilizing fixed control- orientations, while Table 5-7 assumes the ability to reorient the controls for each user. The optimal orientation for each user in each configuration can be found in Table 5-6.

User Configuration Display Position 1 & 2 1 & 3 1, 2, 3 1, 2, 3, 4 S 26% 23% 26% 26% SE 24% 36% 36% 36% E 26% 17% 26% 26% NE 15% 24% 24% 36% N 23% 26% 26% 26% NW 36% 36% 36% 36% W 23% 18% 23% 26% SW 36% 24% 36% 36% Table 5-7. Worst-case performance penalty for each user / display position configuration. The display position is expressed relative to user 1

The contrast between Table 5-5 and Table 5-7 is striking: it is immediately evident that there is significant benefit to providing each user with a personalised control orientation. If the technology of the space does not allow this, however, and the designer is forced to set a single control orientation, the best arrangement for two participants with a shared control orientation is to be seated across from one another (positions 1 and 3 from Figure 5-12) while using a vertical display located on either side (W or E). Worth noting is the dramatic increase in penalties paid when moving from three users to four (Table 5-5). For such environments, it may become necessary that multiple participants be seated on the same side of the table rather than seating at all 4 canonical positions. How control spaces are shared and positioned is best determined by examining the environment, but it is clear that care should be taken to avoid a high-penalty configuration.

Ultimately, the designer must balance user preference and performance. This applies to both the display positions, and control orientations. How this balance is struck will depend largely on the configuration of the room, as well as whether the input technology supports multiple control orientations. 123 5.7 Avenues for Future Research

Our work has explored the impact of display space position and control space orientation on user preference and performance. The results contribute to the literature on transformed input-output spatial mappings by investigating important transformations not previously tested. These results also allow designers to make more informed choices as to layout of shared displays in multi-display environments. In addition to these contributions, the work helps to illuminate several future paths for exploration.

The first consideration is how well the results of the experiments reported in tables 5-5 through 5-7 represent general behaviour in the wild. They are, of course, derived from experimentation which includes some error. The precision with which these results can be applied to room designs is a clear avenue for future research.

Also of interest is whether the results for individuals truly generalise to shared spaces? Clearly, the presence or absence of collaborators is unlikely to affect pointing performance. But, how well does our proposed ideal solution, individualised control orientations for each participant, transition from theory to implementation? Such a scheme would lead to a loss of shared orientation of input in the shared input device – gestures of input by one user, when mimicked by another, would yield different results. This issue bears further investigation.

Also to be considered is whether, and how well, users would adapt to multiple displays with different control orientations. Cunningham and Welch’s (1994) work showed, there is a penalty to be paid when switching frequently between multiple control orientations within the same display position. Further, their work explored mechanisms for signalling the control orientation to the user, including audible beeps and visual cues. Would users be more adept at switching control orientations between display positions? Would the change of display be a sufficient cue, or would an additional signal, such as visual indicators, be required?

The final, and perhaps most important issue to be considered is the importance of a closed-loop movement for adapting to different control orientations. In all of Cunningham’s work, continuous feedback was given to the user about cursor position 124 while making movements, as was done in our experiments. Most touch-table technologies, however, do not detect the position of the hand or stylus when it is not in contact with the table. To get around this, we forced our users to keep the stylus on the table at all times, and employed crossing -based selection of our target. In the wild, however, this may be impractical, since selections on tables are typically land-on based. Therefore, to mimic our technique, designers would be forced to choose between breaking the typical direct-touch metaphor and employing different selection mechanisms on different displays.

In the next chapter, which describes a series of interaction techniques for table-centric interactive spaces, we present an approach to overcome this issue, while applying the other results of this chapter to optimize input by multiple users seated around a table.

125 Chapter 6: Interaction Techniques

As we have seen, the design of table-centric interactive spaces is fraught with contradiction: in order to facilitate face to face collaboration, users should sit across the table from one another, but reading speed is better if users sit on the same side; multiple surfaces facilitate a large homogeneous information space, but perception of information visualization requires specialization of the displays; table-centric interaction reduces workload, but control of ancillary displays suffers when working at a table. To designers, the list of variables to be considered in creating their spaces can be overwhelming. To help designers working with such a space, we have developed a series of interaction designs, intended to provide means of control of a table- centric space by multiple users seated at the table.

We begin with the delineation and examination of a series of issues confronted by interaction designers in a table-centric interactive space. This list is compiled from three sources: our conversations with potential users, our experiences with tabletop and multi-surface systems, and an examination of the underlying assumptions of previous design solutions. This list has provided a means of validation of our design, and is also intended to underline the unique attributes of a table-centric interactive space.

6.1 Design Issues

The following list of design issues is included in order to serve two purposes: first, to help to crystallize the definition of a table-centric interactive space. Second, we aim to provide designers with a list of issues that must be considered when creating alternative interaction schemes.

6.1.1 Non-Linear Alignment of Vertical Displays

Many pointer-based systems that facilitate multiple-display interaction require that the edges of the various displays be linearly or nearly linearly aligned with one another (Baudisch et al. 2004). Although aligning the vertical displays with the edges of the table might simplify the problem space, several factors make this unrealistic:

• The number of displays may outnumber the table edges • Space is required for users to sit between the displays and the table, and for observers not seated at the table • Displays may need to be positioned to optimise viewing angles 126 • The previous chapter identifies situations where non-linear alignment is necessary to improve pointing accuracy • The position of a room’s doors, windows, power outlets, etc. may not permit precise display alignment

Systems designed for our problem space must enable fluid and fast interaction despite the lack of rectilinear alignment of the edges of the displays. In our exploratory environment, two wall displays are located off the corners of the table. While this may seem like a minor requirement, its inclusion in the design space makes the adoption of many existing interaction techniques infeasible, thus necessitating new designs (Baudisch et al. 2004).

6.1.2 Awareness of Input-Output Mapping

Given our requirement of a table-centric interaction system, it is necessary to multiplex the tabletop input technology amongst the various displays, resulting in a modal input paradigm. Thus, visualizing the current mode of the input space is essential. The visualization must indicate that the users’ input is currently being sent to an ancillary display, and also identify that display appropriately. It is also important that mode changes be as smooth as possible to avoid user disorientation. In other words, the input space must transition seamlessly between modes, and allow the users to maintain an understanding of where their input is being sent at all times.

6.1.3 Contextual Association

Previous efforts, such as those of Baudisch et al. (2003), Bezerianos and Balakrishnan (2005), and Shen et al. (2005), have shown the strength of visual contextual associations in large display applications. Objects that are related to one another are drawn together by a visible “context” object, which visually articulates the link between them. The need for visual association of related items becomes all the more important as associated items are separated between displays.

6.1.4 Direct-Touch Input Paradigm

Typically, tabletop systems combine a touch-surface with a projected image, calibrated such that the input and visual space are directly overlaid. This affords an interface where on-screen artefacts are manipulated by direct-touch in an absolute, direct-touch mapping. To maintain a single consistent input paradigm, it is likely desirable to maintain the direct-touch input style while using the interactive table to send input to the ancillary displays.

127 6.1.5 Multiple Control Orientations

As was described in the previous chapter, the need to switch between multiple control / display mappings can be disruptive, and reduce efficiency of input. We therefore take it as a principle that the number of such mappings should be kept to a minimum.

6.1.6 Multiple Granularities of Remote Interaction

A table used as an input device must provide not only fast and coarse interaction, but also precise control of ancillary displays. Thus, multiple granularities of input are necessary in any design solution, and switching between these granularities must be simple and fluid.

6.1.7 Support for Non-Interfering Concurrent Tasks

While collaborative work among multiple users must be supported, it is also important that each user be able to work on a particular sub-task without interfering with the work of others. Two types of interference must be considered: physical interference, where one user’s actions physically “collide” with another; and visual interference, where a task being performed by one user interferes with the visual display space of another. Such interference not only disrupts users seated at the table, but also affects peripheral “audience” members observing ancillary displays.

6.2 Interaction Design

In order to facilitate interaction in our table-centric multi-display space, we designed interaction and visualization techniques to facilitate table-centric interactive spaces (shown in Video, see page vii). Paramount in the designs has been addressing the issues we have described.

6.2.1 Visual Connectivity between Displays

To provide a sense of visual and spatial continuity and connectivity among the various non- aligned displays in our interaction space, we leveraged Gestalt principles, including closure and continuity. On the ancillary display, we placed a repeating pattern on the bottom edge of the screen, symmetrical to the pattern of a proxy to each ancillary display shown on the tabletop (Figure 6-1). This increases the probability of visual alignment from different user perspectives, and also serves to leverage the law of grouping, as explained by Dempsey et al. (2002). Although subtle, this visual connection between the ancillary displays and the tabletop helps to promote a sense of spatial cohesiveness, and so helps to establish the virtual topology of the system.

128 Proxy for Table

Proxy for Wall Figure 6-1. Schematic diagram of our system with screenshots overlaid: the matching colours and shapes of the repeating pattern on the vertical displays and associated proxies on the table allows precognitive connections

To further reinforce the connection between the screens, when a user touches a proxy, the intensity of its colour subtly increases and decreases. This throbbing is mimicked by the proxy on the ancillary display, drawing attention and strengthening the perceived connection. The use of both color-pairing and spatial proximity of the proxies is important: each provides a precognitive connectivity cue, as well as a clear characteristic for searching. The power of each may be reduced in some environments: as more screens are added, the distinctions between colors will become more subtle, reducing the pop-out effect. Relying on proximity is also limiting, since some environments may be ambiguous (e.g., if two ancillary displays are vertically stacked, it is infeasible to distinguish them by proximity to the table’s edge).

6.2.2 Enhanced Contextual Associations

As associated objects become separated across displays, there is an increased need to assist users in understanding this association. In previous work, contextual associations, dubbed CoR 2Ds by Shen et al. (2005), were shown graphically on the same screen. We have built upon their ideas, which were originally intended to support contextual menu functionality, and have extended them to assist in general contextual association. We have maintained the directionality of the relationships between the components, by drawing the associations as growing “out” from some 129 point on the underlying object, to nearly encompass the entirety of the sub-object (Figure 6-2). We also expanded the role of the proxy to allow it to act as a virtual conduit through which cords could be mapped to other displays. In this design, all cords between objects on different displays would begin or end at the proxy for the display on which the associated object was located. Figure 6-2 demonstrates how this was accomplished.

Sub-Object

Cords: Contextual Associations

Object

Figure 6-2. Cords connecting related objects are routed between displays via the proxy objects Note the Cord connecting the website and its URL appears to ‘pass through’ the proxy.

As shown in Figure 6-2, cords passing through the same proxy are distinguished by color. To address the issue of visual clutter, cords are displayed for a few seconds when an associated object is touched and disappears shortly thereafter. Touching the wall’s proxy on the tabletop (see Figure 6-1) displays all cords passing through it. Although no attempt is made to maintain the visual continuity of the cord between displays, having the cords in the same color, as well as appearing and disappearing in unison, provides some precognitive cues. An alternative design would be to attempt to visually align the edges of the cords between displays, but the multitude of viewing angles inherent in a multi-user table-centric system makes this impossible.

Passing cords through the proxy reduces visual clutter, and also helps the user to understand where the cord is passing to, so that the visual connectivity of cords could be maintained across

130 the displays when users and screens are not aligned. The disadvantage of this approach, however, is that ambiguity is created when multiple cords share the same color: two blue cords may connect objects to the proxies on the table and on an ancillary display, but it is impossible to discern the pairings between the objects across the two screens. A designer employing our technique would be well advised to make cords in the system uniquely identifiable by color.

6.2.3 World in Miniature: WIM

Although there exist several techniques that facilitate control over distant areas of a large surface, such as those presented by Baudisch et al. (2003 & 2004), and Bezerianos and Balakrishnan (2004), the demonstrated utility of radar views, as studied by Nacenta et al. (2005), combined with the results of our studies in control orientation, led us to explore their use in our interaction space. A radar view is a world in miniature (WIM), previously explored by Pierce et al. (1999a, 1999b), where a remote environment is displayed in a scaled format in the work area, and manipulations within the scaled miniature view are transferred to the original space. In our environment, interactions performed on the WIM on the tabletop would directly impact the corresponding region of the ancillary display represented by that WIM. This approach has several desirable properties:

• A direct-touch table-centric paradigm is maintained • Fast movement across multiple displays is possible by simply moving between multiple WIMs. • Every point on the remote display is selectable, which is not true of techniques such as drag-and-pick and Vacuum which collapse the white space around objects to make objects more easily selectable (Baudisch et al. 2003, Bezerianos and Balakrishnan 2005). • Users can comfortably view and manipulate screens that might otherwise require body contortions to be able to see while seated at the table. • Direct-touch alleviates control-orientation issues.

The last point in this list is especially important: the results the previous chapter clearly demonstrate the need to address the issue of multiple input / display mappings inherent in a multi-display environment. The presentation of a direct-touch world in miniature completely eliminates this issue: so long as users direct their view to the WIM while making input, the mapping of input to display is completely overlapped, and so there is no offset at all between 131 control and display spaces. As we will later describe, users in our study struck a balance between looking at the WIM for displays behind them, and looking primarily at the large displays when they were positioned in front of them. This not only enabled them to reduce the amount of physical distortion required, it also allowed them to find an almost ideal trade-off between viewing larger content and optimizing their input / display mapping.

WIMs were integrated into the proxy objects, such that a WIM of the ancillary display was shown below the matching circle. To allow users to dynamically reposition and reorient a WIM, we included a control to display a copy, visually tethered, which could be freely moved, resized and rotated about the table. Further, we surrounded the WIM with a graphical bevelled edge, shaded to match the color of the proxy. This bevel also gives the appearance of depth, providing a window-like feel to the WIM view (Figure 6-3). Note that, in Figure 6-3, the larger WIM is shown taking up most of the content of the tabletop. While this mode provides a great deal of precision for touches on wall content, we anticipate that users will generally prefer a smaller WIM, so as not to occupy so much of the table with the WIMs.

In some systems, a WIM approach is already being used to control large ancillary displays from a control terminal using remote desktop software such as VNC (www.vnc.com/). In our work, we have augmented the WIM concept with dynamic orientation and zoom control, as well as multiple-user pointers and a meta-level zoom to be discussed shortly. Combined, these innovations make the WIM more appropriate for a table-centric control system.

Despite its power, there are disadvantages to a WIM approach. Each requires a large amount of space on the table. Second, orienting the WIM for ease of both viewing and controlling a non- aligned display may be difficult. This issue is explored in our user study, presented later in this chapter.

132 Vertical Display

Table

Figure 6-3. Bottom: the contents of the tabletop: three small WIM’s are shown on the edge closest to the ancillary displays they represent. The user has opened and expanded a larger WIM view of the green ancillary display. Top: the right ancillary display

6.2.4 Meta World in Miniature

Although the coloration and positioning of the proxy are helpful in establishing the mapping of control area to display, we found that it was sometimes difficult to achieve a quick understanding of which vertical display a particular WIM was controlling. To improve this, we extended the WIM to represent not only a single ancillary display in miniature, but also the space as a whole.

To achieve this, we added a control to the tethered WIMs to allow the user to “zoom out” to a photograph of the work area. When the zoom-out button is pressed, the bevelled edge of the window disappears, and the WIM animates to the approximate location of the physical display it represents within a photograph of the room (Figure 6-4). The user can then touch any of the displays in the photo to cause the WIM to zoom into that screen.

133

Figure 6-4. Left to Right: the content of a WIM as it zooms out to show a photograph of the room. The user can then zoom the WIM back in by touching a display in the photograph Although we attempted to correct the perspective of the various screens, in some cases this made the screens too small to see clearly, and the targets too small to select. This approach is similar conceptually to ARIS, presented by Biehl and Bailey (2004, 2005), where application windows are presented iconically in an abstract schematic of the system’s screens. Our approach differs in that the entirety of the screen content is shown, rather than simply iconic representations, making it more practical for environments where applications occupy the entirety of the display. Our use of a photograph of the room, rather than a schematic, is intended to help the participant quickly orient to the content of the meta-WIM.

One could envision a layout of screens with our non-linearly aligned topology such that it would not be possible to view them all within a single photograph. This approach could be augmented by multiple images, or stitched photographs, in order to facilitate the viewing of the system as a whole. If this were done, it would likely require an intermediate zoom step: zooming first to a single photo, as we have done here, which includes the display with which the current window is associated, then again to see the system as a whole. It is also possible to imagine a control environment so large, or densely packed with displays, that the content of the screens shown in our meta-WIM would be so small as to be unrecognisable. We imagine that, in such an environment, a meta-WIM approach would require multiple zoom levels, or the ability to pan around the space.

6.2.5 Multiple Control Granularities

When manipulating objects on the ancillary display, users typically use the input space of the WIM, but keep their visual focus on the larger screen. Because movement in the WIM is exaggerated by the scale difference between the displays, fine operations become

134 difficult. To enable fine-grained operations, we must reduce the control-display (CD) gain between the WIM and associated display, without sacrificing favourable aspects of the WIM design. A simple way to do this is to increase the size of the world in miniature: by increasing its size, the CD gain between the WIM and the ancillary display is reduced, and finer control is afforded. To allow this, we provide both a manual size control and a button to toggle between two preset sizes.

We also wanted to provide users with the ability to adjust the gain without increasing the size of the WIM, since a large ancillary display could mean that even a WIM occupying the entire table might not provide sufficiently fine-grained control. One solution is to allow the contents of the WIM, and the associated display, to be zoomed and thus increase/decrease the CD gain. Unfortunately, this affects the content shown on the ancillary display and can interfere with concurrent actions by other users. Our solution was to allow the user to zoom the WIM view, but leave the view on the ancillary display unchanged. By maintaining direct-touch control within a zoomed WIM view, the result is a reduction in the CD gain (Figure 6-5).

Figure 6-5. Top: screenshot of ancillary display, which remains static during a WIM zoom. Bottom: three stages of a zoom of the world in miniature (partial screenshot of table)

135 There are two potential issues with the zoom we ultimately adopted. First, zooming the WIM while leaving the ancillary display static means that users who are concentrating on the wall may move their hands outside the controlling WIM without realising that they are doing so. To mitigate this, we explored adding a visualisation to the vertical display of the zoomed region of the WIM, but found this limiting when multiple WIMs are active. Note that automatic panning of the WIM contents when the finger reaches the edges of the WIM is also not a good alternative as there is no easy way to distinguish between dragging past the edge of to pan vs. dragging an item from the WIM to table.

Second, zooming the WIM without any contextual information on the table means that users would have to look at the wall or zoom in and out to maintain context. We attempted to resolve this by using lenses within the WIM to provide context for users looking at the table. This approach was ultimately discarded for two reasons. First, the space needed around the periphery of the WIM to visualise the zoom of the lens significantly reduced the size of the available control space. Second, our observations of the zooming interface suggested the additional context information was not necessary for effective use of the overall interaction space. Users would zoom their WIM to perform a specific task, then return to a normal zoom level once the task was complete.

6.2.6 Workspace Pointer

As described previously, we wished to facilitate the use of the WIM as an input-only space, allowing users to use the larger displays as their visual focus. We felt it necessary to provide observers not seated at the table with cues to facilitate their understanding of operations being performed on the ancillary displays. To support these goals, we added telepointers, as in Engelbart and English (1968), to the ancillary display. Whenever a user touches the WIM, the corresponding point on the ancillary display would show a pointer, as in Figure 6-6. Because the hardware we used distinguished between different user’s touches, we augmented these pointers with a color coding to uniquely identify each participant. In addition to providing a reference point for the user performing the action, these pointers could be used by other participants as a visual reference (e.g., “look at this”), and also to reduce interference with concurrent tasks performed on the ancillary displays, by providing other users with awareness of the current focus.

136

Figure 6-6. Each user’s point of contact on one or more WIMs is shown on ancillary displays, uniquely identified by color

As with any attempt to promote awareness, these pointers run the risk of becoming a distraction. Also, for many touch-tables, a single pointer is not enough to visualise the input. It is likely that a different visualisation would be needed for each technology.

6.2.7 Mechanism for Moving (‘Teleporting’) Objects Between Displays

Moving objects (or ‘ Teleporting’ them) between displays using the WIM is accomplished by dragging them from one WIM to another, or from a WIM to the general tabletop area and vice-versa. Figure 6-7 shows an object being moved from the tabletop to an ancillary display. Note that the orientation of the object changes once it has been moved off of the table. Although various orientations might be desirable for objects on the tabletop so that they can, for example, face a particular user, once they are moved to an ancillary display all users share a common “up” vector, and thus the object orientation is corrected.

Figure 6-7. An object is moved (‘teleported’) from the tabletop to an ancillary display by dragging it onto a WIM. The orientation of the object is corrected once it is placed on the vertical display 137 6.3 Example Usage Scenario

6.3.1 Description

We have prototyped an application that utilises the design solutions presented in the last section. We used a DiamondTouch multi-touch table (Dietz and Leigh 2001), top projected with a 1248x1024 projector. The vertical displays used include one 62” (measured diagonally) plasma display, and one 36” LCD display.

The particular application scenario we developed is a police emergency management system that would be part of a larger emergency operation control centre. The purpose of this centre is situational assessment and operations deployment to deal with riots and high-priority criminal targets. At our table are seated the primary decision makers of the centre, such as high-ranking police and city officials. Although they are not included in the scenario, the presence of supporting staff to carry out supporting tasks is assumed.

Participants are seated around the interactive, touch-sensitive table, with two ancillary displays (Figure 6-8).

Figure 6-8. The emergency management scenario environment: an interactive table augmented with two large displays: the video wall (left) and deployment wall (right). Figure 6-9, Figure 6-10, and Figure 6-11 show detailed views of each of the displays

138 On one wall (the Video Wall – left vertical display in Figure 6-8), a surveillance camera monitoring system is augmented with geospatial data to allow participants to monitor ongoing field situations using the visible contextual associations included in our system. The display (Figure 6-9) can be controlled via a WIM on the table, and the video feeds can be moved on screen or dragged onto the table for closer viewing.

Figure 6-9. Real-time surveillance video is displayed on the video wall . The video feeds are augmented with geospatial information to aid with field situation assessment

On another wall (the Deployment Wall ), an application which monitors and allows changes to the location of deployed field units is envisioned. As seen in Figure 6-10, the left pane features an annotated satellite photograph, the center is a zoomed portion of that pane, replacing satellite photography with cartographic information. Unit positions can be viewed and changed by adjusting the positions of their icons on this map. As before, control over the deployment wall from the table is facilitated by a WIM view from the table. A new unit is deployed to the field by moving its icon from the table onto the appropriate position on the map using the world in miniature.

The final display surface is the interactive touch-table, shown in Figure 6-11. On the table non-deployed special-forces units are displayed as icons labelled using visible contextual associations. Also on the table is other information sent there by lower-ranking participants in the room, such as special bulletins (bottom-left), as well as the worlds in miniature of the Video and Deployment Walls. 139

Figure 6-10. An application to allow the monitoring and deployment of special police forces is displayed on the “deployment wall” and controlled from the table

Figure 6-11. The contents of the interactive touch-table, including police-forces unit information, special bulletins, and control areas for the other surfaces which make-up the system

140 In this scenario, our interaction techniques are able to facilitate the identification, analysis, and ultimate resolution of a real-world situation. In particular, the stages where discussion is required are enhanced by the table-centric interactive space, while still leveraging computing technologies to make tasks more efficient.

6.3.2 User Feedback

During a visit to the New York Police Department’s (NYPD) Real Time Crime Center (RTCC) we demonstrated our example scenario to over a dozen high-ranking members of the NYPD, led by Deputy Commissioner James Onolfo.

Figure 6-12. The demonstration of our system to senior members of the RTCC team

The system we envision, a table surrounded by several ancillary displays, varies significantly from the current design of the RTCC (Figure 6-13), where all participants sit facing a single, large, shared display.

141

Figure 6-13. The New York Police Department’s Real Time Crime Center in New York City, USA features several workstations facing a shared 10’ x 26’ tiled display

Despite these differences, our scenario implementation was highly praised: it was agreed by those in attendance that providing the higher-ranking officials with a collaborative, table-centric system would allow for them to more fully participate in processes currently being delegated to others. In addition, it was stated that a system such as the one we envisioned could improve emergency management. It was noted in particular that our system included facilities for a more collaborative system that supported better awareness of field situations for participants. In particular, Deputy Commissioner Onolfo told us: “This isn’t the way we do things now, but it’s the way we should be doing them”.

6.4 Interaction Technique Discoverability

We conducted study to evaluate the effectiveness of our designs. Given that the efficacy of a WIM for control of an ancillary display has already been empirically demonstrated by Nacenta et al. (2005), we focused our study on the general usability of the interaction designs. We evaluated three levels of usability: how quickly and easily users can discover functionality without help or guidance, how effectively they can use each of the functions to perform a simple task, and how effectively the users can combine functions to perform a more complex task.

142 6.4.1 Design and Procedure

The study was conducted in three phases. First, we gave the participants minimal instructions and then asked them to explore the space which consisted of our interaction techniques running on four display surfaces: an interactive table, three large-screen plasma displays, and a projected display. On each display three photographs were shown, and nothing else other than the UI elements described above. Each could be moved, rotated, and resized as seen previously in Shen et al. (2004). They were given only the following instructions:

The system you will be using today is designed to allow you to perform basic operations, and move images on and between the various screens and the table. I will now give you 10 minutes to discover the functionality of the system. Please feel free to try anything you like, make comments, and ask questions. Do you have any questions before we begin? We deliberately kept the instructions to a minimum as we wished to determine which of the techniques users would be able to discover on their own. Additionally, we also wanted to learn if the color and positions of the proxies would be sufficient to allow the user to understand the interconnectedness and topology of the system. The specific features tested were:

Basic Interaction: whether or not the subject was able to perform basic system functions: moving, rotating, and resizing objects.

Proxy as Connection: whether or not the subject understood which proxy controlled which display, and generally how to observe that a proxy represented a particular display (colour matching and spatial proximity).

Proxy Teleport: whether or not the subject moved an object from the table to another screen by moving it and dropping it on to the proxy.

Proxy Position Adjust: whether or not the subject repositioned objects on the vertical displays by dragging them within the proxy’s WIM.

Expand WIM: whether or not the subject pushed the button to evoke the larger WIM.

143 Move Expanded WIM: whether or not the subject moved the larger WIM.

Resize Expanded WIM: whether or not the subject resized the larger WIM.

Maximize Expanded WIM: whether or not the subject found and understood the function of the button on the expanded WIM which resized it to a very large size.

Zoom Expanded WIM: whether or not the subject discovered how to zoom the expanded WIM.

Centre of Zoom: whether or not the subject discovered how to centre the zoom when pressing the zoom in button (first touch the WIM to set the centre point).

Telepointer : whether or not the subject discovered the telepointers shown on the remote display when giving input to any of the WIM.

A video camera recorded the users’ actions, and a post-task interview was conducted with each participant.

Once this first phase 10-minute discovery period and post-task interview was complete, all the interface functionality was demonstrated to the participant in order to prepare them for the second phase of the study. Here, they were asked to perform a series of basic tasks on photographs located on each of the displays in order to test their understanding of each of the system functions and interaction techniques.

In the third phase, each participant was given a more complex grouping and sorting task which required the use of several functions in combination. 36 cards from a standard deck (2-10 of each suit) were randomly distributed across each of the ancillary displays and the table, each of which was uniquely labelled with one of the four suits. Participants were asked to move the cards such that each was placed, in numeric order, on the display labelled with its suit.

144 6.4.2 Participants

Six participants (4 male and 2 female) were recruited from the local community. All were between the ages of 25 and 27, and had achieved at least a bachelor’s degree in a non-CS or HCI discipline. None had previous experience working with computer systems beyond commodity desktop and notebook computers.

6.4.3 Results and Discussion

In the initial discovery-phase of the study, users were asked to explore the system, while the experimenter noted which features they were able to uncover and use correctly. If it was unclear from their use whether or not the user understood the function in question, the interviewer asked them to explain its functionality. Table 6-1 shows which features of the system were discovered by each of the participants.

Participant

Feature User 1 User 2 User 3 User 4 User 5 User 6 Basic Interaction X X X X X X Proxy as Connection X X X X X X Proxy Teleport X X X Proxy Position X X Adjustment Expand WIM X X X X X X Move Expanded WIM X X X X X X Resize Expanded WIM X X X X X X Maximize Expanded X X X X X X WIM Zoom Expanded WIM X X X X X Centre of Zoom X Telepointer X X Table 6-1. Each feature tested in the study, and whether each user discovered that feature

Results from the first part of the study indicate that nearly all aspects of the system are discoverable within a 10-minute exploratory period of using our system: all of the basic operations (resize, move, rotate) on system objects and on the WIMs were discovered by all participants. More importantly, participants were all able to discover and understand the proxy as the connection to the ancillary displays, matched by proximity, color, or both. All were able to discover the ability to move objects on and between the ancillary displays using the WIMs.

145 Three features of the system were not discovered by most of the participants, two of which were only minor aspects of the system: only one of the six participants discovered how to control the centre of the zoom on the world in miniature, and only two participants noticed the pointer displayed on the ancillary display while operating in the world in miniature. More troubling was that users generally did not recognize the small worlds in miniature, built-in to the proxy objects, as control centres. Only three of the six users discovered that they could drag objects here to move them to the vertical display, and only two users discovered that they could give input that would be sent to the larger display. A likely explanation was that, because they found how to expand the larger WIM so quickly, they did not use the smaller WIM for these tasks.

In the second phase, where users performed simple tasks following a demonstration of all functions, all participants were able to complete all tasks without further instruction.

In the third phase of the study, in which participants were asked to perform a more complex sorting task that required extensive use of all four displays, all participants were able to complete the task. Worth noting, however, was the trade-off that users seemed to experience between turning their heads and enlarging the WIM: of the three vertical displays, only one was positioned within 45 degrees of the centre of the user’s field of vision when sitting at the table. For this display, participants tended to leave the WIM small, such that the suit of similarly coloured cards could not be distinguished (the two of clubs and the two of spades, for example, were not distinguishable through the WIM at this size). Participants tended to look at the larger screen rather than at the WIM in this case. For the other displays, one situated at approximately 90 o, and the other at approximately 135 o, the participants tended to enlarge the WIM and not look at these screens at all. The disadvantage of enlarging the WIM is that it would be more likely to occlude cards positioned on the table, necessitating frequent repositioning to access those cards – participants seemed more willing to move the WIM using their hands than to leave it reduced and turn their heads away from the table. This result is quite desirable: as we described in detail in the previous chapter, users had difficulty operating controls for a display directly behind them. This result, however, indicates that users will make a reasonable trade-off between efficiency, comfort, and use of space.

146 6.5 Implications for Design

The contributions of this chapter are twofold: identification of interaction and visualization issues that arise in the given problem space of single tabletop augmented with multiple ancillary displays, and the development of a suite of interaction and visualization techniques designed to address those issues. Coupled with a real-world usage scenario and user study, the end result is a better understanding of how such table- centric spaces can be best utilized for collaborative applications, as well as a prototype interface that facilitates such use. Those seeking to use this chapter as an aid for design would best be served not only by taking up the techniques we describe, but by carefully considering the issues we describe in 6.1 , considering how they apply to their designs.

Although the interaction techniques we proposed found great success, there exist several potential avenues for future research.

6.6 Avenues for Future Research

One of the triumphs of the world in miniature technique is its near-elimination of the control orientation problem on two fronts. First, users can choose to either look at the display while giving input to the WIM, or look at the WIM itself, maintaining a balance between display size and awkward viewing. Second, our design allows the user to reduce the offset between control and display spaces, since input while looking at the WIM is direct-touch. This triumph, however, comes at a clear cost: users looking at the WIM lose the shared context of the ancillary display, while simultaneously reducing their functional display size. Future researchers should examine how to encourage and facilitate interaction while looking at the ancillary display, whenever possible, probably by implementing new techniques.

Second, although we have gone to great lengths to help users to interact easily with the ancillary displays via the portals and worlds in miniature, the fluidity of our implementation is sometimes lacking: users were sometimes observed spending several seconds initiating and positioning multiple worlds in miniature, indicating a poor balance between time spent on UI management and the task.

147 Also to be considered is the amount of primary interaction space required for worlds in miniature. When the larger views are opened, they require significant fractions of the table space. At times, the table seems to act primarily as a control mechanism for the ancillary displays, rather than the displays serving to augment the table. There is a need, therefore, to either reduce the amount of table space required, or to provide mechanisms to more easily transition between table and display focus. This will be especially important as ancillary displays grow larger, requiring either larger fractions of the table, or constant zoom in order to allow reasonable control/display gain between the world in miniature and those displays.

Finally, there is a clear need to provide further validation of our interaction techniques. Though informative, the feedback on our application scenario and our user study does not provide sufficient testing to allow designers to take up our techniques without closer examination.

148 Chapter 7: Conclusions

7.1 Summary

Since at least the days of the mythical King Arthur, the table has been the centre of meetings, and at times, the focal point of discussion. As technology evolved, so too did interaction spaces, which were augmented with vertical displays. These displays have been installed, or built into laptops that participants carry with them. Innovations in touch technology have enabled the use of the table itself as an information display, allowing a return to a table-centric focus. Such a refocusing can come at a cost, however, as the net number of pixels available to display information decreased. Designers of table-centric interactive spaces seek to address this issue by augmenting the tabletop with vertical displays. Unlike other multi-display topologies, the focus in a table-centric space is enablement of all interaction directly through the table, rather than requiring participants to change input devices, or move to interact directly with other displays.

As we have seen, table-centric interactive spaces offer both opportunities and challenges. While the use of such spaces promises to extend direct-touch tables by providing additional display area, this extension comes at the cost of needing to consider a number of design and interaction issues. This thesis is a first step towards the identification, classification, and addressing of these issues.

In Chapter 1 of this thesis, we provide a discussion of three new angles between users and display which are introduced in a table-centric interactive space. These are the angle of each user to the surface of the table, the angle between each user and the displayed content on the table, and the angle between the user and an ancillary display. We then identify the issue of how these angles might affect each of three fundamental tasks: reading, perception of graphics, and the control of ancillary display from the table. Crossing the three angles and three fundamental tasks yields the nine-cells of Table 1-2. We then use this table to classify relevant previous efforts in psychology and HCI, and to identify areas not yet examined. These areas represent important human factors questions to explore, and motivated our investigation of these issues in Chapters 3-5.

149 In Chapter 2, we set the stage for this examination with a detailed review of background literature. We discussed existing research in tabletop hardware, and in various extensions to the direct-touch tabletop. We then provided an extensive review of behavioural, perceptual, and physical issues related to tabletop design, and interaction design for tabletops. Finally, we examined interactive spaces, including multi-terminal roomware, and multi-surface systems. While a great deal of this work is valuable to designers, unique properties of table-centric interactive spaces required additional investigation.

Each of Chapters 3-5 examined a subset of the crossing of the three fundamental tasks and the unusual display angles inherent in these spaces. To facilitate a thorough examination, these are addressed in a task-centric way, with each chapter addressing the affects of the angles on one of reading, graphical perception, or control.

Chapter 3 provided the first such examination. It began with a review of previous results in psychology, and pointed to the need to reopen the issue of the effect of orientation on reading. Previous results in HCI have explored two of our angles of concern, but left unexplored is the effect of users trying to read on-screen content oriented towards another user sitting at the table. We then presented the results of our experiment, which tested the effects of this angle, as well as the angle between the user and the physical tabletop, on the speed of reading text. Our results showed that users pay a much smaller penalty than was previously suggested by the results in psychology. In particular, the angle inherent in displaying text on a flat surface receding away from the eyes had only a very small effect on reading speed. Additionally, the orientation of text on the display was found to have a significant but manageable effect on reading speed, indicating increases in reading times of words, phrases, and 6-digit numbers of only a few seconds. From the results of this study, designers are able to decide how to orient on-screen content, and perhaps provide multiple copies when viewers would be subjected to significant performance penalties. Chapter 3 also provided insights into the effect of viewing content from different sides of the table on performance of visual search. It reported that, while viewing text at non- optimal angles did hurt search times, this effect was less than was suggested in the results of the first experiment.

150 In Chapter 4, we turned our attention to the issue of graphical perception. Noting that previous results suggest no effect for viewing content oriented towards another user, we focused on the two remaining angles. We conducted two experiments, both of which found significant effects for the angle of viewing on the accuracy of the decoding of information graphics. These effects were correlated with the vertical distance between the two stimuli being compared, suggesting that it may be important for designers to minimize this distance in the display of information graphics. Of particular interest is that some types of graphical representations proved more robust than others to viewing under these angles. Taken together, the results of these experiments point designers towards particular types of graphical encoding which might be best suited to display in table- centric interactive spaces, such as those which utilize judgements of line length in place of relative position of stimuli, such as in the modified box-plot seen in Figure 4-16.

In Chapter 5, we then examined the issue of control. Because control of the content on the table is generally performed in a direct-touch way, the angles of the user to the table and of content to the user are unlikely to have an effect on accuracy. Instead, we focused on how various seating positions relative to differing canonical positions of vertical displays affect the accuracy of pointing on those displays when using the table for input. In our first experiment, we allowed participants to orient their own controls for input to a display at each of 8 positions. In the second experiment, we forced the user to use each of 8 orientations of control to give input to a display at each of 8 positions around the table. The results of the first study inform the designer as to performance when users are able to set the orientation of their controls, while also hinting at the preferred control orientation for each display position. The results of the second study provide detailed information to inform designers about how fixed controls should be positioned and oriented for users at each display position.

Following our examination of angles and fundamental tasks, we turned our attention in Chapter 6 to the design of interaction techniques. These techniques included mechanisms for overcoming additional issues inherent in table-centric spaces. These issues included the non-linear alignment of ancillary displays, the need to provide awareness of input- output mappings and contextual associations, the need to maintain a direct-touch

151 paradigm, working to minimize the number of control orientations, providing multiple granularities of remote interaction, and supporting non-interfering concurrent tasks. Our techniques addressed each of these issues, as well as providing means of overcoming issues of angle as they relate to control of ancillary displays.

In order to evaluate our interaction techniques, we provided two means of validation: an application scenario, and a user study. The application scenario provided for a potential control room environment for managing police forces in a borough of New York City. This scenario was evaluated and lauded by a Deputy Commissioner of the New York Police Department. The user study focused on how easily our techniques could be discovered, and found that participants were able to quickly learn to use the majority of their functionality without aid.

7.2 Additional Display Parameters

Table-centric interactive spaces describe a specific class of interactive spaces: a multi- user, interactive touch table, surrounded by one or more ancillary displays. Although this class includes a multitude of possible configurations to suit many different environments, there are a number of possible configurations that it does not include. We now review some of these.

7.2.1 Very Large Displays

Very large or wall-sized displays, offer unique affordances and properties for perception, as discussed by Czerwinski et al. (2003). An extension of the table-centric space that should be considered is the combined use of a table and wall-sized display. Similar conceptually to the work described by Malik et al. (2005), such an environment would likely be configured such that multiple users sit around a table, with no user occupying the side of the table adjacent to the wall. Although not specifically addressed in the text of the thesis, implicit in some of the work we have done is that the ancillary displays in an interactive space are large screens, perhaps LCD or Plasma. Some of the work we have done, however, may not apply to extremely large displays, such as wall-sized displays.

152 In particular, the angle of the user’s seating position to a wall-sized display varies across the display surface. On-screen content shown at one side of the display is at a different apparent angle to the user than content on the other side of the display. In some cases, these angles may be somewhat extreme, and relative positions outside of the canonical positions described in Chapter 5. Such extreme angles suggest a need for further study. This would include re-examining all three of the fundamental tasks we have described.

In addition, some of the interaction techniques we have described may not be robust to very large displays. The world in miniature metaphor, for example, would require either extreme control / display gain, or would need to always be zoomed to a portion of the large display. Alternative techniques, such as the one described by Malik et al. (2005) should be considered.

7.2.2 Table Size

Also of interest are alternative table sizes. Previous work in table sizes by Ryall et al. (2004) describes three classes of table sizes: those that are sufficiently small that all points can be reached by all users, those that are large enough that no one user can reach all points, but that all points can be reached by a user, and those that are so large that there exist points that can be reached by no users. Tables of sufficiently large size would introduce extreme angles between users’ eyes and on-screen content, beyond the angles investigated in Chapters 3 and 4. Additionally, the techniques we have described in Chapter 6 do not address the issue of tables that do not allow on-screen content to be passed between participants. Designers working with sufficiently large tables would need to carefully reconsider how the work in this thesis can be applied before making their designs.

7.2.3 Additional Tables

An additional issue is how to address the presence of additional tables in the space. One could imagine a configuration where multiple teams are working, each at a separate table, in a table-centric interactive space. How these teams might share the ancillary displays, and how the tables might behave to support inter-table communication is an open area open for research.

153 7.2.4 Multiple Table-Centric Spaces

Section 2.5.4 describes several systems intended to support tabletop users working remotely, many of which include the use of a vertical display to support collaboration. These systems, however, are dedicated to supporting multiple remote tables and not table-centric spaces. As tabletops are extended into table-centric interactive spaces, the requirements for a system to support remote collaboration must also be extended.

7.2.5 3D Displays

As Grossman and Wigdor (2007) describe, there is a multitude of systems intended to provide interaction in 3D on or around a tabletop. The presence of 3D technologies in a table-centric space introduces a profusion of new viewing angles to be considered, while also creating new challenges for the design of interaction techniques.

In addition to these display parameters, there exist other issues which may be of interest to designers and researchers. We review these for consideration as future research.

7.3 Avenues for Future Research

While we believe that we have identified and begun to offer solutions to a number of issues with table-centric spaces, there remain a number of potential avenues for future research. Several of these avenues have been explored or identified in the relevant previous chapters. In addition to those relating to individual chapters, several research areas have emerged which span the material of multiple chapters. We now review several of these.

7.3.1 Additional Physical Parameters

We have examined three angles between user and display introduced in a table-centric interactive space: the angle of each user to the surface of the table, the angle between each user to the content on the table, and the angle between the user and an ancillary display. While these angles represent an issue of significant concern to designers, additional physical parameters may need to be considered. These might include the distance between the table and the ancillary displays, the size of those displays, and the height at which those displays are positioned.

154 7.3.2 Task Considerations

The fundamental tasks we described, reading, perception of graphics, and control, offer a reasonable cross-section of fundamental tasks performed while working with a digital system. None the less, it is possible that these fundamental tasks, when performed with additional context, or in combination, may interfere with one another, or be relieved from some of the effects we observed. There may also be additional higher-level effects which may mitigate or affect the results we describe, particularly when cognitive and task elements are considered. An examination of higher level tasks is needed to provide a thorough understanding of interaction in a table-centric interactive space.

In addition, it is possible that the experimental tasks we asked our participants to perform were ideal in their mapping to the low-level tasks we wished to examine. In Chapter 3, for example, it was our aim to examine reading speed exclusively, while the experimental task required participants to memorize the text before entering it in to the system. While this is in keeping with experimental practices described in the literature, it is worth noting that there may exist a confound between the reading and memorization components. Careful consideration of experimental tasks may reveal other confounds, and validation with higher level tasks will indeed be important.

7.3.3 Additional Interaction Mechanisms

The techniques described in Chapter 6 provide mechanisms to support most user interaction in a table-centric space. Nonetheless, alternative mechanisms may be required.

7.3.3.1 Non-Tabletop Input

In Chapter 1, we defined table-centric interactive spaces such that input is made only from the tabletop. While this definition forced us to consider issues and interactions in support of that goal, it is likely that such spaces would include a mix of input devices. Designers have considered input from personal devices, as well as input directly to the ancillary displays. Although inputs to these devices have been considered in a non-table- centric space, yet to be considered is how to mix these input mechanisms with the techniques for table-centric input we have described.

155 7.3.3.2 Alternative Table / Wall Couplings

The world in miniature metaphor we described in Chapter 6 was designed to adhere to the design principle of maintaining a direct-touch input paradigm, as described in section 6.1.4 . As we have seen, previous work has described alternative modes of coupling between the table and ancillary displays. An example of one such system is that described by Forlines et al. (2006b), where users adjust the position of virtual cameras on the tabletop to control the view of the vertical displays. How this and other mechanisms for control can coexist with our techniques is an area of future research.

7.3.4 Extending from One to Many Users

A potentially significant limitation of the experiments we have described are their reliance on an assumption of scalability. In all cases, these experiments were conducted with a single participant, acting in isolation. While the presence of additional participants is unlikely to have affected performance of fundamental tasks, there is a clear need to examine issues that arise in multi-surface environments given the presence of additional users. How, for example, does the position of alternative users affect user preference for display position? How does the issue of territoriality described by Scott et al. (2004) extend from the table to table-centric spaces? Indeed, how do any number of multi-user behavioural issues described in section 2.4.2 extend from multi-user tables to table- centric spaces? What new issues arise in behaviour with the addition of the vertical displays? These and other questions will be of critical importance for researchers and designers alike, as table-centric interactive spaces continue to be explored.

7.3.5 Application Building

Section 6.3 describes an application scenario built using the interaction techniques described in that chapter. In that scenario, a table-centric interactive space is used to monitor and deploy police forces throughout a major metropolitan centre. This scenario received high praise from actual police officials in that centre. Although compelling, the application described is just one of many potential uses for table-centric spaces. These spaces are particularly suited to applications that require intensive group collaboration,

156 that benefit from large amounts of display space, and those which have a natural mapping to horizontal and vertical displays. We now examine example applications.

7.3.5.1 Sports Strategy

Typical of sports strategy sessions is a plan-view of the playing field, outlining plays using annotation specific to the sport. A multi-surface environment may well be excellently suited to such an environment. The table serves as a natural surface for the display of the playing field, with ancillary displays dedicated to alternate strategies, in- game footage, or 3D simulations.

7.3.5.2 Surveillance

Many large installations include video monitoring equipment. This typically is comprised of a multitude of cameras installed throughout the facility, with one or more observing stations monitored by security guards. The arrangement of these stations is typically a bank of video monitors, which are often multiplexed to show output of more than one camera. A table-centric interactive space could be employed to serve as a monitoring station, with a plan view of the facility shown on the table, with vertical displays used to show the output of cameras.

7.3.5.3 Research and Exploration Spaces

Groups of students or researchers often come together to share and explore information. A table-centric interactive space for research and exploration would allow participants to use the table as a primary information foraging area, collecting and examining materials from online sources. Vertical displays could serve as both primary interaction areas, displaying content from digital libraries, as well as collection areas for information selected by the users. It is likely that such a space would be greatly aided by providing facilities for users to connect their own laptops to share information.

7.4 Conclusions

We have presented an examination of the issues associated with augmenting digital tabletops with ancillary, vertical displays to create table-centric interactive spaces. By focusing on three problems of angle created in these spaces, and examining how each of 157 these affects the performance of low-level tasks, we were able to provide a thorough exploration of the fundamental challenges faced by designers of such spaces. To further aid these designers, we then presented and evaluated a series of interaction techniques, as constrained by a set of design principles we identified. We then built and evaluated an interaction scenario designed for use in the environment we envision. Finally, we presented a series of additional parameters which might be considered to expand our definition of table-centric interactive spaces.

As technologies continue to evolve, the declining financial costs of constructing such spaces will stimulate their proliferation and increased prevalence in both professional and personal settings. We believe that designers will be well served in considering the issues we have presented and applying the tools we have provided in this thesis.

158 Bibliography

Adam, J. (2002). Perception of Exocentric Distance in a Static Virtual Environment . M.A.Sc. thesis, Department of Mechanical and Industrial Engineering, University of Toronto.

Agrawala, M., Beers, A., McDowall, I., Frohlich, B., Bolas, M., and Hanrahan, P. (1997). The two-user Responsive Workbench: support for collaboration through individual views of a shared space. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques , ACM Press/Addison-Wesley Publishing Co., 327-332.

Albinsson, P. and Zhai, S. (2003). High precision touch screen interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA, April 5 - 10, 2003). CHI '03. ACM Press, New York, NY, 105-112.

Baird, J.C. (1970), Psychophysical Analysis of Visual Space , New York: Pergamon Press.

Ball, R. and North, C. (2005) Effects of tiled high-resolution display on basic visualization and navigation tasks. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA, April 02 - 07, 2005), CHI ’05. ACM Press, New York, NY, 1196-1199.

Baudisch, P., Cutrell, E., Hinckley, K., and Gruen, R. (2004). Mouse ether: accelerating the acquisition of targets across multi-monitor displays. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM Press, New York, NY, 1379-1382.

Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. (2003). Drag-and-pop and drag-and-pick: Techniques for accessing remote screen content on touch- and pen-operated systems. In Proceedings of Interact 2003 (Zurich Switzerland, August 2003), 57-64.

Beattie, V., Jones, M.J. (2002). The impact of graph slope on rate of change judgements in corporate reports. ABACUS 38 (2), 177-199.

Beaudouin-Lafon M. (2001). Novel Interaction Techniques for Overlapping Windows. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM Press, New York, NY, 153-154.

Begeman, M., Cook, P., Ellis, C., Graf, M., Rein, G., and Smith, T. (1986). Project Nick: meetings augmentation and analysis. In Proceedings of the 1986 ACM Conference on Computer-Supported Cooperative Work (Austin, Texas, December 03 - 05, 1986). CSCW '86. ACM Press, New York, NY, 1-6.

159 Benko, H., Ishak, E., and Feiner, S. (2005). Cross-dimensional gestural interaction techniques for hybrid immersive environments. In Proceedings of the IEEE Virtual Reality Conference 2005 (VR 2005), 209-216, 327.

Benko, H., Wilson, A., and Baudisch, P. (2006). Precise selection techniques for multi- touch screens. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM Press, New York, NY, 1263-1272.

Bentley, R., Hughes, J., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., and Sommerville, I. (1992). Ethnographically informed systems design for air traffic control In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work (Toronto, Ontario, Canada, November 01 - 04, 1992). CSCW '92. ACM Press, New York, NY, 123-129.

Berard, F. (2003). The Magic Table: Computer Vision Based Augmentation of a Whiteboard for Creative Meetings. In CD-ROM Proceedings of the IEEE International Conference in Computer Vision , Workshop on Projector-Camera Systems (Nice, France, October 12, 2003) (ICCV ‘03), (no page numbers).

Bertin, J. (1977). La Graphique et le traitement graphique de l'information. Flammarion, Paris, France.

Bezerianos, A. and Balakrishnan, R. (2005). The Vacuum: Facilitating the manipulation of distant objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM Press, New York, NY, 361-370.

Biehl, J. and Bailey, B. (2004). ARIS: an interface for application relocation in an interactive space. In Proceedings of Graphics interface 2004 (London, Ontario, Canada, May 17 - 19, 2004). ACM International Conference Proceeding Series, vol. 62. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, 107-116.

Biehl, J. and Bailey, B. (2005). A scalable, portal-based interface for managing applications in interactive workspaces. Technical Report UIUCDCS-R-2005- 2542, 2005. University of Illinois at Urbana-Champaign.

Bruijn, O. and Spence, R. (2001). Serendipity within a Ubiquitous Computing Environment: A Case for Opportunistic Browsing. Proceedings of the 3rd international Conference on Ubiquitous Computing (Atlanta, Georgia, USA, September 30 - October 02, 2001). Lecture Notes In Computer Science, vol. 2201. Springer-Verlag, London, 362-370.

Carpendale, M.S.T. (2003). Considering Visual Variables as a Basis for Information Visualisation. Computer Science Technical Report #2001-693-16 . Department of Computer Science, University of Calgary.

160 Catford, N. and Jenner, B. (2001). RAF Uxbridge - Battle of Britain Ops. Room. www.subbrit.org.uk/rsg/sites/u/uxbridge/.

Chapman, J. (1923): Chapman-Cook speed of reading test . Ames, IA, Iowa State University Press.

Cleveland, W.S., McGill, R. (1984). Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. In Journal of the American Statistical Association , 79 (387), 531-553.

Cleveland, W.S., McGill, R. (1985). Graphical perception and graphical methods for analyzing and presenting scientific data. In Science, 229 (4716), 828-833.

Chen, F., Close, B., Eades, P., Epps, J., Hutterer, P., Lichman, S., Takatsuka, M. Thomas, B., Wu, M. (2006). ViCAT: visualisation and interaction on a collaborative access table. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 59-62.

Corballis, M.C., McLaren, R. (1984). Minding one's Ps and Qs: Mental rotation and mirror-image discrimination. In Journal of Experimental Psychology: Human Perception and Performance , 10. p. 318-327.

Corballis, M.C., Macadie, L., Beale, I.L. (1985). Mental rotation and visual laterality in normal and reading disabled children. In Cortex , 21, 225-236.

Cotting, D. and Gross, M. (2006). Interactive environment-aware display bubbles. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06, 245-254.

Coutaz, J., Lachenal, C., Dupuy-Chessa, S. (2003). Ontology for Multi-surface Interaction. In Proceedings of Human-Computer Interaction . Interact’03. IOS Press, IFIP, 447-454.

Covi, L., Olson, J., and Rocco, E. (1998). A room of your own: What do we learn about support of teamwork from assessing teams in dedicated project rooms? In Proceedings of the First international Workshop on Cooperative Buildings, integrating information, Organization, and Architecture . Lecture Notes In Computer Science, vol. 1370. Springer-Verlag, London, 53-65.

Cunningham, H. (1989). Aiming error under transformed spatial mappings suggests a structure for visual-motor maps. In Journal of Experimental Psychology: Human Perception and Performance , 15 (3), 493-506.

Cunningham, H.A. and Welch, R.B. (1994). Multiple concurrent visual-motor mappings: implications for models of adaptation. Journal of Experimental Psychology: Human Perception and Performance , 20(5), 987-999.

161 Cutler, L. D., Fröhlich, B., and Hanrahan, P. (1997). Two-handed direct manipulation on the responsive workbench. In Proceedings of the 1997 Symposium on interactive 3D Graphics (Providence, Rhode Island, United States, April 27 - 30, 1997). SI3D '97. ACM Press, New York, NY, 107-114.

Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G., and Starkweather, G. (2003). Toward characterizing the productivity benefits of very large displays. In Proceedings of Human-Computer Interaction . Interact’03. IOS Press, IFIP, 9-16.

Dempsey, C., Laurence, D., and Tuovinen, J. (2002). Gestalt theory in visual screen design: a new look at an old subject.. In Proceedings of the Seventh World Conference on Computers in Education Conference on Computers in Education: Australian Topics - Volume 8 (Copenhagen, Denmark, July 29 - August 03, 2001). ACM International Conference Proceeding Series, vol. 26. Australian Computer Society, Darlinghurst, Australia, 5-12.

Dietz, P. and Leigh, D. (2001). DiamondTouch: a multi-user touch technology. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM Press, New York, NY, 219-226.

Dobbelsteen, J.v.d., Brenner, E., and Smeets, J. (2004). Body-centered visuomotor adaptation . In Journal of Neurophysiology , 92 (1) , 416-423.

Ellermeier, W., Faulhammer, G. (2000). Empirical evaluation of axioms fundamental to Stevens's ratio-scaling approach: I. Loudness production. In Perception & Psychophysics , 62, 1505-1511.

Engelbart, D. and English, W. (1968). A research center for augmenting human intellect. AFIPS Fall Joint Computer Conference , 295-410.

Esenther, A. and Ryall, K. (2006). Fluid DTMouse: better mouse support for touch-based interactions. In Proceedings of the Working Conference on Advanced Visual interfaces (Venezia, Italy, May 23 - 26, 2006). AVI '06. ACM Press, New York, NY, 112-115.

Everitt, K., Shen, C., Forlines, C., and Ryall, K. (2006). MultiSpace: Enabling electronic document micro-mobility in table-centric, multi-device environments. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems . TableTop 2006, 8-15.

Everitt, K., Shen, C., Ryall, K., Forlines, C. (2005). Modal spaces: spatial multiplexing to mediate direct-touch input on large displays. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA, April 02 - 07, 2005). CHI '05. ACM Press, New York, NY, 1359-1362.

Fechner, G.T. (1860). Element der Psychophysik . Leipzig: Breitkopf & Harterl.

162 Fischer, M., Stone M., Liston, K.,m Kunz, J., Singhal, V. (2002). Multi-stakeholder collaboration: The CIFE iRoom. In Proceedings of the CIB W78 Conference 2002 , Aahus, Denmark, 6-13.

Fitzmaurice, G. W., Ishii, H., and Buxton, W. A. (1995). Bricks: laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, May 7 - 11, 1995), 442-449.

Fitzmaurice, G. W., Balakrishnan, R., Kurtenbach, G., and Buxton, B. (1999). An exploration into supporting artwork orientation in the user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI '99. ACM, New York, NY, 167-174.

Forlines, C. and Shen, C. (2005). DTLens: multi-user tabletop spatial data exploration. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 119-122

Forlines, C., Shen, C., Vernier, F. and Wu, M. (2005). Under My Finger: Human Factors in Pushing and Rotating Documents Across the Table. in Proceedings of Human- Computer Interaction - INTERACT 2005: IFIP TC13 International Conference , (Rome, Italy, 2005), 994-997.

Forlines, C., Shen, C., Wigdor, D., and Balakrishnan, R. (2006a). Exploring the effects of group size and display configuration on visual search. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (Banff, Alberta, Canada, November 04 - 08, 2006). CSCW '06. ACM, New York, NY, 11-20.

Forlines, C., Esenther, A., Shen, C., Wigdor, D., and Ryall, K. (2006b). Multi-user, multi-display interaction with a single-user, single-display geospatial application. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06. ACM, New York, NY, 273-276.

Galanter, E. (2004), Extracting magnitude estimations of loudness from pair wise judgments. In The Journal of the Acoustical Society of America , 115(5), 2534.

Gescheider, G.A. (1988). Psychophysical Scaling. In Annual Review of Psychology , 39, 159-200.

Green, D.M., Luce, R.D. (1974). Variability of magnitude estimates: a timing theory analysis. In Perception & Psychophysics , 15(2), 291-300.

Green, M. (1991) Visual search, visual streams, and visual architectures. In Perception and Psychophysics , 50, 388-403. 163 Green, M. (1996). Toward a Perceptual Science of Multidimensional Data Visualization: Bertin and Beyond . http://web.archive.org/web/19991104170601/http://www.ergogero.com/dataviz/d viz0.html. (archive).

Grossman, T., Wigdor, D., and Balakrishnan, R. (2004). Multi-finger gestural interaction with 3d volumetric displays. In Proceedings of the 17th Annual ACM Symposium on User interface Software and Technology (Santa Fe, NM, USA, October 24 - 27, 2004). UIST '04. ACM, New York, NY, 61-70.

Grossman, T., Wigdor, D., and Balakrishnan, R. (2007). Exploring and reducing the effects of orientation on text readability in volumetric displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 483- 492.

Grossman, T., Wigdor, D. (2007). Going Deeper: a Taxonomy of 3D on the Tabletop. In Proceedings of he Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems , 137-144.

Han, J. Y. (2005). Low-cost multi-touch sensing through frustrated total internal reflection. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 115-118.

Hancock, M. S., Shen, C., Forlines, C., and Ryall, K. (2005). Exploring non-speech auditory feedback at an interactive multi-user tabletop. In Proceedings of Graphics interface 2005 (Victoria, British Columbia, May 09 - 11, 2005). ACM International Conference Proceeding Series, vol. 112. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, 41-50.

Hancock, M., Vernier, F. D., Wigdor, D., Carpendale, S., Shen, C (2006). Rotation and Translation Mechanisms for Tabletop Interaction. In Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (IEEE TableTop) , 79-86.

Harrison, C., Amento, B., Kuznetsov, S., Bell, R.( 2007). Rethinking the progress bar. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 115-118.

Heath, C. and Luff, P. (1992). Collaboration and control: Crisis management and multimedia technology in London underground line control rooms . In Journal of Computer Supported Cooperative Work , 1(1), 24-48.

Hedge, A., (2002). Anthropometry and Workspace Design, in DEA 325/651.Cornell, NY.

164 Helmholtz, H. (1866). Treatise on physiological optics .

Hindmarsh, J. and Pilnick, A. (2002). The tacit order of teamwork: Collaboration and embodied conduct in anaesthesia . In Sociological Quarterly , 43(2), 139-164.

Hinrichs, U., Carpendale, S., Scott., S.D., Pattison., E, (2005). Interface Currents: Supporting fluent collaboration on tabletop displays. In Lecture Notes in Computer Science . Springer-Verlag, Germany, 185–197.

Hinrichs, U., Carpendale, S., and Scott, S. D. (2006). Evaluating the effects of fluid interface components on tabletop collaboration. In Proceedings of the Working Conference on Advanced Visual interfaces (Venezia, Italy, May 23 - 26, 2006). AVI '06. ACM, New York, NY, 27-34.

Hodges, S., Izadi, S., Butler, A., Rrustemi, A., and Buxton, B. (2007). ThinSight: versatile multi-touch sensing for thin form-factor displays. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM Press, New York, NY, 259-268.

Huey, E. (1898): Preliminary Experiments in the Physiology and Psychology of Reading. In The American Journal of Psychology , vol. 9, 575-586.

Hutterer, P., Close, B.S., Thomas, B.H. (2006).Supporting mixed presence groupware in tabletop applications. In Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (IEEE TableTop) , 63-70.

Ishii, H. and Kobyashi, M., (1993). Integration of interpersonal space and shared workspace: Clearboard design and experiments. In ACM Transactions on Information Systems , 11 (4), 349-375.

Izadi, S., Agarwal, A., Criminisi, A. (2007). C-Slate: A Multi-Touch and Object Recognition System for Remote Collaboration using Horizontal Surfaces. In Proceedings of he Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems , 3-10.

Johanson, B., Fox, A., Winograd, T., (2002a). The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms. IEEE Pervasive Computing Magazine 1(2), April-June 2002.

Johanson, B., Hutchins, G., Winograd, T., and Stone, M. (2002b). PointRight: experience with flexible input redirection in interactive workspaces. In Proceedings of the 15th Annual ACM Symposium on User interface Software and Technology (Paris, France, October 27 - 30, 2002). UIST '02. ACM, New York, NY, 227-234.

Johanson, B., Ponnekanti, S., Sengupta, C., and Fox, A. (2001). Multibrowsing: Moving Web Content across Multiple Displays. In Proceedings of the 3rd international Conference on Ubiquitous Computing (Atlanta, Georgia, USA, September 30 - 165 October 02, 2001). Notes In Computer Science, vol. 2201. Springer-Verlag, London, 346-353.

Kagerer, F., J.Contreras-Vidal, and Stelmach, G. (1997). Adaptation to gradual as compared with sudden visuo-motor distortions . Experimental Brain Research , 115(3) , 557-561.

Kakehi, Y., Hosomi, T., Iida, M., Naemura, T., Matsushita, M. (2006). Transparent tabletop interface for multiple users on Lumisight table. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 143-150.

Kato, H., Billinghurst, M. Poupyrev, I., Imamoto, K., Tachibana, K. (2000). Virtual Object Manipulation on a Table-Top AR Environment. In IEEE and ACM International Symposium on Augmented Reality (ISAR'00), 2000, 111.

Khandelwal, M. (2006). Teaching Table a tangible mentor for pre-kindergarten math education . MSc thesis, School of Literature, Communication and Culture Ivan Allen College Georgia Institute of Technology.

Kim, W.S., Ellis, S.R., Tyler, M. E., Hannaford, B., and Stark, L.W. (1987).Quantitative evaluation of perspective and stereoscopic displays in a three-axis manual tracking task. In IEEE Transactions on Systems, Man, and Cybernetics , 17, 61-71.

Kim, W.S., Tendrick, F., and Stark, L. (1991). Visual enhancements in pick-and-place tasks: human operators controlling a simulated cylindrical manipulator. In Ellis, S.R. (Ed.), Pictorial Communication in Real and Virtual Environments . London: Taylor and Francis, 265-282.

Koike, H., Nagashima, S., Nakanishi, Y., Sato, Y. (2004). EnhancedTable: Supporting small meetings in ubiquitous and augmented environment. IEEE Pacific-Rim Conf. on Multimedia (PCM2004), 97-104.

Koike, H., Sato, Y., Kobayashi, Y. (2001). Integrating Paper and Digital Information on EnhancedDesk: a Method for Realtime Finger Tracking on an Augmented Desk System. ACM Trans. Comput.-Hum. Interact. 8, 4 (Dec. 2001), 307-322.

Koriat, A. and Norman, J. (1984). What is rotated in mental rotation? Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 10, 1984, 421- 434.

Koriat, A. and Norman, J. (1985). Reading Rotated Words. In Journal of Experimental Psychology; Human Perception and Performance, 11 (4) 1985, 490-508.

Krakauer, J., Pine, Z., Ghilardi, M., and Ghez, C. (2000). Learning of visuomotor transformations for vectorial planning of rearching trajectories . In Journal of Neuroscience , 20(23), 8916-8924.

166 Kruger, R., Carpendale, S., Scott, S. D., and Greenberg, S. (2003). How people use orientation on tables: comprehension, coordination and communication. In Proceedings of the 2003 international ACM SIGGROUP Conference on Supporting Group Work (Sanibel Island, Florida, USA, November 09 - 12, 2003). GROUP '03. ACM, New York, NY, 369-378.

Kruger, R., Carpendale, S., Scott, S. D., and Greenberg, S. (2004). Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication. In Computer Supported Cooperative Work 13, 5-6 (Dec. 2004), 501-537.

Kruger, R., Carpendale, S., Scott, S. D., and Tang, A. (2005). Fluid integration of rotation and translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 601-610.

Krüger, W., Bohn, C., Frohlich, B., Schiith, H., Strauss, W., and Wesche, G. (1995). The responsive workbench: A virtual work environment. In IEEE Computer , July 1995, 42-48.

Krüger, W. and Frohlich, B. (1994). The responsive workbench. In IEEE Computer Graphics and Applications , May 1994, 12-15.

Luckiesh, M. (1944): Light, vision and seeing: a simplified presentation of their relationships and their importance in human efficiency and welfare. New York, New York, D. Van Nostrand Company Inc.

MacKenzie, I. S. and Soukoreff, R. W. (2003). Phrase sets for evaluating text entry techniques. In CHI '03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA, April 05 - 10, 2003). CHI '03. ACM, New York, NY, 754-755.

Malik, S., Ranjan, A., and Balakrishnan, R. (2005). Interacting with large displays from a distance with vision-tracked multi-finger gestural input. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 43- 52.

Mark, G. 2002. Extreme collaboration. Communications. ACM 45, 6 (June 2002), 89-93.

Masliah, M. R. and Milgram, P. 2000. Measuring the allocation of control in a 6 degree- of-freedom docking experiment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 25-32.

Matsuda, M., Matsushita, M., Yamada, T., Namemura, T. (2006). Behavioral analysis of asymmetric information sharing on Lumisight table. In Proceedings of First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06) , 113-122. 167 Matsushita, M., Iida, M., Ohguro, T., Shirai, Y., Kakehi, Y. and Naemura, T. (2004): ‘Lumisight table: a face-to-face collaboration support system that optimizes direction of projected information to each stakeholder’, Proceedings of the 2004 ACM conference on Computer supported cooperative work, Chicago, Illinois, USA ACM Press, 2004, 274-283.

Mazalek, A. (1999). Tangible Interfaces for Interactive Point-of-View Narratives . MSc thesis. Program in Media Arts and Sciences, School of Architecture and Planning, Massachusetts Institute of Technology.

Mazalek, A., Reynolds, M., Davenport, G. (2006). TViews: An Extensible Architecture for Multiuser Digital Media Tables. In IEEE Computer Graphics and Applications , 26 (5), 47-55. Sept/Oct, 2006.

Mazalek, A., Reynolds, M., Davenport, G. (2007). The TViews Table in the Home. In Proceedings of the Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems, 52-59.

Mitchell, G. D. (2003). Orientation on Tabletop Displays, M.Sc. Thesis, Simon Fraser University, Burnaby, BC.

Mohamed, K.A., Haag, S., Peltason, J., Dal-Ri, F., Ottmann, T. (2006) Disoriented pen- gestures for identifying users around the tabletop without cameras and motion sensors. In Proceedings of First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06) , 43-52.

Montgomery, D.C. (2001). Design and Analysis of Experiments . John Wiley and Sons, Fifth Edition. New York, NY.

Morris, M. R., Morris, D., and Winograd, T. (2004a). Individual audio channels with single display groupware: effects on communication and task strategy. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM, New York, NY, 242-251.

Morris, M. R., Ryall, K., Shen, C., Forlines, C., and Vernier, F. (2004b). Beyond "social protocols": multi-user coordination policies for co-located groupware. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM, New York, NY, 262-265.

Morris, M.R.. Supporting Effective Interaction with Tabletop Groupware . (Ph.D. Dissertation) Stanford University Technical Report, April 2006.

Morris, M.R., Paepcke, A., Winograd, T. (2006a). TeamSearch: comparing techniques for co-present collaborative search of digital media. In Proceedings of First IEEE

168 International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06) , 97-104.

Morris, M. R., Paepcke, A., Winograd, T., and Stamberger, J. (2006b). TeamTag: exploring centralized versus replicated controls for co-located tabletop groupware. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 1273-1282.

Morris, M. R., Huang, A., Paepcke, A., and Winograd, T. (2006c). Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 1201- 1210.

Morris, M.R. (2006d). Supporting Effective Interaction with Tabletop Groupware Horizontal Interactive Human-Computer Systems. In Proceedings of First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 55-56.

Morris, M.R., Cassanego, A., Paepcke, A., Winograd, T., Piper, A.M., Huang, A., (2006e). Mediating Group Dynamics through Tabletop Interface Design. In IEEE Computer Graphics and Applications , 26(5) (Sept/Oct, 2006), 65-73.

Morris, M.R. Brush, A.J.B., Meyers, B. R. (2007). Reading Revisited: Evaluating the Usability of Digital Display Surfaces for Active Reading Tasks. In Proceedings of the Second IEEE International Workshop on Horizontal Interactive Human- Computer Systems , 52-59.

Nacenta, M. A., Aliakseyeu, D., Subramanian, S., and Gutwin, C. (2005). A comparison of techniques for multi-display reaching. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 371-380.

Nacenta, M. A., Sallam, S., Champoux, B., Subramanian, S., and Gutwin, C. (2006). Perspective cursor: perspective-based interaction for multi-display environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 289-298.

Nardi, B. A., Schwarz, H., Kuchinsky, A., Leichner, R., Whittaker, S., and Sclabassi, R. 1993. Turning away from talking heads: the use of video-as-data in neurosurgery. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands, April 24 - 29, 1993). CHI '93. ACM, New York, NY, 327-334.

Narens, L. (1996). A theory of magnitude estimation. In Journal of Mathematical Psychology , 40, 109-129. 169 Newman, W. and Wellner, P. (1992). A desk supporting computer-based interaction with paper documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, United States, May 03 - 07, 1992). CHI '92. ACM Press, New York, NY, 587-592.

Nishimoto, K., Amano, K., Usuki, M. (2006) pHotOluck: a home-use table-ware to vitalise mealtime communications by projecting photos onto dishes. In Horizontal Interactive Human-Computer Systems, 2006. TableTop 2006, 9-16.

Nowell, L.T. (1997). Graphical encoding for information visualization: using icon color, shape, and size to convey nominal and quantitative data. PhD thesis, Department of Computer Science, Virginia Polytechnic Institute and State University.

Pangaro, G., Maynes-Aminzade, D., and Ishii, H. (2002). The actuated workbench: computer-controlled actuation in tabletop tangible interfaces. In Proceedings of the 15th Annual ACM Symposium on User interface Software and Technology (Paris, France, October 27 - 30, 2002). UIST '02. ACM, New York, NY, 181-190.

Parker, J. K., Mandryk, R. L., and Inkpen, K. M. (2005a). TractorBeam: seamless integration of local and remote pointing for tabletop displays. In Proceedings of Graphics interface 2005 (Victoria, British Columbia, May 09 - 11, 2005). ACM International Conference Proceeding Series, vol. 112. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, 33-40.

Parker, K., Mandryk, R., Nunes, M., and Inkpen, K., (2005b). TractorBeam selection aids: Improving target acquisition for pointing input on tabletop displays. In Proceedings of the Interact Conference . p. 80-93.

Parker, J.K., Mandryk, R.L., Inkpen, K.M. (2006). Integrating Point and Touch for Interaction with Digital Tabletop Displays. In IEEE Computer Graphics and Applications, 26(5) , 28-35.

Patten, J., Ishii, H., Hines, J., and Pangaro, G. (2001). Sensetable: a wireless object tracking platform for tangible user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, United States). CHI '01. ACM, New York, NY, 253-260.

Perron, R., Laborie, F. (2006). Augmented tabletops, an incentive for distributed collaboration. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 135-142.

Pierce, J. S., Conway, M., van Dantzich, M., and Robertson, G. (1999a). Toolspaces and glances: storing, accessing, and retrieving objects in 3D desktop applications. In Proceedings of the 1999 Symposium on interactive 3D Graphics (Atlanta, Georgia, United States, April 26 - 29, 1999). I3D '99. ACM, New York, NY, 163- 168.

170 Pierce, J. S., Stearns, B. C., and Pausch, R. (1999b). Voodoo dolls: seamless interaction at multiple scales in virtual environments. In Proceedings of the 1999 Symposium on interactive 3D Graphics (Atlanta, Georgia, United States, April 26 - 29, 1999). I3D '99. ACM, New York, NY, 141-145.

Pizlo Z. (1994). A theory of shape constancy based on perspective invariants. In Vision Research 34,1637–1658.

Potter, R. L., Weldon, L. J., and Shneiderman, B. (1988). Improving the accuracy of touch screens: an experimental evaluation of three strategies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Washington, D.C., United States, May 15 - 19, 1988). CHI '88. ACM, New York, NY, 27-32.

Prante, T., Streitz, N., & Tandler, P. (2004). Roomware: Computers disappear and interaction evolves. In IEEE Computer , December 2004 , 47-54.

Rekimoto, J. (1997). Pick-and-drop: a direct manipulation technique for multiple computer environments. In Proceedings of the 10th Annual ACM Symposium on User interface Software and Technology (Banff, Alberta, Canada, October 14 - 17, 1997). UIST '97. ACM, New York, NY, 31-39.

Rekimoto, J. (2002). SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM, New York, NY, 113-120.

Rekimoto, J. and Saitoh, M. (1999). Augmented surfaces: a spatially continuous work space for hybrid computing environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI '99. ACM, New York, NY, 378-385.

Ringel, M., Ryall, K., Shen, C., Forlines, C., and Vernier, F. (2004). Release, relocate, reorient, resize: fluid techniques for document sharing on multi-user interactive tables. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 1441- 1444.

Rogers, Y., Hazlewood, W., Blevis, E., and Lim, Y. (2004). Finger talk: collaborative decision-making using talk and fingertip interaction around a tabletop display. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 1271-1274.

Rogers, Y., Lim, Y., Hazlewood, W.R., (2006). Extending tabletops to support flexible collaborative interactions. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 71-78.

171 Ryall, K., Forlines, C., Shen, C., and Morris, M. R. (2004). Exploring the effects of group size and table size on interactions with tabletop shared-display groupware. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM, New York, NY, 284-293.

Ryall, K. Esenther, E., Forlines, C., Shen, C., Shipman, S., Ringel Morris, M., Everitt, K., Vernier, F.D., (2006a). Identity-Differentiating Widgets for Multiuser Interactive Surfaces. In IEEE Computer Graphics and Applications , 26(5) (Sept/Oct, 2006), 56-64.

Ryall, K., Morris, M., Everitt, K., Forlines, C., and Shen, C., (2006b). Experiences with and observations of direct-touch tabletops. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 89-96.

St. John, M., Cowen, M.B., Smallman, H.S., and Oonk, H.M. (2001). The use of 2d and 3d displays for shape-understanding versus relative-position tasks. In Human Factors , 43(1), 79-98.

Schneider, B., Erlich, D.J., Stein, R., Flaum, M., Mangel, M. (1978). Changes in the apparent lengths of lines as a function of degree of retinal eccentricity. In Perception , 7, 215-223.

Scott, S. D., Sheelagh, M., Carpendale, T., and Inkpen, K. M. (2004). Territoriality in collaborative tabletop workspaces. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM, New York, NY, 294-303.

Scott, S., Carpendale, S., and Habelski, S. (2005). Storage bins: Mobile storage for collaborative tabletop displays. In IEEE Computer Graphics and Applications , 26(5) (Sept/Oct, 2006), 58-65.

Scott, S. (2006). Territoriality in Collaborative Tabletop Workspaces. PhD Dissertation, Department of Computer Science, University of Calgary, Calgary, Alberta, Canada

Seidler, R. (2004). Multiple motor learning experiences enhance motor adaptability . Journal of Cognitive Neuroscience , 16(1), 65-73.

Shen, C. Everitt, K., Ryall, K., (2003a). UbiTable: Impromptu Face-to-Face Collaboration on Horizontal Interactive Surfaces. In Proceedings of the 5th International Conference on Ubiquitous Computing , LNCS 2864, Springer Verlag, 281-288.

Shen, C., Lesh, N., and Vernier, F. (2003b). Personal digital historian: story sharing around the table. In Interactions 10 (2) (Mar. 2003), 15-22.

172 Shen, C., Vernier, F. D., Forlines, C., and Ringel, M. (2004). DiamondSpin: an extensible toolkit for around-the-table interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 167-174.

Shen, C., Hancock, M. S., Forlines, C., and Vernier, F. D. (2005). CoR 2Ds. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 1781-1784.

Shen, C., Ryall, K., Forlines, C., Esenther, A., Vernier, F., Everitt, K., Wu, M., Wigdor, D., Morris, M.R., Hancock, M., Tse, E., (2006). Informing the design of direct- touch tabletops. In IEEE Computer Graphics and Applications , 26(5) (Sept/Oct, 2006), 36-46.

Shepard, R.N., Metzler, J. (1971). Mental rotation of three dimensional objects. In Science , 171, 701-703.

Smart Technologies (DViT) Digital Vision Touch Technology . http://smarttech.com/DViT/. Accessed November, 2006.

Smeaton, A.F., Foley, C., Gurrin, C., Lee, H., McGivney, S., (2006). Collaborative searching for video using the Fischlar system and a DiamondTouch table. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 151-159.

Stefik, M., Bobrow, D., Lanning, S., and Tartar, D., (1987). WYSIWIS revised: early experiences with multiuser interfaces. In ACM Transactions on Information Systems , 5(2) (Apr. 1987), 147-167.

Stefik, M. J., Foster, G., Bobrow, D.G., Kahn, K., Lanning, S., and Suchman, L. (1987b) Beyond the chalkboard: Computer Support for Collaboration and Problem Solving in Meetings. Communications of the ACM, 30(1), 32-47.

Stevens, S.S. (1957). On the Psychophysical law. In Psychology Review , 64, 153-181.

Stratton, G. (1897a). Upright vision and the retinal image. In Psychology Review , 4, 182- 187.

Stratton, G. (1897b). Vision without inversion of the retinal image. In Psychology Review , 4, 341-360, 463-481.

Streitz, N., Geißler, J., Holmer, T., Konomi, S.i., Müller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P., and Steinmetz, R. (1999). i-LAND: an interactive landscape for creativity and innovation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI '99. ACM, New York, NY, 120-127.

173 Streitz, N., Prante, T., Müller-Tomfelde, C., Tandler, P., and Magerkurth, C. (2002). Roomware©: the second generation. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM, New York, NY, 506-507.

Stroop, J. (1935). Studies of interference in serial verbal reactions, Journal of Experimental Psychology: General , 18, 643-662.

Su, R. and Bailey, B. (2005). Towards guidelines for positioning large displays in interactive workspaces. In Proceedings of the Interact Conference , 377-390.

Subramanian, S., Aliakseyeu, D., and Lucero, A. (2006). Multi-layer interaction for digital tables. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06. ACM Press, New York, NY, 269-272.

Tandler, P., Prante, T., Müller-Tomfelde, C., Streitz, N., and Steinmetz, R. (2001). Connectables: dynamic coupling of displays for the flexible creation of shared workspaces. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 11-20.

Tang, J., (1991). Findings from observational studies of collaborative work. International Journal of Man-Machine Studies , 34 (2) (1991), 143-160.

Tang, J. C. and Minneman, S. L., (1991). Videodraw: a video interface for collaborative drawing. ACM Transactions on Information Systems. 9 (2) (Apr. 1991), 170-184.

Teasley, S., Covi, L., Krishnan, M. S., and Olson, J. S. 2000. How does radical collocation help a team succeed?. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (Philadelphia, Pennsylvania, United States). CSCW '00. ACM, New York, NY, 339-346.

Tinker, M. (1972). Effect of angular alignment upon readability of print. In Journal of Educational Psychology , vol. 47, 1972, 358-363.

Tollinger, I., McCurdy, M., Vera, A. H., and Tollinger, P. (2004). Collaborative knowledge management supporting mars mission scientists. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM, New York, NY, 29- 38.

Toney, A., Thomas, B.H. (2006). Considering reach in tangible and table top design. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 57-58.

Tse, E., Greenberg, S., Shen, C. and Forlines, C., (2006a). Multimodal Multiplayer Tabletop Gaming. In Proceedings of the Third International Workshop on 174 Pervasive Gaming Applications (PerGames'06), in conjunction with 4th Intl. Conference on Pervasive Computing, 139-148.

Tse, E., Shen, C., Greenberg, S., and Forlines, C. (2006b). Enabling interaction with single user applications through speech and gestures on a multi-user tabletop. In Proceedings of the Working Conference on Advanced Visual interfaces (Venezia, Italy, May 23 - 26, 2006). AVI '06. ACM, New York, NY, 336-343.

Tse, E., Greenberg, S., and Shen, C. (2006c). GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications. In Proceedings of the 8th international Conference on Multimodal interfaces (Banff, Alberta, Canada, November 02 - 04, 2006). ICMI '06. ACM, New York, NY, 76- 83.

Tse, E. (2007) Multimodal Co-located Interaction . PhD Dissertation, Department of Computer Science, University of Calgary, Calgary, Alberta, Canada.

Tufte, E. (1983). Visual Display of Quantitative Information. Graphics Press, Cheshire, Connecticut.

Ullmer, B. and Ishii, H. (1997). The metaDESK: models and prototypes for tangible user interfaces. In Proceedings of the 10th Annual ACM Symposium on User interface Software and Technology (Banff, Alberta, Canada, October 14 - 17, 1997). UIST '97. ACM, New York, NY, 223-232.

Underkoffler, J. and Ishii, H. (1998). Illuminating light: an optical design tool with a luminous-tangible interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 542-549.

Vernier, F., Lesh, N., and Shen, C. (2002). Visualization techniques for circular tabletop interfaces. In Proceedings of Advanced Visual Interfaces (Trento Italy, May 2002), ACM Press, 257-263.

Voyer, D., Brydon, M.P. (1990). Gender, level of spatial ability, and lateralization of mental rotation. In Brain and Cognition , 13(1), 18-29.

Wellner, P. (1993). Interacting with paper on the DigitalDesk. Communications of the ACM 36, (7) (Jul. 1993), 87-96.

Wesugi, S., Miwa, Y. (2006)"Lazy Susan" communication system for remote, spatial and physical collaborative works. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 35-42.

Wigdor, D., Balakrishnan, R. (2005). Empirical Investigation into the Effect of Orientation on Text Readability in Tabletop Displays. Proceedings of the 9th European Conference on Computer Supported Cooperative Work , 205-224. 175 Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R., (2006a). Table-Centric Interactive Spaces for Real-Time Collaboration: Solutions, Evaluation, and Application Scenarios. In Proceedings of the 2006 conference on Collaborative Technologies (CollabTech), 9-14.

Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R., (2006b). Table-Centric Interactive Spaces for Real-Time Collaboration. In Proceedings of the 2006 conference on Advanced Visual Interfaces (AVI), 103-107.

Wigdor, D., Shen, C., Forlines, C., and Balakrishnan, R. (2006c). Effects of display position and control space orientation on user preference and performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006) CHI '06. ACM Press, New York, NY, 309-318.

Wigdor, D., Leigh, D., Forlines, C., Shipman, S., Barnwell, J., Balakrishnan, R., and Shen, C. (2006d). Under the table interaction. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06. ACM, New York, NY, 259-268.

Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R., (2006e). Table-Centric Interactive Spaces for Real-Time Collaboration: a Video Demonstration. Video Proceedings of the 2006 ACM conference on Computer Supported Cooperative Work.

Wigdor, D., Shen, C., Forlines, C., and Balakrishnan, R. (2007). Perception of elementary graphical elements in tabletop and multi-surface environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 473- 482.

Wigdor, D., Penn, G., Ryall, K., Esenther, A., Shen, C., (2007b). Living with a Tabletop: Analysis and Observations of Long Term Office Use of a Multi-Touch Table. In Proceedings of the Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems , 60-67.

Wilson, A. D. (2005). PlayAnywhere: a compact interactive tabletop projection-vision system. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 83-92.

Wilson, A. (2007). Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. Proceedings of the Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems , 201-204.

Wu, M. and Balakrishnan, R. (2003). Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 193-202. 176 Wu, M., Shen, C., Ryall, K., Forlines, C., and Balakrishnan, R. (2006). Gesture registration, relaxation, and reuse for multi-point direct-touch surfaces. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 183-190.

Yeh, Y.Y., and Silverstein, L.D. (1992). Spatial judgements with monoscopic and stereoscopic presentation of perspective displays. In Human Factors , 34(5), 583- 600.

Yoon, J., Oishi, J., Nawyn, J., Kobayashi, K., and Gupta, N. (2004). FishPong: encouraging human-to-human interaction in informal social environments. Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago, Illinois, USA, November 06 - 10, 2004). CSCW '04. ACM Press, New York, NY. p. 374-377.

Yoshino, T., Matsushita, M., Munemori, J. (2006). Proposal of a multi-layer structure for multilingual display on a Lumisight table. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06), 129-130.

Zhai, S. and Milgram, P. 1998. Quantifying coordination in multiple DOF movement and its application to evaluating 6 DOF input devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 320-327.

Zhang, X., Takatsuka, M. (2007). Put That There NOW: Group Dynamics of Tabletop Interaction under Time Pressure. In Proceedings of the Second IEEE International Workshop on Horizontal Interactive Human-Computer Systems , 37- 43.

Zimmer, K. (2005). Examining the validity of numerical ratios in loudness fractionation. Perception & Psychophysics, 67, 569-579.

177