Designing Multi-sensory Displays for Abstract Data. Keith V. Nesbitt BMath (Newcastle), MComp (Newcastle), Diploma in Technical Analysis (SIA) School of Information Technologies University of Sydney NSW 2006 Australia Submitted in partial fulfilment of the requirements of the degree of Doctor of Philosophy January 2003 Abstract 2 Abstract The rapid increase in available information has lead to many attempts to automatically locate patterns in large, abstract, multi-attributed information spaces. These techniques are often called data mining and have met with varying degrees of success. An alternative approach to automatic pattern detection is to keep the user in the exploration loop by developing displays for perceptual data mining. This approach allows a domain expert to search the data for useful relationships and can be effective when automated rules are hard to define. However, designing models of the abstract data and defining appropriate displays are critical tasks in building a useful system. Designing displays of abstract data is especially difficult when multi-sensory interaction is considered. New technology, such as Virtual Environments, enables such multi-sensory interaction. For example, interfaces can be designed that immerse the user in a 3D space and provide visual, auditory and haptic (tactile) feedback. It has been a goal of Virtual Environments to use multi-sensory interaction in an attempt to increase the human-to-computer bandwidth. This approach may assist the user to understand large information spaces and find patterns in them. However, while the motivation is simple enough, actually designing appropriate mappings between the abstract information and the human sensory channels is quite difficult. Designing intuitive multi-sensory displays of abstract data is complex and needs to carefully consider human perceptual capabilities, yet we interact with the real world everyday in a multi-sensory way. Metaphors can describe mappings between the natural world and an abstract information space. This thesis develops a division of the multi-sensory design space called the MS-Taxonomy. The MS-Taxonomy provides a concept map of the design space based on temporal, spatial and direct metaphors. The detailed concepts within the taxonomy allow for discussion of low level design issues. Furthermore the concepts abstract to higher levels, allowing general design issues to be compared and discussed across the different senses. The MS-Taxonomy provides a categorisation of multi-sensory design options. However, to design effective multi-sensory displays requires more than a thorough understanding of design options. It is also useful to have guidelines to follow, and a process to describe the design steps. This thesis uses the structure of the MS-Taxonomy to develop the MS-Guidelines and the MS-Process. The MS-Guidelines capture design recommendations and the problems associated with different design choices. The MS- Process integrates the MS-Guidelines into a methodology for developing and evaluating multi-sensory displays. A detailed case study is used to validate the MS-Taxonomy, the MS-Guidelines and the MS-Process. The case study explores the design of multi-sensory displays within a domain where users wish to explore abstract data for patterns. This area is called Technical Analysis and involves the interpretation of patterns in stock market data. Following the MS-Process and using the MS-Guidelines some new multi-sensory displays are designed for pattern detection in stock market data. The outcome from the case study includes some novel haptic-visual and auditory-visual designs that are prototyped and evaluated. Acknowledgements 3 Acknowledgements Early motivation for this work was gained while working at BHP Research on a project to investigate the use of Virtual Environments for solving industrial problems. Many people were involved in this early work and I would like to thank Steve Woodyatt, Randall Gallimore, Bernard Orenstein, Harmeet Lamba, Adrian Ugray and Tony Murnain for their support and assistance during that time. Many of the collaborations that began during this early work have carried on during the course of this thesis. I would like to thank a number of people for their input and for providing access to the Virtual Environments necessary for me to complete my work. In particular, Kevin Smith, Duncan Stevens, Chris Gunn, Stephen Barrass, Mathew Hutchins and others at the Interactive Modelling and Visualisation Systems Group, which is part of the Mathematical and Information Sciences Division, CSIRO, Canberra. I would like to thank Martin Göbel, Bernd Fröhlich, John Plate and other members of the IMK-VE group for their help and support. This group is located in Sankt Augustin, Germany and formed part of the German National Research Centre for Information Technology (GMD) before becoming the Fraunhofer Institute for Media Communication. I would also like to express my appreciation for funding support from Rudi Vernik and Tim Patterson at the Information Technology Division of the Defence Science and Technology Organisation (DSTO), Adelaide. A special note of appreciation is extended to my supervisor, Professor Peter Eades for his invaluable assistance and direction during the course of my work. Finally, of course this work would not have been possible with out the support of family, friends and the odd angel or two (metaphorical not metaphysical). To those people I would like to offer my sincere thanks for your faith and patience. Preface 4 Preface All work presented is the original work of the author unless otherwise indicated. Parts of this thesis have been published in the following works. Keith Nesbitt (2002) "Experimenting with Haptic Attributes for Display of Abstract Data", Proceedings of Eurohaptics, 2002. Edinburgh, UK, pp 150-156. Keith Nesbitt and Stephen Barrass (2002) "Evaluation of a Multimodal Sonification and Visualisation of Depth of Market Stock Data", Proceedings of the International Conference of Auditory Display, 2002. Kyoto, Japan, pp 233-238. Keith Nesbitt and Carsten Friedrich (2002) "Applying Gestalt Principles to Animated Visualizations of Network Data", International Symposium on Web Graphics and Visualisation, IV02-WGV 2002, London, pp 737-743. Keith Nesbitt (2001) "Modelling the Multi-sensory Design Space", Information Visualisation, 2001. Proceedings of the Australian Symposium on Information Visualisation, Conferences in Research and Practice in Information Technology. Volume 9, Australian Computer Society, pp 27-36. Keith Nesbitt (2001) "Interacting with Stock Market Data in a Virtual Environment", Data Visualization, 2001. Proceedings of the Joint Eurographics IEEE TCVG Symposium on Visualization, SpringerWein New York, pp273-282 Keith Nesbitt, Randall Gallimore, Bernard Orenstein (2001) "Using Force Feedback for Multi- sensory Display", Proceedings of 2nd Australasian User Interface Conference. AUIC 2001. Australian Computer Science Communications, Vol. 23, No. 5. pp64-68. Keith Nesbitt (2000) "Designing Multi-sensory Models for Finding Patterns in Stock Market Data" Proceedings of the Third International Conference on Multimodal Interfaces 2000, Springer Lecture Notes in Computer Science 1948, pp24-31. Keith Nesbitt (2000) "A Classification of Multi-sensory Metaphors for Understanding Abstract Data in a Virtual Environment", Proceedings of the IEEE Conference on Information Visualisation 2000, IEEE Computer Society, Los Alamitos, pp493-498. Keith Nesbitt, Randall Gallimore, Bernard Orenstein (2000), "Investigating the Application of Virtual Environment Technology for use in the Petroleum Exploration Industry", Proceedings of the 23rd Australasian Computer Science Conference ACSC 2000, Vol 22 (1), pp181-188, Canberra, Australia. Keith Nesbitt and Bernard Orenstein (1999) "Multisensory Metaphors and Virtual Environments applied to Technical Analysis of Financial Markets", Proceedings of the Conference on Advanced Investment Technology 1999, Gold Coast, Australia, pp 195-205. ISBN: 0733100171. This work has not been submitted for a higher degree to any other University or institution. Keith Nesbitt Contents 5 Contents List of Tables . 6 List of Figures . 10 PART I – Introduction Chapter 1 – Multi-sensory Display . 22 Chapter 2 – Metaphors and Senses . 56 PART II – The MS-Taxonomy Chapter 3 – Spatial Metaphors . 78 Chapter 4 – Direct Metaphors . 136 Chapter 5 – Temporal Metaphors . 162 PART III– The MS-Guidelines Chapter 6 – General Guidelines . 192 Chapter 7 – Guidelines for Spatial Metaphors . 206 Chapter 8 – Guidelines for Direct Metaphors . 222 Chapter 9 – Guidelines for Temporal Metaphors . 242 PART IV– The MS-Process Chapter 10 – Designing Multi-sensory Displays . 256 PART V – Case Study Chapter 11 – The Stock Market . 277 Chapter 12 – Haptic-Visual Applications . 288 Chapter 13 – Auditory-Visual Application . 340 PART VI – Conclusion Chapter 14 – Final Remarks . 390 Bibliography . 401 Appendix A - Information Metaphors . 422 Appendix B – Train of Thought Network . 432 Appendix C – The XCM-Taxonomy . 455 List of Tables 6 List of Tables Chapter 1 Table 1-1 Some different modalities of input and output used in Virtual Environments………. 39 Table 1-2 A summary of the questions that this thesis tries to address……...…….………..…... 53 Chapter 2 Table 2-1 Display artefacts used to output information to the different senses………………... 60 Table 2-2 The nine basic classes of the Metaphor Sensory (MS) Taxonomy………………….. 61 Table 2-3 Distances at which the different depth cues are