• Abstract and Figures
• Public Full-text

A University of Sussex DPhil thesis Available online via Sussex Research Online: http://sro.sussex.ac.uk/ This thesis is protected by copyright which belongs to the author. This thesis cannot be reproduced or quoted extensively from without first obtaining permission in writing from the Author The content must not be changed in any way or sold commercially in any format or medium without the formal permission of the Author When referring to this work, full bibliographic details including the author, title, awarding institution and date of the thesis must be given Please visit Sussex Research Online for more information and further details How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences Giles Hamilton-Fletcher Thesis submitted for the degree of Doctor of Philosophy University of Sussex December, 2014 i The thesis conforms to an ‘article format’ in which the middle chapters consist of discrete articles written in a style that is appropriate for publication in peer- reviewed journals in the field. The first and final chapters present synthetic overviews and discussions of the field and the research undertaken. ii Chapter two is published in Multisensory Research as: Hamilton-Fletcher, G., & Ward, W. (2013). Representing Colour in Sound and Touch for Sensory Substitution: A Review. Multisensory Research, 26, 503-532. The author contributions are as follows: Giles Hamilton-Fletcher was responsible writing of the manuscript; Jamie Ward was responsible providing feedback and corrections to the manuscript. Chapter three written in the style of an article appropriate for the Journal of Consciousness Studies The author contributions are as follows: Giles Hamilton-Fletcher was responsible writing of the manuscript, experimental design, all aspects of data collection, data analysis; Jamie Ward was responsible providing feedback on the design and corrections to the manuscript. Chapter four written in the style of an article appropriate for Multisensory Research The author contributions are as follows: Giles Hamilton-Fletcher was responsible writing of the manuscript, experimental design, all aspects of data collection, data analysis and feedback on the experimental device; Thomas Wright was responsible for the initial concept of a touchpad based SSD named the Creole and the programming of the host and client applications for the experimental device in C# using Microsoft Visual Studio; Jamie Ward was responsible providing feedback on the experimental design, experimental device and corrections to the manuscript. Chapter five written in the style of an article appropriate for Attention, Perception and Psychophysics iii The author contributions are as follows: Giles Hamilton-Fletcher was responsible writing of the manuscript, experimental design, data collection for experiment 1 and 2, all data analysis; Christoph Witzel was responsible for the experimental program for experiment 1 and 2; David Reby was responsible for feedback on auditory stimuli; Devin Terhune and David Luke were also responsible for the study design for experiment 3; Mendel Kaelen and Robin Carhart-Harris were responsible for data collection for experiment 3; David Nutt was responsible for obtaining ethics for experiment 3; Jamie Ward was responsible providing feedback on the study and corrections to the manuscript. Chapter six written in the style of an article appropriate for Multisensory Research The author contributions are as follows: Giles Hamilton-Fletcher was responsible writing of the manuscript, experimental design, all aspects of data collection, data analysis as well as alterations to the Creole SSD to include CIE LUV / LCH colour space and output sound samples in through further programming in C# using Microsoft Visual Studio; Ward was responsible providing feedback and corrections to the manuscript. iv Declaration I declare that this thesis has not, in whole or in part, been submitted previously for the award of any degree at any University. Signature: v Acknowledgements Firstly I would like to thank my primary supervisor Jamie Ward who has provided excellent guidance in pursuing a wide variety of research topics while giving detailed feedback for me to improve myself. I am also grateful for the essential help in expanding my research beyond this thesis. I would like to thank Anil Seth, Tom Wright and Keisuke Suzuki for their input on the technological side of the sensory substitution work. I am grateful to David Reby and Christoph Witzel for sharing their expertise on sound and colour research. Also thanks to Kristin Schmidt and Robyn Dillon for their assistance with testing and illustrations. Thanks to all of the participants in my experiments for contributing their time and effort, and in particular the synaesthetes who provided such detailed descriptions of their experiences. Importantly, thank you to D, for everything that made the lows bearable and the highs so much more enjoyable in our journey. I can only try to provide the same level of support to you, that you showed to me. Finally I would like to dedicate this thesis to the memory of my mother, who helped provide the freedom and support necessary for me to follow this path. vi Preface Research reported here has also been presented at the following conferences: Synaesthesia in Perspective: Development, Networks and Multisensory Processing Conference (Hamilton-Fletcher and Ward, 2014); British Academy Conference for Sensory Substitution and Augmentation (Hamilton-Fletcher, Wright and Ward, 2013). This research was funded by a three-year studentship from the ESRC and the University of Sussex vii How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences Giles Hamilton-Fletcher Summary Sensory substitution devices (SSDs) systematically turn visual dimensions into patterns of tactile or auditory stimulation. After training, a user of these devices learns to translate these audio or tactile sensations back into a mental visual picture. Most previous SSDs translate greyscale images using intuitive cross-sensory mappings to help users learn the devices. However more recent SSDs have started to incorporate additional colour dimensions such as saturation and hue. Chapter two examines how previous SSDs have translated the complexities of colour into hearing or touch. The chapter explores if colour is useful for SSD users, how SSD and veridical colour perception differ and how optimal cross-sensory mappings might be considered. After long-term training, some blind users of SSDs report visual sensations from tactile or auditory stimulation. A related phenomena is that of synaesthesia, a condition where stimulation of one modality (i.e. touch) produces an automatic, consistent and vivid sensation in another modality (i.e. vision). Tactile-visual synaesthesia is an extremely rare variant that can shed light on how the tactile-visual system is altered when touch can elicit visual sensations. Chapter three reports a series of investigations on the tactile discrimination abilities and phenomenology of tactile-vision synaesthetes, alongside questionnaire data from synaesthetes unavailable for testing. Chapter four introduces a new SSD to test if the presentation of colour information in sensory substitution affects object and colour discrimination. viii Chapter five presents experiments on intuitive auditory-colour mappings across a wide variety of sounds. These findings are used to predict the reported colour hallucinations resulting from LSD use while listening to these sounds. Chapter six uses a new sensory substitution device designed to test the utility of these intuitive sound-colour links for visual processing. These findings are discussed with reference to how cross-sensory links, LSD and synaesthesia can inform optimal SSD design for visual processing. ix Contents 1. Introduction .................................................................................................................................... 1 1.1. Synaesthesia................................................................................................................................ 2 1.1.1. Defining synaesthesia and its variations ................................................................................. 2 1.1.2. History of synaesthesia in science .......................................................................................... 3 1.1.3. Confirming synaesthesia ......................................................................................................... 4 1.1.4. Synaesthetic prevalence and genetics .................................................................................... 5 1.1.5. Synaesthesia, perception and memory .................................................................................. 6 1.1.6. Theories of synaesthesia ......................................................................................................... 7 1.1.7. Synaesthesia and the present research ................................................................................ 10 1.2. Cross-modal correspondences .................................................................................................. 11 1.2.1. Definition and behavioural effects ....................................................................................... 11 1.2.2. History of correspondences - symbolism, synaesthesia and other senses ........................... 12 1.2.3. Explaining correspondences

Load more

  262

Copy Link