Designing Sensory-Substitution Devices: Principles, Pitfalls and Potential1
Total Page：16
File Type：pdf, Size：1020Kb
Restorative Neurology and Neuroscience 34 (2016) 769–787 769 DOI 10.3233/RNN-160647 IOS Press Designing sensory-substitution devices: Principles, pitfalls and potential1 Arni´ Kristjansson´ a,∗, Alin Moldoveanub, Omar´ I. Johannesson´ a, Oana Balanb, Simone Spagnolc, Vigd´ıs Vala Valgeirsdottir´ a and Runar´ Unnthorssonc aLaboratory of Visual Perception and Visuomotor control, University of Iceland, Faculty of Psychology, School of Health Sciences, Reykjavik, Iceland bUniversity Politehnica of Bucharest, Faculty of Automatic Control and Computers, Computer Science and Engineering Department, Bucharest, Romania cFaculty of Industrial Engineering, Mechanical Engineering and Computer Science, School of Engineering and Natural Sciences, Reykjavik, Iceland Abstract. An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object. Keywords: Sensory substitution, neural plasticity, multisensory perception 1. Introduction common sense (koin¯e aesthesis, see Ackrill, 1987). The individual senses (seeing, hearing, touch and In his seminal epistemological work, De Anima, smell) have their own domains but many percep- the Greek philosopher Aristotle made an important tual functions do not involve simple operations of distinction between the individual senses and the individual senses, but a central holistic represen- tation of the environment. Aristotle realized that perception of the environment is a multifaceted oper- 1This work was supported by the European Union’s Horizon 2020 Research and Innovative Programme under grant agreement ation where all available senses come into play. no. 643636, “Sound of Vision”. Aristotle´s insight highlights not only that our rep- ∗Corresponding author: Arni´ Kristjansson,´ Laboratory of resentation of the world is holistic, but also that this Visual Perception and Visuomotor control, University of Iceland, representation is multimodal since it involves infor- Faculty of Psychology, School of Health Sciences, Reykjav´ık, Iceland. Tel.: +3545255198; Fax: +3545255199; E-mail: ak@ mation from all the senses. Borrowing Aristotle’s hi.is. term, the common sense builds a representation of the 0922-6028/16/$35.00 © 2016 – IOS Press and the authors. All rights reserved This article is published online with Open Access and distributed under the terms of the Creative Commons Attribution Non-Commercial License (CC BY-NC 4.0). 770 A.´ Kristj´ansson et al. / Designing sensory-substitution devices environment that encompasses information from all tioned here undoubtedly apply to other forms of senses. sensory substitution. Our review provides a number There are various examples of how information of novel perspectives pertinent to sensory substitution from one modality influences, or even alters, our that have not been addressed systematically in the perception of stimuli in another modality (McGurk literature. We discuss research on neural activity dur- & McDonald, 1976; Shams, Kamitani, & Shimojo, ing sensory substitution. Additionally we highlight 2000; Hotting¨ & Roder,¨ 2004), showing how impor- the importance of training on sensory substitution tant integration from different sensory modalities can devices, the best ways of implementing such training be. Perhaps most importantly, this holistic represen- and constraints stemming from inherent characteris- tation of the environment is influenced by perceptual tics of sensory mechanisms. We also point out some interpretation (Hoffmann, 1998; Palmer, 1999; Rock, pitfalls that any such enterprise must avoid. 1983). The stimulation from the external world (light, air pressure, tactile stimulation) is only part of the story in perception. From the stimulation, interpre- 2. Key general considerations for sensory tive mechanisms of the nervous system create a useful substitution representation of the external environment. When individuals lose the function of one sense, The neural mechanisms of vision in humans are such as through damage to sense organs, mecha- more complex than for the other senses. The audi- nisms concerned with perceptual interpretation are tory nerve has around 30,000 fibers while the optic not necessarily affected. It should in principle be nerve contains over 1 million fibers (Wurtz & Kan- possible to find a way of conveying the missing infor- del, 2000), and psychophysical measurements show mation, if this information can still be processed. that the information capacity of vision is consider- As in the case of blindness that can be traced to ably higher than of audition (Ash, 1951). Brown, damage to the retinae, there is still a lot of intact Simpson and Proulx (2014) found that there is an neural hardware devoted to visual function. As Bach- upper limit to representation of auditory objects due y-Rita and Kercel (2003) put it: “Persons who become to intrinsic limits of transmission in auditory corti- blind do not lose the capacity to see” (p. 541). cal areas. Investigations of the capacity of the visual Information may be fed to these neural mechanisms system reveal that it has four orders of magnitude via other channels, utilizing the fact that the main greater bandwidth than haptic perception (Kokjer, organ of perception is the nervous system. The brain 1987; Schmidt, 1981) while the information capacity is not just a sensory-machine that responds sepa- of the human ear falls between these two estimates rately to specific stimuli or sensory modalities but (Bialek et al., 1993; Dudel, 1986; Proulx et al., 2016; rather a complex “task-machine”, that can partially Wurtz & Kandel, 2000). Loomis, Klatzky and Giu- restore function with input from other senses (Maid- dice (2012) argue that the bandwidth of touch equals enbaum et al., 2014a; Murray et al., 2016). This is blurred vision through filtering of information in the the domain of sensory substitution (SS), where touch cutaneous system. or audition, for example, convey information that is Vision loss leads to lessened mobility. Various otherwise not available, such as vision. Sensory sub- attempts have been made at generating compensatory stitution devices (SSDs) have been available for a strategies where information about the environment is long time. The white-cane for the blind translates conveyed to the blind through audition and/or touch. environmental structure into haptic and propriocep- The goal has been to implement this feedback in real tive feedback and sign language translates visual time, to enable obstacle avoidance, scene analysis or stimuli into language. In Braille, “verbal” informa- object recognition through interactions with the envi- tion is conveyed through haptic stimulation. Such ronment. One of our central points is that any sensory devices have become increasingly sophisticated with substitution device (SSD) and related training must be advances in technology. based on a thorough understanding of the key aspects Here we present an overview of research on ways that need to be conveyed, and – no less importantly – of conveying information to the blind about the exter- a thorough understanding of the psychophysics of the nal environment through other intact senses with the perceptual channels involved. Notably, although the aim of improving perceptual function and mobil- available evidence suggests that there is great poten- ity. While we limit our focus to haptic and auditory tial in sensory substitution methods (see Bach-y-Rita feedback approaches for the blind, many issues men- & Kercel, 2003; Merabet & Pascual-Leone, 2010; A.´ Kristj´ansson et al. / Designing sensory-substitution devices 771 Proulx et al., 2016; Shull & Damian, 2015; Nagel et to effectively convey this information to the haptic al., 2005, for reviews), such devices are still not in and auditory sensory systems. The basic processing widespread