Opinion Benefits of multisensory learning Ladan Shams1 and Aaron R. Seitz2 1 Department of Psychology, University of California, Los Angeles, CA 90095, USA 2 Department of Psychology, University of California, Riverside, CA 92521, USA Studies of learning, and in particular perceptual learning, We note that the human brain has evolved to learn and have focused on learning of stimuli consisting of a operate in natural environments in which behavior is often single sensory modality. However, our experience in guided by information integrated across multiple sensory the world involves constant multisensory stimulation. modalities. Multisensory interactions are ubiquitous in the For instance, visual and auditory information are inte- nervous system and occur at early stages of perceptual grated in performing many tasks that involve localizing processing. Therefore, unisensory-training protocols used and tracking moving objects. Therefore, it is likely that for skill acquisition in adults can provide unnatural settings the human brain has evolved to develop, learn and and do not tap into multisensory learning mechanisms that operate optimally in multisensory environments. We have evolved to produce optimal behavior in the naturally suggest that training protocols that employ unisensory multisensory environment. Within this framework, we stimulus regimes do not engage multisensory learning argue that multisensory-training protocols, as opposed to mechanisms and, therefore, might not be optimal for unisensory protocols, can better approximate natural set- learning. However, multisensory-training protocols can tings and, therefore, produce greater and more efficient better approximate natural settings and are more effec- learning. However, the extent to which this facilitation tive for learning. occurs depends upon appropriate relations (i.e. congruency; between the information coming into each of the senses. Suppose that an adult individual wishes to learn to discriminate a variety of bird species. What would be The multisensory brain the best training protocol for this kind of learning? Perception has traditionally been viewed as a modular Acquiring this skill can involve many types of learning function with the different sensory modalities operating and here we focus on aspects of perceptual learning; largely as separate and independent processes. However, namely, improvements in the perceptual skills that an overwhelming set of new findings has overturned this enable recognition and discrimination of the birds. Based dogma. Reports of multisensory interactions in various on many studies of visual perceptual learning [1–5],we perceptual tasks and settings indicate that these inter- can hypothesize that this learning might be obtained actions are the rule rather than the exception in human through training with pictures or video clips of the bird processing of sensory information [10,11] and there exists a types labeled with the name of the species. However, rapidly growing literature of the neuroanatomical, electro- mastering perceptual skills is often slow and effortful in physiological and neuroimaging studies that show that adults (although some initial learning can be rapid [6]). multisensory interactions can occur throughout processing Studies of perceptual learning show that a simple task of [11–13]. detecting a subtle visual stimulus can require a month or There are numerous brain areas and pathways for more of training to asymptote, with only a small incre- multisensory interactions, ranging from the brain stem ment of performance [7–9]. [14] to early sensory cortical areas [13,15], to association Would the learning be easier if the training combined the and other cortical areas [12], including feedforward [16] images with the sound of the birds? Most likely ‘yes’ because and feedback pathways [17,18]. Of particular interest are the bird songs are another feature that can be used to recently discovered multisensory modulations, activations distinguish the different species. What if the individual and connectivity at the earliest stages of perceptual pro- must discriminate among the species only based on pictures cessing; areas that have long been viewed as ‘sensory- in the absence of any sound? Would training with auditory– specific’ [12,13] (also see Refs [19–21]). Such findings of visual (i.e. multisensory) stimuli still be beneficial if the multisensory interactions in early brain areas, raise the ultimate goal of learning is to visually discriminate them? question of whether any brain regions can be fully charac- Traditionally, researchers of perception would answer ‘no’, terized through their unisensory response properties [12], with the assumption that if the task to be learned is visual, and motivate our assertion that learning at all processing then sound features that would be absent during the task stages can involve multisensory processes. would not be very helpful for learning, and could even be disruptive to learning by distracting attention away from Multisensory plasticity in development visual features. However, here we discuss findings that Both animal and human studies of the early stages of life show the opposite. show that, during early development, an alteration to the environment or a disruption of processing to one sense, can Corresponding authors: Shams, L. ([email protected]); result in a striking degree of neural plasticity between Seitz, A.R. ([email protected]). the senses [10,11,22–25]. For example, changing spatial 1364-6613/$ – see front matter ß 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.tics.2008.07.006 Available online 18 September 2008 411 Opinion Trends in Cognitive Sciences Vol.12 No.11 correlations between auditory and visual inputs alters the cues demonstrated facilitated learning of subthreshold multisensory representation in the superior colliculus [26], stimuli as compared to drosophila conditioned with only and disruption to the inputs to auditory cortex can result in visual cues. More importantly, unisensory memory retrieval this area being driven by visual inputs [27]. These findings was also enhanced by the multisensory learning conditions. provide evidence that learning and plasticity in early life can Moreover, preconditioning with bimodal stimuli followed by be highly multisensory. unisensory conditioning led to crossmodal memory transfer. Furthermore, multisensory stimulation has been argued These results indicate that multisensory training pro- to provide a redundancy that is crucial for extracting infor- motes more effective learning of the information than mation that would not be possible based on unisensory unisensory training. Although these findings span a large stimulation alone in early infancy. For example, Bahrick range of processing levels and might be mediated by differ- and Lickliter [28] showed that 5-month-old infants could ent mechanisms, it nonetheless seems that the multisen- discriminate visually presented rhythms only if they were sory benefit to learning is an overarching phenomenon. habituated with auditory–visual presentations of the rhythm and not when habituated with visual-only or audi- The importance of congruency tory-only presentations of rhythm. Can training on any arbitrary pair of auditory and visual stimuli have such facilitatory learning effects? We suspect Multisensory facilitation of unisensory learning that the answer is ‘yes and no’. Although training on any pair Although sensory plasticity [5] and perceptual learning in of multisensory stimuli might induce a more effective adults is considerably more restricted compared to devel- representation of the unisensory stimulus, the effects could opment [29], accumulating reports indicate the superiority be substantially more pronounced for congruent stimuli. of bisensory training in adult learners. A recent study Here, we define congruency broadly as the relationship compared auditory–visual and visual training for percep- between stimuli that are consistent with the prior experi- tual learning by using a coherent motion detection and ence of the individual or relationships between the senses discrimination task [30]. Compared to a visual (V) group, found in nature. This spans the basic attributes such as the auditory–visual (AV) group showed greater learning concordance in space and time, in addition to higher-level both within the first session (Figure 1a) and across the features such as semantic content (e.g. object and speech ten training sessions (Figure 1b). The two groups were information). Although some of these relationships might be compared on trials that only contained visual signals and more fundamental than others, we regard congruency as an no auditory signal and, therefore, the advantage of multi- overall function of all these relationships. As such, the sensory training was evident even in the absence of auditory extant data seem to support the role of congruency in information. The advantage of auditory–visual training over learning facilitation. visual-alone training in this study was substantial; it For example, Kim, Seitz and Shams [33] compared learn- reduced the number of sessions required to reach asymptote ing across three groups, one trained with visual motion by as much as 60%, while also raising the maximum stimuli only, one with congruent auditory–visual (both mov- performance. ing in the same direction) and one with incongruent audi- Complementary results were found in a study of audi- tory–visual stimuli (moving in opposite directions). They tory recognition