Multisensory Interaction in Saccadic Reaction Time: A Time-Window-of-Integration Model Hans Colonius1 and Adele Diederich2 Abstract & Saccadic reaction time to visual targets tends to be faster transformations necessary for coordinated head and eye when stimuli from another modality (in particular, audition movements. However, because these models do not capture and touch) are presented in close temporal or spatial the temporal coincidences critical for multisensory integration proximity even when subjects are instructed to ignore the to occur, they cannot easily predict multisensory effects accessory input (focused attention task). Multisensory inter- observed in behavioral data such as saccadic reaction times. action effects measured in neural structures involved in This article proposes a quantitative stochastic framework, the saccade generation (in particular, the superior colliculus) have time-window-of-integration model, to account for the tempo- demonstrated a similar spatio-temporal dependence. Neural ral rules of multisensory integration. Saccadic responses network models of multisensory spatial integration have been collected from a visual–tactile focused attention task are shown to generate convergence of the visual, auditory, and shown to be consistent with the time-window-of-integration tactile reference frames and the sensorimotor coordinate model predictions. & INTRODUCTION an area clearly involved in saccade generation (Robin- Saccades are fast, voluntary movements of the eyes to son, 1972). Multisensory neurons in SC of anesthetized align the high-resolution fovea with objects and events cats (Meredith & Stein, 1986a, 1986b; Stein, Magalhaes- of interest. In a natural environment, saccades are part Castro, & Kruger, 1976) and monkeys (Wallace, Wilkin- of a rapid goal-directed orienting response, often includ- son, & Stein, 1996) show an enhanced response to ing head movements, to stimuli occurring in the periph- particular combinations of visual, auditory, and tactile ery. These stimuli are usually crossmodal (e.g., a rapidly stimuli paralleling the spatial–temporal rules observed in approaching subway train). In addition to visual and behavioral studies. Similar results have recently been auditory inputs, vestibular and somatosensory afferents obtained for recordings from unanesthetized cats by have access to the saccade-generating mechanism. Thus, Wallace, Meredith, and Stein (1998) and from the awake it is no surprise that the oculomotor system has become behaving monkey by Bell, Corneil, Meredith, and Munoz a prominent arena for the analysis of multisensory (2001) and by Frens and Van Opstal (1998). Information integration. For example, it has been found that saccadic about stimulus location is represented topographically reaction time to visual targets (the time between the within the structure by an orderly arrangement of onset of the visual stimulus and the onset of the saccadic neurons according to the location of their respective eye movement) tends to be faster when auditory stimuli receptive fields (RFs). The spatial register among the are presented in close temporal or spatial proximity different sensory maps is formed by multisensory neu- (Colonius & Arndt, 2001; Harrington & Peck 1998; rons whose different RFs are in register with one anoth- Hughes, Nelson, & Aronchick, 1998; Frens, Van Opstal, er (Meredith & Stein, 1986a, 1986b; for a review, see & Van der Willigen, 1995). Similar response enhance- Stein & Meredith, 1993). In addition, the SC contains a ment effects for saccades have been observed for com- motor map composed of output neurons coding appro- bining visual and somatosensory stimuli (cf. Groh & priate eye movements (Sparks, 1986), that is, the locus Sparks, 1996a, for monkeys; Amloˆt, Walker, Driver, & of activity in the motor map encodes a movement Spence, 2003; Diederich, Colonius, Bockhorst, & Tabel- command that reorients the eyes (and the head) a given ing, 2003, for humans). distance in a particular direction. These psychophysical observations are in line with There are a number of explicit neural network models neurophysiological evidence for multisensory integra- of multisensory spatial integration mechanism de- tion in the deep layers of the superior colliculus (SC), scribing the integration of visual, auditory, and tactile reference frames and the sensorimotor coordinate trans- formations necessary for directed head and eye move- 1Universitaet Oldenburg, 2International University Bremen ments (cf. Deneve, Latham, & Pouget, 2001; Xing & D 2004 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 16:6, pp. 1000–1009 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/0898929041502733 by guest on 27 September 2021 Andersen, 2000). Although these models capture many Because the first stage refers to very early sensory aspects of spatial rules of multisensory integration (for processing, random processing times for visual, audito- a recent critical review, see Pouget, Deneve, & Duhamel, ry, and somatosensory stimuli are assumed to be statis- 2002), they do not address the effects of temporal tically independent. coincidences of crossmodal stimulus sets. In this article, we propose a time-window-of-integration (TWIN) model Second Stage Assumption to serve as a framework incorporating the temporal rules of multisensory integration in saccades. A paramet- The second stage comprises neural integration of the ric version of the model is fit to a recent set of data on input and preparation of an ocular motor response. visual–tactile interaction in saccades. Multisensory integration manifests itself in an increase or decrease of second stage processing time. TWIN Model Distinguishing between two stages only is clearly an The classic explanation for a speed-up of responses to oversimplification. But note that the second stage is multisensory stimuli is that subjects are merely respond- defined by default: It includes all subsequent, possibly ing to the first stimulus detected. Formally, observed overlapping, processes that are not part of the periph- reaction time (RT) would then be the minimum of the eral processes in the first stage (for a formal description response time to the visual, auditory, or tactile signal of the model, see Methods section). causing a statistical facilitation effect (Raab, 1962). However, several studies have clearly shown that statis- tical facilitation is not sufficient to explain the speed- TWIN Assumption up in saccadic reaction time (Harrington & Peck, 1998; Multisensory integration occurs only if the periph- Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; eral processes of the first stage all terminate within a Hughes et al., 1998; Corneil & Munoz, 1996). Using given time interval, the ‘‘window of integration’’. Miller’s inequality as a benchmark test (see Diederich, 1995; Colonius, 1990; Miller, 1982), responses to bimodal The window of integration acts like a filter determin- stimuli have been found to be faster than predicted by ing whether afferent information delivered from dif- statistical facilitation, in particular, when the stimuli were ferent sensory organs is registered close enough in spatially aligned. Moreover, the race model has no simple time to allow for multisensory integration. Passing the explanation for the decrease in facilitation observed with filter is a necessary, but not a sufficient, condition for increasing spatial disparity between the stimuli. multisensory integration to occur. The reason is that Although there is increasing evidence for a complex multisensory integration also depends on the spatial network of largely parallel neural subprocesses un- configuration of the stimuli. However, rather than as- derlying performance even in simple crossmodal tasks suming the existence of a joint spatial–temporal window (Driver & Spence, 2000), the initial separation of the of integration permitting interaction to occur only for afferent pathways for the different sensory modalities both spatially and temporally neighbored stimuli, the suggests that one can distinguish at least two serial stages TWIN model allows for multisensory integration to occur of saccadic reaction time: an early, afferent stage of pe- even for rather distant stimuli (of different modalities) as ripheral processing (first stage) followed by a compound long as they fall within the time window. Such interac- stage of converging subprocesses (second stage). As tion will typically be an inhibition or only a small shown below, in conjunction with a number of weak ad- facilitation. This arrangement affords more flexibility in ditional assumptions, some interesting and empirically a complex environment. For example, response depres- testable predictions can be derived from this simple sion may occur with nearly simultaneous but distant setup. stimuli, making it easier for the organism to focus Even under invariant experimental conditions, saccad- attention on the more important event. ic responses typically vary from one trial to the next due to an inherent variability of the underlying neural pro- cesses in both ascending and descending pathways. This Predictions is taken into account in the TWIN model by assuming The TWIN model makes a number of empirical predic- the duration of each of the stages to be a random tions. First, the amount of multisensory integration variable (Figure 1). should depend on the stimulus onset asynchrony (SOA) between the stimuli. Indeed, the effect tends to First Stage Assumption be most prominent