View metadata, citation and similar papers at core.ac.uk brought to you by CORE BEHAVIORAL AND BRAIN SCIENCES (2008) 31, 179–239 provided by RERO DOC Digital Library Printed in the United States of America doi: 10.1017/S0140525X08003804 Visual prediction: Psychophysics and neurophysiology of compensation for time delays Romi Nijhawan Department of Psychology, University of Sussex, Falmer, East Sussex, BN1 9QH, United Kingdom [email protected] http://www.sussex.ac.uk/psychology/profile116415.html Abstract: A necessary consequence of the nature of neural transmission systems is that as change in the physical state of a time-varying event takes place, delays produce error between the instantaneous registered state and the external state. Another source of delay is the transmission of internal motor commands to muscles and the inertia of the musculoskeletal system. How does the central nervous system compensate for these pervasive delays? Although it has been argued that delay compensation occurs late in the motor planning stages, even the earliest visual processes, such as phototransduction, contribute significantly to delays. I argue that compensation is not an exclusive property of the motor system, but rather, is a pervasive feature of the central nervous system (CNS) organization. Although the motor planning system may contain a highly flexible compensation mechanism, accounting not just for delays but also variability in delays (e.g., those resulting from variations in luminance contrast, internal body temperature, muscle fatigue, etc.), visual mechanisms also contribute to compensation. Previous suggestions of this notion of “visual prediction” led to a lively debate producing re-examination of previous arguments, new analyses, and review of the experiments presented here. Understanding visual prediction will inform our theories of sensory processes and visual perception, and will impact our notion of visual awareness. Keywords: biased competition; compensation; coordinate transformation; feedforward control; flash-lag; internal models; lateral interactions; neural representation; reference frames; time delays; visual awareness 1. Introduction Visual processing occurs in a series of hierarchical steps involving photoreceptors, retinal bipolar cells, ganglion 1.1. Time delays in the nervous system cells, the lateral geniculate nucleus (LGN), the primary Time delays are intrinsic to all neural processes. visual cortex (V1), and beyond. Neural delays have been Helmholtz, an eminent physicist and neurophysiologist extensively investigated at various levels within the visual of the nineteenth century, was among the first scientists system. For example, in response to retinal stimulation, to provide a clear measure of the speed of signal trans- significant neural delays (.10 msec) have been measured mission within the nervous system. Using a nerve– within the retina and at the optic nerve (Dreher et al. muscle preparation, he electrically stimulated the motor 1976; Kaplan & Shapley 1982; Ratliff & Hartline 1959). nerve at two different points and noted the time of con- Schiller and Malpeli (1978) electrically stimulated axons traction of the connected muscle. The distance between of the optic nerve at the optic chiasm and measured the the stimulated points divided by the time difference with delay (2–3 msec) in the response of cells in the which the muscle responded, gave the speed of neural magno- and parvo-cellular layers of the LGN. Many transmission along the given section of the motor nerve. Before this seminal experiment in 1850, many well- ROMI NIJHAWAN is a Reader (Associate Professor) known scientists including Johannes Mu¨ ller had specu- in Psychology at the University of Sussex, England. lated that transmission of neural signals would be too He received his Ph.D. in 1990 from Rutgers Univer- fast to allow experimental measurement. However, to sity–New Brunswick. During his post-doctorate at the surprise of the scientific community, the experiment the University of California–Berkeley, Nijhawan revealed not only that the speed of neural conduction “re-discovered” what he termed as the flash-lag effect was measurable, but also that it was an order of magnitude and subsequently published a brief paper on it (in slower than the speed of sound through air! In this article, Nature, 1994). This paper provocatively introduced I focus on visual delays, the problem of measurement of the notion of prediction in sensory pathways, giving visual delays, and the effect these delays have on neural rise both to active empirical research on the flash-lag phenomenon, and to a spirited debate regarding the representations of change – such as those that result existence and nature of visual prediction. Since then, from visual motion – and on perception and behavior. In Nijhawan has published multiple papers in the area addition, the focus is on potential mechanisms that of sensory prediction. might compensate for visual delays. # 2008 Cambridge University Press 0140-525X/08 $40.00 179 Downloaded from https:/www.cambridge.org/core. University of Basel Library, on 30 May 2017 at 18:09:29, subject to the Cambridge Core terms of use, available at https:/www.cambridge.org/core/terms. https://doi.org/10.1017/S0140525X08003907 Nijhawan: Visual prediction studies on the macaque have recorded the delay with (CNS), the types of the intervening synapses, the strength which neurons in the primary visual cortex (V1) respond of stimulation, and to an extent certain qualitative aspects to retinal stimulation (Raiguel et al. 1989; Maunsell & of the stimulus. Gibson 1992; Schmolesky et al. 1998). An estimate based Several investigators have directly studied the relation- on a large database yields an average delay of approxi- ship between neuronal latency and reaction time. In mately 72 msec of V1 neurons (Lamme & Roelfsema measurements of reaction time, presentation of a discrete 2000). However, there is usually a range of delays with stimulus starts a time-counter, which the subject’s overt which neurons in a given area of the cortex respond. motor response stops. In measurements of neuronal Different neurons in the macaque inferotemporal cortex, latency, however, discerning the neural event used to for example, respond to complex visual information with stop the time-counter can be more challenging. In one delays ranging from 100 to 200 msec (Nakamura et al. study, researchers recorded from single neurons in the 1994). monkey motor cortex while the animal performed a learned flexion/extension arm movement in response to a visual, somatosensory, or auditory stimulus (Lamarre 1.2. Neuronal and behavioral delays for discrete stimuli et al. 1983). They reported a constant delay between the Discreteness of stimulation is key to defining time delays change in the firing rate of the motor cortical neuron for both neural responses and in behavioral tasks. For and arm movement, irrespective of the stochastic and visual neurons, time delays are typically measured in modality contingent variation in reaction times. Whether response to light flashes, light onsets/offsets, or short elec- change in a neuron’s spike rate following the stimulus, trical pulses applied to some point in the visual pathway. on a single trial, can be used in the measurement of neur- Neural delay is defined as the time interval between the onal latency (i.e., to stop the time-counter) is debatable. discrete change in stimulation and change in neural Individual spikes provide a sparse sample of the activity at the target site. The onset time of a discrete assumed underlying rate function of the neuron. Thus, stimulus is easily determined. However, the definition of even if one were to assume a step change in the underlying neural delays becomes more complex for time-varying rate function triggered by a discrete stimulus, the variance stimuli that are a continuous function of time. For such in spike times dictates averaging over many trials to deter- stimuli, neural delay can only be defined with respect to mine the precise time of the “true” change in the firing an instantaneous value of the stimulus. rate triggered by sensory stimulation (DiCarlo & Maunsell Discreteness of stimulation is central also to defining 2005). behavioral delays. The paradigm that formed the corner- stone of experimental psychology in the latter half of the 1.3. Neuronal and behavioral delays for continuous nineteenth century with the work of Helmholtz, Donders, stimuli Wundt, and Cattell, involved measurement of simple reaction times (Meyer et al. 1988). In simple reaction- Animals encounter both discrete and continuous environ- time tasks, the participant produces a prespecified overt mental events. At one extreme are discrete stimuli result- response – for example a button press – in response to a ing from unexpected events in nature (Walls 1942). At the discrete suprathreshold stimulus such as a flash of light, opposite extreme are stationary stimuli, for example, a sound burst, or a tactile stimulus. The stimulus–response nearby objects such as trees, that can be just as behavio- interval for the light stimulus is 200–250 msec, whereas for rally relevant as changing stimuli, but for which the issue sound or touch it is about 150 msec. The minimum latency of neural delays arises only when the animal itself moves for a voluntary learned motor response appears to be (see further on). Between the two extremes of the around 100–120 msec (Woodworth & Schlosberg 1954, discrete–continuous continuum, there are a multitude of p 9).