REVIEWS Parallel processing strategies of the primate visual system Jonathan J. Nassi* and Edward M. Callaway‡ Abstract | Incoming sensory information is sent to the brain along modality-specific channels corresponding to the five senses. Each of these channels further parses the incoming signals into parallel streams to provide a compact, efficient input to the brain. Ultimately, these parallel input signals must be elaborated on and integrated in the cortex to provide a unified and coherent percept. Recent studies in the primate visual cortex have greatly contributed to our understanding of how this goal is accomplished. Multiple strategies including retinal tiling, hierarchical and parallel processing and modularity, defined spatially and by cell type-specific connectivity, are used by the visual system to recover the intricate detail of our visual surroundings. Percept Our richest life experiences and longest-lasting memo- Within just a few hundred micrometres of retinal thick- The perception that arises ries are formed through our interactions with the sen- ness, this initial signal encoded by our photo receptors internally, in the mind, based sory world around us and have a profound impact on must be transformed into an adequate representation on an external stimulus, such our personality and sense of self. Each of these sensations of the entire visual scene. This representation, however, as a visual stimulus. is transmitted to our brains through distinct biological must also be sufficiently condensed so that the axons Parallel processing machinery such as that found in the nose or under the carrying it can pass through the optic nerve, which Simultaneous processing of skin. Yet, rather than perceiving the world as a disjointed forms an anatomical bottleneck along the route from the information through collection of attributes, we most often experience a single eye to the brain. Probably owing to these constraints, independent circuits. unified percept. Parallel processing of sensory information incoming visual signals are processed by at least 80 ana- Photoreceptor is a commonly used strategy in the mammalian brain, tomically and physiologically distinct neural cell popula- A specialized cell in the retina not only between sensory modalities but across features tions and 20 separate circuits in the retina. These circuits that detects light and responds of a single sense as well. Gasser and Erlanger1 first dem- comprise at least a dozen parallel pathways that project with a change in membrane onstrated that the sensations of pain and temperature are to the brain for further processing10. The visual cortex potential and a change in transmitted through axons of different calibre from those has the job of extracting the relevant information from neurotransmitter release. that transmit touch. This work was shortly followed up this reduced signal and of further elaborating and inte- Tiling by Bishop2, who proposed that the three different classes grating the information into a unified and coherent Relatively uniform and of axons he found in the optic nerve process different perceptual experience. complete coverage of space. sensory qualities related to vision. These early discover- Many years of research have uncovered important ies laid the foundation for our current understanding of details concerning the anatomy and functional organi- the nervous system’s parallel processing strategies3–8. zation of the primate visual system. This Review focuses *Harvard Medical School, Department of Neurobiology, The need for parallel processing in the visual sys- on recent advances in our understanding of how the pri- 220 Longwood Avenue, tem is immediately appreciated when one considers mary visual cortex (V1) integrates parallel inputs and Boston, Massachusetts the multitude of qualities that are present in the visual constructs new, parallel outputs. These findings have 02115, USA. environment and the physical limitations of the way been particularly helpful in elucidating the complex rela- ‡The Salk Institute for Biological Studies, this information is initially encoded and signalled to tionship between early parallel pathways of the retina Systems Neurobiology, the brain. Colour, depth, shape and motion are just a and the processing streams in the visual cortex. Instead 10010 North Torrey Pines few of the many dimensions through which we inter- of attempting to provide a comprehensive analysis at Road, La Jolla, California pret our visual environment and generate appropriate each level of the visual system, we highlight key princi- 92037, USA. behaviour9. Remarkably, this complexity in our visual ples, such as retinal tiling, hierarchical processing, parallel Correspondence to E.M.C. modularity e-mail: [email protected] surroundings is first encoded as a pattern of light on a processing and (defined spatially and by cell doi:10.1038/nrn2619 two-dimensional array of photoreceptors, with little direct type-specific connectivity), with the aim of providing a Published online 8 April 2009 resemblance to the original input or the ultimate percept. unified and coherent understanding of the processing 360 | MAY 2009 | VOLUME 10 www.nature.com/reviews/neuro © 2009 Macmillan Publishers Limited. All rights reserved REVIEWS Hierarchical processing strategies in the visual system. We hope that this will representation across the entire visual field of the pri- 12 Processing that takes place in inform the understanding not only of visual perception, mary sensory cues it conveys to the brain . These cues serial order, with more but of sensory processing in general. include different spatial and temporal frequencies, lumi- sophisticated properties nance and colour contrasts in the image. At any given emerging at higher levels Parallel pathways from the retina to the cortex through the build-up of simpler point in the visual field, multiple ganglion cell types con- properties at lower levels. The first steps in seeing begin in the retina, where a dense vey different aspects of the visual input simultaneously array of photoreceptors convert the incoming pattern of and in parallel to the brain. Modularity light into an electrochemical signal11. The photorecep- Each ganglion cell type has its own distinct set of When repeating modules are tor mosaic encodes the intensity of light as a function of morphological features, such as soma size and dendritic used to conduct similar (BOX 1) 13 operations. Typically, in the position (two dimensions), wavelength and time . field size and density . Many of these features vary sub- visual cortex each module will Much information is lost from the outset, such as the stantially as a function of retinal eccentricity, but at any perform an operation related exact spectral composition of the image. Nevertheless, given eccentricity they allow for nearly unambiguous to visual information from a computations carried out in our visual system are sup- cell type classification. Each ganglion cell type also has portion of the visual space. Together the modules cover plied with enough information to support highly precise a distinct pattern of dendritic stratification in the inner the space so that the operation hue discriminations and other perceptual abilities that plexiform layer (IPL), allowing for highly specific pat- is conducted over the entire inform our everyday behaviour. Remarkably, many of terns of synaptic connectivity with functionally and/or visual scene. these computations are carried out in the retina, before anatomically defined bipolar and amacrine cell types8 the visual signals even leave the eye. Specialized circuits (FIG. 1a,b). These bipolar and amacrine cell types, in turn, Eccentricity Distance from the centre. It is extract basic sensory cues, such as spatial contrast and have their own unique connections with photoreceptors typically used to describe the temporal frequency, from the initial intensity distribu- and horizontal cells in the outer plexiform layer (OPL), distance of a visual receptive tion and encode these properties across approximately forming distinct anatomical circuits from photorecep- field from the centre of gaze 1.5 million ganglion cells, which form the optic nerve tors to ganglion cells (FIG. 1b). Finally, the somas of each and is expressed as an angle, in degrees. that connects the eye to the brain. ganglion cell type seem to be regularly spaced across the Owing to the anatomical bottleneck of the optic retina so that, collectively, their dendritic fields cover Colour-opponent signal nerve, retinal output must be efficiently condensed. the retina uniformly and with constant overlap (FIG. 1c). The signal that results when a The strategy used by the mammalian visual system is This highly ordered mosaic architecture holds up only visual receptive field is excited to reduce the representation of the visual scene to a among ganglion cells of the same type: subsets of a type in response to one colour and inhibited in response to limited number of specialized, parallel output channels. do not cover the entire visual field and different types 12 another. Rather than send visual signals from the eye to the brain display no systematic spatial relationship . Together, along a homogeneous population of ganglion cells, in these structural features suggest that each ganglion cell the primate at least 17 distinct ganglion cell types exist type is a fundamental unit of the retina and underlies a in the retina and at least 13 of these project in parallel unique channel of visual information.