Sensory Cue Integration SECTION I Introduction to Section I: Theory and Fundamentals The chapters in Section I formalize the com- evidence necessary to test the predictions arising putational problems that need to be solved for from these models. Wei and Körding focus successful cue combination. They focus on the on ideal-observer models for cases where it issue of uncertainty and the Bayesian ways of is not a priori certain that the cues belong solving such problems. Through the sequence together. In this particular case, the nervous of chapters in Section I, the reader will get a system needs to determine whether the cues thorough overview and introduction into the belong together or, in other words, to determine current Bayesian formalization of sensory cue their causal relationship. Battaglia, Kersten, and combination. Schrater focus on object perception. They ask This section highlights the fundamental sim- how the kinds of causal knowledge we have ilarity between seemingly distinct problems of about the way objects are made can be used cue combination. The computational objectives to constrain estimates of attributes such as and the algorithms that can be used to find the shape and size. Vijayakumar, Hospedales, and optimal solution do not depend on the modality Haith extend these ideas to model cue com- or kind of cue that is considered. The solutions bination for sensorimotor integration. This is to progressively more complicated problems complicated because sensorimotor tasks depend are largely derived using essentially the same on a set of variables that must be estimated statistical techniques. simultaneously. Lastly, Sahani and Whiteley The first five chapters of Section I are extend these concepts to cue integration in concerned with Bayesian theories of optimal cluttered environments. They point out that cue combination. The book starts with an complicated visual scenes are a special case introductory chapter by Landy, Banks, and Knill of situations in which we are uncertain about that gives an overview of the basic philosophy the causal relationship between cues. These and the mathematics that is used to calculate chapters use a coherent probabilistic language how an ideal observer would combine cues. to develop methods appropriate for a wide range These models form the backbone of much of of problems. the cue-combination research presented in this The last three chapters of Section I highlight book. The following computational chapters limitations of the standard ideas of probabilistic provide more detailed insights into the basic cue combination. They point out interesting techniques, computational tools, and behavioral ways in which human observers fall short of 3 4 THEORY AND FUNDAMENTALS optimal predictions. Backus explains how novel scale factor) and yet, perhaps surprisingly, is cues can be recruited for perception. In the closely related to the Bayesian weighted-linear next chapter of the section, Domini and Caudek model discussed in Chapter 1. Finally, Rosas propose an alternative to the Bayesian ideal and Wichmann discuss limits of sensory cue observer for combining cues to depth. The integration, suggesting that simple ideas of simple algorithm they present results in an affine optimality may be inappropriate in a complex estimate of depth (i.e., depth up to an unknown world. CHAPTER 1 Ideal-Observer Models of Cue Integration Michael S. Landy, Martin S. Banks, and David C. Knill When an organism estimates a property of the plan (and a fumbled grasp) or incorrect motor environment so as to make a decision (“Do I flee decisions (and a risky descent possibly leading or do I fight?”) or plan an action (“How do I grab to a fall). Thus, estimation accuracy can be very that salt shaker without tipping my wine glass important, so the observer should use all sources along the way?”), there are typically multiple of information effectively. sources of information (signals or “cues”) that The sensory information available to an are useful. These may include different features observer may come in the form of multiple of the input from one sense, such as vision, where visual cues (the pattern of binocular disparity, a variety of cues—texture, motion, binocular linear perspective and foreshortening, shading, disparity, and so forth—aid the estimation etc.) as well as haptic cues (feeling the surface of the three-dimensional (3D) layout of the with the hand, testing the slope with a foot). If environment and shapes of objects within it. one of the cues always provided the observer Information may also derive from multiple with a perfect estimate, there would be no senses such as visual and haptic information need to incorporate information from other about object size, or visual and auditory cues cues. But cues are often imperfectly related to about the location of a sound. In most cases, environmental properties because of variability the organism can make more accurate estimates in the mapping between the cue value and a of environmental properties or more beneficial given property and because of errors in the decisions by integrating these multiple sources of nervous system’s measurement of the cue value. information. In this chapter, we review models of Thus, measured cue values will vary somewhat cue integration and discuss benefits and possible unpredictably across viewing conditions and pitfalls in applying these ideas to models of scenes. For example, stereopsis provides more behavior. accurate estimates of surface orientation for Consider the problem of estimating the 3D near than for far surfaces. This is due to orientation (i.e., slant and tilt) of a smooth the geometry underlying binocular disparity: A surface (Hillis, Ernst, Banks, & Landy, 2002; small amount of measurement error translates Hillis, Watt, Landy, & Banks, 2004; Knill into a larger depth error at long distances than & Saunders, 2003; Rosas, Wagemans, Ernst, at short ones. In addition, estimates may be & Wichmann, 2005). An estimate of surface based on assumptions about the scene and will orientation is useful for guiding a variety of be flawed if those assumptions are invalid. For actions, ranging from reaching for and grasping example, the use of texture perspective cues is an object (Knill, 2005) to judging whether generally based on the assumption that texture one can safely walk or crawl down an incline is homogeneously distributed across the surface, (Adolph, 1997). Errors in the estimate may so estimates based on this assumption will lead to failures of execution of the motor be incorrect if the texture itself varies across 5 6 THEORY AND FUNDAMENTALS the surface. For example, viewing a frontoparallel organism has evolved mechanisms that utilize photograph of a slanted, textured surface could the available information optimally. Therefore, yield the erroneous estimate that the photograph the hypothesis that sensory information is used is slanted. Unlike stereopsis, the reliability of optimally in tasks that are important to the texture perspective as a cue to surface orientation organism is a reasonable starting point. Indeed, does not diminish with viewing distance. given the efficacy of natural selection and Because of this uncertain relationship developmental learning mechanisms, it seems between a cue measurement and the environ- unlikely to us that the nervous system would mental property to be estimated, the observer perform suboptimally in an important task with can generally improve the reliability of an stimuli that are good exemplars of the natural estimate of an environmental property by environment (as opposed to impoverished or combining multiple cues in a rational fashion. unusual stimuli that are only encountered in The combination rule needs to take into account the laboratory). Third, using optimality as a the uncertainties associated with the individual starting point, the observation of suboptimal cues, and those depend on many factors. behavior can be particularly informative. It can Along with the benefit of improving the indicate flaws in our characterization of the reliability of perceptual estimates, there is also a perceptual problem posed to or solved by the clear benefit of knowing how uncertain the final observer; for example, it could indicate that estimate is and how to make decisions given that the perceptual system is optimized for tasks uncertainty. Consider, for example, estimating other than one we have studied or that the the distance to a precipitous drop-off (Maloney, assumptions made in our formulation of an 2002). An observer can estimate that distance ideal-observer model fail to capture the problem most reliably by using all available cues, but posed to observers in naturalistic situations. Of knowing the uncertainty of that estimate can course, there remains the possibility that we have be crucial for guiding future behavior. If the characterized the sensory information and the future task is to toss a ball as close as possible task correctly, but the nervous system simply has to the drop-off, one would use the most likely not developed the mechanisms for performing distance estimate to plan the toss; the plan would optimally (e.g., Domini & Braunstein, 1998; be unaffected by the uncertainty of the distance Todd, 2004). We expect that such occurrences estimate. If, however, the future task is to walk are rare, but emerging scientific investigations blindfolded toward the drop-off, the decision of will ultimately determine this. how far to meander toward the drop would most In this way, “ideal-observer”