Depth Perception 0

Depth Perception

Suggested reading:!

! Stephen E. Palmer (1999) Vision Science.

! Read JCA (2005) Early computational processing in binocular vision and depth perception. Progress in Biophysics and Molecular Biology 87: 77-108

Depth Perception 1 Depth Perception!

Contents: Contents: • The problem of depth perception • • WasDepth is lerning cues ? • • ZellulärerPanum’s Mechanismus fusional area der LTP • • UnüberwachtesBinocular disparity Lernen • • ÜberwachtesRandom dot Lernen stereograms • • Delta-LernregelThe correspondence problem • • FehlerfunktionEarly models • • GradientenabstiegsverfahrenDisparity sensitive cells in V1 • • ReinforcementDisparity tuning Lernen curve • Binocular engergy model • The role of higher visual areas Depth Perception 2 The problem of depth perception!

Our 3 dimensional environment gets reduced to two-dimensional images on the two retinae.

Viewing Retinal geometry image

The “lost” dimension is commonly called depth.

As depth is crucial for almost all of our interaction with our environment and we perceive the 2D images as three dimensional the brain must try to estimate the depth from the retinal images as good as it can with the help of different cues and mechanisms.

Depth Perception 3 Depth cues - monocular! Texture Accretion/Deletion Accommodation (monocular, optical) (monocular, ocular)

Motion Parallax Convergence of Parallels (monocular, optical) (monocular, optical) Depth Perception 4 Depth cues - monocular !

Position relative to Horizon Familiar Size (monocular, optical) (monocular, optical)

Relative Size Texture Gradients (monocular, optical) (monocular, optical)

Depth Perception 5 Depth cues - monocular !

! Edge Interpretation (monocular, optical)

! Shading and Shadows (monocular, optical)

! Aerial Perspective (monocular, optical) Depth Perception 6 Depth cues – binocular!

Convergence Binocular Disparity (binocular, ocular) (binocular, optical)

As the eyes are horizontally displaced we see the world from two slightly different viewpoints.

The visual fields of the two eyes overlap in the central region of vision but points that are not on the so-called horopter fall onto different retinal positions.

This lateral displacement which is relative to the fixation point is called binocular disparity.

Depth Perception 7 Panum's fusional area!

If we see different views of the world with both eyes why don't we experience double vision?

Points in an area near the horopter (called Panum's fusional area) are fused perceptually into a single experienced image. Depth Perception 8 Panum's fusional area!

You can easily check this perceptual fact for yourself

Hold one forefinger in front of you and the other forefinger 30cm behind it. If you focus on one finger you will see the other finger twice because it is not within the limits of Panum's fusional region.

Depth Perception 9 Binocular disparity!

The point that we fixate on (here P) by definition has no binocular disparity as it falls directly onto the fovea at the same retinal position in both eyes.

Fixating a point P the image of points like C that are closer to the viewer than P are displaced outwardly on the two retinae in a so-called crossed disparity, whereas farther points as F are displaced inwardly in a so-called uncrossed disparity.

The direction and amount of binocular disparity therefore specify relative depth information in regard to the horopter. Depth Perception 10 Binocular disparity!

Example of the use of binocular disparity for the experience of depth - Anaglyphs

Nothing new, discovered in the 1850s by Joseph D’Almeida and Louis Du Hauron, William Friese-Green created the first 3D anaglyphic motion pictures in 1889.

Depth Perception 11 Random dot stereograms!

Until about 40 years ago, a viable theory was that the visual system operates on the two retinal images separately, performing scene segmentation and object recognition, and then finally, for each object in the scene, compares where its images fall in the two retinae.

However, in the 1960s, Bela Julesz drew attention to the fact that stereopsis works perfectly well with random dot stereograms. Depth Perception 12 Random dot stereograms!

Julesz, 1971 1. two identical images with random dots 2. cut out a shape that you want to appear in a different depth plane and slide it to the right in one image – the amount will determine the depth 3. fill the empty hole that has appeared through the sliding with random dots 4. put the original image left to the newly created one – that’s it!

Random dot stereograms show that binocular disparity is such a strong depth cue that it does not need any monocular cue for depth perception.

Depth Perception 13 The correspondence problem!

Something is missing – the correspondence problem!

Until now we have talked about how the direction and amount of disparity between corresponding image features in the two eyes can encode depth information but the problem we have excluded until now is how to determine which features in one retinal image correspond to which features in the other?

Computer Vision: match contents of image parts using global cost functions. Depth Perception 14 The brains solution to correspondence problem!

Depth Perception 15 Early models!

First Marr-Poggio Algorithm (MIT 1977) Approach: matching individual pixels in the left and right image, inverse projection Constraints 1. Color constancy: corresponding points have the same color

2. Surface opacity: most surfaces in the world are opaque so only the nearest can be seen ! If A10 is a correct match, then B10, C10, D10 etc. cannot be correct.

3. Surface continuity: surfaces in the world tend to be locally continuous in depth ! excitatory connections between close matches

Note! The constraints are heuristic assumptions that are Depth Perception 16 Early models!

First Marr-Poggio Algorithm – dynamic behavior of the network

Depth Perception 17 Disparity sensitive cells in V1!

In the human visual system the inputs of the two eyes converge upon single cells in V1. Position disparity Phase disparity , Neuron, , 1997. Qian From: From:

Recent studies indicate that most disparity sensitive cells are hybrid, showing both position and phase disparity (Ohzawa, Freeman 1997/1999 – Livingstone, Tsao 1999 – Prince, Cumming, Parker 2002). Depth Perception 18 Disparity tuning curve! Schematic Drawing of a Disparity Tuning Curve for a Binocular Cell:

Three representative retinal stimulus configurations correspond to a point in space that is nearer than, at, or further away from the fixation point.

Depth Perception 19 Disparity tuning curve!

Idealized tuning curves after characterization by Poggio & Fischer (1977)

Disparity tuning: tuned-excitatory, near, tuned-inhibitory, and far cells.

In each graph, the horizontal axis is disparity and the vertical axis is response rate.

The curves are Gabor functions (the product of a cosine and a Gaussian); the angle represents the phase of the cosine component. Depth Perception 20 Binocular Energy model!

Ohzawa et al., 1990; Qian, 1994, Read and Cumming 2007

A complex cell Cx receives input from binocular simple cells (BS). Receptive fields of two BS that differ in their phase by 90° get combined to a complex cell Cx.

BS =

Cx = BS + BS

Depth Perception 21 Binocular Energy model !

Image value at position (x,y)

Receptive field of equivalent left eye simple cell

With the Gabor filter

preferred preferred position phase disparity disparity with standard and deviation Depth Perception 22 Solution to the correspondence problem!

V1 neurons are sensitive to local matches, not to global solutions of the correspondence problem.

Anti-correlated stereograms

With the discovery of V1 disparity sensitive cells the question arose if the correspondence problem was solved in V1 already or in a higher brain region.

One approach to test for this were anti-correlated stereograms. These consist of normal stereograms but the brightness information in one of the images is reversed in comparison to the other image. Because of this there is no global- match solution to the stereo correspondence problem in anti-correlated stereograms.

Nevertheless the disparity sensitive cells in V1 respond to the local false matches in anti-correlated stereograms which is a strong indication that the correspondence problem does not get solved in V1.

Depth Perception 23 The role of higher visual areas!

V2: relative disparity and disparity-defined edges (Thomas et al. 2002, von der Heydt et al., 2000, Qiu & von der Heydt, 2005) V5/MT: Microstimulation can bias the monkey’s judgement in a far-near stereo task that relies on absolute disparity (DeAngelis et al. 1998), but no strong support for other aspects of stereovision.

V4: sensitive to the orientation of disparity defined planes (Hinkle & Connor, 2005). Depth Perception 24 Vergence Control!

Depth Perception 25

Additional reading:!

• Qian N. (1997) Binocular disparity and the perception of depth. Neuron, 18:359-68.! • Qian N, Zhu Y. (1997) Physiological computation of binocular disparity. Vision Res, 37:1811-27. ! Depth Perception 26

References:!

• Hinkle, D.A. and Connor, C.E. (2005) Quantitative characterization of disparity tuning in ventral pathway area V4. Journal of Neurophysiology, 94:2726-37. • Ohzawa I, DeAngelis GC, and Freeman RD (1990) Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors. Science 249:1037-1041. • Poggio GF, Fischer B. (1977) Binocular interaction and depth sensitivity in striate and prestriate cortex of behaving rhesus monkey. J Neurophysiol, 40:1392-405. • Qian, N (1994) Computing stereo disparity and motion with known binocular cell properties, Neural Computation, 6: 390-404.! • Qiu, F.T. and von der Heydt, R. (2005) Figure and ground in the visual cortex: V2 combines stereoscopic cues with Gestalt rules. Neuron, 47:155-166.! • Read JCA, Cumming BG (2007) Sensors for impossible stimuli may solve the stereo correspondence problem. Nature Neuroscience, 10:1322-1328. ! • Thomas OM, Cumming BG, and Parker AJ (2002) A specialization for relative disparity in V2. Nat Neurosci 5: 472–478.! • von der Heydt, R., Zhou, H. and Friedman, H.S. (2000) Representation of stereoscopic edges in monkey visual cortex. Vision Research, 40:1955-1967.!