CS4442/9542b Artificial Intelligence II prof. Olga Veksler Lecture 11 Computer Vision Stereo Some slides are from S. Seitz, S. Narasimhan, K. Grauman Outline • Cues for 3D reconstruction • Stereo Cues • Stereo Reconstruction 1) camera calibration and rectification • an easier, mostly solved problem 2) stereo correspondence • a harder problem 2D Images • World is 3D • In 2D images, depth (the third coordinate) is largely lost • includes human retina 2D Images • Depth is inherently ambiguous from a single view Z ? Y ? X ? P Street Pavement Art • Viewed from the “right” side Street Pavement Art • Viewed from the “wrong” side Babies and Animals Perceive Depth • Yet we perceive the world in 3D The Visual Cliff, by William Vandivert, 1960 3D Shape from Images • What image cues provide 3D information? • Cues from a single image • Cues from multiple images • Motion cues • Stereo cues • Can we use these cues in a computer vision system? Single Image 3D Cues: Shading • Pixels covered by shadow are perceived to be further away Merle Norman Cosmetics, Los Angeles Single Image 3D Cues: Linear Perspective • The further away are parallel lines, the closer they come together Single Image 3D Cues: Relative Size • If objects have the same size, those further away appear smaller Single Image 3D Cues: Texture • Further away texture appears finer (smaller scale) Single Image 3D Cues: Known Size • Ducks are smaller than elephants, duck is closer Illusions: Linear Perspective + Relative Size Illusions: Linear Perspective + Relative Size Illusions: Linear Perspective + Relative Size Illusions: Ames Room Cues from Multiple Image: Motion Parallax • Closer objects appear to move more than further away objects http://psych.hanover.edu/KRANTZ/MotionParallax/MotionParallax.html 3D Shape from X • X = shading, texture, motion, ... • We will focus on stereo • depth perception from two stereo images Why Two Eyes? Cylopes? Why Two Eyes? • Charles Wheatstone first explained stereopsis in 1838 left image right image (x,y) (x-d,y) left eye right eye 3D Scene Why Two Eyes? • Disparity d is the difference in x coordinates of corresponding points left image right image (x,y) (x-d,y) left eye right eye 3D Scene Stereoscopes • Wheatstone invented the first stereoscope Anaglyph Images • Encodes left and right image into a single picture • left eye image is transferred to the red channel • right eye image to the green+blue = cyan channel • Red filter lets through only the left image • Cyan filter lets through only theright eye image • Brain fuses into 3D • Similar technology for 3D movies • Works for most of us What is Needed for Stereopsis? • Need monocular cues for stereopsis? Need object cues? Answered by Julesz in 1960 • Image with no monocular cues and no recognizable objects: random dots Need Object Recognition for Stereopsis? • Answered by Julesz in 1960 • Make a copy of it Need Object Recognition for Stereopsis? • Answered by Julesz in 1960 • Select a square Need Object Recognition for Stereopsis? • Answered by Julesz in 1960 • Copy square the right image, shifting by d to the left • random dot stereogram Need Object Recognition for Stereopsis? • Answered by Julesz in 1960 • Random dot stereogram • Humans perceive square floating in front of background 3D Shape from Stereo • Use two cameras instead of two eyes left image right image (x,y) (x-d,y) left camera right camera 3D Scene Stereo System • Unlike eyes, usually stereo cameras are not on the same plane • better numerical stability 3D scene point optical center optical center left camera right camera Stereo System: Triangulation 3D scene point P optical center optical center left camera right camera • Depth by triangulation • given two corresponding points in the left and right image • cast the rays through the optical camera centers • ray intersection is the corresponding 3D world point P • depth of P is based on camera positions and parameters • Triangulation ideas can be traced to ancient Greece document from 1533 What is needed for Triangulation 1. Distance between cameras, camera focal length • Solved through camera calibration, essentially a solved problem • We will not talk about it • Code available on the web • OpenCV http://www.intel.com/research/mrl/research/opencv/ • Matlab, J. Bouget http://www.vision.caltech.edu/bouguetj/calib_doc/index.html • Zhengyou Zhang http://research.microsoft.com/~zhang/Calib/ 2. Pairs of corresponding pixels in left and right images • Called stereo correspondence problem, still much researched Formula: Depth from Disparity • Top down view on geometry (slice through XZ plane) • from camera calibration, know the distance between camera optical centers called baseline B, and camera focal length f P = (X,Y,Z) Z right image left image point point Z xl xr f f X Cl Cr left optical right optical center baseline B center Formula: Depth from Disparity Z • Height to base ratio of triangle C P C : l r B P = (X,Y,Z) Z right image left image point point Z xl xr f f X Cl Cr left optical right optical center baseline B center Formula: Depth from Disparity Z - f • Height to base ratio of triangle xl P xr : B - xl + xr • xl is positive, xr is negative P = (X,Y,Z) Z right image left image point point Z xl xr f f X Cl Cr left optical right optical center baseline B center Formula: Depth from Disparity Z Z - f • C P C and x P x are similar: = l r l r B B - x + x l r P = (X,Y,Z) Z right image left image point point Z xl xr f f X Cl Cr left optical right optical center baseline B center Formula: Depth from Disparity B f • Rewriting: Z = x - x l r • xl - xr is the disparity P = (X,Y,Z) Z right image left image point point Z xl xr f f X Cl Cr left optical right optical center baseline B center Stereo Correspondence: Epipolar Lines • Which pairs of pixels correspond to the same scene element ? optical center optical center left camera right camera • Epipolar constraint • Given a left image pixel, the corresponding pixel in the right image must lie on a line called the epipolar line • reduces correspondence to 1D search along conjugate epipolar lines • demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html Stereo Rectification • Epipolar lines can be computed from camera calibration • Usually they are not horizontal • Can rectify stereo pair to make epipolar lines horizontal Stereo Correspondence (x,y) (x-d,y) left image right image • From now on assume stereo pair is rectified • How to solve the correspondence problem? • Corresponding pixels should be similar in intensity • or color, or something else Difficulties in Stereo Correspondence • Image noise • corresponding pixels have similar, but not exactly the same intensities 90 90 98 left image patch right image patch • Matching each pixel individually is unreliable Difficulties in Stereo Correspondence • regions with (almost) constant intensity ? ? ? • Matching each pixel individually is unreliable Window Matching Correspondence • Use a window (patch) of pixels • more likely to have enough intensity variation to form a distinguishable pattern • also more robust to noise Window Matching Correspondence • Use a window (patch) of pixels • more likely to have enough intensity variation to form a distinguishable pattern • also more robust to noise Window Matching: Basic Algorithm • for each epipolar line • for each pixel p on the left line • compare window around p with same window shifted to many right window locations on corresponding epipolar line • pick location corresponding to the best matching window Which Locations to Try? (x,y) (x-maxDisp,y) (x-1,y)( x,y) • Disparity cannot be negative • Maximum possible disparity is limited by the camera setup • assume we know maxDisp • Disparity can range from 0 to maxDisp • consider only (x,y), (x-1,y),…(x-maxDisp,y) in the right image Window Matching Cost • How to define the best matching window? • Define window cost • sum of squared differences (SSD) • or sum of absolute differences (SAD) • many other possibilities • Pick window of best (smallest) cost SSD Window Cost left image right image 3 5 4 4 2 4 2 3 5 4 4 2 4 2 7 4 1 4 4 2 6 7 4 1 4 4 2 6 2 7 46 46 46 6 7 46 46 46 3 6 6 7 5 9 46 46 44 9 7 48 46 44 6 4 9 7 4 7 47 47 47 2 4 47 47 47 7 4 2 4 4 7 56 56 46 6 7 58 56 46 5 6 6 7 3 4 4 1 4 3 2 3 4 4 1 4 3 2 46442 4662 4442 47472 4772 4742 56462 5652 4662 12454 Algorithm with SSD Window Cost left image right image 3 5 4 4 2 4 2 3 5 4 4 2 4 2 7 4 1 4 4 2 6 7 4 1 4 4 2 6 2 7 46 46 46 6 7 46 46 46 3 6 6 7 5 9 46 46 44 9 7 48 46 44 6 4 9 7 4 7 47 47 47 2 4 47 47 47 7 4 2 4 4 7 56 56 46 6 7 58 56 46 5 6 6 7 3 4 4 1 4 3 2 3 4 4 1 4 3 2 46442 4662 4442 47472 4772 4742 56462 5652 4662 12454 • This shift corresponds to disparity 0 Algorithm with SSD Window Cost left image right image 3 5 4 4 2 4 2 3 5 4 4 2 4 2 7 4 1 4 4 2 6 7 4 1 4 4 2 6 2 7 46 46 46 6 7 46 46 46 3 6 6 7 5 9 46 46 44 9 7 48 46 44 6 4 9 7 4 7 47 47 47 2 4 47 47 47 7 4 2 4 4 7 56 56 46 6 7 58 56 46 5 6 6 7 3 4 4 1 4 3 2 3 4 4 1 4 3 2 46462 46442 4462 47472 4772 4772 56562 56462 4652 6425 • This shift corresponds to disparity 1 Algorithm with SSD Window Cost left image right image 3 5 4 4 2 4 2 3 5 4 4 2 4 2 7 4 1 4 4 2 6 7 4 1 4 4 2 6 2 7 46 46 46 6 7 46 46 46 3 6 6 7 5 9 46 46 44 9 7 48 46 44 6 4 9 7 4 7 47 47 47 2 4 47 47 47 7 4 2 4 4 7 56 56 46 6 7 58 56 46 5 6 6 7 3 4 4 1 4 3 2 3 4 4 1 4 3 2 46482 46462 44442 47472 47472 47472 56582 56562 46462 8 • This shift corresponds to disparity 2 Algorithm with SSD Window Cost left image right image 3 5 4 4 2 4 2 3 5 4 4 2 4 2 7 4 1 4 4 2 6 7 4 1 4 4 2 6 2 7 46 46 46 6