Scalable Multi-view Stereo Camera Array for Real World Real-Time Image Capture and Three-Dimensional Displays Samuel L. Hill B.S. Imaging and Photographic Technology Rochester Institute of Technology, 2000 M.S. Optical Sciences University of Arizona, 2002 Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning in Partial Fulfillment of the Requirements for the Degree of Master of Science in Media Arts and Sciences at the Massachusetts Institute of Technology June 2004 © 2004 Massachusetts Institute of Technology. All Rights Reserved. Signature of Author:<_- Samuel L. Hill Program irlg edia Arts and Sciences May 2004 Certified by: / Dr. V. Michael Bove Jr. Principal Research Scientist Program in Media Arts and Sciences ZA Thesis Supervisor Accepted by: Andrew Lippman Chairperson Department Committee on Graduate Students MASSACHUSETTS INSTITUTE OF TECHNOLOGY Program in Media Arts and Sciences JUN 172 ROTCH LIBRARIES Scalable Multi-view Stereo Camera Array for Real World Real-Time Image Capture and Three-Dimensional Displays Samuel L. Hill Submitted to the Program in Media Arts and Sciences School of Architecture and Planning on May 7, 2004 in Partial Fulfillment of the Requirements for the Degree of Master of Science in Media Arts and Sciences Abstract The number of three-dimensional displays available is escalating and yet the capturing devices for multiple view content are focused on either single camera precision rigs that are limited to stationary objects or the use of synthetically created animations. In this work we will use the existence of inexpensive digital CMOS cameras to explore a multi- image capture paradigm and the gathering of real world real-time data of active and static scenes. The capturing system can be developed and employed for a wide range of applications such as portrait-based images for multi-view facial recognition systems, hypostereo surgical training systems, and stereo surveillance by unmanned aerial vehicles. The system will be adaptable to capturing the correct stereo views based on the environmental scene and the desired three-dimensional display. Several issues explored by the system will include image calibration, geometric correction, the possibility of object tracking, and transfer of the array technology into other image capturing systems. These features provide the user more freedom to interact with their specific 3-D content while allowing the computer to take on the difficult role of stereoscopic cinematographer. Thesis Supervisor: V. Michael Bove Jr. Title: Principal Research Scientist of Media Laboratory Scalable Multi-view Stereo Camera Array for Real World Real-Time Image Capture and Three-Dimensional Displays Samuel L. Hill Accepted by: D cMfhalHaeII MIT Visiting Scholar Brigham and Women's Hospital - Surgical Planning Laboratory Accepted by: 7 7 Dr. Wendy Plesniak MIT Research Affiliate Research Fellow - Harvard Center of Neurodegeneration and Repair Acknowledgments I dedicate this work to Professors Dr. Stephen Benton. I would also like to extend my gratefulness to my advisor Dr. Bove and my other two thesis readers Dr. Michael Halle and Dr. Wendy Plesniak, all of whom I have relied heavily on in the process of completing my degree. Thank you all. I would also like to acknowledge the support of my sister Jennifer Hill, my brother- in- law John Desisto, along with my parents. This work would not be possible without the tremendous support of Steven Smith, Jacky Mallet, and Tyeler Quentmeyer. Lastly I would like to thank the Spatial Imaging Group community for their warm support. TABLE OF CONTENTS CHAPTER 1. INTRODUCTION 9 1.1 Introduction .......................................................................... 9 1.2 M otivation ........................................................................ ... 9 1.2.1 Previous Work .......................................................... 9 1.2.2 Multi-camera Paradigm ................................................. 10 1.3 Target Applications ............................................................... 12 1.3.1 Aerial Vehicles .......................................................... 12 1.3.2 Facial Recognition ...................................................... 12 1.3.3 Surgical Applications ................................................... 13 1.4 C om m on Issues .................................................................... 14 1.5 Goals and Contributions ...................................................... 15 CHAPTER 2. THE STEREOSCOPIC VALUE CHAIN 16 2.1 Physiology of the Eye .......................................................... 16 2.1.1 Binocular Disparity ..................................................... 16 2.1.2 Accommodation and Convergence ................................... 17 2.2 Projective Transformation ...................................................... 17 2.3 Display Parameters ............................................................... 19 2 .3 .1 P arallax ..................................................................... 19 2.3.2 D ivergence ............................................................... 21 2.3.3 Magnification and Orthoscopy ........................................ 21 2.4 D epth R ange ...................................................................... 22 2.5 Keystoning ........................................................................ 24 2.5.1 Keystone Correction .................................................. 25 2.6 Lens D istortion ................................................................... 28 2.7 Stereoscopic Tools .................................................................. 29 CHAPTER 3. CAMERA CAPTURE FOR 3D 30 3.1 Stereoscopic Cameras .......................................................... 30 3.1.1 Single-lens, Single-sensor Designs .................................. 30 3.1.2 Single-lens, Multi-sensor Designs ....................................... 31 3.1.3 Multi-lens, Single-sensor Designs ................................... 32 3.1.4 Multi-lens, Multi-sensor Designs ....................................... 32 3.2 Image Based Modeling and Rendering ........................................... 34 3.3 Three-Dimensional Displays .................................................. 35 3.3.1 Stereoscopic Displays ................................................. 35 3.3.1.1 Color- multiplexed (anaglyph) Displays ................. 35 3.3.1.2 Polarization- multiplexed Displays ........................ 35 3.3.1.3 Time- multiplexed Displays ................................. 35 3.3.1.4 Time-sequentially Controlled Polarization .................. 35 3.3.1.5 Location- multiplexed Displays ............................. 36 3.3.2 Autostereoscopic Displays ............................................. 36 3.3.2.1 Electro- holography .......................................... 36 3.3.2.2 Volumetric Displays ....................................... 36 3.3.2.3 Direction-multiplexed Displays .......................... 37 3.3.2.3.1 Diffraction .................................... 37 3.3.2.3.2 Refraction ....................................... 37 3.3.2.3.3 Reflection and Occlusion ..................... 38 3.4 Display Applications ............................................................... 38 CHAPTER 4. SPATIAL IMAGING SCALABLE CAMERA ARRAY 39 4.1 Previous System ............................................................... 39 4.2 Design of SISCA ................................................................ 39 4.2.1 Experimental Parameters ................................. 39 4.2.1.1 Unmanned Aerial Vehicle .................................. 40 4.2.1.2 Studio/Facial Recognition ................................ 42 4.2.1.3 Surgical Applications ...................................... 43 4.2.1.4 Review of Parameters ................................... 44 4.2.2 Physical Apparatus ....................................................... 45 4.2.2.1 Camera ....................................................... 47 4.2.2.2 Display ....................................................... 48 4.2.2.3 Software Platform/User Interface ............................. 49 4.3 Testing Procedures ............................................................... 51 4.3.1 Camera Calibration ..................................................... 51 4.3.2 Camera Alignment ...................................................... 57 4.3.3 Recording ................................................... ........ 57 4.3.3.1 McAllister ..................................................... 57 4.3.3.2 Projective Transformation .................................. 58 4.3.3.3 General Method 1 ............................................. 58 4.3.3.4 General Method 2 .......................................... 58 4.3.3.5 OpenGL ....................................................... 58 4.4 Vertical Disparity Limits ........................................................ 58 4.5 Performance .................................................................. 60 4.6 Sum m ary .................................................. .......... ....... 63 CHAPTER 5. FUTURE APPLICATIONS 64 5.1 Program Environment ........................................................... 64 5.2 Object Detection and Tracking ............................................... 65 5.3 Stereo Image Compression .................................................... 66 5.4 Image Based Rendering ....................................................... 67 5.5 Integral Imaging .............................................................