ACHIEVING NEAR-CORRECT FOCUS CUES USING MULTIPLE IMAGE PLANES A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Kurt Akeley June 2004 c Copyright by Kurt Akeley 2004 All Rights Reserved ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Patrick Hanrahan (Principal Adviser) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Martin S. Banks (Visual Space Perception Laboratory University of California, Berkeley) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Mark A. Horowitz Approved for the University Committee on Graduate Studies. iii Abstract Typical stereo displays stimulate incorrect focus cues because the light comes from a single surface. The consequences of incorrect focus cues include discomfort, difficulty perceiving the depth of stereo images, and incorrect perception of distance. This thesis describes a prototype stereo display comprising two independent volumetric displays. Each volumetric display is implemented with three image planes, which are fixed in position relative to the viewer, and are arranged so that the retinal image is the sum of light from three different focal distances. Scene geometry is rendered separately to the two fixed-viewpoint volumetric displays, using projections that are exact for each eye. Rendering is filtered in the depth dimension such that object radiance is apportioned to the two nearest image planes, in linear proportion based on reciprocal distance. Fixed-viewpoint volumetric displays with adequate depth resolution are shown to generate near- correct stimulation of focus cues. Depth filtering, which is necessary to avoid visible artifacts, also improves the accuracy of the stimulus that directs changes in the focus of the eye. Specifically, the stimulus generated by depth filtering an object whose simulated distance falls between image-plane distances closely matches the stimulus that would be generated by an image plane at the desired distance. Viewers of the prototype display required substantially less time to perceive the depth of stereo images that were rendered with depth filtering to approximate correct focal distance. Fixed-viewpoint volumetric displays are shown to be a potentially practical solution for virtual reality viewing. In addition to near-correct stimulation of focus cues, and unlike more familiar autostereoscopic volumetric displays, fixed-viewpoint volumetric displays retain important qualities of 3-D projective graphics. These include correct depiction of occlusions and reflections, utilization of modern graphics processor and 2-D display technology, and implementation of realistic fields of iv view and depths of field. The design and verification of the prototype display are fully described. While not a practical solution for general-purpose viewing, the prototype is a proof of concept and a platform for ongoing vision research. v Acknowledgement I have been blessed with wonderful parents, teachers, mentors, and colleagues. My parents, David and Marcy Akeley, inspired my appreciation of education, allowed me to disassemble the family car and washing machine, and brought our family together for thousands of evening meals where we shared the excitements of our day. Peter Warter, chairman of the University of Delaware elec- trical engineering department when I was an undergraduate there during the late 70’s, trusted me with project work at the graduate-student level, took me into his family, and encouraged me to at- tend graduate school at Stanford. James Clark, a new faculty member in the Stanford electrical engineering department when I met him in 1980, created the professional opportunities at Silicon Graphics that made my career in computer graphics possible. When I left Silicon Graphics after 20 years, Mark Horowitz and Pat Hanrahan made it possible for me to return to Stanford to complete my degree; and Martin Banks provided me with my thesis topic and the laboratory infrastructure at Berkeley in which much of the work was done. Without the support of all of these people this thesis would not have been possible. Many colleagues contributed directly to my thesis work. Simon Watt taught me the basics of vision-science experimental methods, and ran many of the experiments that are described in this thesis. Ahna Reza Girshick picked up where Simon left off, designing and running experiments that will be reported in future publications. Sergei Gepshtein improved my MATLAB proficiency with many useful tips and techniques. Ian Buck shared his deep knowledge of modern graphics systems with me, and set up the environment that allowed this old-time Unix user to run make and vi on his PC. Mike Cammarano helped to define the external image interface to the prototype software, vi and provided the ray traced images that are included in this thesis. And NVIDIA, my part-time employer, gave me the schedule flexibility I needed to get my thesis work done. My immediate family supported me emotionally, financially, and parentally throughout this work. Jian Zhao and Manli Wei, my parents-in-law, created an extended family that gave my chil- dren four parents, allowing me to spend many long days in the labs at Stanford and Berkeley. My deepest appreciation and love go to my wife, Jenny Zhao, and to my children, David and Scarlett. They gave me the freedom to go back to school, and the love and support that allowed me to enjoy every moment of it. vii Contents Abstract iv Acknowledgement vi 1 Introduction 1 1.1RelatedWork..................................... 2 1.2MyContribution................................... 4 2 Fixed-viewpoint Volumetric Display 6 2.1 Principles and Optimizations ............................. 6 2.1.1 Non-homogeneous Voxel Distribution . ................... 8 2.1.2 CollapsingtheImageStack......................... 10 2.1.3 Solid-angle Filtering ............................. 13 2.1.4 ReducedDepthResolution.......................... 18 2.1.5 WideFieldofView.............................. 23 2.2Attributes....................................... 24 2.2.1 Tractable Voxel Count ............................ 24 2.2.2 Standard Components ............................ 25 2.2.3 Multiple Focal Distances ........................... 25 2.2.4 NoEyeTracking............................... 27 2.3Concerns....................................... 29 2.3.1 Aperture-related Lighting Errors ....................... 29 viii 2.3.2 Viewpoint Instability ............................. 30 2.3.3 Silhouette Artifacts .............................. 32 2.3.4 IncorrectRetinalFocusCues......................... 35 2.3.5 HeadMounting................................ 35 3 Prototype Display 37 3.1DesignDecisions................................... 38 3.2ImplementationDetails................................ 41 3.2.1 OverlappingFieldsofView......................... 41 3.2.2 Ergonomics . ................................ 43 3.2.3 Software................................... 44 3.2.4 Construction................................. 46 3.3Validation....................................... 47 3.3.1 IntensityConstancy.............................. 48 3.3.2 ImageAlignment............................... 49 3.3.3 Silhouette Visibility ............................. 52 3.3.4 High-quality Imagery ............................. 54 3.4Issues......................................... 56 4 User Performance 58 4.1TheFuseExperiment................................. 58 4.2FuseExperimentResults............................... 61 4.3ResultsRelatedtoAnalysis.............................. 63 5 Discussion and Future Work 69 A Apparent Focal Distance 73 B Blur Radius 78 C Incorrect Accommodation 81 ix D Summed Images 90 E Dilated Pupil 98 F Software Configuration 104 G Design Drawings 139 H Prototype Display Specifications 147 I Glossary 149 Bibliography 154 x List of Tables 2.1 Box depth-filter discontinuity as a function of depth resolution. ........... 21 2.2Relationshipbetweenvergenceangleandfixationdistance.............. 28 3.1Prototypeimageplanedistances............................ 40 D.1 ATF-sum maxima distances for various intensity ratios and spatial frequencies. 95 H.1Prototypespecifications................................ 147 H.2Prototypespatialresolution.............................. 148 xi List of Figures 2.1Viewdependentlightingeffects............................ 7 2.2Actualandideallinespreadfunctions......................... 11 2.3 Modulation transfer functions of the actual and ideal linespread functions. ..... 11 2.4Idealdisplay...................................... 12 2.5 Box filter shapes. ................................ 14 2.6 Voxel lighting with box and tent depth filters. ................... 16 2.7 Box and tent depth-filter shapes. ........................... 18 2.8 Modeling the visual discontinuity due to box depth filtering. ............ 19 2.9 Tent depth filtering eliminates visual discontinuities. ................ 20 2.10 Multiple focal distances along a visual line. ................... 27 2.11Alignmenterrorduetoopticalcentermovement................... 32 2.12 Intensity discontinuity at foreground/background transitions.