Vision-Realistic Rendering Using Hartmann-Shack Wavefront Aberrations

RAYS (Render As You See): Vision-Realistic Rendering Using Hartmann-Shack Wavefront Aberrations

Category: Research

Abstract However, even the most compelling realistic synthetic images, generated by techniques such as ray tracing and radiosity, do not RAYS (Render As You See) is a system for “vision-realistic render- model the optics of the camera nor of the human visual system. ing” which can simulate the vision of actual patients. Patient data The so-called “camera model” in computer graphics is in fact a is derived from sensors, called Hartmann-Shack devices, that cap- misnomer, meaning little more than the specification of the posi- ture the wavefront aberrations present in the patient’s entire optical tion and orientation of the perspective projection. The complexities pathway of a single point source of light in the retina. of an individual’s visual system are not taken into account. In fact, The input to RAYS is an image, corresponding depth informa- almost all images in computer graphics are generated on the basis tion, and a wavefront derived from the Hartmann-Shack device. of the oversimplified pinhole camera model. For example, an effect Given a focusing distance, our vision-realistic rendering algorithm such as the blur of the background of a scene is usually handled in then blurs the scene appropriately. The result is an image that an ad hoc manner. closely approximates what the actual patient would have seen if One of the main goals of this research is the introduction of op- focused at that distance. tics as well as details of the human visual system and specific data Vision-realistic rendering is particularly interesting in the con- about a specific individual’s visual system. We introduce a new text of laser refractive eye surgeries such as PRK and LASIK. Cur- concept, that of vision-realistic rendering. The primary intention is rently, almost a million Americans per year are choosing to undergo to develop new rendering techniques for the computer generation such elective surgeries. RAYS could convey to doctors the vision of of synthetic images that incorporate accurate optics, especially for a patient before and after surgery. In addition, RAYS could provide the human visual system, from specific patient data. There are two accurate and revealing visualizations of predicted acuity and sim- distinct impacts of this research, one from the perspective of com- ulated vision to potential candidates for such surgeries to facilitate puter graphics and the other from the point of view of optometry educated decisions about the procedure. Still another application and ophthalmology. would be to show such candidates the possible visual anomalies Using the eye model allows us to generate images based on an in- that could arise from the surgery (such as glare at night). dividual’s actual visual system. In addition to the goal of producing CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional vision-realistic synthetic images in computer graphics, this has im- Graphics and Realism—Color, shading, shadowing, and texture; portant applications in optometry and ophthalmology. This enables I.3.3 [Computer Graphics]: Picture/Image Generation—Display al- the generation of images that demonstrate specific defects in what gorithms, Viewing algorithms; a patient sees instead of the currently commonplace thinking that is usually limited to simple blur. Images are generated using the op- Keywords: vision-realistic rendering, photo-realistic render- tics of various ophthalmic conditions such as cataracts, glaucoma, ing, Hartmann-Shack, wavefront aberrations, optics, ray tracing, keratoconus, macular degeneration, and diplopia. This would be image synthesis, human visual system, blur, optometry, ophthal- valuable in educating doctors and patients about the effects of these mology, PRK (photorefractive keratectomy), LASIK (laser in-situ visual defects. keratomileusis), cornea, crystalline lens, pupil, visual acuity, Point Spread Function (PSF), Prentice’s rule, high dynamic range (HDR) Vision-realistic rendering is particularly interesting in the con- maps, text of laser corneal refractive eye surgeries such as PRK (photorefractive keratectomy) and LASIK (laser in-situ keratomileusis). Currently, almost a million Americans per year are choosing to un- 1 Introduction dergo such elective surgeries. This technique could be used to convey to doctors the vision of a patient before and after surgery. In The field of computer graphics is concerned with techniques for the addition, this could provide accurate and revealing visualizations generation of realistic synthetic images using computers. One of of predicted acuity and simulated vision to potential candidates for the primary goals has been photo-realistic rendering, that is, the such surgeries to facilitate educated decisions about the procedure. computer creation of synthetic images that are visually indistin- Still another application would be to show such candidates the pos- guishable from photographs of real scenes. This quest for visual sible visual anomalies that could arise from the surgery (such as realism in computer graphics has been remarkably successful as glare at night). With the increasing popularity of these surgeries, it the field has developed and matured since the mid-1960s. is possible that the current procedure of patients signing a consent form which is relatively incomprehensible to a layperson could be supplemented by the required viewing of a computer-generated an- imation showing the possible visual problems. Our goal is to blur a crisp, rendered image (with corresponding depth information) based on the optical aberrations of a patient. We begin by extracting wavefront aberration information from a Hartmann-Shack device. The wavefront captures all the optical aberrations present in the patient’s entire visual pathway for a point source of light emanating from the retina of the eye. We place a virtual lens in front of the wavefront and focus at a particular distance. If the wavefront were initially perfect (i.e., an aberration-free plane- wave), then after passing through our lens all rays would converge Due to the multilayered structure of the lens, the index of refraction sclera decreases gradually from the inner core to the less dense cortex

[16]. The sclera and choroid are the two layers of tissue that provide

iris ciliary muscle protection for the eye. The sclera is the visible layer seen as the

¦ “white of the eye,” and the choroid is the inner layer that cannot

retina ¤ § be seen from an exterior view. The interior of the eyeball contains pupil

lens fovea the aqueous humor and vitreous humor. The aqueous humor is a

¨ cornea visual axis transparent ﬂuid layer between the cornea and lens. The vitreous optical axis humor is located between the lens and retina, and is a jelly-like substance that ﬁlls the eyeball. The iris is the colored front of the

¥ eye that is positioned behind the cornea to form an opening, which

¢ vitreous humor aqueous is also known as the pupil. £ The total retina is a circular disc of approximately 42 mm diam-

humor ¡ optic eter [32] [23] [39]. The retina is important because it contains the nerve photoreceptor cells. There are two types of retinal photoreceptors in vertebrates, called rods and cones. The fovea is the reddish area with no blood vessels composed primarily of cones. It is located at the center of the area known as the macula, near the optic disk. Figure 1: A side view of the human eye. Moving away from the fovea, the number of rods increases dramati- cally, but the number of cones decreases. The central retina is cone- dominated retina whereas the peripheral retina is rod-dominated. at the focusing distance. We then sample the wavefront slope at a fixed number of focusing distances (determined by just-noticable- difference blur that would result from a perfect eye). These de- 2.2 Brief Review of Optics fine the object-space blur filter that will be present at each depth. Finally, we “stratify” the image into these same focusing distance According to the wave theory of light, a point light source in air will depths, convolve each blur filter with each image depth layer and emit light waves (rays) of identical speed in all directions. This re- composite them together to form the final image. sults in a spherical diverging wavefront, much like a pebble dropped The input image can be anything: a photograph, a computer- vertically onto a pond creates circular ripples. The rays that consti- generated 2D image, or even a standard Snellen acuity eye chart, as tute the wavefront are oriented normally to it; the wavefront ap- long as there is accompanying depth information. This last stimulus proach is related to the ray tracing approach in this manner. As the is very revealing, since it shows what the patient would see during waves travel farther away from the source, the wavefront will lose an eye examination, and provides an accurate picture of his or her curvature [19]. A wavefront emitted from a distant point source vision. will be practically flat by the time it reaches the observer. Vision-realistic rendering has been implemented in RAYS The speed of light changes as it passes through different trans- (Render As You See), which can simulate the vision of actual pa- parent media. These speed changes cause the wavefronts to bend. A tients. We demonstrate our approach on two computer-generated lens is designed to bend wavefronts in a predictable manner, often scenes using both simulated data from an ideal eye and actual pa- so that it converts a diverging wavefront into a converging wave- tient data from two patients. These are the first images in computer front. A spherical converging wavefront will refocus the light en- graphics generated on the basis of the specific vision characteristics ergy to a point, called the focal point.

of a particular patient. The curvature of a wavefront is called vergence, and is equal to

, where is the index of refraction ( =1 in air) and is the distance from the wavefront to the focal point. Vergence is typi- 2 Background cally measured in units of inverse meters called diopters (D); by convention, a diverging wavefront is considered to have negative 2.1 Human Eye Anatomy and Optical Physiology vergence, whereas a converging wavefront is considered to have positive vergence. For example, if a point light source is 2 m away Humans and other vertebrates perceive images that are produced by and separated by air, the wavefront would have a vergence of -0.5 light rays that pass through the cornea and crystalline lens of each D. eye. The amount of light that can enter is regulated by the pupil, The refractive power of a lens is the reciprocal of its focal which is located between the cornea and lens. Ultimately, an im- length, also measured in diopters, and denotes the degree to which age forms on the retina where the photoreceptors are located. The it changes the curvature of wavefronts passing through it. The lo-

nerves attached to these receptors then transmit stimuli to the visual cation of an image formed by the lens in an ideal optical system is

cortex of the brain where further processing occurs for interpreta- given by where is the vergence of the waves leaving

tion. the lens, is the power of the lens, and is the vergence of light

The eye has three principal components as shown in Figure 1: entering the lens. The image will be in focus at a distance the cornea, lens, and retina. The cornea is a transparent membrane from the lens. Given the above example of a light source with -0.5 with a tear layer and it admits light into the interior of the eye. D vergence, a +2.5 D lens would focus an image at +2.0 D, or 0.50 The total thickness of a normal cornea is about 0.51 mm at the m away. center, increasing to about 0.70 mm in the periphery. The cornea is To use a single optical system to focus objects at various dis- the principal refractive element since it is where two-thirds of the tances, either the distance of the lens to the focusing surface or the refraction occurs. power of the lens must be adjusted. Adjusting the focus on a cam- The lens is a flattened sphere comprising crystalline fibers. The era moves the lens to keep light focused on the film. The human eye remaining one third of refraction takes place in the lens, focusing keeps light focused on the retina by taking the other approach; the light rays onto the retina. The lens shape is controlled by ciliary ciliary muscles actually change the curvature of the ocular lens in a muscles, surrounding the lens, which change the shape of the lens to process called accommodation. In both cases, objects at sufficiently vary its focal length so as to focus on objects at different distances. different locations from the point of focus will appear blurred. The

2 depth range for which the image is sufﬁciently focused is called Lenslet depth of ﬁeld.

array E Diffraction is the phenomenon of waves (such as light) bending Mirror around corners. This prevents the formation of a “perfect” image even if aberrations in an optical system are negligible [19]. An optical system is said to be diffraction-limited if aberrations are cor- rected and diffraction is the principal source of blurring. In systems with a variable aperture (e.g., a camera lens or the pupil of the eye), smaller apertures become more diffraction-limited, whereas larger F CCD apertures become more aberration-limited. sensor In bright light – when the pupil is small and the human eye ap- Laser proximates a diffraction-limited system – a person with 20/20 vision can, by deﬁnition, perceive a minimum subtended angle of source one minute, i.e., 1/60th of a degree [3]. In clinical terms, this means that this person can stand 20 feet away from a Snellen letter chart [3] and just discern a letter E that has a thickness and limb spacing Figure 2: A side view of a Hartmann-Shack device. A laser projects of approximately 1/16 of an inch. a spot on the back of the cornea. This spot serves as a point light Refractive error refers to the difference between the power of a source, originating a wavefront out of the eye. This wavefront given lens system and the optimal power needed for a given situa- passes through a lattice of small lenslets which focus the wavefront tion. For example, if a +5.0 D lens were used to focus an object that onto a CCD sensor. optimally required +5.1 D, the refractive error would be 0.1 D.

2.3 Zernike Polynomials Zernike polynomials are derived from the orthogonalization of the

Taylor series. The resulting polynomial basis corresponds to or- # thogonal wavefront aberrations. The coefﬁcients "! weighting each polynomial have easily derived relations with meaningful pa-

rameters in optics. The index $ refers to the aberration type, while

distinguishes between ind ividual aberrations within a harmonic.

%&$ $ For a given index $ , ranges from to in steps of two.

Speciﬁcally,

')( ( '+* * ',* *

! ! -

! is displacement, is horizontal tilt, is vertical

(

'/. '/. . '/. .

! ! -

tilt, ! is average power, is horizontal cylinder, is

'/0

2%43657%98:5;8<5=3 1

oblique cylinder, ! are four terms ( ) related

' >

1 ?%&@65;%"AB5=CB5DAB5=@ to coma, and ! are ﬁve terms ( ) related to Figure 3: Hartmann-Shack sensors measuring a perfect eye with no spherical aberration. aberrations. Image courtesy Larry Thibos [37].

2.4 Ray Tracing vs. Wavefronts 2.5 Hartmann-Shack device Traditionally, computer graphics professionals characterize light transport as rays traveling through space. We recall from physics The Hartmann-Shack Sensor [31] (HSS) is a device that [37] pro- that light is both a particle and a wave, thus an equivalent represen- vides a way to precisely measure the wavefront aberrations of an tation is to to think of light in terms of its dual, a series of wave- eye. It is believed that the HSS is the most effective device for the fronts. measurement of human eye aberration [25]. A low-power 1 mm Traditional ray tracing techniques do not provide depth of ﬁeld; laser beam is directed at the retina of the eye by means of a half- i.e., all parts of a rendered image are equally in focus, regardless silvered mirror as in Figure 2. The retinal image of that laser now of distance from the observer. This is due to the simplistic pinhole serves as a point source of light for a wavefront that passes through camera model used for rendering. Our vision-realistic rendering the eye’s internal optical structures, past the pupil, and eventually technique uses wavefronts which allows us to capture depth of ﬁeld out of the eye. quite accurately. The wavefront then goes through a Hartmann-Shack lenslet ar- As we previously stated in Section 2.2, a spherical converging ray (typically between 50 and 200 lenslets are common) to focus wavefront would refocus rays from a point source to a single point the wavefront onto a CCD image array, which records it. This pro- on the image plane. However, real lenses are subject to spherical cess is somewhat similar to the compound eye of an insect. Image aberrations that blur the produced image. Real lenses cannot out- processing techniques are employed to determine the center of each put perfectly spherical wavefronts since rays passing through the image blur centroid to sub-pixel resolution. The local slope of the nonparaxial (i.e., peripheral) region of a lens do not behave exactly wavefront is determined by the lateral offset of the focus from each as predicted by ideal lens properties; large camera apertures and lenslet. Phase information is then derived from the slope [21]. Fig- pupil sizes are especially subject to these aberrations since more ures 3 and 4 show the Hartmann-Shack output for eyes with and nonparaxial rays pass through the lens in these cases. This explains without aberrations. Figure 5 illustrates the actual output of a HSS the distortions found in photographs produced by extremely wide- for a sample refractive surgery patient. angle lenses. Additionally, imperfections in lens shape and material can be 2.6 Point Spread Function (PSF) other sources of wavefront aberrations. Coupled with spherical aberrations, they are the basis for symptoms of such ocular con- One traditional method of measuring the acuity of an optical sys- ditions as astigmatism, cataracts, and night glare. tem is called the point spread function (PSF), which is a two-

3 dimensional retinal energy histogram from a point source [2, 4, 18, 20, 29, 41, 42]. It provides a great deal of information about predicted acuity, as it is akin to the impulse response of a cir- cuit. In order to calculate the PSF for an eye, light rays are traced through a model of the internal structures and onto the retina [6, 7, 15, 17, 26, 27, 34, 40]. The PSF is then used as a 2D blur ﬁlter and convolved with images to provide a ﬁrst-order approximation of visualization acuity.

2.7 High Dynamic Range (HDR) Most rendering algorithms default to 8-bits-per-channel output, which are called low dynamic range (LDR) images. Conversely, high dynamic range (HDR) images have increased channel depth, and capture much more realistically the true range of brightness of a given scene, real or simulated. It is critically important to use them when synthesizing blur [10], and our algorithm does exactly that. Mapping the resulting blurred HDR image back to LDR for printing can be accomplished to maintain realistic tone reproduction [30].

Figure 4: Hartmann-Shack sensors measuring a normal eye with some aberrations. Image courtesy Larry Thibos [37]. 3 Previous and Related Work

A realistic camera model is an important aspect of rendering photo- realistic images. However, with only a handful of exceptions, this has received almost no attention in the computer graphics litera- ture. The first synthetic images with depth of field were computed using the aperture camera model presented at SIGGRAPH’81 by Potmesil and Chakravarty [33]. Several years later, images with motion blur as well as depth of field were demonstrated using stochastic sampling techniques, as developed in [8][9][11][24]. More recently, Kolb et al. [22] described a realistic camera model. They model components such as the aperture stop and film irradi- ance; such features are common among cameras in photography, but they have been absent from computer graphics. This model renders images that contains effects that are characteristic of images produced by real cameras, such as depth of field as well as chromatic and spherical aberration. Similar to the situation with the camera models, relatively little attention has been paid to the human visual system in the generation of synthetic images. One of the earliest contributions was Upstill’s 1985 Ph.D. dissertation [38], which considered the problem of viewing synthetic images on a CRT and derived techniques for post-processing synthetic images for improved display. Spencer et al. [36] investigated image-based techniques of adding simple ocular and camera effects such as glare, bloom, and lenticular halo of the human eye for digital images. In [5], Bolin and Meyer use a perceptually-based sampling algorithm to sample images as they are being rendered (using wavelets) to determine if there are artifacts that would indicate the need for a change in rendering technique. Pattanaik et al. [30] developed a model of adaptation and spatial vision (based on the human visual system) to map a HDR scene to the LDR scene required for realistic tone reproduction on a CRT or printer. Whereas the majority of similar vision-simulating computer graphics research has centered on the human eye and on models Figure 5: Hartmann-Shack output for a sample eye. The green over- of perceptual effects, we are taking a different route. We are ex- lay lattice is registered to correspond to each lenslet in the array. ploiting the benefits of the Hartmann-Shack device which provides The centroids of the focus are indicated with a red dot. Image and an accurate model of what extremely complex aberrations were in- centroid locations courtesy David Williams. troduced by the optics of the patient’s eye. We no longer need to attempt to model the first-, second- and third- order effects, as those are contained in the Hartmann-Shack data. With the advent of instruments to measure corneal topography and compute accurate corneal reconstruction, several vision science researchers have produced images simulating what a patient would see. Principally, they used ray tracing techniques to generate retinal

4 light distribution, and used them to modify 2D test images. Camp information we are capturing the full optical aberrationsof the eye et al. [6, 7] created a ray tracing algorithm and computer model with no need for assumptions regarding unknown properties of the for evaluation of optical performance. Maguire et al. [26, 27] em- crystalline lens. ployed these techniques to analyze post-surgical corneas using their optical bench software. Greivenkamp [15] created a sophisticated model which included the Stiles-Crawford effect [28], diffraction, 5 Algorithm and contrast sensitivity. A shortcoming of all of these approaches is that they do not take into account the contribution of “hidden” 5.1 Overview optical elements, such as the eye’s lens. Given a 2D color image and a depth plane that corresponds to it, we propose to create a 2D image that simulates human vision. We note 4 Motivation that as we focus on an object at a certain distance, other objects in the field of vision located closer or farther away appear out of 4.1 Early Work focus. This is the essence of what we are attempting to simulate. The amount of blurring is a function of the distance to the object For almost a decade, they have been investigating accurate mea- position and of the distance to the point at which the eye is focused. surement, representation, modeling, reconstruction, and visualiza- Figure 6 graphically illustrates our algorithm. tion of the shape of the human cornea, as well as visualization and prediction of corneal visual acuity. During that time, their work 5.2 Hartmann-Shack has involved developing solutions to problems in optometry and ophthalmology that are in the spirit of geometric modeling and The output from the Hartmann-Shack sensors is an image of bright computer graphics algorithms. Now, their new concept of vision- points where each lenslet has focused the wavefront. Image- realistic rendering represents the inverse endeavor; that is, the in- processing algorithms are applied to derive the position of these corporation of optics and actual patient eye data for the accurate centroids and deviation from where they were intended to land. rendering of computer graphics imagery. This information encodes the gradient, or slope of the wavefront. Recently, they developed the CWhatUC (pronounced “see what The limited number of lenslets provides only a scarce sampling of you see”) system, which uses the reconstructed corneal shape to the overall wavefront; we then fit a Zernike-polynomial surface to simulate a patient’s visual acuity. They used traditional techniques these samples. This provides a continuous surface, allowing us to as discussed in Section 2.6 to blur 2D images exactly to close ap- sample the wavefront at a much higher rate. proximation of how a patient would have seen them [12, 13, 14]. They then developed an ocular model of the human eye for vision-realistic rendering [1]. They used a ray tracing approach 5.3 Object-Space Point Spread Function (OSPSF) similar to Kolb’s camera to trace rays from an image plane through While a PSF is perfectly suited for capturing the convergence of a lens system. However, their lens system is not a camera lens, but rays onto a retinal or film plane in image-space, we have a different a corneal surface extracted from actual patient data. Rendered im- situation. We need to encode to what extent a wavefront that began ages contain visual artifacts, such as abnormal double images, that deep within the eye is converging once it has left the eye and enters is consistent with what is seen, for example, by patients suffering object-space. We introduce the object-space point spread function from eye conditions such as keratoconus. (OSPSF), an analog of the traditional PSF used for wavefronts. We sample the wavefront very finely (typically sending over a

4.2 Limitations of Existing Approaches million queries) and extract slope (gradient) information. Our OS- GON;PM5=NRQ:L PSF, rather than encoding GIH5KJML locations, now encodes There are three major drawbacks to the approach in [1]: 1) Com- slope information. In the PSF domain, a “perfect” set of focused putation time Ray tracing is a time-intensive technique dependent rays would all converge to a single point, yielding a delta function. on the complexity of the scene being rendered. Additionally, a In the OSPSF domain, a “perfect” converging wavefront, after pass- complex refractive surface is placed between the image plane and ing through a tuned diverging lens, would become a plane wave. In the outside geometry, adding more complexity to the ray trace. 2) this case all slopes would be zero, yielding an analogous delta func- Small Light Sources The luminance of any point on the image tion. plane is computed from a weighted cosine function of the rays em- We use the OSPSF as a blur ﬁlter much as vision scientists use anating from that point. Therefore any small light sources or sharp the traditional PSF. The key difference is they are convolving the transitions in color may be blurred or missed. 3) Artifacts from image-space PSF with a rendered image of the world. We are con- the model Because the ocular model is inherently imperfect, with volving our object-space PSF with the true object, since we have approximations such as the spherical approximations to the cornea depth information for each pixel. Thus, we need to create an array and single indices of refraction for the lens, we introduce artifacts of OSPSFs at different depth planes; the choice of depth planes is which are due to our approximation of the eye. covered in Section 5.5. The algorithm of the present paper overcomes these problems. A more minor difference is they have a single spatial parame- 1) Computation time The computation time is very fast because ter, namely where to place the PSF plane. We also have a single only 41 OSPSFs are calculated. For a 1024x1024 image, the ap- parameter, which is the virtual lens we place in front of the wave- proach in [1] would effectively need to calculate more than a mil- front to compensate for the requested focusing distance. For a small lion OSPSFs. 2) Small Light Sources The undersampling problem aperture single-lens camera there is no difference between the dis- whereby thin lines could be missed, is avoided by using many more tribution on a PSF and our new OSPSF. However, they differ with a rays to generate the OSPSF than was possible in [1]. 3) Artifacts multi-lens system with a larger aperture as is the case with a human from the model Probably the main item that distinguishes our al- eye. gorithm from that in [2] and from that of the previous research dis- The process of creating the OSPSFs begins by placing a focusing cussed in Section 3 is that we base the optical properties of the eye lens in front of the wavefront emerging from within the eye. This on the full optics, not just on the information from corneal topogra- is one of the parameters we can tune in our algorithm: we control phy supplemented by guesses about the optical elements within the where the simulated person was focusing. We then record the slope eye. By calculating the OSPSF from the Hartmann-Shack sensor information into gradient histograms at different depth planes. Each

5 depth plane has a corresponding tuned diverging lens which would take a wavefront which exactly converged at its distance and convert it to a plane wave, hence a perfect OSPSF. Any aberrations present manifest themselves as a larger blur at that depth plane. As we move to planes farther away from the focusing depth, the narrow peaks indicating good focus and little blur give way to wider peaks, more blur and glare effects.

] 3D ^ Geometry 5.4 Stratiﬁcation The depth plane that we previously introduced has a distance value

Eye measured for each GIcK5edfL pixel in the original image. These depth values are used to separate pixels of the image into “depth strata.” Since the

depths are continuous, they must be quantized into discrete values. T S Hartmann-Shack Renderman The closest strata corresponds to a distance of 0.1 meter and in- crease by 0.25 D up to a distance of inﬁnity. The result is a division Wavefront of the image into 41 strata. [ centroids

U 5.5 Depth Planes Z

Surface fitting X Image Depth

W Y and sampling Y (RGB) (z) The number and spacing of depth planes needs to be chosen in order to provide the illusion of smooth transition. We can take advantage High-density of the limits of human visual acuity. We previously stated in Sec- tion 2.2 that a person with 20/20 vision can perceive a subtended

Wavefront -ml hB8jik8;C

angle of one minute, or ABg radians. To approximate the \ U acceptable refractive error for these speciﬁcations, we can use the

Generate OSPSFs Stratify n

following relation: noqpsrut where is the subtended angle in rut

radians, p is the pupil diameter in meters, and is refractive er-

-wl

hB8vik87C n

ror in diopters [35]. We use ABg radians for , a typical

p rut _ daylight pupil diameter of 2.4 mm [19] for , and solve for . OSPSF Image

This yields rxtyC6gz8;A D.

` `

depth depth Thus a person with 20/20 vision would not be able to perceive a

a a _

OSPSF layer n X Image layer n refractive error of less than approximately 0.12 D. (In darker envi- ` ` rons, the pupil diameter enlarges, further increasing this threshold.)

depth depth a a Due to diffraction and the non-ideal human visual system, we chose layer 1 layer 1 a precision of slightly more than twice this value, 0.25 D. This may seem too imprecise; however, consider a point that is

1.6 m away from a lens. To focus that point, the lens must have a V

power of { m or 0.625 D. This is the worst case scenario, exactly } Convolve OSPSF depth layer i with Image depth layer i 0.125 D aw{K| ay from both 0.5 D and 0.75 D. A slightly higher diopter value would be rounded to 0.75 D, while a slightly lower value would be rounded to 0.50 D. Therefore the maximum error is half the chosen precision. Blur It follows that with our depth planes spaced at 0.25 D incre-

` depth ments, the difference between adjacent depth planes is just above a b Blur layer n the threshold of perception. It is interesting to note that optometrists

` typically use 0.25 D increments in prescribing corrective lenses for depth

a similar reasons; the pairs of lenses presented to a patient during an layer 1 eye exam differ in power by this value [3]. Finally, we chose the range of depths to correspond to the range of sharp human vision, approximately 10 cm to infinity [3]. There- fore, we used 41 depth planes spaced from 0 D (infinitely far) to Acccumulate 10 D (0.10 m) in 0.25 D increments. This necessarily implies that the concentration of planes is heaviest closer to the observer, taper- ing off with increasing distance. In fact, the second farthest depth plane occurs at 4 m – much closer than infinity. This may seem like Final a surprising leap, but little additional power is necessary to focus Blurred an object only 4 m away as opposed to one on the horizon. This X Image is familiar to photographers who use the concept of depth of field and hyperfocal distance to keep both a relatively close object and a distant background in focus. Figure 6: A schematic flow diagram for our entire algorithm. 5.6 Convolution

Once we have a series of OSPSFs (blur ﬁlters) and Images for B~

each depth plane, we simply convolve OSPSF pw with Im-

B~ pMK age . This is a relatively straightforward operation in most 6.3 Another Sample Image: Road to Pt. RAYS numerical packages, such as MATLAB. The result is a series of blurred images at each depth. Figures C5-14 illustrate a high resolution image of a toy road rendered through a high-end CG system.1 The key aspect of this image is that all geometry is in focus, regardless of depth. The high frequency road marks on either side of the road are used to track the 5.7 Accumulation current fixation point and adjacent blur as a function of their distance from that point. In order to view the road scene at the proper scale one must posi- Once we have the blurred images for each depth layer, the only tion one’s eye a distance of 6.7 picture widths. At this distance each remaining step is to combine them. We use a straightforward sum- pixel subtends 0.5 min of arc. If one is at the more conventional mation, where each GIcK5edfL pixel in the final image is a sum of all the distance of one picture width then each pixel subtends 1/1024 of a corresponding GIcK5OdfL pixels in the depth images. We are left with radian (since the picture is 1024 pixels across) which corresponds a vision-realistic rendering which correlates with what the patient to 3.35 min of arc. would have seen had they focused at the chosen depth. The road scene has E posts at distances corresponding to different acuity levels. The most distant post is the 20/20 E. A person with 20/20 acuity standing at the 6.7 picture width distance will just be able to resolve the E when accommodating to it. Try it. For 6 Examples most people the smallest E will be slightly legible since most people have better than 20/20 acuity. The other E posts are for 20/40, 20/80, 20/160, 20/320, and 20/640 acuity. Figure C5 shows the 6.1 Image Calibration scene through a pinhole pupil with its infinite depth of field, so ev- erything is in focus. Figs. C6-8 shows the road as seen with an eye In rendering proper images to be convolved with the OSPSFs, the with zero aberrations, focused at infinity and with a pupil diameter images need to be calibrated in accordance with the resolution of of 3 mm. Figs. 6-8 are respectively for the cases where the observer the human eye. The spatial frequency cutoff of 60 cycles/degree [3] is accommodating on the 20/640 E, the next E (20/320), and at in- requires that each pixel represent 0.5 minutes of arc, equivalent to finity. Figs. C9-11 are three similar figures for the astigmatic eye 120 pixels/degree. and Figs. C12-14 are the three figures for the LASIK eye. Fig. 8 with the normal 3 mm pupil focusing at infinity is instruc- tive in that it illustrates the surprising result that the defocus blur on the object is independent of object distance. In fact the blur on the 6.2 A Sample Image: Cubes object has a blur diameter equal to the pupil diameter (an exercise for the reader). Thus the angular size of the blur is equal to the Figures C1-4 are for the same scene of four cubes as seen through pupil size divided by the object distance, which is equal to the pupil four visual systems. Figure C1 shows a group of white, red, green size times the diopters of defocus. For our scene each letter E is and blue cubes are at dIstances of 10, 20, 30, and 40 cm respec- 5.25 mm from top to bottom. Thus a 3 mm blur diameter severely tively from a pinhole eye. The pinhole allows all the cubes to be in degrades each sign when the person is accommodating to infinity. focus, independent of viewing distance. Fig. C2 is the same scene In Figures C6 and C7, the accommodation is on the 20/640 and except now the eye’s pupil has a 3 mm diameter. This eye has no 20/320 E. They are the fixated signs slightly blurred since the signs aberrations. The four panels correspond to the cases of focusing are not exactly placed on one of the 41 locations where the point on the white, red, green and blue cubes. In some cases some blur- spread functions are calculated. When viewed from the proper 6.7 ring is apparent even when the cube is in focus due to our sparse picture width distance the slight blur error should not be visible. sampling of the dioptric range. If the picture is viewed from the When the road scene is viewed through an astigmatic eye in Figs. proper distance of 6.7 picture diameters (see discussion in connec- 9-11, there is no way to have an E fully in focus. Fig. C11 is for the tion with the road scene) the blur is not perceptible. Fig. C3 shows case where the observer is focusing on the horizontal horizon. As the scene as viewed with an astigmatic eye. The eye is astigmatic discussed in connection with the colored cubes, when a horizontal with an extra +2 D of power in the vertical meridian. This is called line is in focus (using the power in a vertical meridian) then at a “with the rule” astigmatism since the most common shape for an distance closer by 2 D (for our with-the-rule 2 D astigmatism) the astigmatic cornea is to have more curvature in the vertical than in vertical lines will be in focus. This would correspond to a distance the horizontal direction. The lower left panel illustrates the effect of 50 cm. It turns out that 20/160 sign is at a distance of 45 cm, and of astigmatism. In this panel the eye’s vertical meridian is focused as expected the vertical lines of this sign are close to being in focus. on the green square, making the top and bottom green edges in fo- Figs. C12-14 show the road scene as viewed by a person who has cus. The green vertical edges are out of focus by 2 diopters. The undergone LASIK refractive surgery. This person has quite good vertical edges of the red square are close to being in focus because optics in the central 4-5 mm of his pupil and quite degraded optics the dioptric distance between the red and green squares (at 20 and outside that region. For daytime vision when the pupil is small, he 30 cm distances) is 1/0.2 - 1/0.3 = 5-3.33 = 1.67 D, which is close would be expected to have normal vision. However, under night to the 2 diopters of the eye. Thus the vertical edge of the red square driving conditions with a large pupil there will be problems. Fig. is close to being in focus at the same time as the horizontal edge of 12 shows the case for a 7 mm diameter pupil (not unreasonable the green square. The edges with greater amounts of dioptric dis- for night conditions) looking at the 20/640 sign. In comparing the tance have greater amounts of blur. An interesting nonlinearity of quality of this E with the E of Fig. C6 for a normal person with a the printing process is seen in the upper right panel where the eye 3 mm pupil one sees that the problem isn’t a problem of sharpness is focused on the horizontal edge of the white square. The vertical (again one should view it from 6.7 picture diameters). Rather it is edge of the red square will be 7 D out of focus (5 D due to distance a problem of contrast in that the 20/640 E in Fig. 12 looks sharp and 2 D due to astigmatism). The left edge of the white square is but washed out. That is because the central 4-5 mm of the pupil seen to redden. This effect is due to the nonlinear gamma of the printing process where the blurred white and red squares summate 1We used Pixar PhotoRealistic Renderman 3.9.1 to render and compute nonlinearly to enhance the redness of the overlap region. z values for all our geometry.

7 produces a sharp image. However, the outer rim of the pupil pro- the human visual system, such as the Stiles-Crawford effect of pupil duces a PSF with a broad tail that causes a substantial reduction in apodization, that should be included. The Stiles-Crawford effect the contrast of the image. Another interesting feature of the LASIK reduces the effectiveness of rays entering from peripheral cornea PSF is that there is a slight improvement in the depth of field as since these rays have a lower probability of being caught by the compared to the perfect eye. This can be seen by looking at the waveguide nature of cones. 20/320 E in Fig. 12. It is very blurred, but it is still visible and the E’s orientation can be discerned. Now look at the 20/320 E of Fig. 6 where the person with normal vision is looking at the 20/640 E. 8 Conclusion The 20/640 E is sharp and high contrast, but the 20/320 E is blurred to a greater extent than is the case of the LASIK eye. However, it We introduce a new concept, that of vision-realistic rendering. The is doubtful that the slight improvement of depth of field for a nar- primary intention is to develop new rendering techniques for the row range of stimuli compensates for the loss of contrast for the computer generation of synthetic images that incorporate accurate attended stimulus. optics, especially for the human visual system, from specific patient data. This approach enables the computer generation of images based 7 Future Work on an individual’s actual visual system. In addition to the goal of producing vision-realistic synthetic images in computer graphics, Off-axis aberrations (distortion, curvature of field, coma, off-axis this has important applications in optometry and ophthalmology. astigmatism) are not properly taken into account in our present al- This enables the generation of images that demonstrate specific de- gorithm since we are basing the OSPSF on the Hartmann-Shack fects in what a patient sees instead of the currently commonplace information from a single foveal point. A more realistic rendering thinking that is usually limited to simple blur. Images are generated of peripheral vision could be achieved by producing a multiplicity using the optics of various ophthalmic conditions such as cataracts, of Hartmann-Shack OSPSFs for many peripheral points. However, glaucoma, keratoconus, macular degeneration, and diplopia. This this would not necessarily be a good idea for two reasons. First, pe- would be valuable in educating doctors and patients about the ef- ripheral vision is severely degraded by neural factors in the periph- fects of these visual defects. ery, whereas in the fovea acuity is limited by optical factors such Vision-realistic rendering has been implemented in RAYS as diffraction and aberrations. Second, we typically view extended (Render As You See), which can simulate the vision of actual pa- scenes by moving our eyes, so that our foveal OSPSF is actually tients. Patient data is derived from sensors, called Hartmann-Shack appropriate for the full image. devices, that capture the wavefront aberrations present in the pa- One limitation is that occlusion is not presently handled. Imag- tient’s entire optical pathway of a single point source of light in ine a scene containing a rear-view mirror. If we choose to focus the retina. This is significant because before such information was on the plastic edges of the mirror, the image reflected by the mir- available, researchers had to construct rough approximating models ror (which contains distant objects) would become blurred, just as of the human eye and apply a modified camera model to simulate it should. However, the region of blur would extend slightly past vision. the edges of the mirror. This would not be the case in reality since The input to RAYS is an image such as a synthetic image, cor- the mirror would occlude the rays that cause the extraneous blur. responding depth information, and a wavefront derived from the This issue could be resolved by using occlusion information in post- Hartmann-Shack device. Given a focusing distance, our vision- processing. realistic rendering algorithm then blurs the scene appropriately. The Blur discontinuities may be apparent in certain scenes using our result is an image that closely approximates what the actual patient approach. For example, overhead power lines beginning near the would have seen had he or she accommodated (that is, focused) at observer and fading into the horizon might reveal the 41 discrete that distance. depth planes. The blur discontinuities should only be visible when Vision-realistic rendering is particularly interesting in the con- viewing the image from a closer distance than the intended viewing text of laser corneal refractive eye surgeries such as PRK and distance of 6.7 picture diameters. Since in fact, images are typically LASIK. Currently, almost a million Americans per year are choos- viewed from closer distances, a greater density of depth planes (i.e., ing to undergo such elective surgeries. RAYS could convey to doc- finer diopter increments) would be needed. This would make the tors the vision of a patient before and after surgery. In addition, blur difference between adjacent depth planes less distinguishable RAYS could provide accurate and revealing visualizations of pre- by the naked eye. dicted acuity and simulated vision to potential candidates for such The calculations we have presented are based on geometric (ray surgeries to facilitate educated decisions about the procedure. Still tracing) optics. For an eye with no aberrations geometric optics another application would be to show such candidates the possible produces a defocus OSPSF (and PSF) that is a uniform disc with visual anomalies that could arise from the surgery (such as glare at sharp edges. We could improve on this OSPSF by including phys- night). ical (wave) optics effects that serve to blur the OSPSF. Typically We demonstrate our algorithm on two computer-generated the physical optics effects become especially important for small scenes. The first is simple scene of four cubes, and the second pupils. For the visual degradation produced by refractive surgery is ”Road to Point RAYS”, depicting in small scale road. For each with large pupils the ray tracing assumptions of the present paper scene, we generate images using both simulated data from an ideal should be satisfactory. eye and actual patient data from two patients. These are the first In rendering our images using a color printer we did not com- images in computer graphics generated on the basis of the specific pensate for nonlinearities between print intensity and intended in- vision characteristics of a particular patient. tensity. This distortion produced some interesting color bleeding effects when a blurred image overlapped a white image. In the future we would use “gamma correction” to avoid this problem. 9 Acknowledgements In order to render the images so that they appear similar to the original scene, we must compensate for nonlinearities in the human We would like to thank Koichi Tsunoda, a student at our institution visual system that are relevant because the conditions for viewing for his contributions to the work described in this paper. the printed image can be different from original conditions, as for This work was supported in part by the National Science Foun- example occurs for night driving scenes. There are other effects of dation grant number ASC-9720252, “Visualization and Simula-

8 tion in Scientiﬁc Computing for the Cornea,” and grant number [14] Daniel D. Garcia, Brian A. Barsky, and Stanley A. Klein. The CDA-9726362, “Virtual Environments for Telesurgery and Surgi- OPTICAL project at UC Berkeley: Simulating visual acuity. cal, Training: Efﬁcient Computation, Visualization, and Interac- Medicine Meets Virtual Reality: 6 (Art, Science, Technology: tion.” Healthcare (r)Evolution), January 28–31 1998.

[15] J. E. Greivenkamp, J. Schweigerling, J. M. Miller, and M. D. References Mellinger. Visual acuity modeling using optical raytracing of schematic eyes. Am. J. Ophthalmol., 120:227–240, 1995. [1] Anonymous. Incorporating camera models, ocular models, and actual patient eye data for photo-realistic and vision- [16] Eugene Hecht. Optics, Second Edition. Addison-Wesley Pub- realistic rendering. Submitted, 2001, 2001. lishing Co., 1987.

[2] Pablo Artal, Javier Santamar´ia, and Julian Bescoos.´ Retrieval [17] Richard P. Hemenger, Alan Tomlinson, and Katherine Oliver. of wave aberration of human eyes from actual point-spread- Corneal optics from videokeratographs. Ophthal. Physiol. function data. J. Opt. Soc. Am. A, 5:1201–1206, 1988. Opt., 15(1):63–68, 1995. [18] J. K. Ijspeert, T. J. T. P. Van Den Berg, and H. Spekreijse. [3] Arthur G. Bennett and Ronald B. Rabbetts. Clinical Visual An improved mathematical description of the foveal visual Optics. Butterworth-Heinemann Ltd., Woburn, MA, 1988. point spread function with parameters for age, pupil size and [4] Frans J. J. Blommaert, Henny G. M. Heynen, and Jacques pigmentation. Vision Res., 33(1):15–20, 1993. A. J. Roufs. Point spread functions and detail detection. Spa- [19] Michael P. Keating. Geometric, Physical, and Visual Optics. tial Vision, 2(2):99 – 115, 1987. Butterworth-Heinemann Ltd., Woburn, MA, 1988.

[5] Mark R. Bolin and Gary W. Meyer. A perceptually based [20] Miles V. Klein. Optics. John Wiley & Sons, New York, NY, adaptive sampling algorithm. In Proceedings of ACM SIG- 1970. GRAPH 1998, pages 299–309, Los Angeles, July 19–24 1998. SIGGRAPH, ACM. [21] Stanley A. Klein. Optimal corneal ablation for eyes with arbitrary Hartmann-Shack aberrations. J. Opt. Soc. Am. A, [6] Jon J. Camp, Leo J. Maguire, Bruce M. Cameron, and 15(9):2580–2588, 1998. Richard A. Robb. A computer model for the evaluation of the effect of corneal topography on optical performance. Am. [22] Craig Kolb, Don Mitchell, and Pat Hanrahan. A realistic cam- J. Ophthalmol., 109:379–386, 1990. era model for computer graphics. In Proceedings of ACM SIGGRAPH 1995, pages 317–324, Los Angeles, August 6– [7] Jon. J. Camp, Leo J. Maguire, and Richard A. Robb. An ef- 11 1995. SIGGRAPH, ACM. ﬁcient ray tracing algorithm for modeling visual performance from corneal topography. In First Conference on Visualiza- [23] H. Kolb. The neural organization of the human retina. In tion in Biomedical Computing, pages 279–285, Atlanta, GA, J.R. Hecklenlively and G.B. Arden, editors, Principles and May 22–25 1990. The Institute of Electrical and Electronics Practices of Clinical Electrophysiology of Vision, pages 25– Engineers, Inc., IEEE Computer Society Press. 52. Mosby Year Book Inc., St. Louis, 1991.

[8] Robert L. Cook. Stochastic sampling in computer graphics. [24] Mark E. Lee, Richard A. Redner, and Samuel P. Uselton. Sta- ACM Transactions on Graphics, 5(1):51–72, January 1986. tistically optimized sampling for distributed ray tracing. In Proceedings of ACM SIGGRAPH 1985, pages 61–67, Los An- [9] Robert L. Cook, Thomas Porter, and Loren Carpenter. Dis- geles, July 22–26 1985. SIGGRAPH, ACM. tributed ray tracing. In Proceedings of ACM SIGGRAPH 1984, pages 137–145, Los Angeles, July 23–27 1984. SIG- [25] Junzhong Liang. A New Method to Precisely Measure the GRAPH, ACM. Wave Aberrations of the Human Eye with a Hartmann-Shack Wavefront Sensor. PhD thesis, Department of Mathemat- [10] Paul E. Debevec and Jitendra Malik. Recovering high dy- ics, Universitat¨ Heidelberg, Heidelberg, Germany, December namic range radiance maps from photographs. In Proceed- 1991. ings of ACM SIGGRAPH 1997, pages 369–378, Los Angeles, August 3–8 1997. SIGGRAPH, ACM. [26] Leo J. Maguire, Jon J. Camp, and Richard A. Robb. Inform- ing interested parties of changes in the optical performance of [11] Mark A. Z. Dippe and Erling H. Wold. Antialiasing through the cornea caused by keratorefractive surgery — a ray tracing stochastic sampling. In Proceedings of ACM SIGGRAPH model that tailors presentation of results to ﬁt the level of so- 1985, pages 69–78, Los Angeles, July 22-26 1985. SIG- phistication of the audience. In SPIE Vol. 1808 Visualization GRAPH, ACM. in Biomedical Computing 1992, pages 601 – 609, Chapel Hill, NC, October 13 – 16 1992. SPIE — The International Society [12] Daniel D. Garcia. CWhatUC : Software Tools for Predict- for Optical Engineering, SPIE. ing, Visualizing and Simulating Corneal Visual Acuity. PhD thesis, Computer Science Division, University of California, [27] Leo J. Maguire, Ralph W. Zabel, Paula Parker, and Richard L. Berkeley, CA, May 2000. Lindstrom. Topography and raytracing analysis of patients with excellent visual acuity 3 months after excimer laser pho- [13] Daniel D. Garcia, Brian A. Barsky, and Stanley A. Klein. torefractive keratectomy for myopia. Refract. Corneal Surg., CWhatUC: A visual acuity simulator. In Proceedings of Oph- 8:122–128, March/April 1991. thalmic Technologies VIII, SPIE International Symposium on Biomedical Optics, pages 290–298, San Jose, CA, January [28] Parry Moon and Domina Eberle Spencer. On the Stiles- 24–30 1998. SPIE. Crawford effect. J. Opt. Soc. Am., 34(6):319–329, 1944.

9 [29] Donald C. O’Shea. Elements of Modern Optical Design. John Wiley & Sons, New York, NY, 1985. [30] Sumanta N. Pattanaik, James A. Ferwerda, Mark D. Fairchild, and Donald P. Greenberg. A multiscale model of adaptation and spatial vision for realistic image display. In Proceed- ings of ACM SIGGRAPH 1998, pages 287–298, Los Angeles, 1998. SIGGRAPH, ACM.

[31] B. Platt and R. V. Shack. Lenticular Hartmann-screen. Newsletter 5, 15, Optical Science Center, University of Ari- zona, 1971. [32] S.L. Polyak. The Retina. University of Chicago Press, Chicago, 1941.

[33] Michael Potmesil and Indranil Chakravarty. A lens and camera model for synthetic image generation. Computer Graph- ics, 15(5):297–305, August 1981. [34] T. Seiler, W. Reckmann, and R. K. Maloney. Effective spherical aberration of the cornea as a quantitative descrip- tor in corneal topography. Journal of Cataract and Refractive Surgery, 19(Supplement):155–165, 1993.

[35] George Smith and David A. Atchison. The Eye and Visual Op- tical Instruments. Cambridge University Press, Cambridge, UK, 1997. [36] Greg Spencer, Peter Shirley, Kurt Zimmerman, and Donald P. Greenberg. Physically-based glare effects for digital images. In Proceedings of ACM SIGGRAPH 1995, pages 325–334, Los Angeles, August 6–11 1995. SIGGRAPH, ACM.

[37] Larry N. Thibos. Principles of hartmann-shack aberrometry. In V. Lakshminarayanan, editor, Vision Science and its Appli- cations, volume 15, pages 163–169. OSA-TOPS, Washing- ton, DC, 2000. [38] Steven D. Upstill. The Realistic Presentation of Synthetic Im- ages: Image Processing in Computer Graphics. PhD thesis, Computer Science Division, University of California, Berke- ley, CA, August 1985.

[39] J.M. van Buren. The retinal ganglion cell layer. Charles C. Thomas, Springﬁeld, IL, 1963. [40] Corina van de Pol, Henry H. Tran, Daniel D. Garcia, and Stan- ley A. Klein. Three-dimensional analysis of corneal image forming properties: A monocular diplopia example. In Pre- sented at Vision Science and Its Applications Meeting, Santa Fe, NM, Februrary 6–9 1998.

[41] W. T. Welford. Aberrations of Optical Systems. Adam Hilger, Ltd., Bristol, England, 1986. [42] Gerald Westheimer and Fergus W. Campbell. Light distribution in the image formed by the living human eye. J. Opt. Soc. Am., 52(9):1040–1045, 1962.