RAYS (Render As You See): Vision-Realistic Rendering Using Hartmann-Shack Wavefront Aberrations Category: Research Abstract However, even the most compelling realistic synthetic images, generated by techniques such as ray tracing and radiosity, do not RAYS (Render As You See) is a system for “vision-realistic render- model the optics of the camera nor of the human visual system. ing” which can simulate the vision of actual patients. Patient data The so-called “camera model” in computer graphics is in fact a is derived from sensors, called Hartmann-Shack devices, that cap- misnomer, meaning little more than the specification of the posi- ture the wavefront aberrations present in the patient’s entire optical tion and orientation of the perspective projection. The complexities pathway of a single point source of light in the retina. of an individual’s visual system are not taken into account. In fact, The input to RAYS is an image, corresponding depth informa- almost all images in computer graphics are generated on the basis tion, and a wavefront derived from the Hartmann-Shack device. of the oversimplified pinhole camera model. For example, an effect Given a focusing distance, our vision-realistic rendering algorithm such as the blur of the background of a scene is usually handled in then blurs the scene appropriately. The result is an image that an ad hoc manner. closely approximates what the actual patient would have seen if One of the main goals of this research is the introduction of op- focused at that distance. tics as well as details of the human visual system and specific data Vision-realistic rendering is particularly interesting in the con- about a specific individual’s visual system. We introduce a new text of laser refractive eye surgeries such as PRK and LASIK. Cur- concept, that of vision-realistic rendering. The primary intention is rently, almost a million Americans per year are choosing to undergo to develop new rendering techniques for the computer generation such elective surgeries. RAYS could convey to doctors the vision of of synthetic images that incorporate accurate optics, especially for a patient before and after surgery. In addition, RAYS could provide the human visual system, from specific patient data. There are two accurate and revealing visualizations of predicted acuity and sim- distinct impacts of this research, one from the perspective of com- ulated vision to potential candidates for such surgeries to facilitate puter graphics and the other from the point of view of optometry educated decisions about the procedure. Still another application and ophthalmology. would be to show such candidates the possible visual anomalies Using the eye model allows us to generate images based on an in- that could arise from the surgery (such as glare at night). dividual’s actual visual system. In addition to the goal of producing CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional vision-realistic synthetic images in computer graphics, this has im- Graphics and Realism—Color, shading, shadowing, and texture; portant applications in optometry and ophthalmology. This enables I.3.3 [Computer Graphics]: Picture/Image Generation—Display al- the generation of images that demonstrate specific defects in what gorithms, Viewing algorithms; a patient sees instead of the currently commonplace thinking that is usually limited to simple blur. Images are generated using the op- Keywords: vision-realistic rendering, photo-realistic render- tics of various ophthalmic conditions such as cataracts, glaucoma, ing, Hartmann-Shack, wavefront aberrations, optics, ray tracing, keratoconus, macular degeneration, and diplopia. This would be image synthesis, human visual system, blur, optometry, ophthal- valuable in educating doctors and patients about the effects of these mology, PRK (photorefractive keratectomy), LASIK (laser in-situ visual defects. keratomileusis), cornea, crystalline lens, pupil, visual acuity, Point Spread Function (PSF), Prentice’s rule, high dynamic range (HDR) Vision-realistic rendering is particularly interesting in the con- maps, text of laser corneal refractive eye surgeries such as PRK (pho- torefractive keratectomy) and LASIK (laser in-situ keratomileusis). Currently, almost a million Americans per year are choosing to un- 1 Introduction dergo such elective surgeries. This technique could be used to con- vey to doctors the vision of a patient before and after surgery. In The field of computer graphics is concerned with techniques for the addition, this could provide accurate and revealing visualizations generation of realistic synthetic images using computers. One of of predicted acuity and simulated vision to potential candidates for the primary goals has been photo-realistic rendering, that is, the such surgeries to facilitate educated decisions about the procedure. computer creation of synthetic images that are visually indistin- Still another application would be to show such candidates the pos- guishable from photographs of real scenes. This quest for visual sible visual anomalies that could arise from the surgery (such as realism in computer graphics has been remarkably successful as glare at night). With the increasing popularity of these surgeries, it the field has developed and matured since the mid-1960s. is possible that the current procedure of patients signing a consent form which is relatively incomprehensible to a layperson could be supplemented by the required viewing of a computer-generated an- imation showing the possible visual problems. Our goal is to blur a crisp, rendered image (with correspond- ing depth information) based on the optical aberrations of a patient. We begin by extracting wavefront aberration information from a Hartmann-Shack device. The wavefront captures all the optical aberrations present in the patient’s entire visual pathway for a point source of light emanating from the retina of the eye. We place a vir- tual lens in front of the wavefront and focus at a particular distance. If the wavefront were initially perfect (i.e., an aberration-free plane- wave), then after passing through our lens all rays would converge Due to the multilayered structure of the lens, the index of refraction sclera decreases gradually from the inner core to the less dense cortex [16]. The sclera and choroid are the two layers of tissue that provide iris ciliary muscle protection for the eye. The sclera is the visible layer seen as the ¦ “white of the eye,” and the choroid is the inner layer that cannot retina ¤ § be seen from an exterior view. The interior of the eyeball contains pupil lens fovea the aqueous humor and vitreous humor. The aqueous humor is a © © ¨ cornea visual axis transparent fluid layer between the cornea and lens. The vitreous optical axis humor is located between the lens and retina, and is a jelly-like substance that fills the eyeball. The iris is the colored front of the ¥ eye that is positioned behind the cornea to form an opening, which ¢ vitreous humor aqueous is also known as the pupil. £ The total retina is a circular disc of approximately 42 mm diam- humor ¡ optic eter [32] [23] [39]. The retina is important because it contains the nerve photoreceptor cells. There are two types of retinal photoreceptors in vertebrates, called rods and cones. The fovea is the reddish area with no blood vessels composed primarily of cones. It is located at the center of the area known as the macula, near the optic disk. Figure 1: A side view of the human eye. Moving away from the fovea, the number of rods increases dramati- cally, but the number of cones decreases. The central retina is cone- dominated retina whereas the peripheral retina is rod-dominated. at the focusing distance. We then sample the wavefront slope at a fixed number of focusing distances (determined by just-noticable- difference blur that would result from a perfect eye). These de- 2.2 Brief Review of Optics fine the object-space blur filter that will be present at each depth. Finally, we “stratify” the image into these same focusing distance According to the wave theory of light, a point light source in air will depths, convolve each blur filter with each image depth layer and emit light waves (rays) of identical speed in all directions. This re- composite them together to form the final image. sults in a spherical diverging wavefront, much like a pebble dropped The input image can be anything: a photograph, a computer- vertically onto a pond creates circular ripples. The rays that consti- generated 2D image, or even a standard Snellen acuity eye chart, as tute the wavefront are oriented normally to it; the wavefront ap- long as there is accompanying depth information. This last stimulus proach is related to the ray tracing approach in this manner. As the is very revealing, since it shows what the patient would see during waves travel farther away from the source, the wavefront will lose an eye examination, and provides an accurate picture of his or her curvature [19]. A wavefront emitted from a distant point source vision. will be practically flat by the time it reaches the observer. Vision-realistic rendering has been implemented in RAYS The speed of light changes as it passes through different trans- (Render As You See), which can simulate the vision of actual pa- parent media. These speed changes cause the wavefronts to bend. A tients. We demonstrate our approach on two computer-generated lens is designed to bend wavefronts in a predictable manner, often scenes using both simulated data from an ideal eye and actual pa- so that it converts a diverging wavefront into a converging wave- tient data from two patients. These are the first images in computer front. A spherical converging wavefront will refocus the light en- graphics generated on the basis of the specific vision characteristics ergy to a point, called the focal point. of a particular patient. The curvature of a wavefront is called vergence, and is equal to , where is the index of refraction ( =1 in air) and is the distance from the wavefront to the focal point.