Journal of Microscopy, Vol. 235, Pt 2 2009, pp. 144–162 Received 7 November 2008; accepted 7 April 2009 Recording and controlling the 4D light field in a microscope using microlens arrays M.LEVOY ∗,Z.ZHANG ∗ &I.MCDOWALL † ∗Computer Science Department, Stanford University, Stanford, California, U.S.A. Fakespace Labs, 241 Polaris Ave., Mountain View, California, U.S.A. † Key words. Illumination, light field, light field microscope, microlens array, plenoptic function, synthetic aperture imaging, spatial light modulator, spherical aberration, Shack–Hartmann sensor. Summary same way. We call this a light field microscope (LFM) (Levoy, 2006). From the image captured by this device, one can By inserting a microlens array at the intermediate image plane employ light field rendering (Levoy & Hanrahan, 1996) to of an optical microscope, one can record four-dimensional generate oblique orthographic views or perspective views, at light fields of biological specimens in a single snapshot. Unlike least up to the angular limit of rays that have been captured. a conventional photograph, light fields permit manipulation Since microscopes normally record orthographic imagery, of viewpoint and focus after the snapshot has been taken, perspective views represent a new way to look at specimens. subject to the resolution of the camera and the diffraction Figure 1 shows three such views computed from a light field limit of the optical system. By inserting a second microlens of mouse intestine villi. array and video projector into the microscope’s illumination Startingfromacapturedlightfield,onecanalternativelyuse path, one can control the incident light field falling on the synthetic aperture photography (Isaksen et al., 2000; Levoy specimen in a similar way. In this paper, we describe a et al., 2004) to produce views focused at different depths. prototype system we have built that implements these ideas, Two such views, computed from a light field of Golgi-stained and we demonstrate two applications for it: simulating exotic rat brain, are shown in Fig. 10. The ability to create focal microscope illumination modalities and correcting for optical stacks from a single input image allows moving or light- aberrations digitally. sensitive specimens to be recorded. Finally, by applying 3D deconvolution to these focal stacks (Agard, 1984), one can Introduction reconstruct a stack of cross-sections, which can be visualized using volume rendering (Levoy, 1988). The light field is a four dimensional (4D) function representing Summarizing, the LFM allows us to capture the 3D structure radiance along rays as a function of position and direction of microscopic objects in a single snapshot (and therefore at in space. Over the past 10 years our group has built several a single instant in time). The sacrifice we make to obtain devices for capturing light fields (Levoy, 2000; Levoy, 2005; this capability is a reduction in image size. Specifically, if Ng, 2005b; Wilburn et al., 2002). In particular, Ng et al. is a each microlens subimage contains N N pixels, then our handheldcamerainwhichamicrolensarrayhasbeeninserted × computed images will contain N2 fewer pixels than if the between the sensor and main lens. A photograph taken by microlenses were not present. In return, we can compute N2 this camera contains a grid of circular subimages, one per unique oblique views of the specimen, and we can generate a microlens. Each subimage records one point in the scene, and focal stack containing N slices with non-overlapping depths of within a subimage each pixel records one direction of view field (Levoy, 2006). Note that this trade-off cannot be avoided of that point. Thus, each pixel in the photograph records the merely by employing a sensor with more pixels, because radiance along one ray in the light field. diffraction places an upper limit on the product of spatial and Recently,wehaveshownthatbyinsertingamicrolensarray angular bandwidth for a given aperture size and wavelength, at the intermediate image plane of an optical microscope, regardless of sensor resolution. Despite this limit, light fields we can capture light fields of biological specimens in the contain much useful information that is lost when an object is photographed with an ordinary microscope. Correspondence to: Marc Levoy. Tel: 1 650 725 4089; fax: 1 650 723 0033; While technologies for recording light fields have existed + + e-mail: [email protected] for more than a century, technologies for generating light C # 2009 The Authors C Journal compilation # 2009 The Royal Microscopical Society RECORDING AND CONTROLLING THE 4D LIGHT FIELD 145 Fig. 1. Three oblique orthographic views from the indicated directions, computed from a light field captured by our LFM. The specimen is a fixed whole mount showing L. monocytogenes bacteria in a mouse intestine villus 7 h post infection. The bacteria are expressing green fluorescent protein, and the actin filaments in the brush border of the cells covering the surface of the villus are labelled with rhodamine-phalloidin. Scale bar is 10 µm. Imaging employed a 60 / 1.4NA oil objective and an f/30 microlens array (see section ‘Prototype light field illuminator and microscope’). Contrast of the bacteria × was enhanced by spatially masking the illumination as described in Fig. 7. The inset at left shows a crop from the light field, corresponding to the square in the first oblique view. In this inset, we see the circular subimages formed by each microlens. The oblique views were computed by extracting one pixel from each subimage – a pixel near the bottom of each subimage to produce the leftmost view and a pixel near the top to produce the rightmost view. fields have been limited until recently by the low resolution as a ‘guide star’ and the LFM as a Shack–Hartmann sensor. and high cost of spatial light modulators (SLMs). With the Finally, we can use this information to correct digitally for advent of inexpensive, high resolution liquid crystal displays the optical aberrations induced by these changes in index of and digital micromirror devices, interest in this area has refraction. burgeoned. In microscopy, SLMs have been used to control the distribution of light in space (Hanley et al., 1999; Smith et al., 2000; Chamgoulov et al., 2004) or in angle (Samson The four-dimensional light field & Blanca, 2007). The former is implemented by making the SLM conjugate to the field of the microscope, and the latter We begin by briefly reviewing the theory of light fields. by making it conjugate to the aperture. Systems have also In geometrical optics, the fundamental carrier of light is been proposed for manipulating the wave front using an SLM a ray (Fig. 2(a)). The radiance travelling along all such placed in the aperture plane (Neil et al., 2000). However, in rays in a region of 3D space illuminated by an unchanging these systems the illumination must be coherent, a significant arrangement of lights has been dubbed the plenoptic function limitation. (Adelson & Wang, 1992). Since rays in space can be As an extension of our previous work, we show in this parametrized by three coordinates and two angles as shown in paper that by inserting a microlens array at the intermediate Fig. 2(b), the plenoptic function is 5D. However, if we restrict image plane of a microscope’s illumination path, one can ourselves to regions of space that are free of occluders, then programmatically control the spatial and angular distribution this function contains redundant information, because the of light (i.e. the 4D light field) arriving at the microscope’s radiance of a ray remains constant from point to point along object plane. We call this a light field illuminator (LFI). its length. In fact, the redundant information is exactly one Although diffraction again places a limit on the product of dimension, leaving us with a 4D function historically called spatial and angular bandwidth in these light fields, we can, the light field (Gershun, 1936). nevertheless, exercise substantial control over the quality of Although the 5D plenoptic function has an obvious light incident on a specimen. parametrization, the 4D light field can be parametrized in a In particular, we can reproduce exotic lighting effects such variety of ways (Levoy, 2006c). In this paper, we parametrize as darkfield, oblique illumination, and the focusing of light at rays by their intersection with two planes (Levoy & Hanrahan, planesotherthanwherethemicroscopeisfocused.Inaddition, 1996) as shown in Fig. 2(c). Although this parametrization by generating structured light patterns and recording the cannot represent all rays – for example rays parallel to the appearance of these patterns with our LFM after they have two planes if the planes are parallel to each other – it has passed through a specimen, we can measure the specimen’s the advantage of relating closely to the analytic geometry of index of refraction, even if this index changes across the field perspective imaging. Indeed, a simple way to think about a of view. In this application we are essentially using the LFI two-plane light field is as a collection of perspective views of C # 2009 The Authors C Journal compilation # 2009 The Royal Microscopical Society, Journal of Microscopy, 235, 144–162 146 M.LEVOY ETAL. cross-sectional area L(x,y,z,θ,φ) t L(u,v,s,t) θ v L (x,y,z) s φ u solid angle (a) radiance along a ray (b) 5D plenoptic function (c) 4D light field Fig. 2. Plenoptic functions and light fields. (a) The radiance L of a ray is the amount of light travelling along all possible straight lines through a tube of a given solid angle and cross-sectional area. The units of L are watts (W) per steradian (sr) per meter squared (m2). (b) In 3D space, this function is 5D. Its rays can be parametrized by three spatial coordinates x, y, and z and two angles θ and ϕ.