Eurographics Symposium on Rendering (2004) H. W. Jensen, A. Keller (Editors) Progressively-Refined Reflectance Functions from Natural Illumination Wojciech Matusik†, Matthew Loper, and Hanspeter Pfister Mitsubishi Electric Research Laboratories, Cambridge, MA. Abstract In this paper we present a simple, robust, and efficient algorithm for estimating reflectance fields (i.e., a description of the transport of light through a scene) for a fixed viewpoint using images of the scene under known natural illumination. Our algorithm treats the scene as a black-box linear system that transforms an input signal (the incident light) into an output signal (the reflected light). The algorithm is hierarchical – it progressively refines the approximation of the reflectance field with an increasing number of training samples until the required precision is reached. Our method relies on a new representation for reflectance fields. This representation is compact, can be progressively refined, and quickly computes the relighting of scenes with complex illumination. Our representation and the corresponding algorithm allow us to efficiently estimate the reflectance fields of scenes with specular, glossy, refractive, and diffuse elements. The method also handles soft and hard shadows, inter-reflections, caustics, and subsurface scattering. We verify our algorithm and representation using two measurement setups and several scenes, including an outdoor view of the city of Cambridge. Categories and Subject Descriptors (according to ACM CCS): I.2.10 [Artificial Intelligence]: Vision and Scene Understanding; I.3.3 [Computer Graphic]: Picture/ Image Generation 1. Introduction that illuminates the scene. Figure 1 shows a few results from our method. We are interested in acquiring a representation of arbitrary objects and scenes that allows us to render new images of the The problem of natural scene acquisition has received scene using natural illumination captured in real-world envi- a lot of attention in computer graphics and computer ronments. Such a representation has many applications in re- vision. In general, we would like to acquire the re- ∗ alistic image synthesis (offline or real-time) and scene analy- flectance field [DHT 00], also known as the bidirectional sis, such as 3D scanning of cultural artifacts, object and face scattering-surface reflectance distribution function (BSS- ∗ recognition, or relighting of image-based models. Current RDF) [NRH 77]. It is defined as the ratio of incoming to approaches require devices that generate controlled incident outgoing radiance at two different scene points. Formally we illumination (either point light sources or structured light). use fr(ωi,xi,ωo,xo), where ωi is the direction of the inci- Even for small objects these devices are typically large, ex- dent illumination at pointxi, and ωo is the observation direc- pensive, and not very portable. tion of radiance emitted at point xo. The function is eight- dimensional, assuming a two-dimensional parameterization We propose a novel approach that works with known nat- for the points in the scene. ural illumination and requires a small number of measure- ments to compute a good estimation of the scene reflectance. Because sampling of an eight-dimensional function is Note that we use the term natural illumination freely, since challenging, we focus on a four-dimensional subspace or images of natural scenes can also be displayed on a monitor 4D slice of the reflectance field. First, we assume that the incident illumination is far away and arrives at scene points from direction ωi. Second, we acquire the reflectance field for each image pixel (x,y), which implies both scene † [matusik,pfister]@merl.com position xo and viewing direction ωo. The resulting 4D c The Eurographics Association 2004. W. Matusik, M. Loper, and H. Pfister / Progressively-Refined Reflectance Functions from Natural Illumination Figure 1: Top row: Actual image of a scene under new natural illumination. Bottom row: Prediction using our method. function fw(ωi,x,y) has been called the weighting func- flectance field can be well represented with the storage tion [ZWCS99]orreflectance function [DHT∗00] of light required for a few 2D slices. transport from distant illumination to the observation point. Progressive Refinement: Our algorithm improves the ap- It is a view-dependent slice of the full 8D reflectance field. proximation of the reflectance field as more measure- ments are made. However, robustly acquiring this 4D reflectance func- Fast Evaluation: Our representation of reflectance fields tion for arbitrary scenes is also difficult. Consider a scene allows us to rapidly perform the integration with complex × where the surface coordinates are discretized into 1000 incident illumination. 1000 points and the incident illumination is represented Simplicity: The algorithm is very simple and can be × by a 1000 1000 pixel image. Sampling and tabulating quickly implemented. the reflectance function directly requires 1012 values, pos- ing a challenge for storage. Moreover, the acquisition time 6 would be prohibitively long to take 10 high-resolution pho- 2. Previous Work tographs. As mentioned above, a reflectance function fw(ωi,x,y) re- In practice, most direct sampling approaches use only lates incoming illumination to observed radiance. We can low-resolution (low-frequency) incident lighting. Conse- classify the methods for estimating reflectance functions into quently, they cannot represent high-frequency effects, such forward and inverse methods. as specularities, refraction, and shadows. On the other hand, environment matting techniques focus on reflectance func- Forward Methods Most forward methods sample the re- tions for specular, refractive, and glossy materials, but they flectance functions exhaustively and tabulate the results. have difficulty representing hard shadows or a combination For each incident illumination direction they store the re- of diffuse and specular materials. flectance function weights for a fixed observation direc- We propose a novel approach for reflectance function es- tion. In practice, only low-resolution incident illumination timation and representation with the following properties: can be used, since one reflectance table has to be stored per scene point. Debevec et al. [DHT∗00] use the high- Natural Illumination Input: Our algorithm does not re- est resolution incident illumination with roughly 2000 di- quire structured illumination patterns or point light source rections. Polynomial texture maps [MGW01] improve the input. It works with arbitrary (known) natural illumina- compactness of the representation by expressing each re- tion. flectance field table with a bivariate polynomial. These di- All-Frequency Robustness: Our algorithm is equally effi- rect approaches work very well for diffuse or glossy ob- cient for both the high frequency (e.g., specular) and low jects, such as human faces [GBK99, DHT∗00], cultural ar- frequency (e.g., diffuse) components of reflectance fields. tifacts [HCD01, MGW01, MDA02], and other objects with It also handles scenes with discontinuous reflectance complex appearance [MPN∗02]. However, they can not rep- fields (e.g., hard shadows). resent high-frequency phenomena (such as refractions) or Compact Representation: Our underlying representation discontinuities in the reflectance field (such as hard shad- of reflectance fields is compact. One 4D slice of a re- ows). Furthermore, these methods do not provide progres- c The Eurographics Association 2004. W. Matusik, M. Loper, and H. Pfister / Progressively-Refined Reflectance Functions from Natural Illumination sive refinement of the approximation with an increasing functions, such as a small specular lobe inside a larger number of samples. diffuse component. Wavelet environment matting [PD03] addresses some of Wexler et al. [WFZ02] were the first to use natural il- these shortcomings. Images with wavelets patterns are used lumination to estimate environment mattes. However, their as incident illumination, and a feedback loop determines the method can only capture very specular and refractive scenes. next pattern to emit based on the error of the current approx- Our method works for scenes with arbitrary materials. imation. The reflectance function is progressively refined as ∗ more measurements are made. However, this algorithm re- Hybrid Methods Matusik et al. [MPZ 02] combine a ∗ quires many wavelet patterns for highly specular and re- forward method [DHT 00] for low-frequency components ∗ fractive materials or scenes with hard shadows (up to 2400 and high-quality environment matting [CZH 00] for high- pattern images are reported by Peers and Dutré [PD03]). frequency reflections and refractions. The low-frequency Since each pixel stores the coefficients of the corresponding data is compressed using principal component analysis wavelet patterns, this representation becomes rather large (PCA). However, their representation is not compact, and the (2.5 GB per environment matte with lossless compression). approach does not address any of the problems mentioned Rendering is also expensive, since it requires a weighted sum above. of many wavelet-basis images of the new incident illumina- tion. Pre-Computed Radiance Transport Reflectance func- tions have become very popular as a way to increase the Masselus et al. [MPDW03] measure six-dimensional realism in real-time rendering, where they are known as ra- slices of the eight-dimensional reflectance field by varying diance transport functions. In these applications, the radi- both the position and direction