ToappearintheSIGGRAPH98conferenceproceedings RenderingSyntheticObjectsintoRealScenes: BridgingTraditionalandImage-basedGraphicswithGlobalIllumination andHighDynamicRangePhotography PaulDebevec UniversityofCaliforniaatBerkeley 1 ABSTRACT 1Introduction Renderingsyntheticobjectsintoreal-worldscenesisanimportant Wepresentamethodthatusesmeasuredsceneradianceandglobal applicationofcomputergraphics,particularlyinarchitecturaland illuminationinordertoaddnewobjectstolight-basedmodelswith visualeffectsdomains.Oftentimes,apieceoffurniture,aprop,or correctlighting.Themethodusesahighdynamicrangeimage- adigitalcreatureoractorneedstoberenderedseamlesslyintoa basedmodelofthescene,ratherthansyntheticlightsources,toil- realscene.Thisdif®culttaskrequiresthattheobjectsbelitcon- luminatethenewobjects.Tocomputetheillumination,thesceneis sistentlywiththesurfacesintheirvicinity,andthattheinterplayof consideredasthreecomponents:thedistantscene,thelocalscene, lightbetweentheobjectsandtheirsurroundingsbeproperlysimu- andthesyntheticobjects.Thedistantsceneisassumedtobepho- lated.Speci®cally,theobjectsshouldcastshadows,appearinre¯ec- tometricallyunaffectedbytheobjects,obviatingtheneedforre- tions,andrefract,focus,andemitlightjustasrealobjectswould. ¯ectancemodelinformation.Thelocalsceneisendowedwithes- timatedre¯ectancemodelinformationsothatitcancatchshadows andreceivere¯ectedlightfromthenewobjects.Renderingsare createdwithastandardglobalilluminationmethodbysimulating Distant theinteractionoflightamongstthethreecomponents.Adifferen- Scene tialrenderingtechniqueallowsforgoodresultstobeobtainedwhen light−based onlyanestimateofthelocalscenere¯ectancepropertiesisknown. (no reflectance model) Weapplythegeneralmethodtotheproblemofrenderingsyn- theticobjectsintorealscenes.Thelight-basedmodelisconstructed fromanapproximategeometricmodelofthesceneandbyusinga lightprobetomeasuretheincidentilluminationatthelocationof thesyntheticobjects.Theglobalilluminationsolutionisthencom- light positedintoaphotographofthesceneusingthedifferentialrender- ingtechnique.Weconcludebydiscussingtherelevanceofthetech- Local Synthetic niquetorecoveringsurfacere¯ectancepropertiesinuncontrolled Scene Objects lightingsituations.Applicationsofthemethodincludevisualef- estimated known reflectance reflectance fects,interiordesign,andarchitecturalvisualization. model model CRDescriptors:I.2.10[Arti®cialIntelligence]:Visionand SceneUnderstanding-Intensity,color,photometryandthreshold- ing;I.3.7[ComputerGraphics]:Three-DimensionalGraphicsand Realism-Color,shading,shadowing,andtexture;I.3.7[Computer Figure1:TheGeneralMethodInourmethodforaddingsynthetic Graphics]:Three-DimensionalGraphicsandRealism-Radiosity; objectsintolight-basedscenes,thesceneispartitionedintothree I.4.1[ImageProcessing]:Digitization-Scanning;I.4.8[Image components:thedistantscene,thelocalscene,andthesyntheticob- jects.Globalilluminationisusedtosimulatetheinterplayoflight Processing]:SceneAnalysis-Photometry,SensorFusion. amongstallthreecomponents,exceptthatlightre¯ectedbackatthe distantsceneisignored.Asaresult,BRDFinformationforthedis- tantsceneisunnecessary.Estimatesofthegeometryandmaterial 1ComputerScienceDivision,UniversityofCaliforniaatBerke- propertiesofthelocalsceneareusedtosimulatetheinteractionof ley,Berkeley,CA94720−1776. Email: [email protected]. lightbetweenitandthesyntheticobjects. More information and additional results may be found at: http://www.cs.berkeley.edu/Ädebevec/Research Currentlyavailabletechniquesforrealisticallyrenderingsyn- theticobjectsintoscenesarelaborintensiveandnotalwayssuccess- ful.Acommontechniqueistomanuallysurveythepositionsofthe lightsources,andtoinstantiateavirtuallightofequalcolorandin- tensityforeachreallighttoilluminatethesyntheticobjects.An- othertechniqueistophotographareferenceobject(suchasagray sphere)inthescenewherethenewobjectistoberendered,and useitsappearanceasaqualitativeguideinmanuallycon®guringthe lightingenvironment.Lastly,thetechniqueofre¯ectionmappingis usefulformirror-likere¯ections.Thesemethodstypicallyrequire considerablehand-re®nementandnoneofthemeasilysimulatesthe effectsofindirectilluminationfromtheenvironment. To appear in the SIGGRAPH 98 conference proceedings Accurately simulating the effects of both direct and indirect light- technique that produces perceptually accurate results even when the ing has been the subject of research in global illumination. With a estimated BRDF is somewhat inaccurate. global illumination algorithm, if the entire scene were modeled with We demonstrate the general method for the speci®c case of ren- its full geometric and re¯ectance (BRDF) characteristics, one could dering synthetic objects into particular views of a scene (such as correctly render a synthetic object into the scene simply by adding it background plates) rather than into a general image-based model. In to the model and recomputing the global illumination solution. Un- this method, a light probe is used to acquire a high dynamic range fortunately, obtaining a full geometric and re¯ectance model of a panoramic radiance map near the location where the object will be large environment is extremeley dif®cult. Furthermore, global il- rendered. A simple example of a light probe is a camera aimed at a lumination solutions for large complex environments are extremely mirrored sphere, a con®guration commonly used for acquiring envi- computationally intensive. ronment maps. An approximate geometric model of the scene is cre- Moreover, it seems that having a full re¯ectance model of the ated (via surveying, photogrammetry, or 3D scanning) and mapped large-scale scene should be unnecessary: under most circumstances, with radiance values measured with the light probe. The distant a new object will have no signi®cant effect on the appearance of scene, local scene, and synthetic objects are rendered with global most of the of the distant scene. Thus, for such distant areas, know- illumination from the same point of view as the background plate, ing just its radiance (under the desired lighting conditions) should and the results are composited into the background plate with a dif- suf®ce. ferential rendering technique. Recently, [9] introduced a high dynamic range photographic technique that allows accurate measurements of scene radiance to 1.1 Overview be derived from a set of differently exposed photographs. This tech- nique allows both low levels of indirect radiance from surfaces and The rest of this paper is organized as follows. In the next section high levels of direct radiance from light sources to be accurately we discuss work related to this paper. Section 3 introduces the ba- recorded. When combined with image-based modeling techniques sic technique of using acquired maps of scene radiance to illuminate (e.g. [22, 24, 4, 10, 23, 17, 29]), and possibly active techniques for synthetic objects. Section 4 presents the general method we will use measuring geometry (e.g. [35, 30, 7, 27]) these derived radiance to render synthetic objects into real scenes. Section 5 describes a maps can be used to construct spatial representations of scene ra- practical technique based on this method using a light probe to mea- diance. sure incident illumination. Section 6 presents a differential render- We will use the term light-based model to refer to a repre- ing technique for rendering the local environment with only an ap- sentation of a scene that consists of radiance information, possi- proximate description of its re¯ectance. Section 7 presents a sim- bly with speci®c reference to light leaving surfaces, but not neces- ple method to approximately recover the diffuse re¯ectance char- sarily containing material property (BRDF) information. A light- acteristics of the local environment. Section 8 presents results ob- based model can be used to evaluate the 5D plenoptic function [1] tained with the technique. Section 9 discusses future directions for 1 this work, and we conclude in Section 10. P ; ; V ;V ;V y z x for a given virtual or real subset of space .A material-based model is converted to a light-based model by com- puting an illumination solution for it. A light-based model is differ- 2 Background and Related Work entiated from an image-based model in that its light values are ac- tual measures of radiance2, whereas image-based models may con- The practice of adding new objects to photographs dates to the early tain pixel values already transformed and truncated by the response days of photography in the simple form of pasting a cut-out from one function of an image acquisition or synthesis process. picture onto another. While the technique conveys the idea of the new object being in the scene, it usually fails to produce an image In this paper, we present a general method for using accurate that as a whole is a believable photograph. Attaining such realism measurements of scene radiance in conjunction with global illumi- requires a number of aspects of the two images to match. First, the nation to realistically add new objects to light-based models. The camera projections should be consistent, otherwise the object may synthetic objects may have arbitrary material properties and can be seem too foreshortened or skewed relative to the rest of the picture. rendered with appropriate illumination in arbitrary lighting environ- Second, the patterns of ®lm grain and ®lm response should match. ments. Furthermore, the objects can correctly interact with the en- Third, the lighting on the object needs to be consistent with other vironment around them: they cast the appropriate shadows,