DOCSLIB.ORG

Faces and Image-Based Lighting

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Faces and Image-Based Lighting Outline • Image-based lighting • 3D acquisition for faces Faces and Image-Based Lighting • Statistical methods (with application to face super-resolution) • 3D Face models from single images Digital Visual Effects • Image-based faces Yung-Yu Chuang • Relighting for faces with slides by Richard Szeliski, Steve Seitz, Alex Efros, Li-Yi Wei and Paul Debevec Rendering • Rendering is a function of geometry, reflectance, lighting and viewing. • To synthesize CGI into real scene, we have to mathtch the above four fac tors. Image-based lighting • Viewing can be obtained from calibration or structure from motion. •Geometryypg can be captured using 3D photography or made by hands. • How to capture lighting and reflectance? Reflectance Rendering equation • The Bidirectional Reflection Distribution Function – Given an incoming ray and outgoing ray Li (p,ωi ) what proportion of the incoming light is reflected along outgoing ray? ωi surface normal ωo p Lo (p,ωo ) 5D light field Lo (p,ωo ) Le (p,ωo ) 2 (p,ωo ,ωi )Li (p,ωi ) cosθi dωi Answer given by the BRDF: s Complex illumination Point lights Lo (p,ωo ) Le (p,ωo ) Classically, rendering is performed assuming point light sources f (p,ωo ,ωi )Li (p,ωi ) cosθi dωi s2 B(p,ωo ) 2 f (p,ωo ,ωi )Ld (p,ωi ) cosθi dωi s reflectance lighting directional source Natural illumination Natural illumination People perceive materials more easily under Rendering with natural illumination is more natural illumination than simplified illumination. expensive compared to using simplified illumination ItRDdTdAdlImages courtesy Ron Dror and Ted Adelson directional source natural illumination Environment maps Acquiring the Light Probe Miller and Hoffman , 1984 HDRI Sky Probe Clipped Sky + Sun Source Lit byyy sun only Lit byyy sky only Lit byyy sun and sky Illuminating a Small Scene Real Scene Example • Goal: place synthetic objects on table Light Probe / Calibration Grid Modeling the Scene light-based model real scene The Light-Based Room Model Rendering into the Scene • Background Plate Rendering into the scene Differential rendering • Objects and Local Scene matched to Scene • Local scene w/o objects, illuminated by model Differential rendering Differential rendering - = + Differential Rendering Environment map from single image? • Final Result Eye as light probe! (Nayar et al) Results Application in “Superman returns” Capturing reflectance Application in “The Matrix Reloaded” 3D acquisition for faces Cyberware scanners Making facial expressions from photos • Similar to Façade, use a generic face model and view-dependent texture mapping • Procedure 1. Take multiple photographs of a person 2. Establish corresponding feature points 3. Recover 3D points and camera parameters 4. Deform the generic face model to fit points 5. Extract textures from photos face & head scanner whole body scanner Reconstruct a 3D model Mesh deformation input photographs – Compute displacement of feature points – Apply scattered data interpolation generic 3D pose more deformed face model estimation features model generic model displacement deformed model Texture extraction Texture extraction • The color at each point is a weighted combination of the colors in the photos • Texture can be: – view-independent – view-dependent • Considerations for weighting – occlusion – smoothness – ppyositional certainty –view similarity Texture extraction Texture extraction view-independent view-dependent Model reconstruction Creating new expressions • In addition to global blending we can use: – Reg iona l blen ding – Painterly interface Use images to adapt a generic face model. Creating new expressions Creating new expressions New expressions are created with 3D morphing: xx+ = + = /2 /2 Applying a global blend Applying a region-based blend Creating new expressions Drunken smile + + + = Using a painterly interface Animating between expressions Video Morphing over time creates animation: “neutral” “joy” Spacetime faces Spacetime faces black & white cameras color cameras video projectors time time Face surface time time stereo stereo active stereo time Spacetime Stereo time surface motion stereo active stereo spacetime stereo time=1 Spacetime Stereo time Spacetime Stereo time surface motion surface motion time=2 time=3 Spacetime Stereo time Spacetime Stereo time surface motion surface motion time=4 time=5 Spacetime Stereo time Spacetime stereo matching surface motion Better • spatial resolution • temporal stableness time Video Fitting FaceIK Animation 3D face applications: The one 3D face applications: Gladiator extra 3M Statistical methods para- observed zyf(z)+ meters signal Example: Statistical methods z* max P(z | y) super-resolution z P(y | z)P(z) de-noising max de-blocking z P(y) Inpainting … min L(y | z) L(z) z Statistical methods Statistical methods para- observed There are approximately 10240 possible 1010 zyf(z)+ gray-level images. Even human being has not signal meters seen them all yet. There must be a strong statistical bias. z* minL(y |z) L(z) Takeo Kanade z 2 data y f(z) a-priori Approximately 8X1011 blocks per day per person. evidence 2 knowledge Generic priors Generic priors “Smoo th images are good images. ” L(z) (V(x)) x Gaussian MRF (d) d 2 d 2 d T Huber MRF (d) 2 T 2T(d T) d T Example-based priors Example-based priors “Ex is ting images are good images. ” L(z) six 200200 Images 2,000,000 pairs Example-based priors Model-based priors “Face images are good images when working on face images …” Parametric high-resolution model Z=WX+ L(X) z* minL(y |z) L(z) z X* minL(y |WX ) L(X) x low-resolution z* WX * PCA PCA on faces: “eigenfaces” • Principal Components Analysis (PCA): First principal component approximating a high-dimensional data set Average with a lower-dimensional subspace face Other * * components Second principal component * * First principal component * * * * * * * * * * * * * Original axes * * * * * For a ll e xcept av er age, * * “gray” = 0, Data points “white” > 0, “black” <0< 0 Model-based priors Super-resolution “Face images are good images when working on face images …” Parametric model Z=WX+ L(X) z* minL(y |z) L(z) z X* minL(y |WX ) L(X) x (a) (b) (c) (d) (e) (f) z* WX * (a) Input low 24×32 (b) Our results (c) Cubic B-Spline (d) Freeman et al. (e) Baker et al. (f) Original high 96×128 Morphable model of 3D faces • Start with a catalogue of 200 aligned 3D Cyberware scans Face models from single images • Build a model of average shape and texture, and principal variations using PCA Morphable model Morphable model of 3D faces shape examplars texture examplars • Adding some variations Reconstruction from single image Reconstruction from single image prior shape and texture priors are learnt from database Rendering must ρ is the set of parameters for shading including be similar to camera pp,ggose, lighting and so on the input if we ggguess right Modifying a single image Animating from a single image Video Exchanging faces in images Exchange faces in images Exchange faces in images Exchange faces in images Exchange faces in images Morphable model for human body Image-based faces (lip sync.) Video rewrite (analysis) Video rewrite (synthesis) Results Morphable speech model • Video database – 2 minu tes of JFK • Only half usable • Head rotation training video RdRead my lips. I never met Forest Gump. Preprocessing Prototypes (PCA+k-mean clustering) We fidfind Ii and Ci for each protttotype image. Morphable model Morphable model analysis analysis I α β synthesis synthesis Synthesis Results Results Relighting faces Light is additive Light stage 1.0 lamp #1 lamp #2 Light stage 1.0 Input images 64x32 lighting directions Reflectance function Relighting occlusion flare Results Changing viewpoints Results Video 3D face applications: Spiderman 2 Spiderman 2 real synthetic Spiderman 2 Light stage 3 video Light stage 6 Application: The Matrix Reloaded Application: The Matrix Reloaded References •Paul Debevec, Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography, SIGGRAPH 1998. • F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski. Synthesizing realistic facial expressions from photographs. SIGGRAPH 1998, pp75-84. • Li Zhang, Noah Snavely, Brian Curless, Steven M. Seitz, Space time Faces: High Reso lu tion Cap ture for Mo de lin g and Animation, SIGGRAPH 2004. • Blanz, V. and Vetter, T., A Morphable Model for the SthiSynthesis of 3D Faces, SIGGRAPH 1999, pp187-194. • Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Sagar, Acquiring the RfltReflectance FildField of a Human Face, SIGGRAPH 2000. • Christoph Bregler, Malcolm Slaney, Michele Covell, Video Rewrite: Driving Visual Speeach with Audio, SIGGRAPH 1997. • Tony Ezzat, Gadi Geiger, Tomaso Poggio, Trainable Videorealistic Speech Animation, SIGGRAPH 2002. .
Load more

Recommended publications
  • Programmable Image-Based Light Capture for Previsualization
    ii Abstract Previsualization is a class of techniques for creating approximate previews of a movie sequence in order to visualize a scene prior to shooting it on the set. Often these techniques are used to convey the artistic direction of the story in terms of cinematic elements, such as camera movement, angle, lighting, dialogue, and char- acter motion. Essentially, a movie director uses previsualization (previs) to convey movie visuals as he sees them in his ”minds-eye”. Traditional methods for previs include hand-drawn sketches, Storyboards, scaled models, and photographs, which are created by artists to convey how a scene or character might look or move. A recent trend has been to use 3D graphics applications such as video game engines to perform previs, which is called 3D previs. This type of previs is generally used prior to shooting a scene in order to choreograph camera or character movements. To visualize a scene while being recorded on-set, directors and cinematographers use a technique called On-set previs, which provides a real-time view with little to no processing. Other types of previs, such as Technical previs, emphasize accurately capturing scene properties but lack any interactive manipulation and are usually employed by visual effects crews and not for cinematographers or directors. This dissertation’s focus is on creating a new method for interactive visualization that will automatically capture the on-set lighting and provide interactive manipulation of cinematic elements to facilitate the movie maker’s artistic expression, validate cine- matic choices, and provide guidance to production crews. Our method will overcome the drawbacks of the all previous previs methods by combining photorealistic ren- dering with accurately captured scene details, which is interactively displayed on a mobile capture and rendering platform.
    [Show full text]
  • Capturing and Simulating Physically Accurate Illumination in Computer Graphics
    Capturing and Simulating Physically Accurate Illumination in Computer Graphics PAUL DEBEVEC Institute for Creative Technologies University of Southern California Marina del Rey, California Anyone who has seen a recent summer blockbuster has witnessed the dramatic increases in computer-generated realism in recent years. Visual effects supervisors now report that bringing even the most challenging visions of film directors to the screen is no longer a question of whats possible; with todays techniques it is only a matter of time and cost. Driving this increase in realism have been computer graphics (CG) techniques for simulating how light travels within a scene and for simulating how light reflects off of and through surfaces. These techniquessome developed recently, and some originating in the 1980sare being applied to the visual effects process by computer graphics artists who have found ways to channel the power of these new tools. RADIOSITY AND GLOBAL ILLUMINATION One of the most important aspects of computer graphics is simulating the illumination within a scene. Computer-generated images are two-dimensional arrays of computed pixel values, with each pixel coordinate having three numbers indicating the amount of red, green, and blue light coming from the corresponding direction within the scene. Figuring out what these numbers should be for a scene is not trivial, since each pixels color is based both on how the object at that pixel reflects light and also the light that illuminates it. Furthermore, the illumination does not only come directly from the lights within the scene, but also indirectly Capturing and Simulating Physically Accurate Illumination in Computer Graphics from all of the surrounding surfaces in the form of bounce light.
    [Show full text]
  • SIGGRAPH 2008 Course: Computational Photography: Advanced Topics
    SIGGRAPH 2008 Course: Computational Photography: Advanced Topics http://computationalphotography.org Speakers Paul Debevec (USC, USA) (debevec (at) ict.usc.edu) Ramesh RASKAR (MIT Media Lab, USA) (raskar (at) media.mit.edu) Jack TUMBLIN (Northwestern University, USA) (jet (at) cs.northwestern.edu) Course Abstract Computational photography combines plentiful computing, digital sensors, modern optics, many varieties of actuators, probes and smart lights to escape the limitations of traditional film cameras and enables novel imaging applications. Unbounded dynamic range, variable focus, resolution, and depth of field, hints about shape, reflectance, and lighting, and new interactive forms of photos that are partly snapshots and partly videos, performance capture and interchangeably relighting real and virtual characters are just some of the new applications emerging in Computational Photography. The computational techniques encompass methods from modification of imaging parameters during capture to sophisticated reconstructions from indirect measurements. We will bypass basic and introductory material presented in earlier versions of this course (Computational Photography 2005,6,7) and expand coverage of more recent topics. Emphasizing more recent work in computational photography and related fields (2006 or later) this course will give more attention to advanced topics only briefly touched before, including tomography, heterodyning and Fourier Slice applications, inverse problems, gradient illumination, novel optics, emerging sensors
    [Show full text]
  • The Light Stages and Their Applications to Photoreal Digital Actors
    The Light Stages and Their Applications to Photoreal Digital Actors Paul Debevec∗ University of Southern California Institute for Creative Technologies Figure 1: Left to Right: Light Stage 1’s spiraling spotlight records a reflectance field in 60 seconds; Light Stage 2 records actor Alfred Molina for Spider-Man 2; Light Stage 3 illuminates an actor with a reproduction of the colorful light of the Grace Cathedral HDRI map; Light Stage 5 uses high-speed photography to record an actor’s reflectance with time-multiplexed illumination; Light Stage 6, at 8m in diameter, allows performance relighting for the whole human body. Abstract the computer-generated object to create a spatially modulated area light which mimics the color, intensity, and directionality of the The Light Stage systems built at UC Berkeley and USC ICT have recorded incident illumination. When the light from this IBL en- enabled a variety of facial scanning and reflectance measurement vironment is simulated with global illumination, a rendering of the techniques that have been explored in several research papers and object is produced as if it were illuminated by the recorded real- used in various commercial applications. This short paper presents world lighting. This can add an additional level of realism to the the evolutionary history of the Light Stage Systems and some of the appearance of the object, adding subtle variation to its shading, techniques and applications they have enabled. highlights, and shadows, and is also useful for lighting the object consistently with a scene into which it is being composited, allow- Keywords: 3D scanning, facial animation, computational illumi- ing it to appear as if it were really there.
    [Show full text]
  • Acquiring the Reflectance Field of a Human Face
    To appear in the SIGGRAPH 2000 Conference Proceedings Acquiring the Reflectance Field of a Human Face Ý Ý Ý Ý Paul DebevecÝ Tim Hawkins Chris Tchou Haarm-Pieter Duiker Westley Sarokin and Mark SagarÞ ½ Þ Ý University of California at Berkeley LifeF/X, Inc. ABSTRACT models of individual faces. Work to animate facial expressions through morphing [2, 4, 29], performance-driven animation [38], We present a method to acquire the reflectance field of a human motion capture [14], and physics-based simulation [34, 20, 30] has face and use these measurements to render the face under arbitrary produced examples of realistic facial motion. changes in lighting and viewpoint. We first acquire images of the An outstanding problem is the lack of a method for capturing the face from a small set of viewpoints under a dense sampling of in- spatially varying reflectance characteristics of the human face. The cident illumination directions using a light stage. We then con- traditional approach of texture-mapping a photograph of a face onto struct a reflectance function image for each observed image pixel a geometric model usually fails to appear realistic under changes from its values over the space of illumination directions. From the in lighting, viewpoint, and expression. The problem is that the reflectance functions, we can directly generate images of the face reflectance properties of the face are complex: skin reflects light from the original viewpoints in any form of sampled or computed both diffusely and specularly, and both of these reflection com- illumination. To change the viewpoint, we use a model of skin re- ponents are spatially varying.
    [Show full text]
  • Acquiring the Reflectance Field of a Human Face
    To appear in the SIGGRAPH 2000 Conference Proceedings Acquiring the Reflectance Field of a Human Face Ý Ý Ý Ý Paul DebevecÝ Tim Hawkins Chris Tchou Haarm-Pieter Duiker Westley Sarokin and Mark SagarÞ ½ Þ Ý University of California at Berkeley LifeF/X, Inc. ABSTRACT models of individual faces. Work to animate facial expressions through morphing [2, 4, 29], performance-driven animation [38], We present a method to acquire the reflectance field of a human motion capture [14], and physics-based simulation [34, 20, 30] has face and use these measurements to render the face under arbitrary produced examples of realistic facial motion. changes in lighting and viewpoint. We first acquire images of the An outstanding problem is the lack of a method for capturing the face from a small set of viewpoints under a dense sampling of in- spatially varying reflectance characteristics of the human face. The cident illumination directions using a light stage. We then con- traditional approach of texture-mapping a photograph of a face onto struct a reflectance function image for each observed image pixel a geometric model usually fails to appear realistic under changes from its values over the space of illumination directions. From the in lighting, viewpoint, and expression. The problem is that the reflectance functions, we can directly generate images of the face reflectance properties of the face are complex: skin reflects light from the original viewpoints in any form of sampled or computed both diffusely and specularly, and both of these reflection com- illumination. To change the viewpoint, we use a model of skin re- ponents are spatially varying.
    [Show full text]
  • The Digital Emily Project
    CREATING A PHOTOREAL DIGITAL ACTOR: THE DIGITAL EMILY PROJECT Oleg Alexander1 Mike Rogers1 William Lambeth1 Matt Chiang 2 Paul Debevec2 1 Image Metrics 1918 Main St, 2nd Floor, Santa Monica, CA 90405, USA e-mail: foleg.alexander,mike.rogers,[email protected] 2 University of Southern California Institute for Creative Technologies 13274 Fiji Way, Marina del Rey, CA 90292, USA e-mail: fchiang,[email protected] 1 Introduction Abstract Creating a photoreal digital actor with computer graphics The Digital Emily Project is a collaboration between has been a central goal of the field for at least thirty years facial animation company Image Metrics and the Graphics [Parke 1972]. The Digital Emily project undertaken by Image Laboratory at the University of Southern California’s Institute Metrics and the University of Southern California’s Institute for Creative Technologies to achieve one of the world’s for Creative Technologies (USC ICT) attempted to achieve an first photorealistic digital facial performances. The project animated, photoreal digital face by bringing together latest- leverages latest-generation techniques in high-resolution face generation results in 3D facial capture, modeling, animation, scanning, character rigging, video-based facial animation, and rendering. The project aimed to cross the ”uncanny valley” and compositing. An actress was first filmed on a studio set [20], producing a computer-generated face which appeared speaking emotive lines of dialog in high definition. The lighting to be a real, relatable, animated person. Some of the key on the set was captured as a high dynamic range light probe technologies employed included a fast high-resolution digital image.
    [Show full text]
  • Multi-Scale Capture of Facial Geometry and Motion
    Multi-Scale Capture of Facial Geometry and Motion The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Bickel, Bernd, Mario Botsch, Roland Angst, Wojciech Matusik, Miguel Otaduy, Hanspeter Pfister, and Markus Gross. 2007. Multi-scale capture of facial geometry and motion. Proceedings International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH 2007 papers: August 05-09, 2007, San Diego, California, 33-41. New York, NY: ACM. Also published in ACM Transactions on Graphics 26(3): 33-41. Published Version doi:10.1145/1275808.1276419;doi:10.1145/1276377.1276419 Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:4726184 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Other Posted Material, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#LAA Multi-Scale Capture of Facial Geometry and Motion Bernd Bickel∗ Mario Botsch∗ Roland Angst† Wojciech Matusik† Miguel Otaduy∗ Hanspeter Pfister† Markus Gross∗ Figure 1: Animation of a high-resolution face scan using marker-based motion capture and a video-driven wrinkle model. From left to right: video frame, large-scale animation without wrinkles, synthesis of medium-scale wrinkles, realistic skin-rendering, different expression. Abstract 1 Introduction Capturing the likeness and dynamic performance of a human face We present a novel multi-scale representation and acquisition with all its subtleties is one of the most challenging problems in method for the animation of high-resolution facial geometry and computer graphics.
    [Show full text]
  • Capturing and Simulating Physically Accurate Illumination in Computer Graphics
    Capturing and Simulating Physically Accurate Illumination in Computer Graphics PAUL DEBEVEC Institute for Creative Technologies University of Southern California Marina del Rey, California Anyone who has seen a recent summer blockbuster has witnessed the dramatic increases in computer-generated realism in recent years. Visual effects supervisors now report that bringing even the most challenging visions of film directors to the screen is no longer a question of what’s possible; with today’s techniques it is only a matter of time and cost. Driving this increase in realism have been computer graphics (CG) techniques for simulating how light travels within a scene and for simulating how light reflects off of and through surfaces. These techniques—some developed recently, and some originating in the 1980’s—are being applied to the visual effects process by computer graphics artists who have found ways to channel the power of these new tools. RADIOSITY AND GLOBAL ILLUMINATION One of the most important aspects of computer graphics is simulating the illumination within a scene. Computer-generated images are two-dimensional arrays of computed pixel values, with each pixel coordinate having three numbers indicating the amount of red, green, and blue light coming from the corresponding direction within the scene. Figuring out what these numbers should be for a scene is not trivial, since each pixel’s color is based both on how the object at that pixel reflects light and also the light that illuminates it. Furthermore, the illumination does not only come directly from the lights within the scene, but also indirectly Capturing and Simulating Physically Accurate Illumination in Computer Graphics from all of the surrounding surfaces in the form of “bounce” light.
    [Show full text]
  • A Lighting Reproduction Approach to Live-Action Compositing Paul Debevec Andreas Wenger† Chris Tchou Andrew Gardner Jamie Waese Tim Hawkins
    To appear at SIGGRAPH 2002, San Antonio, July 21-26, 2002 A Lighting Reproduction Approach to Live-Action Compositing Paul Debevec Andreas Wenger† Chris Tchou Andrew Gardner Jamie Waese Tim Hawkins University of Southern California Institute for Creative Technologies 1 ABSTRACT We describe a process for compositing a live performance of an ac- tor into a virtual set wherein the actor is consistently illuminated by the virtual environment. The Light Stage used in this work is a two-meter sphere of inward-pointing RGB light emitting diodes fo- cused on the actor, where each light can be set to an arbitrary color and intensity to replicate a real-world or virtual lighting environ- ment. We implement a digital two-camera infrared matting system to composite the actor into the background plate of the environ- ment without affecting the visible-spectrum illumination on the ac- tor. The color reponse of the system is calibrated to produce correct color renditions of the actor as illuminated by the environment. We demonstrate moving-camera composites of actors into real-world environments and virtual sets such that the actor is properly illumi- nated by the environment into which they are composited. Keywords: Matting and Compositing, Image-Based Lighting, Ra- diosity, Global Illumination, Reflectance and Shading Figure 1: Light Stage 3 focuses 156 red-green-blue LED lights to- ward an actor, who is filmed simultaneously with a color camera and an infrared matting system. The device composites an actor 1 Introduction into a background environment as illuminated by a reproduction of the light from that environment, yielding a composite with consis- Many applications of computer graphics involve compositing, the tent illumination.
    [Show full text]