Flash Photography Enhancement via Intrinsic Relighting Elmar Eisemann Frédo Durand MIT/Artis-INRIA MIT IntroductionIntroduction Satisfactory photos in dark environments are challenging! IntroductionIntroduction Available light: + nice lighting - noise/blurriness - color No-flash IntroductionIntroduction Flash: + details + color - flat/artificial - flash shadows - red eyes Flash IntroductionIntroduction Our approach: Use no-flash image relight flash image No-flash Flash Result IntroductionIntroduction Our approach: Use no-flash image relight flash image No-flash + original lighting + details/sharpness + color Result IntroductionIntroduction One approach: Blend the two photos No-flash + Flash Blending IntroductionIntroduction One approach: Blend the two photos Blending Our result IntroductionIntroduction One approach: Blend the two photos Our solution: Blendingmore details and less noise Our result OverviewOverview • Related Work • Our Approach • Results • Conclusion and Future Work OverviewOverview • Related Work • Our Approach • Results • Conclusion and Future Work RelatedRelated WorkWork Relighting: Decouple luminance and texture • Barrow and Tennenbaum [1978] • Weiss [2001] – using image sequences • Tappen et al. [2003] – from a single image • Oh et al. [2001] Input Image Luminance Texture Images courtesy of Oh et al. RelatedRelated WorkWork Tone Mapping of High Dynamic Range Images Decouple detail / large-scale information • Tumblin et al. [1999] • Durand et al. [2002] • Choudhury et al. [2003] + Input Detail Large-Scale Output RelatedRelated WorkWork Jia et al. [2004]: • Solve problem of blur in long exposures • Image pair with different exposure times Short exposure Result Color manipulation Long exposure ¾ Details from short exposure Images courtesy of Jia et al. RelatedRelated WorkWork Petschnigg et al.[2004]: • many similarities • previous talk • discussion at the end OverviewOverview • Related Work • Our Approach • Results • Conclusion and Future Work OurOur ApproachApproach--MainMain IdeaIdea OurOur ApproachApproach OurOur ApproachApproach OurOur ApproachApproach Registration Registration RegistrationRegistration • Compensate for camera motion (image translation) • Difficult because lighting changes • Edge detection No-flash Flash • See also Ward[2004], Kang[2003] OurOur ApproachApproach Registration Registration OurOur ApproachApproach Decomposition DecompositionDecomposition Color / Intensity: = * original intensity color OurOur ApproachApproach Decomposition OurOur ApproachApproach Decoupling DecouplingDecoupling • Lighting : Large-scale variation • Texture : Small-scale variation Lighting Texture LargeLarge--scalescale LayerLayer • Bilateral filter – edge preserving filter Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002 Input Output LargeLarge--scalescale LayerLayer • Bilateral filter CrossCross BilateralBilateral FilterFilter • Similar to joint bilateral filter by Petschnigg et al. • When no-flash image is too noisy • Borrow similarity from flash image ¾ edge stopping from flash image • See detail in paper Bilateral Cross Bilateral DetailDetail LayerLayer / = Intensity Large-scale Detail Recombination: Large scale * Detail = Intensity RecombinationRecombination * = Large-scale Detail Intensity No-flash Flash Result Recombination: Large scale * Detail = Intensity RecombinationRecombination shadows * ~ Intensity Color Result Result Flash Recombination: Intensity * Color = Original OurOur ApproachApproach Our Approach Our Approach Shadow Detection/Treatment ShadowShadow CorrectionCorrection––WhyWhy?? No flash Flash GlobalStraightforward white balance recombination in shadows ShadowShadow CorrectionCorrection––WhyWhy?? Global white balance in shadows Several artifacts: • at shadow boundary • inside shadows (color bleeding) ¾ shadows need to be corrected ! ... and detected ShadowShadow DetectionDetection Flash • Umbra • Penumbra ¾ Detection in two steps ShadowShadow DetectionDetection Umbra detection • No direct light from flash ¾ Difference of the two photos ∆Ι reveals these regions - = Flash No-flash ∆Ι ShadowShadow DetectionDetection ? Umbra detection • Difference ∆Ι = light from the flash • Goal: Find a threshold for ∆Ι Automatic Threshold Detection (see paper) ∆Ι ShadowShadow DetectionDetection Penumbra detection ¾ strong gradient at boundary • no strong gradient in no-flash image • connected to umbra No-flash flash Umbra Penumbra ShadowShadow CorrectionCorrection • Correct color and detail Shadow mask ShadowShadow ColorColor CorrectionCorrection Correct color Wrong color Flash color Fill in shadow from similar surrounding No-flash colors ShadowShadow ColorColor CorrectionCorrection No-flash Flash ShadowShadow ColorColor CorrectionCorrection No-flash Flash ShadowShadow ColorColor CorrectionCorrection No-flash Flash Outside shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Select pixel in shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Corresponding pixel ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Spatial weights ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Spatial and Color weights ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Spatial and Color weights ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Use weights on flash color ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Replace shadow pixel ShadowShadow ColorColor CorrectionCorrection No-flash Flash Inside shadow Outside shadow Proceed for all shadow pixels OurOur ApproachApproach OurOur ApproachApproach OverviewOverview • Related Work • Our Approach • Results • Conclusion and Future Work ResultsResults No-flash Flash ResultsResults No correction Our result ResultsResults No-flash Flash ResultsResults No-flash Flash ResultsResults No-flash Flash Our result ResultsResults Our result ResultsResults Our result EmphasizeEmphasize ForegroundForeground Our result Deduce “distance“ to camera Exploit 1/r2 flash intensity falloff (see paper) Emphasized foreground (Inverse)(Inverse) WhiteWhite BalanceBalance No-flash Flash Retain warm tones from available lighting (see paper) ResultsResults No-flash Flash Result OverviewOverview • Related Work • Our Approach • Results • Conclusion and Future Work ConclusionConclusion Improving photography in dim environments • Capture original lighting • Add sharpness/details No-flash • Cross bilateral filter • Correct flash shadows Result • Pseudo distance (emphasize foreground) • (Inverse) white balance FutureFuture WorkWork • Local coherence for shadow detection • Different recombinations • Bilateral filter parameters • Infra-red flash AknowledgementsAknowledgements • École Normale Supérieure - Paris • Deshpande Center • MIT-France •NSF CISE ThankThank youyou veryvery muchmuch forfor youryour attention!attention! Comparison:Comparison: SimilaritiesSimilarities Petschnigg et al. Eisemann et al. • Detail transfer or relighting • Use bilateral filter • Joint or cross bilateral filter Comparison:Comparison: ColorColor Petschnigg et al. Eisemann et al. • Color from no-flash • Color from flash • More noisy • Less noisy • Less need for shadow • Need correction in shadow correction • White balance • Inverse white balance no-flash image flash image Comparison:Comparison: DifferencesDifferences Petschnigg et al. Eisemann et al. Algorithmic differences • Color from no-flash • Color from flash • Semi-manual • Automatic shadows shadows/highlights Additional tools • Continuous flash • Pseudo distance, • Red-eye removal emphasize foreground ThankThank youyou againagain forfor youryour attention!attention!.