Flash Photography Enhancement Via Intrinsic Relighting

Home , Flash (photography)

Flash Photography Enhancement via Intrinsic Relighting

Elmar Eisemann Frédo Durand MIT/Artis-INRIA MIT IntroductionIntroduction

Satisfactory photos in dark environments are challenging! IntroductionIntroduction

Available light: + nice lighting

- noise/blurriness - color

No-flash IntroductionIntroduction

Flash: + details + color

- flat/artificial - flash shadows - red eyes

Flash IntroductionIntroduction Our approach: Use no-flash image relight flash image

No-flash

Flash Result IntroductionIntroduction Our approach: Use no-flash image relight flash image

No-flash

+ original lighting + details/sharpness + color Result IntroductionIntroduction

One approach: Blend the two photos

No-flash +

Flash Blending IntroductionIntroduction

One approach: Blend the two photos

Blending Our result IntroductionIntroduction

One approach: Blend the two photos

Our solution: Blendingmore details and less noise Our result OverviewOverview

• Related Work • Our Approach • Results • Conclusion and Future Work OverviewOverview

• Related Work • Our Approach • Results • Conclusion and Future Work RelatedRelated WorkWork Relighting: Decouple luminance and texture • Barrow and Tennenbaum [1978] • Weiss [2001] – using image sequences • Tappen et al. [2003] – from a single image

• Oh et al. [2001]

Input Image Luminance Texture

Images courtesy of Oh et al. RelatedRelated WorkWork Tone Mapping of High Dynamic Range Images Decouple detail / large-scale information • Tumblin et al. [1999] • Durand et al. [2002] • Choudhury et al. [2003]

Input Detail Large-Scale Output RelatedRelated WorkWork Jia et al. [2004]: • Solve problem of blur in long exposures • Image pair with different exposure times Short exposure Result Color manipulation

Long exposure

¾ Details from short exposure Images courtesy of Jia et al. RelatedRelated WorkWork Petschnigg et al.[2004]: • many similarities • previous talk • discussion at the end OverviewOverview

• Related Work • Our Approach • Results • Conclusion and Future Work OurOur ApproachApproach--MainMain IdeaIdea OurOur ApproachApproach OurOur ApproachApproach OurOur ApproachApproach Registration Registration RegistrationRegistration

• Compensate for camera motion (image translation) • Difficult because lighting changes • Edge detection

No-flash Flash • See also Ward[2004], Kang[2003] OurOur ApproachApproach Registration Registration OurOur ApproachApproach Decomposition DecompositionDecomposition

Color / Intensity:

= *

original intensity color OurOur ApproachApproach Decomposition OurOur ApproachApproach Decoupling DecouplingDecoupling

• Lighting : Large-scale variation • Texture : Small-scale variation

Lighting Texture LargeLarge--scalescale LayerLayer • Bilateral filter – edge preserving filter Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

Input Output LargeLarge--scalescale LayerLayer

• Bilateral filter CrossCross BilateralBilateral FilterFilter

• Similar to joint bilateral filter by Petschnigg et al. • When no-flash image is too noisy • Borrow similarity from flash image ¾ edge stopping from flash image • See detail in paper

Bilateral Cross Bilateral DetailDetail LayerLayer

/ =

Intensity Large-scale Detail

Recombination: Large scale * Detail = Intensity RecombinationRecombination

* =

Large-scale Detail Intensity No-flash Flash Result Recombination: Large scale * Detail = Intensity RecombinationRecombination shadows

* ~

Intensity Color Result Result Flash

Recombination: Intensity * Color = Original OurOur ApproachApproach Our Approach Our Approach Shadow Detection/Treatment ShadowShadow CorrectionCorrection––WhyWhy??

No flash

Flash GlobalStraightforward white balance recombination in shadows ShadowShadow CorrectionCorrection––WhyWhy??

Global white balance in shadows Several artifacts: • at shadow boundary • inside shadows (color bleeding) ¾ shadows need to be corrected ! ... and detected ShadowShadow DetectionDetection

Flash

• Umbra • Penumbra ¾ Detection in two steps ShadowShadow DetectionDetection

Umbra detection • No direct light from flash

¾ Difference of the two photos ∆Ι reveals these regions

- =

Flash No-flash ∆Ι ShadowShadow DetectionDetection ? Umbra detection • Difference ∆Ι = light from the flash

• Goal: Find a threshold for ∆Ι

Automatic Threshold Detection (see paper)

∆Ι ShadowShadow DetectionDetection

Penumbra detection ¾ strong gradient at boundary • no strong gradient in no-flash image • connected to umbra

No-flash flash Umbra Penumbra ShadowShadow CorrectionCorrection

• Correct color and detail

Shadow mask ShadowShadow ColorColor CorrectionCorrection

Correct color

Wrong color Flash color Fill in shadow from similar surrounding

No-flash colors ShadowShadow ColorColor CorrectionCorrection No-flash Flash ShadowShadow ColorColor CorrectionCorrection No-flash Flash ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Outside shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Select pixel in shadow ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Corresponding pixel ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Spatial weights ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Spatial and Color weights ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Use shadow mask ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Use weights on flash color ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Replace shadow pixel ShadowShadow ColorColor CorrectionCorrection No-flash Flash

Inside shadow Outside shadow Proceed for all shadow pixels OurOur ApproachApproach OurOur ApproachApproach OverviewOverview

• Related Work • Our Approach • Results • Conclusion and Future Work ResultsResults

No-flash

Flash ResultsResults

No correction Our result ResultsResults

No-flash

Flash ResultsResults

No-flash

Flash ResultsResults

No-flash

Flash Our result ResultsResults

Our result ResultsResults

Our result EmphasizeEmphasize ForegroundForeground

Our result

Deduce “distance“ to camera Exploit 1/r2 flash intensity falloff (see paper) Emphasized foreground (Inverse)(Inverse) WhiteWhite BalanceBalance

No-flash Flash Retain warm tones from available lighting (see paper) ResultsResults

No-flash

Flash Result OverviewOverview

• Related Work • Our Approach • Results • Conclusion and Future Work ConclusionConclusion Improving photography in dim environments • Capture original lighting • Add sharpness/details No-flash • Cross bilateral filter

• Correct flash shadows Result

• Pseudo distance (emphasize foreground) • (Inverse) white balance FutureFuture WorkWork

• Local coherence for shadow detection • Different recombinations • Bilateral filter parameters • Infra-red flash AknowledgementsAknowledgements

• École Normale Supérieure - Paris • Deshpande Center • MIT-France •NSF CISE ThankThank youyou veryvery muchmuch forfor youryour attention!attention! Comparison:Comparison: SimilaritiesSimilarities Petschnigg et al. Eisemann et al.

• Detail transfer or relighting • Use bilateral filter • Joint or cross bilateral filter Comparison:Comparison: ColorColor

Petschnigg et al. Eisemann et al.

• Color from no-flash • Color from flash • More noisy • Less noisy • Less need for shadow • Need correction in shadow correction • White balance • Inverse white balance no-flash image flash image Comparison:Comparison: DifferencesDifferences Petschnigg et al. Eisemann et al.

Algorithmic differences • Color from no-flash • Color from flash • Semi-manual • Automatic shadows shadows/highlights

Additional tools • Continuous flash • Pseudo distance, • Red-eye removal emphasize foreground ThankThank youyou againagain forfor youryour attention!attention!