• Abstract and Figures
• Public Full-text

Content Authentication for Neural Imaging Pipelines: End-to-end Optimization of Photo Provenance in Complex Distribution Channels Pawe lKorus1,2 and Nasir Memon1 New York University1, AGH University of Science and Technology2 http://kt.agh.edu.pl/~korus Abstract which exploits inherent statistical properties result- ing from the photo acquisition pipeline. While the Forensic analysis of digital photo provenance relies former provides superior performance and allows for on intrinsic traces left in the photograph at the time advanced protection features (like precise tampering of its acquisition. Such analysis becomes unreliable af- localization [38], or reconstruction of tampered con- ter heavy post-processing, such as down-sampling and tent [25]), it failed to gain widespread adoption due to re-compression applied upon distribution in the Web. the necessity to generate protected versions of the pho- This paper explores end-to-end optimization of the en- tographs, and the lack of incentives for camera vendors tire image acquisition and distribution workflow to fa- to modify camera design to integrate such features [4]. cilitate reliable forensic analysis at the end of the dis- Passive forensics, on the other hand, relies on our tribution channel. We demonstrate that neural imag- knowledge of the photo acquisition pipeline, and statis- ing pipelines can be trained to replace the internals of tical artifacts introduced by its successive steps. While digital cameras, and jointly optimized for high-fidelity this approach is well suited for potentially analyzing photo development and reliable provenance analysis. In any digital photograph, it often falls short due to the our experiments, the proposed approach increased im- complex nature of image post-processing and distribu- age manipulation detection accuracy from 45% to over tion channels. Digital images are not only heavily com- 90%. The findings encourage further research towards pressed, but also enhanced or even manipulated before, building more reliable imaging pipelines with explicit during or after dissemination. Popular images have provenance-guaranteeing properties. many online incarnations, and tracing their distribu- tion and evolution has spawned a new field of image phylogeny [13, 14] which relies on visual differences be- 1. Introduction tween multiple images to infer their relationships and editing history. However, phylogeny does not provide Ensuring integrity of digital images is one of the any tools to reason about the authenticity or history most challenging and important problems in multime- of individual images. Hence, reliable authentication of dia communications. Photographs and videos are com- real-world online images remains untractable [39]. monly used for documentation of important events, and At the moment, forensic analysis often yields useful as such, require efficient and reliable authentication results in near-acquisition scenarios. Analysis of native protocols. Our current media acquisition and distribu- images straight from the camera is more reliable, and tion workflows are built with entertainment in mind, even seemingly benign implementation details - like the and not only fail to provide explicit security features, rounding operators used in the camera’s image signal but actually work against them. Image compression processor [1] - can provide useful clues. Most forensic standards exploit heavy redundancy of visual signals traces quickly become unreliable as the image under- to reduce communication payload, but optimize for hu- goes further post-processing. One of the most reliable man perception alone. Security extensions of popular tools at our disposal involves analysis of the imaging standards lack in adoption [20]. sensor’s artifacts (the photo response non-uniformity Two general approaches to assurance and verifica- pattern) which can be used both for source attribution tion of digital image integrity include [24, 30, 32]: (1) and content authentication problems [10]. pro-active protection methods based on digital signa- In the near future, rapid progress in computational tures or watermarking; (2) passive forensic analysis imaging will challenge digital image forensics even in 8621 Cross entropy loss Post­proc. A Classification labels: pristine, post­proc. A ­ Z Post­proc. B RAW Developed JPEG NIP MUX Downscale FAN = measurements image Compression Post­proc. L2 loss Y = Desired Post­proc. output image Z Acquisition Post-processing / editing Distribution Analysis Figure 1. Optimization of the image acquisition and distribution channel to facilitate photo provenance analysis. The neural imaging pipeline (NIP) is trained to develop images that both resemble the desired target images, but also retain meaningful forensic clues at the end of complex distribution channels. near-acquisition authentication. In the pursuit of bet- ited by the necessity to modify camera hardware, our ter image quality and convenience, digital cameras and approach exploits the flexibility of neural networks to smartphones employ sophisticated post-processing di- learn relevant integrity-preserving features within the rectly in the camera, and soon few photographs will expressive power of the model. We believe that with resemble the original images captured by the sensor(s). solid security-oriented understanding of neural imaging Adoption of machine learning has recently challenged pipelines, and with the rare opportunity of replacing many long-standing limitations of digital photogra- the well-established and security-oblivious pipeline, we phy, including: (1) high-quality low-light photogra- can significantly improve digital image authentication phy [9]; (2) single-shot HDR with overexposed con- capabilities. tent recovery [15]; (3) practical high-quality digital We aim to inspire discussion about novel camera de- zoom from multiple shots [37]; (4) quality enhancement signs that could improve photo provenance analysis ca- of smartphone-captured images with weak supervision pabilities. We demonstrate that it is possible to opti- from DSLR photos [18]. mize an imaging pipeline to significantly improve de- These remarkable results demonstrate tangible ben- tection of photo manipulation at the end of a complex efits of replacing the entire acquisition pipeline with real-world distribution channel, where state-of-the-art neural networks. As a result, it will be necessary to deep-learning techniques fail. The main contributions investigate the impact of the emerging neural imaging of our work include: pipelines on existing forensics protocols. While impor- tant, such evaluation can be seen rather as damage 1. The first end-to-end optimization of the imaging assessment and control and not as a solution for the pipeline with explicit photo provenance objectives; future. We believe it is imperative to consider novel 2. The first security-oriented discussion of neural possibilities for security-oriented design of our cameras imaging pipelines and the inherent trade-offs; and multimedia dissemination channels. In this paper, we propose to optimize neural imaging 3. Significant improvement of forensic analysis per- pipelines to improve photo provenance in complex dis- formance in challenging, heavily post-processed tribution channels. We exploit end-to-end optimization conditions; of the entire photo acquisition and distribution chan- 4. A neural model of the entire photo acquisition and nel to ensure that reliable forensics decisions can be distribution channel with a fully differentiable ap- made even after complex post-processing, where clas- proximation of the JPEG codec. sical forensics fails (Fig. 1). We believe that immi- nent revolution in camera design creates a unique op- To facilitate further research in this direction, and portunity to address some of the long-standing limi- enable reproduction of our results, our neural imaging tations of the current technology. While adoption of toolbox is available at https://github.com/pkorus/ digital watermarking in image authentication was lim- neural-imaging. 8622 2. Related Work of the emerging neural imaging pipelines, or to exploit this opportunity to improve photo reliability. Trends in Pipeline Design Learning individual steps of the imaging pipeline (e.g., demosaicing) has a long history [21] but regained momentum in the re- 3. End-to-end Optimization of Photo cent years thanks to adoption of deep learning. Nat- Provenance Analysis urally, the research focused on the most difficult op- Digital image forensics relies on intrinsic statisti- erations, i.e., demosaicing [17, 23, 34, 35] and denois- cal artifacts introduced to photographs at the time of ing [6, 26, 40]. Newly developed techniques delivered their acquisition. Such traces are later used for reason- not only improved performance, but also additional fea- ing about the source, authenticity and processing his- tures. Gharbi et al. proposed a convolutional neural tory of individual photographs. The main problem is network (CNN) trained for joint demosaicing and de- that contemporary media distribution channels employ noising [17]. A recent work by Syu et al. proposes to heavy compression and post-processing which destroy exploit CNNs for joint optimization of the color filter the traces and inhibit forensic analysis. array and a corresponding demosaicing filter [34]. The core of the proposed approach is to model the Optimization of digital camera design can go even entire acquisition and distribution channel, and opti- further. The recently proposed L3 model by Jiang et al.

Load more

  9

Copy Link