Dissecting Image Crops Basile Van Hoorick Carl Vondrick Columbia University Columbia University New York, NY, USA New York, NY, USA [email protected] [email protected] Abstract The elementary operation of cropping underpins nearly every computer vision system, ranging from data augmen- tation and translation invariance to computational photog- raphy and representation learning. This paper investigates the subtle traces introduced by this operation. For exam- Figure 1: We show two infamous image crops, visualized by the ple, despite refinements to camera optics, lenses will leave red box. (left) An Ugandan climate activist had been cropped out behind certain clues, notably chromatic aberration and vi- of the photo before it was posted in an online news article, the gnetting. Photographers also leave behind other clues re- discovery of which sparked controversy [16]. (right) A news net- lating to image aesthetics and scene composition. We study work had cropped out a large stick being held by a demonstrator how to detect these traces, and investigate the impact that during a protest [14]. Cropping dramatically alters the message of cropping has on the image distribution. While our aim is the photographs. to dissect the fundamental impact of spatial crops, there are also a number of practical implications to our work, such context, altering the message of the image. Twitter’s auto- as revealing faulty photojournalism and equipping neural crop feature relied on a saliency prediction network that was network researchers with a better understanding of short- racially biased [10]. cut learning. Code is available at https://github.com/ The guiding question of this paper is to understand the basilevh/dissecting-image-crops. traces left behind from this fundamental operation. What impact does image cropping have on the visual distribution? 1. Introduction Can we determine when and how a photo has been cropped? Despite extensive refinements to the manufacturing pro- The basic operation of cropping an image underpins cess of camera optics and sensors, nearly every modern nearly every computer vision paper that you will be reading camera pipeline will leave behind subtle lens artefacts onto this week. Within the first few lectures of most introduc- the photos that it captures. For example, vignetting is tory computer vision courses, convolutions are motivated caused by a lens focusing more light at the center of the as enabling feature invariance to spatial shifts and crop- sensor, creating images that are slightly brighter in the mid- ping [52, 31,2]. Neural networks rely on image crops as arXiv:2011.11831v4 [cs.CV] 5 Sep 2021 dle than near its borders [36]. Chromatic aberration, also a form of data augmentation [28, 50, 21]. Computational known as purple fringing, is caused by the lens focusing photography applications will automatically crop photos each wave length differently [5]. Since these artefacts are in order to improve their aesthetics [47, 12, 60]. Predic- correlated with their spatial position in the image plane, tive models extrapolate pixels out from crops [51, 57, 55]. they cause image crops to have trace signatures. Even the latest self-supervised efforts depend on crops for Physical aberrations are not the only traces left behind contrastive learning to induce rich visual representations during the operation. Photographers will prefer to take pho- [13, 20, 45, 49]. tos of interesting objects and in canonical poses [53,4, 22]. This core visual operation can have a significant impact Aesthetically pleasing shots will have sensible composi- on photographs. As Oliva and Torralba told us twenty years tions that respect symmetry and certain ratios in the scene. ago, scene context drives perception [44]. Recently, image Violating these principles leaves behind another trace of the cropping has been at the heart of media disinformation. Fig- cropping operation. ure1 shows two popular photographs where the photogra- These traces are very subtle, and the human eye often pher or media organization spatially cropped out part of the cannot detect them, which makes studying and characteriz- speediness of videos [6], and visual chirality [34]. For ex- Achromatic lens Sensor plane ample, insight into image crops could enable detection of soft tampering operations, or spur developments to mitigate shortcut learning. In Section 2, we briefly review related work in lens im- perfections and self-supervised learning. In Section 3, we Transverse CA describe the principles that govern our curation of an appro- priate dataset. In Section 4, we learn a representation for de- tecting absolute patch locations, and subsequently motivate the complete network architecture that incorporates both ag- Lateral CA gregated patch-wise and global context. In Section 5, we (a) Lens with transverse and longitudinal chromatic aberration. In this il- analyze what our model has learned by measuring its per- lustration, the red and blue channels are aligned (hence the magenta rays), but green-colored light is magnified differently in addition to having a sep- formance under various controlled circumstances. Finally, arate in-focus plane. in Section 6, we visualize and interpret how the model ad- dresses several interesting examples of image crops. 2. Background and Related Work Optical aberrations. No imaging device is perfect, and every step in the imaging formation pipeline will leave traces behind onto the final picture. The origins of these signatures range from the physics of light in relation to the camera hardware, to the digital demosaicing and compres- (b) Close-up of two photos, revealing visible transverse chromatic aberra- sion algorithms used to store and reconstruct the image. tion (TCA) artefacts. Lenses typically suffer from several aberrations, including chromatic aberration, vignetting, coma, and radial distor- Figure 2: The origin behind, and examples of, chromatic aberra- tion [26,5, 33, 24]. As shown in Figure 2a, chromatic tion. aberration is manifested in two ways: transverse (or lat- eral) chromatic aberration (TCA) refers to the spatial dis- ing them challenging. However, neural networks are excel- crepancies in focus points across color channels perpendic- lent at identifying these patterns. Indeed, extensive effort ular to the optical axis, while longitudinal chromatic aber- goes into preventing neural networks from learning such ration (LCA) refers to shifts in focus along the optical axis shortcuts enabled by image crops [15, 43]. instead [23, 24]. TCA gives rise to color channels that ap- In this paper, we flip this around and declare that these pear to be scaled slightly differently relative to each other, shortcuts are not bugs, but instead an opportunity to dis- while LCA causes the distance between the focal surface sect and understand the subtle clues left behind from image and the lens to be frequency-dependent, such that the degree cropping. Capitalizing on a large, high-quality collection of blurring varies among color channels. Chromatic aberra- of natural images, we train a convolutional neural network tion can be leveraged to extract depth maps from defocus to predict the absolute spatial location of a patch within an blur [19, 54, 24], although the spatial sensitivity of these image. This is only possible if there exist visual features cues is often undesired [15, 42, 43, 40]. TCA is leveraged that are not spatially invariant. Our experiments analyze the by [59] to measure the angle of an image region relative to types of features that this model learns, and we show that the lens as a means to detect cropped images. We instead it is possible to detect traces of the cropping operation. We present a learning-based approach that discovers additional can also use the discovered artefacts, along with semantic clues without the need for carefully tailored algorithms. information, to recover where the crop was positioned in Patch localization. While one of the first major works in the original sensor plane. self-supervised representation learning focused on predict- While the aim of this paper is to analyze the fundamental ing the relative location of two patches among eight possi- traces of image cropping in order to question conventional ble configurations [15], it was also discovered that the abil- assumptions about translational invariance and the crucial ity to perform absolute localization seemed to arise out of role of data augmentation pervading the field, we believe chromatic aberration. For the best-performing 10% of im- our investigation could have a large practical impact as well. ages, the mean Euclidean distance between the ground truth Historically, asking fundamental questions has spurred sig- and predicted positions of single patches is 31% lower than nificant insight into core computer vision problems, such chance, and this gap narrowed to 13% if every image was as invariances to scale [9], asymmetries in time [46], the pre-processed to remove color information along the green- 2 magenta axis. Although there are reasons to believe that dataset. Our underlying dataset has around 700; 000 high- modern network architectures might perform better, these resolution photos from Flickr, which were scraped during rather modest performance figures suggest a priori that the the fall of 2019. We impose several constraints on the train- attempted task is a difficult one. Note that the learnabil- ing images, most importantly that they should not already ity of absolute location is often regarded as a bug; treat- have been cropped and that they must maintain a constant, ments used in practice include random color channel drop- fixed aspect ratio and resolution. Appendix A describes this ping [15], projection [15], grayscale conversion [42, 43], selection and collection process in detail. jittering [42], and chroma blurring [40]. We generate image crops by first defining the crop rect- 4 Visual crop detection. In the context of forensics, al- angle (x1; x2; y1; y2) 2 [0; 1] as the relative boundaries most all existing research has centered around ’hard’ tam- of a cropped image within its original camera sensor plane, pering such as splicing and copy-move operations.