Fast and Stable Color Balancing for Images and Augmented Reality Thomas Oskam 1,2 Alexander Hornung 1 Robert W. Sumner 1 Markus Gross 1,2 1 Disney Research Zurich 2 ETH Zurich Abstract This paper addresses the problem of globally balanc- ing colors between images. The input to our algorithm is a sparse set of desired color correspondences between a source and a target image. The global color space trans- formation problem is then solved by computing a smooth Source Image Target Image Color Balanced vector field in CIE Lab color space that maps the gamut of the source to that of the target. We employ normalized ra- dial basis functions for which we compute optimized shape parameters based on the input images, allowing for more faithful and flexible color matching compared to existing RBF-, regression- or histogram-based techniques. Further- more, we show how the basic per-image matching can be Rendered Objects efficiently and robustly extended to the temporal domain us- Tracked Colors balancing Augmented Image ing RANSAC-based correspondence classification. Besides Figure 1. Two applications of our color balancing algorithm. Top: interactive color balancing for images, these properties ren- an underexposed image is balanced using only three user selected der our method extremely useful for automatic, consistent correspondences to a target image. Bottom: our extension for embedding of synthetic graphics in video, as required by temporally stable color balancing enables seamless compositing applications such as augmented reality. in augmented reality applications by using known colors in the scene as constraints. 1. Introduction even for different scenes. With today’s tools this process re- quires considerable, cost-intensive manual efforts. Similar Pablo Picasso highlighted the importance of color in issues arise in augmented reality applications, where com- painting with the famous quote, “They’ll sell you thousands puter generated graphics must be embedded seamlessly and of greens. Veronese green and emerald green and cadmium in real-time into a video stream. Here, achieving consis- green and any sort of green you like; but that particular tent colors is critical but highly challenging, since white- green, never.” Today, the same struggle to capture the per- balance, shutter time, gamma correction, and other factors fect nuance of color lives on in digital photography and may continually change during acquisition and cannot be image-based applications. The colors that are ultimately perfectly controlled or addressed through calibration alone. recorded by both consumer-grade and high-end digital cam- Therefore, an efficient correction and adaptation of colors eras depend on a plethora of factors and are influenced that is agnostic to the underlying capture hardware is highly by complex hardware and software processing algorithms, desirable in such AR-related applications. However, aside making the precise color quality of captured images difficult for a few preliminary efforts [13, 14] practically applicable to predict and control. As a result, one generally has to rely solutions to this problem are yet to be developed. on post-processing algorithms to fix color-related issues. This paper contributes a real-time color balancing algo- This problem is generally known as color balancing or rithm that is able to globally adapt the colors of a source color grading, and plays a central role in all areas involving image to match the look of a target image, requiring only capturing, processing and displaying image data. In digi- a sparse set of color correspondences. Fundamentally, our tal photography and filmmaking, specifically trained artists approach is based on radial basis function (RBF) interpo- work hard to achieve consistent colors for images captured lation. However, we show that, for an optimal RBF-based with different cameras, often under varying conditions and color matching, it is crucial to optimize for the shape of the 4321 employed basis functions. As a second main contribution, of user controlled edits is typically targeted towards local we then extend the per-image computation to be temporally color changes. Our work, in contrast, targets global color consistent, enabling the application of the basic balancing balancing using only sparse and possibly incomplete input. mechanism in video-based applications such as augmented Li et al. [15] provide an approach for image and video reality. A key component to the success of this system is editing with instant results. They employ radial basis func- a reliable RANSAC-based classification algorithm that se- tion interpolation in a feature space to propagate sparse ed- lects a temporally stable set of correspondences used for the its locally. In contrast to this work, we are formulating the basic color balancing. We present a fast GPU-assisted im- problem as a global color space transformation independent plementation of our color balancing algorithm that achieves from the image space and we optimize the shape of basis run times of 1ms and below on real-world applications. In functions to achieve color balancing behavior found in ex- this paper and the accompanying video, we show results for ample images. We show that the basis function used by Li et interactive color editing of images as well as for improved al. is suboptimal for the purpose of global image balancing. visual quality and consistency of video and computer gen- Color calibration and balancing: Color calibration is erated images in augmented reality applications. the process of computing the color distribution function of a camera to allow recorded images to be consistently colored. 2. Related Work Adams et al. [3] gives an overview of standard camera cal- In this section, we discuss related work on histogram ibration techniques using linear transformations. Usually, matching, user controlled color transfer, and color calibra- a 3x3 matrix is computed using regression. A related area tion. is color consistency. Humans perceive colors to be consis- Statistics and Histogram Matching: This is the pro- tent even under different lighting conditions. This property cess of transferring the color distribution statistics from one is not available for digital images. Colors have different image to another image, pioneered by Reinhard et al. [20]. numerical values under different illuminations. In order to They proposed computing the mean and standard devia- counter this issue, different algorithms have been proposed. tion of the color distribution in the color space. Then, by An overview over different algorithms is provided by Ar- scaling and translating the parameters onto the ones of the gawal et al. [4] and Cohen [7]. In contrast to our approach, target image, they achieve automatic color transfer. The many of these methods do not apply additional information approach has been applied to histograms and extended to such as correspondences or known colors in a scene to bal- achieve a mapping in different color spaces [25], minimum ance the images. Therefore, these methods are less interac- cost linear mapping [19], and per-region execution of the tive and less robust when applied to videos. algorithm [11]. Further extensions have been proposed that There is also significant work that enhances the quality optimize the images after matching to avoid discontinuities of imagery using a database of example images [8, 12, 21, in color gradients [18, 26]. 23]. The work of Yang et al. [27] achieve color compensa- While statistics and histogram matching methods for au- tion by replicating the non-linear relationship between the tomatic color transfer can produce very appealing results, camera color and luminance. Wang et al. [24] transfer color they provide only limited control. Direct control over color and tone style of expert photographed images onto the im- changes is often required in image editing and difficult to ages from cheap cameras by learning the complex transfor- achieve with these methods. mation encoded in local descriptors. Recent work of Lin Color transfer with user interaction: Abadpour et et al. [16] revisits the radiometric calibration of cameras al. [2] first proposed an image color transfer algorithm with by investigating 10,000 images from over 30 cameras and user interaction. In their work, the user is allowed to de- introduces a new imaging model for computing the color note corresponding regions between images. Colors are response curves. Our parameter optimization of the radial then locally matched using fuzzy principal component anal- basis functions is also based on extracting image proper- ysis. Other methods have been proposed that aid the user in ties form data sets. However, we design our functions to local color matching [17, 22]. The user is allowed to se- be sparsely sampled so that they provide appealing results lect a local region, and the actions are propagated through- when interpolated in order to considerably increase perfor- out adjacent parts of the image. An et al. [6] propose a mance. scribble-based framework for color transfer between im- Color Balancing in Augmented Reality: Only recently ages. Fast and interactive methods for editing images and there has been research in improving the compositing of im- video have also been proposed. An et al. [5] propagate lo- ages in augmented reality. Klein et al. [13] examine and cal edits by assisting a user with iteratively refining rough reproduce several image degradation effects of digital cam- edits to areas of spatial and appearance similarity. Farbman eras to increase the realism of augmentation. However, they et al. [9] propose the use of diffusion maps to propagate are not considering color shifts due to camera parameter image editing information. In these approaches the impact changes. The works of Knecht et al. [14] propose to bal- 4322 1 avg. function norm. Shepard deviation Gaussian inv. quadric 0 Color Space Distance 0 30 Image Gamut + Smooth Transformed Correspondences Vector Field Image Gamut Figure 3. Plots of the basis functions we consider for our color transfer. On the left, the three functions are shown in relative scale.