Unmixing-Based Soft Color Segmentation for Image Manipulation YAGIZ˘ AKSOY ETH Zurich¨ and Disney Research Zurich¨ TUNC¸ OZAN AYDIN and ALJOSAˇ SMOLIC´ Disney Research Zurich¨ and MARC POLLEFEYS ETH Zurich¨ We present a new method for decomposing an image into a set of soft color Additional Key Words and Phrases: Soft segmentation, digital composit- segments that are analogous to color layers with alpha channels that have ing, image manipulation, layer decomposition, color manipulation, color been commonly utilized in modern image manipulation software. We show unmixing, green-screen keying that the resulting decomposition serves as an effective intermediate image ACM Reference Format: representation, which can be utilized for performing various, seemingly unrelated, image manipulation tasks. We identify a set of requirements that Yagız˘ Aksoy, Tunc¸ Ozan Aydin, Aljosaˇ Smolic,´ and Marc Pollefeys. 2017. soft color segmentation methods have to fulfill, and present an in-depth Unmixing-based soft color segmentation for image manipulation. ACM theoretical analysis of prior work. We propose an energy formulation for Trans. Graph. 36, 2, Article 19 (March 2017), 19 pages. producing compact layers of homogeneous colors and a color refinement DOI: http://dx.doi.org/10.1145/3002176 procedure, as well as a method for automatically estimating a statistical color model from an image. This results in a novel framework for automatic and high-quality soft color segmentation that is efficient, parallelizable, and 1. INTRODUCTION scalable. We show that our technique is superior in quality compared to previous methods through quantitative analysis as well as visually through The goal of soft color segmentation is to decompose an image into a an extensive set of examples. We demonstrate that our soft color segments set of layers with alpha channels, such as in Figure 1(b). These lay- can easily be exported to familiar image manipulation software packages ers usually consist of fully opaque and fully transparent regions, as and used to produce compelling results for numerous image manipulation well as pixels with alpha values between the two extremes wherever applications without forcing the user to learn new tools and workflows. multiple layers overlap. Ideally, the color content of a layer should be homogeneous, and its alpha channel should accurately reflect the Categories and Subject Descriptors: I.3.8 [Computer Graphics]: Appli- color contribution of the layer to the input image. Equally impor- cations; I.4.6 [Image Processing and Computer Vision]: Segmentation— tant is to ensure that overlaying all layers yields the input image. Pixel classification; I.4.8 [Image Processing and Computer Vision]: Scene If a soft color segmentation method satisfies these and a number Analysis—Color of other well-defined criteria that we discuss in detail later, then 19 General Terms: Image Processing the resulting layers can be used for manipulating the image content conveniently through applying per-layer modifications. These im- age manipulations can range from subtle edits to give the feeling that the image was shot in a different season of the year (Figure 1(c)) Authors’ addresses: Y. Aksoy and M. Pollefeys, Department of Com- to more pronounced changes that involve dramatic hue shifts and puter Science, ETH Zurich, Universitatstrasse 6, CH-8092, Zurich, Switzer- replacing the image background (Figure 1(d)). In this article, we land; emails: {yaksoy, marc.pollefeys}@inf.ethz.ch; T. Aydin, Disney Re- propose a novel soft color segmentation method that produces a search Zurich, Stampfenbachstrasse 48, CH-8006, Zurich, Switzerland; powerful intermediate image representation, which in turn allows email: [email protected]; A. Smolic, Graphics Vision and Visu- a rich set of image manipulations to be performed within a unified alisation Group, School of Computer Science and Statistics, Trinity College framework using standard image editing tools. Dublin, Dublin 2, Ireland; email: [email protected]. Obtaining layers that meet the demanding quality requirements Permission to make digital or hard copies of part or all of this work for of image manipulation applications is challenging, as even barely personal or classroom use is granted without fee provided that copies are visible artifacts on individual layers can have a significant negative not made or distributed for profit or commercial advantage and that copies impact on quality when certain types of image edits are applied. That show this notice on the first page or initial screen of a display along with said, once we devise a soft color segmentation method that reliably the full citation. Copyrights for components of this work owned by others produces high-quality layers, numerous image manipulation tasks than ACM must be honored. Abstracting with credit is permitted. To copy can be performed with little extra effort by taking advantage of this otherwise, to republish, to post on servers, to redistribute to lists, or to use image decomposition. Importantly, the resulting layers naturally in- any component of this work in other works requires prior specific permission tegrate into the layer-based workflows of widely used image manip- and/or a fee. Permissions may be requested from Publications Dept., ACM, ulation packages. By using soft color segmentation as a black box, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 and importing the resulting layers into their favorite image manipu- (212) 869-0481, or [email protected]. lation software, users can make use of their already-existing skills. c 2017 ACM 0730-0301/2017/03-ART19 $15.00 While the traditional hard segmentation is one of the most active DOI: http://dx.doi.org/10.1145/3002176 fields of visual computing, soft color segmentation has received ACM Transactions on Graphics, Vol. 36, No. 2, Article 19, Publication date: March 2017. 19:2 • Y. Aksoy et al. Fig. 1. Our method automatically decomposes an input image (a) into a set of soft segments (b). In practice, these soft segments can be treated as layers that are commonly utilized in image manipulation software. Using this relation, we achieve compelling results in color editing (c), compositing (d), and many other image manipulation applications conveniently under a unified framework. surprisingly little attention so far. In addition to direct investigations ground/background probabilities [Yang et al. 2010b], and inter- of soft color segmentation [Tai et al. 2007; Tan et al. 2016], certain active image segmentation utilizing soft input constraints [Yang natural alpha matting and green-screen keying methods presented et al. 2010a]. In fact, generally speaking, even the traditional k- soft color segmentation methods—without necessarily calling them means clustering algorithm can be considered as a soft segmen- as such—as a component in their pipeline. While it may seem at a tation method, as it computes both a label as well as a confi- first glance that one can simply use any of these previous methods dence value for each point in the feature space [Tai et al. 2007]. for practical and high-quality photo manipulation, a closer look In contrast to these approaches, we seek to compute proper al- reveals various shortcomings of the currently available soft color pha values rather than arbitrarily defined confidence values or segmentation methods. To this end, in this article, we provide a probabilities. theoretical analysis of the problems with previous work, which The goals of our method are similar to those of the alternating we also supplement with quantitative evaluation, as well as visual optimization soft color segmentation technique [Tai et al. 2007] examples that support our arguments. and the decomposition technique via RGB-space geometry [Tan We address the two main challenges of soft color segmentation: et al. 2016], which decompose input images into a set of soft devising a color unmixing scheme that results in high-quality soft color segments and demonstrate their use in various image manip- color segments and automatically determining a content-adaptive ulation applications. A number of techniques proposed in natural color model from an input image. We propose a novel energy for- matting and green-screen keying literature, such as the ones by Sin- mulation that we call sparse color unmixing (SCU) that decomposes garaju and Vidal [2011], Chen et al. [2013], and Aksoy et al. [2016], the image into layers of homogeneous colors. The main advantage can also be regarded as soft color segmentation methods. While the of SCU when compared to similar formulations used in different goal of natural matting is to compute the alpha channel of the fore- applications such as green-screen keying [Aksoy et al. 2016] is ground in an image, the aforementioned methods can also be used that it produces compact color segments by favoring fully opaque to obtain soft color layers from the input image. Our method has or transparent pixels. We also enforce spatial coherency in opacity several advantages when compared to these works in terms of the channels and accordingly propose a color refinement procedure that quality of the opacity channels, the color content of individual layers is required for preventing visual artifacts while applying image ed- and the absence of need for user input. We present an in-depth dis- its. We additionally propose a method for automatically estimating cussion of our differences to and the shortcomings of these methods a color model corresponding to an input image, which comprises in Sections 4 and 6. a set of distinct and representative color distributions. Our method While we focus on soft color segmentation for the purpose of determines the size of the color model automatically in a content image manipulation, other methods in the literature have been pro- adaptive manner. We show that the color model estimation can ef- posed that utilize spatial information [Levin et al. 2008b; Tai et al. ficiently be performed using our novel projected color unmixing 2005], as well as texture [Yeung et al.