Pixelated Image Abstraction 1, 1 2 3 1,4 1 Timothy Gerstner ⇤ Doug DeCarlo Marc Alexa Adam Finkelstein Yotam Gingold Andrew Nealen 1Rutgers University 2TU Berlin 3Princeton University 4Columbia University (a) original (b) naive nearest (c) naive cubic (d) superpixels (e) our result Figure 1: Pixel art images simultaneously use very few pixels and a tiny color palette. Attempts to represent image (a) using only 22 32 pixels and 8 colors using (b) nearest-neighbor or (c) cubic downsampling (both followed by median cut color quantization), result in detail⇥ loss and blurriness. We optimize over a set of superpixels (d) and an associated color palette to produce output (e) in the style of pixel art. Abstract pixel art to convey information on compact screens. Companies like Coca-Cola, Honda, Adobe, and Sony use pixel art in their We present an automatic method that can be used to abstract high advertisements [Vermehr et al. 2012]. It is used to make icons for resolution images into very low resolution outputs with reduced desktops and avatars for social networks. While pixel art stems color palettes in the style of pixel art. Our method simultaneously from the need to optimize imagery for low resolution displays, it has solves for a mapping of features and a reduced palette needed to emerged as a contemporary art form in its own right. For example, construct the output image. The results are an approximation to the it has been featured by MoMA, and there are a number of passionate results generated by pixel artists. We compare our method against online communities devoted to it. The “Digital Orca” by Douglas the results of a naive process common to image manipulation Coupland is a popular sight at the Vancouver Convention Center. programs, as well as the hand-crafted work of pixel artists. Through France recently was struck by a “Post-it War”1, where people use a formal user study and interviews with expert pixel artists we show Post-It notes to create pixel art on their windows, competing with that our results offer an improvement over the naive methods. their neighbors across workplaces, small businesses, and homes. What makes pixel art both compelling and difficult is the limitations CR Categories: I.3.3 [Computer Graphics]: Picture/Image imposed on the medium. With a significantly limited palette and Generation— [I.3.4]: Computer Graphics—Graphics Utilities resolution to work with, the task of creating pixel art becomes carefully choosing the set of colors and placing each pixel such Keywords: pixel art, image abstraction, non-photorealistic ren- that the final image best depicts the original subject. This task is dering, image segmentation, color quantization particularly difficult as pixel art is typically viewed at a distance where the pixel grid is clearly visible, which has been shown to Links: DL PDF contribute to the perception of the image [Marr and Hildreth 1980]. As seen in Figure 2, creating pixel art is not a simple mapping 1 Introduction process. Features such as the eyes and mouth need to be abstracted and resized in order to be represented in the final image. The We see pixel art every day. Modern day handheld devices such as end product, which is no longer physically accurate, still gives the the iPhone, Android devices and the Nintendo DS regularly utilize impression of an identifiable person. However, few, if any methods exist to automatically create effective ⇤e-mail:[email protected] pixel art. Existing downsampling methods, two of which are shown in Figure 1, do not accurately capture the original subject. Artists often turn to making pieces by hand, pixel-by-pixel, which can take a significant amount of time and requires a certain degree of skill not easily acquired by novices of the art. Automated and semi-automated methods have been proposed for other popular art forms, such as line drawing [DeCarlo et al. 2003; Judd et al. 2007] and painting [Gooch et al. 2002]. Methods such as [DeCarlo and Santella 2002] and [Winnemoller¨ et al. 2006] not only abstract images, but do so while retaining salient features. 1http://www.postitwar.com/ Mass-constrained deterministic annealing (MCDA) [Rose 1998] is a method that uses a probabilistic assignment while clustering. Similar to k-means, it uses a fixed number of clusters, but unlike k-means it is independent of initialization. Also, unlike simulated annealing [Kirkpatrick et al. 1983], it does not randomly search the solution space and will converge to the same result every time. We use an adapted version of MCDA for color palette optimization. Puzicha et al. [2000] proposed a method that reduces the palette of an image and applies half-toning using a model of human visual perception. While their method uses deterministic annealing and the CIELAB space to find a solution that optimizes both color reduction and dithering, our method instead emphasizes palette Figure 2: “Alice Blue” and “Kyle Red” by Alice Bartlett. Notice reduction in parallel with the reduction of the output resolution. how faces are easily distinguishable even with this limited resolu- tion and palette. The facial features are no longer proportionally Kopf and Lischinski [2011] proposed a method that extracts vector accurate, similar to deformation in a caricature. art representations from pixel art. This problem is almost the inverse of the one presented in this paper. However, while their We introduce an automated process that transforms high resolution solution focuses on interpolating unknown information, converting images into low resolution, small palette outputs in a pixel art style. an image to pixel art requires compressing known information. At the core of our algorithm is a multi-step iterative process that Finally, we show that with minor modification our algorithm can simultaneously solves for a mapping of features and a reduced produce “posterized” images, wherein large regions of constant palette to convert an input image to a pixelated output image. In the color are separated by vectorized boundaries. To our knowledge, first part of each iteration we use a modified version of an image little research has addressed this problem, though it shares some segmentation proposed by Achanta et al. [2010] to map regions of aesthetic concerns with the artistic thresholding approach of Xu the input image to output pixels. In the second step, we utilize and Kaplan [2007]. an adaptation of mass-constrained deterministic annealing [Rose 1998] to find an optimal palette and its association to output pixels. These steps are interdependent, and the final solution is 3 Background an optimization of both the physical and palette sizes specified by the user. Throughout this process we utilize the perceptually Our technique for making pixel art builds upon two existing tech- uniform CIELAB color space [Sharma and Trussell 1997]. The niques, which we briefly describe in this section. end result serves as an approximation to the process performed by SLIC. Achanta et al. [2010] proposed an iterative method to pixel artists. Aside from assisting a class of artists in this medium, segment an image into regions termed “superpixels.” The algorithm applications for this work include automatic and semi-automatic is analogous to k-means clustering [MacQueen 1967] in a five design of low-resolution imagery in handheld, desktop, and online dimensional space (three color and two positional), discussed for contexts like Facebook and Flickr, wherever iconic representations example in Forsyth and Ponce [2002]. Pixels in the input image pi of high-resolution imagery are used. are assigned to superpixels ps by minimizing 2 Related Work N d(pi,ps)=dc(pi,ps)+m dp(pi, ps) (1) r M One aspect of our problem is to reproduce an image as faithfully as possible while constrained to just a few output colors. Color where dc is the color difference, dp is the positional difference, M quantization is a classic problem wherein a limited color palette is the number of pixels in the input image, N is the number of is chosen based on an input image for indexed color displays. superpixels, and m is some value in the range [0, 20] that controls A variety of methods were developed in the 1980’s and early the relative weight that color similarity and pixel adjacency have 1990’s prior to the advent of inexpensive 24-bit displays, for on the solution. The color and positional differences are measured example [Gervautz and Purgathofer 1990; Heckbert 1982; Orchard using Euclidean distance (as are all distances in our paper, unless and Bouman 1991; Wu 1992]. A similar problem is that of selecting otherwise noted), and the colors are represented in LAB color a small set of custom inks to be used in printing an image [Stollnitz space. Upon each iteration, superpixels are reassigned to the et al. 1998]. These methods rely only on the color histogram of the average color and position of the associated input pixels. input image, and are typically coupled to an independent dithering Mass Constrained Deterministic Annealing. MCDA [Rose 1998] (or halftoning) method for output in a relatively high resolution is a global optimization method for clustering that draws upon an image. In our problem where the spatial resolution of the output is analogy with the process of annealing a physical material. We use also highly constrained, we optimize simultaneously the selection this method both for determining the colors in our palette, and for and placement of colors in the final image. assigning one of these palette colors to each pixel—each cluster corresponds to a palette color. The problem of image segmentation has been extensively stud- ied. Proposed solutions include graph-cut techniques, such as MCDA is a fuzzy clustering algorithm that probabilistically assigns the method proposed by Shi and Malik [1997], and superpixel- objects to clusters based on their distance from each cluster.