Texture Synthesis for Digital Painting John Peter Lewis Massachusetts Institute of Technology Abstract The problem of digital painting is considered from a signal processing viewpoint, and is reconsidered as a problem of directed texture synthesis. It is an important characteristic of nat- ural texture that detail may be evident at many scales, and the de- tail at each scale may have distinct characteristics. A “sparse con- volution” procedure for generating random textures with arbitrary spectral content is described. The capability of specifying the tex- ture spectrum (and thus the amount of detail at each scale) is an improvement over stochastic texture synthesis processes which are scalebound or which have a prescribed 1/f spectrum. This spectral texture synthesis procedure provides the basis for a digital paint system which rivals the textural sophistication of traditional artistic media. Applications in terrain synthesis and texturing computer- rendered objects are also shown. CR Categories and Subject Descriptors: I.3.3 [Computer Graph- ics]: Picture /Image generation; I.3.7 [Computer Graphics]: Three Figure 2: A watercolor brushstroke. Dimensional Graphics and Realism - color, shading, shadowing , and texture. Additional Key Words and Phrases: texture synthesis, painting, ter- rain models, fractals. may be said that , whereas the paint program ignores the nuances of the artists gesture to produce a uniform, analytical mark, an ex- pressive (physical) painting medium amplifies the artists gesture. Figure 2 is presented to emphasize that digital painting is far from 1 Introduction a “solved problem”, as a single brushstroke in an traditional (and ostensibly primitive) medium produces a quantity and character of textural information which would be difficult to digitally generate. The motivation for texture synthesis lies in the observation that the absence of texture is often responsible for the characteristic stark Computer artists make the point that digital simulation of traditional or geometric appearance of computer generated images, and more artistic media is not an appropriate goal. While this is agreed, it is generally, in the recognition that current image synthesis techniques also inappropriate to identify the intrinsic character of computer im- are inappropriate for rendering many natural phenomena. [Reeves, ages with hardware and software primitives which were originally 1983]. developed to accommodate information display rather than image synthesis. Since this medium is essentially defined in software, or- The lack of texture is felt particularly in digital painting. While ganic effects are no less intrinsic than geometrical effects though it appears initially that a “put-that-color-there” paint program (in the former will require new techniques. which any region of the display screen may be shaded with any desired color) is a comprehensive definition of painting, its limi- The ultimate form of computer-resisted image creation will cer- tations are severe, as can be seen by the following argument: The tainly depart from the painting metaphor. The immediate potential number of distinct color regions in a digitized (non-computer) im- of a programmable painting is nevertheless attractive, remember- age is generally found to be a significant fraction of the total number ing that a good illustrator can render an arbitrary scene in the time of pixels. In a typical medium resolution (250,000 pixel) digitized sometimes required to ray trace relatively simple scenes. As an ini- image there are on the order of 105 distinct color regions, depend- tial approximation the problem of developing an expressive digital ing on the color resolution of the digitizer and image buffer. This painting medium will be viewed as a problem of texture synthesis. can be visually demonstrated by applying a unique mapping of the Texture is defined here as object detail which we do not care to pixel values of a digitized image onto relatively incoherent or ran- explicitly specify though some of its aggregate characteristics may dom values [Figure 1]. Thus, on the order of 105 manual operations be known. This definition encompasses many natural phenomena would be required to digitally paint a medium resolution image of as well as the small-scale detail usually associated with the term. comparable complexity using a put-that-color-there procedure. As The size and location of a geographic feature may be of interest, a result many digital paintings incorporate digitized images or use for example, but it is not desirable to design or measure its surface geometrical patterns to achieve visual complexity, while in other beyond a certain scale. The textural threshold is the scale beyond cases the jaggie, low resolution appearance resulting from the put- which image detail may be replaced by other, similar detail without that-color-there procedure provides the “digital” character of the affecting the viewers perception of subject. painting. While this is a broad definition of texture, the consideration of tex- In contrast much of the detail in traditional media is generated as ture as an essentially planar phenomenon is restrictive. It will be a desired effect of the painting process. Paints, brushes, and paint- seen that texture fields may be interpreted as height fields for ter- ing surfaces are carefully selected for their characteristic effects. It rain synthesis, and the development of three-dimensional general- Computer Graphics Volume 18, Number 3 July 1984 Copyright ACM, see last page 2 izations of planar texture synthesis methods is conceivable. 2 Texture Models Texture models trade depth for generality in attempting to repro- duce the appearance of a texture without considering its underlying structure. Natural textures exhibit local variations which are mod- elled as a random process although the responsible phenomena are not random from other viewpoints. A number of stochastic texture models have been proposed and are discussed in the references. A brief description of several important models follows. Time-series modelling considers periodicities in the row scan of an image; this approach is fundamentally limit in that it cannot easily describe the structure of the texture in the dimensions(s) perpendic- Figure 3: A field of pebbles. ular to the scan. Planar random point processes generate a texture consisting of a distribution of points on a planar background. Bombing processes generalize the planar point process by replacing the point primitive with a shape possessing orientation, color and other characteristics [Schacter & Ahuja, 1979]. Cell growth processes partition the plane into cells. The Voronoi tesselation is an exemplary cell grown process which distributes cell nuclei by a point process; a cell is defined as the boundary of the collection of points which are closest to a particular nucleus. The resulting texture resembles natural cellular structures. [Mezei, Puzin, and Conroy]. Syntactic texture models equate tokens of a formal grammar with structural primitives of the texture. A highly structured but nonde- terministic texture may be generated if probabilities are assigned to the rewrite or expansion rules of the grammar [Lu and Fu, 1978]. Figure 4: Windowed plot of the grey levels in one row of Figure Two-dimensional Markove random field models consider the condi- 3 (bottom), auto correlation, power spectrum . and ten pole auto tional probability of color values over a sample region of the texture regressive spectrum estimate obtained by linear prediction (top). [Hassner and Sklansky, 1980; Cross and Jain, 1983]. The Marko- vian property is modified by defining the transition probabilities on a neighborhood of adjacent or non-adjacent pixels, so the term de- scribes a finite memory process rather than a planar Markov chain. character while close examination can reveal only the constituent this method has achieved good results in simulating prototype tex- pixels of the texture. tures. Unfortunately it is not practical for textures sampled at high The Brownian model of terrain does have the property that closer resolution since the number of conditional probabilities is Gs if G observation yields more detail. This application of Brownian mo- is the number of quantized color levels and S is the number of ad- tion resulted from Mandelbrots interest in fractals, a collection of jacent pixels to be considered. Several authors have used Brownian mathematical objects united by the criterion of recursive self- sim- sheets to model rugged terrain [Fournier, Fussell, and Carpenter, ilarity: the objects are defined as analytic or probabilistic functions 1983; Mandelbrot, 1982]. Though not originally conceived of as of scale [Mandelbrot, 1982]. Brownian motion can be considered a texture model, the height values of the Brownian sheet may be self-similar since the statistical moments of any sample are similar reinterpreted as color values of a planar texture. to any other sample, when adjusted by a scaling factor. In fact un- modified Brownian motion generates a very rugged terrain. Man- delbrot has proposed a revised model in which B(t) is filtered to 3 Texture and Scale adjust its amplitude spectrum to f-q (the unfiltered B(t) has the pa- rameter q=1). The filtering is conceived as a Riemann-Liouville integral of B(t) [Oldham and Spanier, 1974]: (equation here) (The An important characteristic of natural texture is that textural de- dB(x) form of the integral is used to sidestep the problems of for- tail may occur at more than one scale. Detail may be noticeable at mally defining the derivative of B(t) or the integral thereof using all scales from the textural threshold to the limit of visual activity. the limit calculus). This is seen to be a convolution of the derivative For example a field of pebbles has an overall shape; closer inspec- of B(t) (white noise) with a filter h(t)+t q-1. The (equation) scaling tion shows the contour of individual pebbles, each of which has its factor cancels the numerator in the transform of h(t), (equation here) own surface texture [Figure 3]. With the exception of the Brownian (for integer values n).