Chapter 1 NPR in the Wild Holger Winnemöller Abstract During the early years of computer graphics, the results were arguably not as realistic as the intended goal set forth. However, it was not until sometime later that non-realism was accepted as a goal worthwhile pursuing. Since then, NPR has matured considerably and found uses in numerous applications, ranging from movies and television, production tools, and games, to novelty and casual creativity apps on mobile devices. This chapter presents examples from each of these cate- gories within their historical and applied context. 1.1 Introduction In this chapter, we take a look at applications of NPR in the real world. This survey is neither intended to be comprehensive in scope nor in technical detail (for back- ground on specific techniques, please consult the paper references at the end of this chapter). Instead, we focus on representative techniques and applications of various NPR approaches that have found use in production software and entertainment. To further focus our attention, we shall use a somewhat more restrictive definition of NPR than might be applied elsewhere in this book. Specifically, we shall define NPR as “computer-enabled synthesis and tools for Art creation and reproduction”. That is, we include systems and tools that enable digital synthesis and creation of traditional artistic styles, but we exclude examples that are clearly non-realistic, yet are not obviously linked to any particular traditional art style (such as the movie “Tron”, 1982, which makes heavy use of non-realistic visual effects and computer graphics, but does not represent an established artistic style). The chapter starts with a historical review of artistic styles that were invented or refined as answers to certain production challenges, particularly print reproduc- Holger Winnemöller Adobe Systems, Inc., Seattle, USA. e-mail: [email protected] PREPRINT1 2 Holger Winnemöller tion. While modern printing techniques obviate the need for traditional reproduction methods, their artistic merits have prompted NPR researchers to revive them in the digital age. Next, uses of NPR for entertainment, namely movies and games, are discussed, followed by visualization and presentation applications. Finally, a case study focusing on design considerations covers an example of NPR on mobile de- vices. The chapter closes with some forward looking suggestions to further increase the impact of NPR, particularly to reach a larger audience of artistically untrained users. 1.2 Production Tools NPR has found use in Art and other production environments in a number of ways. For example, there are art creation tools, such as the digital paint brushes discussed in Chapter ??. These art creation tools mimic tools in the real world (e.g. a painter’s brush), but often add some unique digital affordances, such as undo, copy/paste, scaling, etc. In many cases, art creation tools straddle the border between NPR and realistic techniques, as they commonly rely on highly realistic, physically-based simulations, or approximations thereof. Another production use of NPR is automatic or user-assisted style reproduction. While an artist could be employed to create an artifact manually using an art cre- ation tool, it may sometimes be cheaper, more convenient, and less time consuming to replicate an artistic style using an NPR system. For example, creating a cartoon animation commonly requires between 12–24 individually painted frames per sec- ond of the final movie. Automatically converting a live-action video or 3D animation into a cartoon style [27, 34, 51] can significantly save on artists’ time and thereby lower production costs. Finally, there are NPR assistive technologies, which help in the style-specific production process, but which are not necessarily contributing to the stylization it- self. Examples of such technologies include tweening for animations (Chapter ??), cartoon-optimized flood-filling [45], and animation motion capture and retarget- ing [7]. 1.2.1 Artistic Styles in Printing Several media reproduction techniques, such as woodcut, engraving, stamping, and stippling are of particular interest to this section, as they were themselves conceived primarily as answers to production problems, but have since then been recognized as artistic styles in their own right, and subsequently been subject of several NPR investigations. Before the invention of printing processes like woodblock printing (xylography), PREPRINTmovable type, and the Gutenberg printing press, production and copying of writ- 1 NPR in the Wild 3 (a) Traditional Woodcut (b) Detail of Dürer’s “Melencolia I” (c) Modern Hedcut stipple Fig. 1.1 Art (Re)production: (a) Woodcut of a block-cutter by Jost Amman (1568), (b) Detail of Dürer’s engraving “Melencolia I” (1514). Note the combined use of hatching and stippling, the masterful texture of numerous materials, and the orientation of marks along surface features. (c) Modern Hedcut stipple by Kevin Sprouls, creator of the WSJ portrait style.((c) Copyright © 2012 Kevin Sprouls, used with permission) ten materials was a fully manual process, performed by skilled scribes and artists. Images, illustrations, and visual decorations were drawn and copied by hand. As printing techniques revolutionized the dissemination of information, new methods had to be found to replicate the accompanying figures and images. One of the oldest samples of Eastern woodcut printing dates back to 220 A.D., with variations of the technique being discovered and re-discovered throughout antiquity. The basic building block of all so-called relief printing techniques is the ‘relief matrix’. For woodcut, this is simply a level piece of wood. Areas that are intended to appear as ‘white’ are chiseled or scraped away. The remaining parts appear as ‘black’ by applying dark ink onto the wood surface and pressing or stamping the wood onto cloth, parchment, or paper (see Fig. 1.1a). It is easy to see how the char- acters comprising text could be chiseled into wood and replicated this way. Even simple images akin to line drawings were reasonably straightforward to achieve in woodcut. However, more intricate or lifelike images proved to be more challenging, because the complexity of shape and particularly shading was difficult to express in terms of simple presence or absence of ink. As a solution, printmakers adopted artistic techniques to approximate continu- ous tone (grayscale) images with dot and hatching patterns of a single color. In stippling, dark dots of varying size and spacing are placed on a light background. When viewed from a sufficiently large distance, the individual dots are fused into a continuous tone by the human perceptual system. For hatching or cross-hatching, lines (parallel or intersecting, respectively) of varying thickness and spacing are used to achieve a similar effect. Both hatching and especially stippling have found extensive use in printing, engraving, etching, tattooing, and scrimshaw. Common- day examples of these techniques include newspaper printing, currency design, and PREPRINTtechnical illustrations (particularly biology). 4 Holger Winnemöller The manual creation of high-quality stippling images or engravings requires sig- nificant artistic training and, even then, is a time-consuming and laborious process. As such, various NPR works have addressed the need to simplify or automate these production processes. Woodcut-like images are addressed in the works of Mello et al. [31] and Winnemöller et al. [50]. Mizuno et al. [33] have presented a sys- tem to virtually model, carve, and print with digital woodblocks1. Igarashi and Igarashi [18] developed a method to allow novice users (even children) to produce physically realizable stencils from 2D drawings. A system for digital facial engrav- ing has been presented by Ostromokhov [35] (Chapter ??). 1.2.2 Stippling – The Story of a Virtuous Cycle An interesting example for its interplay between art and production is stippling2. Technically, hatching and cross-hatching predate stippling both in terms of art use and in printmaking, possibly due to hatching requiring fewer marks (lines) and therefore being faster to produce. A noteworthy aspect of hatching marks is that they are commonly aligned along directions of principal curvature on the perceived surfaces of a drawing. As such, hatching lines suggest surface shape not only by simulating shading, but also by visually encoding important geometric properties of a surface. In artistic use, stippling is often used to depict soft surfaces, such as faces, cloth, or sand, while hatching is used to depict rougher or textured surfaces, such as wood, rocks, or wallpaper. It is common to intermix both techniques within the same work of art (Fig. 1.1b). Use of these techniques in printmaking followed the same chronological order, with hatching being employed in early woodcuts, while stippling became more com- mon only in later printing methods, such as etching and Mezzotint. Both stippling and hatching served as inspiration for a printing technique called halftoning, which allows photographs to be reproduced by automatically mapping them onto one or more3 screens, which are then applied similar to other relief printing techniques. Commonly, halftone screens consist of grids of regularly spaced dots, where the size of each dot correlates to the darkness of the photograph at the position of the dot. Halftone printing became commercially successful in the late 1800’s and, in its modern digital form, is still used in print production and computer printers today. Despite their historical origins, halftoning and the related dithering are not generally considered artistic techniques, as they follow fairly simple, deterministic algorithms. One of the few newspapers that used to frown upon the extensive use of photo- graphic reproduction was the Wall Street Journal (WSJ). One of its past editors, Fred Taylor, has been quoted as ascribing to a “one word is worth a thousand pictures” policy. It was felt that the liberal use of photographs would detract from the journal- 1 As such, their work is closer related to physical modeling and simulation than NPR. 2 For technical details on stippling, refer to Chapter ??.