Higher Level Techniques for the Artistic Rendering of Images and Video submitted by John Philip Collomosse for the degree of Doctor of Philosophy of the University of Bath 2004 COPYRIGHT Attention is drawn to the fact that copyright of this thesis rests with its author. This copy of the thesis has been supplied on the condition that anyone who consults it is understood to recognise that its copyright rests with its author and that no quotation from the thesis and no information derived from it may be published without the prior written consent of the author. This thesis may be made available for consultation within the University Library and may be photocopied or lent to other libraries for the purposes of consultation. Signature of Author . John Philip Collomosse Higher Level Techniques for the Artistic Rendering of Images and Video John Philip Collomosse i SUMMARY This thesis investigates the problem of producing non-photorealistic renderings for the purpose of aesthetics; so called Artistic Rendering (AR). Specifically, we address the problem of image-space AR, proposing novel algorithms for the artistic rendering of real images and post-production video. Image analysis is a necessary component of image-space AR; information must be ex- tracted from two-dimensional content prior to its re-presentation in some artistic style. Existing image-space AR algorithms perform this analysis at a \low" spatiotemporal level of abstraction. In the case of static AR, the strokes that comprise a rendering are placed independently, and their visual attributes set as a function of only a small image region local to each stroke. In the case of AR animation, video footage is also rendered on a temporally local basis; each frame of animation is rendered taking ac- count of only the current and preceding frame in the video. We argue that this low-level processing paradigm is a limiting factor in the development of image-space AR. The process of deriving artwork from a photograph or video demands visual interpretation, rather than localised filtering, of that source content | a goal challenging enough to warrant application of higher level image analysis techniques, implying interesting new application areas for Computer Vision (and motivating new Computer Vision research as a result). Throughout this thesis we develop a number of novel AR algorithms, the results of which demonstrate a higher spatiotemporal level of analysis to benefit AR in terms of broadening range of potential rendering styles, enhancing temporal coherence in ani- mations, and improving the aesthetic quality of renderings. We introduce the use of global salience measures to image-space AR, and propose novel static AR algorithms which seek to emphasise salient detail, and abstract away unimportant detail within a painting. We also introduce novel animation techniques, describing a \Video Paint- box" capable of creating AR animations directly from video clips. Not only do these animations exhibit a wide gamut of potential styles (such as cartoon-style motion cues and a variety of artistic shading effects), but also exhibit a significant improvement in temporal coherence over the state of the art. We also demonstrate that consideration of the AR process at a higher spatiotemporal level enables the diversification of AR styles to include Cubist-styled compositions and cartoon motion emphasis in animation. ii ACKNOWLEDGEMENTS First, I would like to thank my supervisor, Peter Hall, for taking me on, and wish to ex- press my sincere gratitude for all his hard work, patience and encouragement. Thanks are also due to members of the Media Technology Research Centre at the University of Bath: Phil Willis, Dan Su, Emmanuel Tanguy, and in particular to David Duke, whose advice and feedback over the last three years has been invaluable. Many thanks also go to \our animator" David Rowntree (and others at Nanomation Ltd.) who, on numerous occasions, supplied useful advice and assisted with edits of both conference papers and the Video Paintbox show-reel. Thanks also to Catriona Price, our ballerina in the show-reel, and the artists who have commented on our work over the course of this project. I am also grateful to the numerous conference attendees and anonymous referees, in both the Computer Graphics and Vision communities, who have provided encouragement and suggestions regarding this work. Thanks to all the postgraduate students and research officers, with whom I have shared the lab, for the numerous coffees and distractions from work: Adam B., Andy H., Dan, Emma, Marc, James, Owen, and Vas. Thanks also to Ben for teaching me backgam- mon, and occasionally allowing me to win. Special mentions go those who \star" in the videos in this thesis: Emmanuel, Adam D., Siraphat, and of course Martin who was coerced into a bear costume to co-star in the sequences at Sham castle. I am also indebted to our lab support staff Mark and Jim for turning a blind eye to the copious amount of disk storage occupied by this work. Finally, I would like to thank my family for their emotional and financial support throughout this project. Thanks also to Becky for her love and support. I gratefully acknowledge the financial support of the EPSRC who funded this research under grant GR/M99279. John P. Collomosse, May 2004. iii PUBLICATIONS Portions of the work described in this thesis have appeared in the following papers: Chapter 3 [23] Collomosse J. P. and Hall P. M. (October 2003). Cubist style Rendering from Photographs. IEEE Transactions on Visualization and Computer Graphics (TVCG) 4(9), pp. 443{453. [22] Collomosse J. P. and Hall P. M. (July 2002). Painterly Rendering using Image Salience. In Proc. 20th Eurographics UK Conference, pp. 122{128. (awarded Terry Hewitt Prize for Best Student Paper, 2002) Chapter 4 [24] Collomosse J. P. and Hall P. M. (October 2003). Genetic Painting: A Salience Adaptive Relaxation Technique for Painterly Rendering. Technical Report, University of Bath. Report No. CSBU-2003-02. [66] Hall P. M. and Owen M. J. and Collomosse J. P (2004). A Trainable Low-level Feature Detector. Proc. Intl. Conference on Pattern Recognition (ICPR), to appear. Chapter 6 [27] Collomosse J. P., Rowntree D. and Hall P. M. (September 2003). Video Analysis for Cartoon-like Special Effects. In Proc. 14th British Machine Vision Conference (BMVC), pp. 749{758. (awarded BMVA Industry Prize, 2003) [25] Collomosse J. P., Rowntree D. and Hall P. M. (July 2003). Cartoon-style Rendering of Motion from video. In Proc. 1st Intl. Conference on Video, Vision and Graphics (VVG), pp. 117{124. [28] Collomosse J. P. and Hall P. M. (2004). Automatic Rendering of Cartoon-style Motion Cues in Post-production Video. submitted to Journal on Graphical Models and Image Processing (CVGIP). iv Chapter 8 [26] Collomosse J. P., Rowntree D. and Hall P. M. (June 2003). Stroke Surfaces: A Spatio-temporal Framework for Temporally Coherent Non-photorealistic Animations. Technical Report, University of Bath. Report No. CSBU-2003-01. NON-PUBLICATIONS Collomosse J. P. and Hall P. M. (March 2004). Stroke Surfaces: A Spatio-temporal Framework for Temporally Coherent Non-photorealistic Animations. Presented at BMVA Symposium on Spatiotemporal Processing. Collomosse J. P. and Hall P. M. (July 2002). Applications of Computer Vision to Non-photorealistic Rendering. Poster at 8th EPSRC/BMVA Summer School on Computer Vision. In some cases, these papers describe work in an earlier stage of development than presented in this thesis. Electronic versions are available on the DVD-ROM in Appendix C and at http://www.cs.bath.ac.uk/~jpc/research.htm. v Contents I Introduction 1 1 Introduction 2 1.1 Contribution of this Thesis . 3 1.2 Motivation for a Higher level of Analysis . 4 1.2.1 The Low-level Nature of Image-space AR . 4 1.2.2 Limitations of a Low-Level Approach to AR . 5 1.3 Structure of the Thesis . 8 1.4 Application Areas . 12 1.5 Measuring Success in Artistic Rendering . 13 2 The State of the \Art" 14 2.1 Introduction . 14 2.2 Simulation and Modelling of Artistic Materials . 15 2.2.1 Brush models and simulations . 15 2.2.2 Substrate and Media models . 18 2.3 Interactive and Semi-automatic AR Systems . 19 2.3.1 User assisted digital painting . 20 2.3.2 User assisted sketching and stippling . 22 2.4 Fully Automatic AR Systems . 23 2.4.1 Painterly Rendering Techniques . 24 2.4.2 Sketchy and Line-art Techniques . 28 2.5 Non-photorealistic Animation . 28 2.5.1 Animations from Object-space (3D) . 29 2.5.2 Artistic rendering from video (2D) . 29 2.5.3 Rendering Motion in Image Sequences . 33 2.6 Observations and Summary . 34 II Salience and Art: the benefits of vi Higher level spatial analysis 40 3 Painterly and Cubist-style Rendering using Image Salience 41 3.1 Introduction . 41 3.2 A Global Measure of Image Salience . 45 3.3 Painterly Rendering using Image Salience . 48 3.3.1 Results and Qualitative Comparison . 52 3.4 Cubist-style Rendering from Photographs . 54 3.4.1 Identification of Salient Features . 55 3.4.2 Geometric Distortion . 57 3.4.3 Generation of Composition . 60 3.4.4 Applying a Painterly Finish . 66 3.4.5 Results of Cubist Rendering . 67 3.5 Personal Picasso: Fully Automating the Cubist Rendering System . 70 3.5.1 An Algorithm for Isolating Salient Facial Features . 70 3.5.2 Tracking the Isolated Salient Features . 73 3.6 Summary and Discussion . 77 4 Genetic Painting: A Salience Adaptive Relaxation Technique for Painterly Rendering 80 4.1 Introduction . 80 4.2 Background in Evolutionary Computing . 83 4.2.1 Genetic Algorithms in Computer Graphics . 84 4.3 Determining Image Salience .