DOCSLIB.ORG

Higher Level Techniques for the Artistic Rendering of Images and Video Submitted by John Philip Collomosse

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Higher Level Techniques for the Artistic Rendering of Images and Video Submitted by John Philip Collomosse Higher Level Techniques for the Artistic Rendering of Images and Video submitted by John Philip Collomosse for the degree of Doctor of Philosophy of the University of Bath 2004 COPYRIGHT Attention is drawn to the fact that copyright of this thesis rests with its author. This copy of the thesis has been supplied on the condition that anyone who consults it is understood to recognise that its copyright rests with its author and that no quotation from the thesis and no information derived from it may be published without the prior written consent of the author. This thesis may be made available for consultation within the University Library and may be photocopied or lent to other libraries for the purposes of consultation. Signature of Author . John Philip Collomosse Higher Level Techniques for the Artistic Rendering of Images and Video John Philip Collomosse i SUMMARY This thesis investigates the problem of producing non-photorealistic renderings for the purpose of aesthetics; so called Artistic Rendering (AR). Specifically, we address the problem of image-space AR, proposing novel algorithms for the artistic rendering of real images and post-production video. Image analysis is a necessary component of image-space AR; information must be ex- tracted from two-dimensional content prior to its re-presentation in some artistic style. Existing image-space AR algorithms perform this analysis at a \low" spatiotemporal level of abstraction. In the case of static AR, the strokes that comprise a rendering are placed independently, and their visual attributes set as a function of only a small image region local to each stroke. In the case of AR animation, video footage is also rendered on a temporally local basis; each frame of animation is rendered taking ac- count of only the current and preceding frame in the video. We argue that this low-level processing paradigm is a limiting factor in the development of image-space AR. The process of deriving artwork from a photograph or video demands visual interpretation, rather than localised filtering, of that source content | a goal challenging enough to warrant application of higher level image analysis techniques, implying interesting new application areas for Computer Vision (and motivating new Computer Vision research as a result). Throughout this thesis we develop a number of novel AR algorithms, the results of which demonstrate a higher spatiotemporal level of analysis to benefit AR in terms of broadening range of potential rendering styles, enhancing temporal coherence in ani- mations, and improving the aesthetic quality of renderings. We introduce the use of global salience measures to image-space AR, and propose novel static AR algorithms which seek to emphasise salient detail, and abstract away unimportant detail within a painting. We also introduce novel animation techniques, describing a \Video Paint- box" capable of creating AR animations directly from video clips. Not only do these animations exhibit a wide gamut of potential styles (such as cartoon-style motion cues and a variety of artistic shading effects), but also exhibit a significant improvement in temporal coherence over the state of the art. We also demonstrate that consideration of the AR process at a higher spatiotemporal level enables the diversification of AR styles to include Cubist-styled compositions and cartoon motion emphasis in animation. ii ACKNOWLEDGEMENTS First, I would like to thank my supervisor, Peter Hall, for taking me on, and wish to ex- press my sincere gratitude for all his hard work, patience and encouragement. Thanks are also due to members of the Media Technology Research Centre at the University of Bath: Phil Willis, Dan Su, Emmanuel Tanguy, and in particular to David Duke, whose advice and feedback over the last three years has been invaluable. Many thanks also go to \our animator" David Rowntree (and others at Nanomation Ltd.) who, on numerous occasions, supplied useful advice and assisted with edits of both conference papers and the Video Paintbox show-reel. Thanks also to Catriona Price, our ballerina in the show-reel, and the artists who have commented on our work over the course of this project. I am also grateful to the numerous conference attendees and anonymous referees, in both the Computer Graphics and Vision communities, who have provided encouragement and suggestions regarding this work. Thanks to all the postgraduate students and research officers, with whom I have shared the lab, for the numerous coffees and distractions from work: Adam B., Andy H., Dan, Emma, Marc, James, Owen, and Vas. Thanks also to Ben for teaching me backgam- mon, and occasionally allowing me to win. Special mentions go those who \star" in the videos in this thesis: Emmanuel, Adam D., Siraphat, and of course Martin who was coerced into a bear costume to co-star in the sequences at Sham castle. I am also indebted to our lab support staff Mark and Jim for turning a blind eye to the copious amount of disk storage occupied by this work. Finally, I would like to thank my family for their emotional and financial support throughout this project. Thanks also to Becky for her love and support. I gratefully acknowledge the financial support of the EPSRC who funded this research under grant GR/M99279. John P. Collomosse, May 2004. iii PUBLICATIONS Portions of the work described in this thesis have appeared in the following papers: Chapter 3 [23] Collomosse J. P. and Hall P. M. (October 2003). Cubist style Rendering from Photographs. IEEE Transactions on Visualization and Computer Graphics (TVCG) 4(9), pp. 443{453. [22] Collomosse J. P. and Hall P. M. (July 2002). Painterly Rendering using Image Salience. In Proc. 20th Eurographics UK Conference, pp. 122{128. (awarded Terry Hewitt Prize for Best Student Paper, 2002) Chapter 4 [24] Collomosse J. P. and Hall P. M. (October 2003). Genetic Painting: A Salience Adaptive Relaxation Technique for Painterly Rendering. Technical Report, University of Bath. Report No. CSBU-2003-02. [66] Hall P. M. and Owen M. J. and Collomosse J. P (2004). A Trainable Low-level Feature Detector. Proc. Intl. Conference on Pattern Recognition (ICPR), to appear. Chapter 6 [27] Collomosse J. P., Rowntree D. and Hall P. M. (September 2003). Video Analysis for Cartoon-like Special Effects. In Proc. 14th British Machine Vision Conference (BMVC), pp. 749{758. (awarded BMVA Industry Prize, 2003) [25] Collomosse J. P., Rowntree D. and Hall P. M. (July 2003). Cartoon-style Rendering of Motion from video. In Proc. 1st Intl. Conference on Video, Vision and Graphics (VVG), pp. 117{124. [28] Collomosse J. P. and Hall P. M. (2004). Automatic Rendering of Cartoon-style Motion Cues in Post-production Video. submitted to Journal on Graphical Models and Image Processing (CVGIP). iv Chapter 8 [26] Collomosse J. P., Rowntree D. and Hall P. M. (June 2003). Stroke Surfaces: A Spatio-temporal Framework for Temporally Coherent Non-photorealistic Animations. Technical Report, University of Bath. Report No. CSBU-2003-01. NON-PUBLICATIONS Collomosse J. P. and Hall P. M. (March 2004). Stroke Surfaces: A Spatio-temporal Framework for Temporally Coherent Non-photorealistic Animations. Presented at BMVA Symposium on Spatiotemporal Processing. Collomosse J. P. and Hall P. M. (July 2002). Applications of Computer Vision to Non-photorealistic Rendering. Poster at 8th EPSRC/BMVA Summer School on Computer Vision. In some cases, these papers describe work in an earlier stage of development than presented in this thesis. Electronic versions are available on the DVD-ROM in Appendix C and at http://www.cs.bath.ac.uk/~jpc/research.htm. v Contents I Introduction 1 1 Introduction 2 1.1 Contribution of this Thesis . 3 1.2 Motivation for a Higher level of Analysis . 4 1.2.1 The Low-level Nature of Image-space AR . 4 1.2.2 Limitations of a Low-Level Approach to AR . 5 1.3 Structure of the Thesis . 8 1.4 Application Areas . 12 1.5 Measuring Success in Artistic Rendering . 13 2 The State of the \Art" 14 2.1 Introduction . 14 2.2 Simulation and Modelling of Artistic Materials . 15 2.2.1 Brush models and simulations . 15 2.2.2 Substrate and Media models . 18 2.3 Interactive and Semi-automatic AR Systems . 19 2.3.1 User assisted digital painting . 20 2.3.2 User assisted sketching and stippling . 22 2.4 Fully Automatic AR Systems . 23 2.4.1 Painterly Rendering Techniques . 24 2.4.2 Sketchy and Line-art Techniques . 28 2.5 Non-photorealistic Animation . 28 2.5.1 Animations from Object-space (3D) . 29 2.5.2 Artistic rendering from video (2D) . 29 2.5.3 Rendering Motion in Image Sequences . 33 2.6 Observations and Summary . 34 II Salience and Art: the benefits of vi Higher level spatial analysis 40 3 Painterly and Cubist-style Rendering using Image Salience 41 3.1 Introduction . 41 3.2 A Global Measure of Image Salience . 45 3.3 Painterly Rendering using Image Salience . 48 3.3.1 Results and Qualitative Comparison . 52 3.4 Cubist-style Rendering from Photographs . 54 3.4.1 Identification of Salient Features . 55 3.4.2 Geometric Distortion . 57 3.4.3 Generation of Composition . 60 3.4.4 Applying a Painterly Finish . 66 3.4.5 Results of Cubist Rendering . 67 3.5 Personal Picasso: Fully Automating the Cubist Rendering System . 70 3.5.1 An Algorithm for Isolating Salient Facial Features . 70 3.5.2 Tracking the Isolated Salient Features . 73 3.6 Summary and Discussion . 77 4 Genetic Painting: A Salience Adaptive Relaxation Technique for Painterly Rendering 80 4.1 Introduction . 80 4.2 Background in Evolutionary Computing . 83 4.2.1 Genetic Algorithms in Computer Graphics . 84 4.3 Determining Image Salience .
Load more

Recommended publications
  • Linear Algebra and Numerical Techniques
    Appendix A Linear algebra and numerical techniques A.1 Matrix decompositions .............................646 A.1.1 Singular value decomposition .....................646 A.1.2 Eigenvalue decomposition .......................647 A.1.3 QR factorization ............................649 A.1.4 Cholesky factorization .........................650 A.2 Linear least squares ...............................651 A.2.1 Total least squares ...........................653 A.3 Non-linear least squares .............................654 A.4 Direct sparse matrix techniques .........................655 A.4.1 Variable reordering ...........................656 A.5 Iterative techniques ...............................656 A.5.1 Conjugate gradient ...........................657 A.5.2 Preconditioning .............................659 A.5.3 Multigrid ................................660 646 A Linear algebra and numerical techniques In this appendix, we introduce some elements of linear algebra and numerical techniques that are used elsewhere in the book. We start with some basic decompositions in matrix algebra, including the singular value decomposition (SVD), eigenvalue decompositions, and other matrix decompositions (factorizations). Next, we look at the problem of linear least squares, which can be solved using either the QR decomposition or normal equations. This is followed by non-linear least squares, which arise when the measurement equations are not linear in the unknowns or when robust error functions are used. Such problems require iteration to find a solution. Next, we look at direct solution (factorization) techniques for sparse problems, where the ordering of the variables can have a large influence on the computation and memory requirements. Finally, we discuss iterative techniques for solving large linear (or linearized) least squares problems. Good general references for much of this material include the work by Bjorck¨ (1996), Golub and Van Loan (1996), Trefethen and Bau (1997), Meyer (2000), Nocedal and Wright (2006), and Bjorck¨ and Dahlquist (2010).
    [Show full text]
  • Optimal Transport for Computer Graphics and Temporal Coherence of Image Processing Algorithms
    No d’ordre NNT : 041 2018 HABILITATION À DIRIGER DES RECHERCHES DE L’UNIVERSITÉ DE LYON opérée au sein de l’Université Claude Bernard Lyon 1 École Doctorale ED01 École Doctorale InfoMaths Soutenue publiquement le 09/11/2018, par : Nicolas Bonneel Optimal Transport for Computer Graphics and Temporal Coherence of Image Processing Algorithms Devant le jury composé de : Julie DELON, Professeur, Université Paris Descartes Rapporteure Quentin MÉRIGOT, Professeur, Université Paris-Sud Rapporteur Filippo SANTAMBROGIO, Professeur, Université de Lyon Rapporteur Nicolas COURTY, Professeur, Université de Bretagne Sud Président Justin SOLOMON, Assistant Professor, Massachusetts Institute of Technology Examinateur 2 Remerciements Je remercie d’abord les rapporteurs et examinateurs de mon jury pour leur lecture attentive du manuscrit, les questions pertinentes lors de ma soutenance, mais aussi les riches discussions qui s’en sont suivies par email. Je remercie mes co-auteurs (j’ai pu en dénombrer une soixantaine que je ne listerai pas ici!), étudiants et collègues du LIRIS. En particulier, je ne serais pas arrivé au CNRS sans l’accueil qui m’a été fourni auparavant en master, thèse et postdoctorats, par Matthias Paulin, George Drettakis, Michiel van de Panne, Bruno Lévy et Hanspeter Pfister. Je remercie finalement mon père Stéphane pour son intérêt continuel, et ma femme Lucie pour m’avoir soutenu et accompagné lors de plusieurs années de vadrouille post-doctoresques en attente du Graal du poste permanent. Et je dédie ce manuscrit à Evan, notre fils, né le 19 Novembre 2017, presqu’un an avant cette soutenance. 3 4 Contents 1 Background 13 1.1 The Optimal Transport Problem ..........................
    [Show full text]
  • Abstracts Sitis 2012 Abstracts
    WG 2.6 Ernesto Damiani, Giuseppe De Pietro, Luigi Gallo Albert Dipanda, Richard Chbeir, Kokou Yetongnon ABSTRACTS SITIS 2012 ABSTRACTS Track Signal and Image Technology (SIT) SIT-S1: Video Processing and Coding 33 Adapting the Streaming Video Based on the Estimated Position of the Region of Interest Hussein Muzahim Aziz, Markus Fiedler, Håkan Grahn, Lars Lundberg [email protected] Keywords: streaming video, region of interest, sum of absolute differences, mean opinion scores Abstract Streaming real time video over wireless networks cannot guarantee that all the frames could meet their deadlines. Wireless networks may suffer from bandwidth limitations. To reduce the streaming data over wireless networks, we propose a technique to identify, and extract the Region Of Interest (ROI), and drop the non-ROI from the frames that are between the reference frames. The Sum of Absolute Differences (SAD) will compute the consecutive video frames to identify the ROI as it considered the most motion and important region. The reconstruction mechanism to the non-ROI is performed on the mobile side by applying linear interpolation between the reference frames. We evaluate the proposed approach by using Mean Opinion Score (MOS) measurements. MOS are used to evaluate the two scenarios with equivalent encoding size, where the users observe the first scenario with a low bit rate for the original videos, while for the second scenario the users observe our proposed approach. The results show that our technique significantly reduces the amount of data, while
    [Show full text]
  • Expanded Cinema
    EXPANDED CINEMA ARTSCILAB 2001 2 Blank Page ARTSCILAB 2001 Gene Youngblood became a passenger of Spaceship Earth on May 30, 1942. He is a faculty member of the California Institute of the Arts, School of Critical Studies. Since 1961 he has worked in all aspects of communications media: for five years he was reporter, feature writer, and film critic for the Los Angeles Herald-Examiner; in 1965 he conducted a weekly program on film and the arts for KPFK, Pacifica Radio in Los Angeles; in 1967 he wrote, produced, directed, edited, and on-camera reported "human interest" filmed news features for KHJ-TV in Los Angeles; since 1967 his column “Intermedia” has appeared weekly in the Los Angeles Free Press on subjects ranging from film and the arts to science, technology, and the cultural revolution. Mr. Youngblood currently is working on two books: The Videosphere, about global television in the 1970s as a tool for conscious evolution, and Earth Nova, a philosophical novel and screenplay about the new consciousness, the new lifestyle, and their relation to technology. ARTSCILAB 2001 4 Blank Page ARTSCILAB 2001 EXPANDED CINEMA by Gene Youngblood Introduction by R. Buckminster Fuller A Dutton Paperback P. Dutton & Co., Inc., New York 1970 ARTSCILAB 2001 Copyright © 1970 by Gene Youngblood Introduction and poem, "Inexorable Evolution and Human Ecology," copyright © 1970 by R. Buckminster Fuller All rights reserved. Printed in the U.S.A. First Edition No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system now known or to be invented, without permission in writing from the publishers, except by a reviewer who wishes to quote brief passages in connection with a review written for inclusion in a magazine or newspaper or broadcast.
    [Show full text]
  • 2011 18Th IEEE International Conference on Image Processing
    2011 18th IEEE International Conference on Image Processing (ICIP 2011) Brussels, Belgium 11 – 14 September 2011 Pages 1-740 IEEE Catalog Number: CFP11CIP-PRT ISBN: 978-1-4577-1304-0 1/5 Monday, September 12 MA.L1: Modern Shape from Shading and Beyond (Special Session) Modern Shape From Shading and Beyond Michael Breuss (Saarland University, Germany); Oliver Vogel (Saarland University, Germany); Ariel Tankus (Technion - Israel Institute of Technology, Israel) pp. 1-4 Numerical Schemes for Advanced Reflectance Models for Shape From Shading Oliver Vogel (Saarland University, Germany); Emiliano Cristiani (SAPIENZA - Universita di Roma, Italy) pp. 5-8 Break Shape From Shading with Specular Highlights: Analysis of the Phong Model Michael Breuss (Saarland University, Germany); Yong Chul Ju (Saarland University, Germany) pp. 9-12 Shape-from-shading Under Complex Natural Illumination Rui Huang (University of York, United Kingdom); William Smith (University of York, United Kingdom) pp. 13-16 Reconstruction of non-Lambertian Surfaces by Fusion of Shape From Shading and Active Range Scanning Steffen Herbort (Dortmund University of Technology, Germany); Arne Grumpe (Dortmund University of Technology, Germany); Christian Wöhler (Dortmund University of Technology, Germany) pp. 17-20 Shape From Specular Reflection in Calibrated Environments and the Integration of Spatial Normal Fields Jonathan Balzer (King Abdullah University of Science and Technology, Saudi Arabia) pp. 21-24 MA.L2: Structural Models (Lecture) Joint Pose Estimation and Action Recognition in Image Graphs Raja Kumar (IRISA & INRIA Rennes, France); Ivan Laptev (INRIA Paris - Rocquencourt, France); Patrick Pérez (INRIA, France); Lionel Oisel (Thomson, France) pp. 25-28 Inferring 3D Body Pose Using Variational Semi-parametric Regression Yan Tian (Hikvision Digital Technology Co.
    [Show full text]
  • Digital Video Processing
    DIGITAL VIDEO PROCESSING Second Edition This page intentionally left blank DIGITAL VIDEO PROCESSING Second Edition A. Murat Tekalp New York • Boston • Indianapolis • San Francisco Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic versions; custom cover designs; and content particular to your business, training goals, marketing focus, or branding interests), please contact our corporate sales department at [email protected] or (800) 382-3419. For government sales inquiries, please contact [email protected]. For questions about sales outside the United States, please contact [email protected]. Visit us on the Web: informit.com/ph Library of Congress Cataloging-in-Publication Data Tekalp, A. Murat. Digital video processing / A. Murat Tekalp.—Second edition. pages cm Includes bibliographical references and index. ISBN 978-0-13-399100-0 (hardcover : alk. paper)—ISBN 0-13-399100-8 (hardcover : alk. paper) 1.
    [Show full text]
  • Hardware Architectures for Infrared Pedestrian Detection Systems
    School of Engineering and the Built Environment Hardware Architectures for Infrared Pedestrian Detection Systems by Robert Walczyk A thesis submitted in partial fulfilment of the requirements of Edinburgh Napier University, for the award of Doctor of Philosophy May 2013 Contact: [email protected] This document was set with the help of KOMA-Script and LATEX Abstract Infrared pedestrian detection systems struggle with real-time processing performance. Known solutions are limited to either low resolution systems with basic functionality running at full frame rate, or software based techniques featuring higher detection rates with full set of features, however running only in off-line mode for statistical analysis. Here, a comprehensive solution for real-time pedestrian detection is described. This research project includes investigation of possible solutions, design, development and implementation of a pedestrian detection system, processing data from infrared video source in real-time. Design requirements include processing at full frame rate as well as low memory and system resource consumption. The memory utilization is one of the major concerns since high demand for memory resources is a critical aspect in most image processing applications. For the purpose of this task, a number of general purpose image processing techniques were revised, taking into consideration the suitability for infrared pedestrian detection. These tasks include background separation, acquisition noise removal and object de- tection through connected component labelling. They are discussed and addressed in individual chapters. Various techniques for background segmentation are discussed. A chronological review of popular techniques is provided. The proposed architecture for background subtrac- tion is based on selective running average for adaptive background model, supported by adaptive thresholding based on histogram calculation.
    [Show full text]
  • What to Submit And
    Digital Video Processing, 2E Table of Contents ©A. Murat Tekalp January 2015 TABLE OF CONTENTS Preface 1. MULTI-DIMENSIONAL SIGNALS AND SYSTEMS 1.1 Multi-dimensional Signals 1.1.1. Finite Extent Signals and Periodic Signals 1.1.2. Symmetric Signals 1.1.3. Special Multi-dimensional Signals 1.2 Multi-dimensional Transforms 1.2.1 Fourier Transform of Continuous Signals 1.2.2 Fourier Transform of Discrete Signals 1.2.3 Discrete Fourier Transform 1.2.4 Discrete Cosine Transform 1.3 Multi-dimensional Systems 1.3.1 Impulse Response and 2D Convolution 1.3.2 Frequency Response 1.3.3 FIR Filters and Symmetry 1.3.4 IIR Filters and Partial Difference Equations 1.4 Multi-dimensional Sampling Theory 1.4.1 Sampling on a Lattice 1.4.2 Spectrum of Signals Sampled on a Lattice 1.4.3 Nyquist Criterion for Signals Sampled on a Lattice 1.4.4 Reconstruction from Samples on a Lattice 1.5 Sampling Lattice Conversion References Exercises 2. DIGITAL IMAGES AND VIDEO 2.1 Human Visual System and Color 2.1.1 Color Vision and Models 2.1.2 Contrast Sensitivity 2.1.3 Spatio-Temporal Frequency Response 2.1.4 Stereo/Depth Perception 2.2 Analog Video 2.2.1 Progressive vs. Interlaced Scanning 2.2.2 Analog Video Signal Formats 2.2.3 Analog-to-Digital Video Conversion 1 Digital Video Processing, 2E Table of Contents ©A. Murat Tekalp January 2015 2.3 Digital Video 2.3.1 Spatial Resolution and Frame Rate 2.3.2 Color, Dynamic Range and Bit Depth 2.3.3 Color Image Processing 2.3.4 Digital Video Standards 2.4 3D Video 2.4.1 3D Display Technologies 2.4.2 Stereoscopic Video 2.4.3 Multi-view Video 2.5 Digital Video Applications 2.5.1 Digital TV 2.5.2 Digital Cinema 2.5.3 Video Streaming over the Internet 2.5.4 Computer Vision and Scene/Activity Understanding 2.6 Image and Video Quality 2.6.1 Visual Artifacts 2.6.2 Subjective Quality Assessment 2.6.3 Objective Quality Assessment References 3.
    [Show full text]