Texture Synthesis Presenter: Kaijian Chen Course Advisor: Prof. Hui Huang 2020-05-25 Visual Computing 2020 Texture • Definition of Texture - Images containing repeating patterns Texture example Texture mapping 可视计算研究中心 Visual Computing Research Center 2 Texture acquisition Can we acquire texture in another way? 1. Hand-drawn texture - hard to make them photo-realistic; 2. Scanned texture image • Restricted to fixed resolution; • non-uniform lighting, shadows, etc., lead to visible seams or repetition; Not photo-realistic Visible seams and repetitions 可视计算研究中心 Visual Computing Research Center 3 Outline • Example-based Texture Synthesis • MRF – pixel-based, patch-based, optimization-based • Auxiliary algorithm – Nearest Neighbor Search: K-coherence, PatchMatch • Deep-based Texture Synthesis • Decoupling content and style • Texture synthesis with Generative Adversarial Network • Procedural Texture Synthesis • Perlin noise & Turing patterns • Inverse Procedural Texture Synthesis 可视计算研究中心 Visual Computing Research Center 4 Example-based Texture Synthesis • Goal • Given a texture sample, synthesize a new texture that, when perceived by a human observer, appears to be generated by the same underlying process. input output 可视计算研究中心 Visual Computing Research Center 5 Example-based Texture Synthesis • Analysis - Markov random field (MRF) • model a texture as a realization of a local and stationary random process. �(��|��, ��, ��, … , ��+�) Local Non-stationary Stationary 可视计算研究中心 Visual Computing Research Center 6 Example-based Texture Synthesis • More specific • For each output pixel in generated texture, its spatial neighborhood is similar to at least one neighborhood at the input. input output 可视计算研究中心 Visual Computing Research Center 7 Example-based | Pixel-based method • Main idea - Given input texture and output size, Generate target texture pixel-by-pixel in raster-scan order. Input � Output � Treating input image as datasets Synthesis pixel based of pixel with neighborhood N; on pixel’s neighborhood; 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 8 Example-based | Pixel-based method • Method illustration Inappropriate White noise with neighborhood size histogram matching 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 9 Example-based | Pixel-based method • An efficient way to represent large-scale structure • Gaussian pyramid - represent large scale structures more compactly by a few pixels in a certain lower resolution pyramid level; Gaussian pyramid 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 10 Example-based | Pixel-based method • Coarse-to-fine synthesis • Constructing pyramid for Z & X, synthesis X from low resolution to high resolution; • Large-scale structure constrain - When searching for a match for pixel X, the neighborhood vector is constructed that includes the O’s, Q’s, and Y, in scanline order. Coarsest generation (like before) Gaussian Pyramid Z & X Results - 1 layer, 2 layers, 3 layers Extended Neighborhood 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 11 Example-based | Pixel-based method Results 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 12 Example-based | Pixel-based method Texture replacement Extrapolation Applications Disadvantage - Slow synthesis 可视计算研究中心 Visual Computing Research Center [Wei L. Y., et al.: Fast texture synthesis using tree-structured vector quantization] 13 Example-based | Patch-based method • Main idea – like pixel-based method, but unit of synthesis = block. • Much faster: synthesis all pixels in a block at once. Input Output 可视计算研究中心 Visual Computing Research Center [Alexei Efros, Bill Freeman: Image Quilting for Texture Synthesis & Transfer] 14 block Example-based | Patch-based method • Overlapping region - combining patches with minimal error boundary cut (dynamic programing); B1 B2 B1 B2 Neighboring blocks Minimal error constrained by overlap boundary cut Input texture Min. error boundary 可视计算研究中心 Visual Computing Research Center [Alexei Efros, Bill Freeman: Image Quilting for Texture Synthesis & Transfer] 15 Example-based | Patch-based method Generation process Texture Transfer Results 可视计算研究中心 Visual Computing Research Center [Alexei Efros, Bill Freeman: Image Quilting for Texture Synthesis & Transfer] 16 Example-based | Optimization-based method • Main idea • rather than greedy generation, optimization method tends to directly optimize the error of each patch in output X with its nearest neighbor in input Z. • Energy function • Xp – patch of pixel p in X • Zp – nearest patch of Xp in Z 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 17 Example-based | Optimization-based method • Iterative optimization - For i in range(n_iters): • Expectation – fixed {�0}, modify output X according to {�0}; • Maximization – fixed {�0}, finding their nearest neighborhood {�0}; �0 �0 �2 �2 �2 Source Z �0 Source Z Target Frame X Target Frame X Expectation Maximization 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 18 Example-based | Optimization-based method • Flow-guided Texture Animation + = Input Input Output Warp only With OT Texture Flow Field Sequence 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 19 Example-based | Optimization-based method • Flow-guided Texture Animation • Texture similarity – Shape, size, orientation of texture elements similar to source; • Flow consistency – Perceived motion similar to flow; Flow Consistency Texture Similarity Source Texture Flowing Target 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 20 Example-based | Optimization-based method • Flow-guided Texture Animation 6 6 • E – Find X by solving linear equation: Minimize |�� − �0| + �|�� − �0| ; �0 Blend �0 Source Z Target Frame X w0 Warped Frame W 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 21 Example-based | Optimization-based method • Flow-guided Texture Animation 6 6 • M – Find �0 by searching nearest neighborhood: Minimize |�� − �0| + �|�� − �0| ; �0 �0 Source Z Target Frame X w0 Warped Frame W 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 22 Example-based | Optimization-based method • Flow-guided Texture Animation 6 6 • E – Minimize |�� − �0| + �|�� − �0| ; 6 6 • M – Minimize |�� − �0| + �|�� − �0| ; • Initialize – W -> X; • Multiple 1. Resolution Levels; 2. Neighborhood Sizes; 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 23 Example-based | Optimization-based method Results 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 24 Example-based | Optimization-based method Input Wei-Levoy Image Texture [Wei’00] Quilting Optimization [Efros’01] [Kwatra’05] Results 可视计算研究中心 Visual Computing Research Center [Vivek Kwatra, et al.: Texture Optimization for Example-based Synthesis] 25 Example-based | Optimization-based method • K-Coherence • Observation – Pixels that are together in the input ought to have a tendency to be also together in the output. • Main idea – Propagate the neighbors’ correspondence vector to target pixel; 可视计算研究中心 Visual Computing Research Center [X. Tong, et al.: Synthesis of bidirectional texture functions on arbitrary surfaces] 26 Example-based | Neighborhood search • Main components of K-Coherence <=>?@ABC • Nearest neighborhood search – search from K candidate of �0 − ���� ; • Analysis – compute K nearest neighbor (candidate) for each z0 in {z0}; <=>?@ABC {�0 } <=>?@ABC ^ <=>?@ABC �0 − ����� : neighbor s NN of pixel O， �0 − ���� : shifted by relatiVe position K = 2 可视计算研究中心 Visual Computing Research Center [X. Tong, et al.: Synthesis of bidirectional texture functions on arbitrary surfaces] 27 Example-based | Neighborhood search • PatchMatch • Basic idea – augments the previous coherence (or propagation) stage with a random search process, which can search for good correspondences across the entire exemplar image. • Optimize NNF – Iteratively refine Nearest Neighbor Field (NNF) until converge. Method overview 可视计算研究中心 Visual Computing Research Center [Connelly Barnes, et al.: PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing] 28 Example-based | Neighborhood search • Method of PatchMatch 1. Initialization – initialize random NN vector {�} for each pixel in X; 可视计算研究中心 Visual Computing Research Center [Connelly Barnes, et al.: PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing] 29 Example-based | Neighborhood search • Method of PatchMatch <=>?@ABC 2. Propagation – calculate neighbor {�0 } correspondence loss and current argmin <=>?@ABC pixel NN (�0) loss, propagate �(�0 ) to �0; �0 Similarity Criterion f Propagate NN vector ü minimal 可视计算研究中心 Visual Computing Research Center [Connelly Barnes, et al.: PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing] 30 Example-based