!"#$%&'&()*"%*"&+*,$#-.- Style & Content Analysis § What is content, what is style in an image? § What are the methods of classification for each? § With computation-generated images, how can the distinction be defined between style and content? § What if the style is the content? -- § Visual Modality – the ability to see: How to evaluate the degree to which pictorial expression through color, detail, depth, tonal shades are used? !"#$%&'()*+,%(&-.)"#+/ 0&1%2()$&0$34(5"67&4,&0("5+"58&!"#$% !"#$%&'%()*+,- &."/)$0"1%2'%345"1- 6)**78),% 9"*7:" ;<=>?@ A$%B8$"%)1*-%",C"48)..+%C)8$*8$:-%7DE)$,%7)F"% E),*"1"0%*7"%,58..%*#%41")*"%D$8GD"%F8,D).% "/C"18"$4",%*71#D:7%4#EC#,8$:%)%4#EC."/% 8$*"1C.)+%H"*I""$%*7"%4#$*"$*%)$0%,*+."%#B%)$% 8E):"'%J7D,%B)1%*7"%).:#18*7E84%H),8,%#B%*78,% C1#4",,%8,%D$5$#I$%)$0%*7"1"%"/8,*,%$#%)1*8B848).% ,+,*"E%I8*7%,8E8.)1%4)C)H8.8*8",'%K#I"F"1-%8$% #*7"1%5"+%)1"),%#B%F8,D).%C"14"C*8#$%,D47%),% #HL"4*%)$0%B)4"%1"4#:$8*8#$%$")1M7DE)$% C"1B#1E)$4"%I),%1"4"$*.+%0"E#$,*1)*"0%H+%)% 4.),,%#B%H8#.#:84)..+%8$,C81"0%F8,8#$%E#0".,%4).."0% N""C%O"D1).%O"*I#15,'% K"1"%I"%8$*1#0D4"%)$%)1*8B848).%,+,*"E%H),"0%#$%)% N""C%O"D1).%O"*I#15%*7)*%41")*",%)1*8,*84%8E):",% #B%78:7%C"14"C*D).%GD).8*+'%J7"%,+,*"E%D,",% $"D1).%1"C1","$*)*8#$,%*#%,"C)1)*"%)$0% 1"4#EH8$"%4#$*"$*%)$0%,*+."%#B%)1H8*1)1+%8E):",-% C1#F808$:%)%$"D1).%).:#18*7E%B#1%*7"%41")*8#$%#B% )1*8,*84%8E):",'%6#1"#F"1-%8$%.8:7*%#B%*7"%,*1858$:% ,8E8.)18*8",%H"*I""$%C"1B#1E)$4"M#C*8E8,"0 )1*8B848).%$"D1).%$"*I#15,%)$0%H8#.#:84).%F8,8#$-% #D1%I#15%#BB"1,%)%C)*7%B#1I)10%*#%)$%).:#18*7E84% D$0"1,*)$08$:%#B%7#I%7DE)$,%41")*"%)$0 C"14"8F"%)1*8,*84%8E):"1+' !"#$%&'()*+,%(&-.)"#+/ Style Transfer § Style transfer – Transferring the style of the source image, while preserving the content of the target image § Style specific to Neural Style Transfer is defined as the Texture of an image - Texture of an image captures the geometric shapes, patterns and transitions § Goal: To synthesize a tecture from a source image while constraining the texture synthesis in order to preserve the semantic content of the target image !"#$%&'()*+,%(&-.)"#+/ Style Transfer (Intel 4004 processor into city map by Fabian Offert) Created by training a pix2pix GAN (implementations are freely available on GitHub, originally proposed in this paper: https://arxiv.org/abs/1611.07004 on pairs of map data and corresponding satellite images, scraped from Google Maps with the help of a script that I wrote. Video by Fabian translating Stockhausen’s scores into cities: https://vimeo.com/235286541 Computational Decomposition of Style for Controllable and Enhanced Style Transfer Minchao Li Shikui Tu Lei Xu Shanghai Jiao Tong University marshal lmc,tushikui,leixu @sjtu.edu.cn { } Abstract area of computer graphics, which focuses on enabling artis- tic styles such as oil painting and drawing for digital images. Neural style transfer has been demonstrated to be pow- However, NPR is usually limited to specific styles and hard erful in creating artistic image with help of Convolutional to generalize to produce styled images for any other artistic Neural Networks (CNN). However, there is still lack of com- styles. putational analysis of perceptual components of the artistic One significant advancement was made by Gatys et al. in style. Different from some early attempts which studied the 2015 [7], called neural style transfer, which could separate style by some pre-processing or post-processing techniques, the representations of the image content and style learned we investigate the characteristics of the style systematically by deep CNN and then recombine the image content from based on feature map produced by CNN. First, we compu- one and the image style from another to obtain styled im- tationally decompose the style into basic elements using not ages. During this neural style transfer process, fantastic only spectrum based methods including Fast Fourier Trans- stylized images were produced with the appearance simi- form (FFT), Discrete Cosine Transform (DCT) but also la- lar to a given real artistic work, such as Vincent van Gogh’s tent variable models such Principal Component Analysis “The Starrry Night”. The success of the style transfer indi- (PCA), Independent Component Analysis (ICA). Then, the cates that artistic styles are computable and are able to be decomposition of style induces various ways of controlling migrated from one image to another. Thus, we could learn the style elements which could be embedded as modules in to draw like some artists apparently without being trained state-of-the-art style transfer algorithms. Such decomposi- for years. tion of style brings several advantages. It enables the com- Following Gatys et al.’s pioneering work, a lot of efforts putational coding of different artistic styles by our style ba- have been made to improve or extend the neural style trans- sis with similar styles clustering together, and thus it facili- fer algorithm.[25] considered the semantic content and in- tates the mixing or intervention of styles based on the style troduced the semantic style transfer network.[15] combined basis from more than one styles so that compound style or the discriminatively trained CNN with the classical Markov new style could be generated to produce styled images. Ex- Random Field (MRF) based texture synthesis for better periments demonstrate the effectiveness of our method on mesostructure preservation in synthesized images. Seman- not only painting style transfer but also sketch style transfer tic annotations were introduced by [1] to achieve seman- which indicates possible applications on picture-to-sketch tic transfer. To imporve the efficiency, [14] as well as [22] arXiv:1811.08668v2 [cs.CV] 24 Nov 2018 problems. introduced a fast neural style transfer method, which is a feed-forward network to deal with a large set of images per training. With help of an adversarial training network, re- 1. Introduction sults were further improved in [16]. For a systematic review on neural style transfer, please refer to [13]. Painting art, like Vincent van Gogh’s “The Starry Night”, The success of recent progress on style transfer relies on have attracted people for many years. It is one of the most the separable representation learned by deep CNN, in which popular art forms for creative expression of the conceptual the layers of convolutional filters automatically learns low- intention of the practitioner. Since 1990’s, researches have level or abstract representations in a more expressive fea- been made by computer scientists on the artistic work, in ture space than the raw pixel-based images. However, it is order to understand art from the view of computer or to turn still challenging to use CNN representations for style trans- a camera photo into an artistic image automatically. One fer due to their uncontrollable behavior as a black-box, and early attempt is Non-photorealistic rendering (NPR)[18], an thus it is still difficult to select appropriate composition of 1 Image Style Transfer § Find image representations that independently model variations in the semantic image content and the style which it is presented § This calls for a need to develop a method for disentangling style and content in images § Tutorial: https://pytorch.org/tutorials/advanced/neural_style_tutorial. html !"#$%&'()*+,")-./+*,"%012*'.3'+42*.5'6,+4'.07-,$'("( High-Resolution Multi-Scale Neural Texture Synthesis Xavier Snelgrove Toronto, Ontario, Canada [email protected] (a) Source texture (b) Synthesized image Figure 1: High resolution texture synthesis matching CNN texture statistics at 5 image scales ABSTRACT ACM Reference Format: We introduce a novel multi-scale approach for synthesizing high- Xavier Snelgrove. 2017. High-Resolution Multi-Scale Neural Texture Syn- thesis. In resolution natural textures using convolutional neural networks SA ’17 Technical Briefs: SIGGRAPH Asia 2017 Technical Briefs, No- vember 27–30, 2017, Bangkok, Thailand. ACM, New York, NY, USA, 4 pages. trained on image classication tasks. Previous breakthroughs were https://doi.org/10.1145/3145749.3149449 based on the observation that correlations between features at inter- mediate layers of the network are a powerful texture representation, however the xed receptive eld of network neurons limits the 1 INTRODUCTION maximum size of texture features that can be synthesized. We show that rather than matching statistical properties at many There have been recent signicant improvements in the quality of layers of the CNN, better results can be achieved by matching example-based texture synthesis techniques by taking advantage a small number of network layers but across many scales of a of intermediate representations in a convolutional neural network Gaussian pyramid. This leads to qualitatively superior synthesized (CNN) trained to classify images [Gatys et al. 2015b]. high-resolution textures. Correlations between features maps at these intermediate layers, represented by a Gram matrix, turn out to be a powerful represen- tation of texture. By synthesizing new images whose Gram matrix CCS CONCEPTS is close to that of an exemplar image (for instance via gradient • Computing methodologies → Texturing; descent), we get images with similar texture. However, this only works well when the semantically signicant KEYWORDS features in the image are at the correct scale for the network, and in practice the receptive eld of a feature at an intermediate layer for texture synthesis, neural networks, Gaussian pyramid common CNN architectures is relatively small [Luo et al. 2016]. The popular VGG architectures from Simonyan and Zisserman [2014], used by Gatys et al. and others, are trained on 224 224 pixel images, ⇥ SA ’17 Technical Briefs, November 27–30, 2017, Bangkok, Thailand in which relevant features will be quite a bit smaller.