The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) Adversarial Turing Patterns from Cellular Automata Nurislam Tursynbek1, Ilya Vilkoviskiy1, Maria Sindeeva1, Ivan Oseledets1 1Skolkovo Institute of Science and Technology [email protected], [email protected], [email protected], [email protected] Abstract where authors proposed iterative strategy of gradually push- ing a data point to the decision boundary. However, to con- State-of-the-art deep classifiers are intriguingly vulnerable struct a successful perturbation thousands of images were to universal adversarial perturbations: single disturbances of needed, whereas Khrulkov et. al (Khrulkov and Oseledets small magnitude that lead to misclassification of most in- puts. This phenomena may potentially result in a serious se- 2018) proposed an efficient algorithm of constructing UAPs curity problem. Despite the extensive research in this area, with a very small number of samples. The proposed univer- there is a lack of theoretical understanding of the structure of sal perturbations construct complex and interesting unusual these perturbations. In image domain, there is a certain vi- patterns. Studying how these patterns emerge will allow bet- sual similarity between patterns, that represent these pertur- ter understanding the nature of adversarial examples. bations, and classical Turing patterns, which appear as a so- We start from an interesting observation that patterns gen- lution of non-linear partial differential equations and are un- erated in (Khrulkov and Oseledets 2018) visually look very derlying concept of many processes in nature. In this paper, we provide a theoretical bridge between these two different similarly to the so-called Turing patterns (Figure 1) which theories, by mapping a simplified algorithm for crafting uni- were introduced by Alan Turing in the seminal paper “The versal perturbations to (inhomogeneous) cellular automata, Chemical Basis of Morphogenesis” (Turing 1952). It de- the latter is known to generate Turing patterns. Furthermore, scribes the way in which patterns in nature such as stripes we propose to use Turing patterns, generated by cellular au- and spots can arise naturally out of a homogeneous uni- tomata, as universal perturbations, and experimentally show form state. The original theory of Turing patterns, a two- that they significantly degrade the performance of deep learn- component reaction-diffusion system, is an important model ing models. We found this method to be a fast and efficient in mathematical biology and chemistry. Turing found that a way to create a data-agnostic quasi-imperceptible perturba- stable state in the system with local interactions can become tion in the black-box scenario. The source code is available at unstable in the presence of diffusion. Reaction–diffusion https://github.com/NurislamT/advTuring. systems have gained significant attention and was used as a prototype model for pattern formation. Introduction In this paper, we provide an explanation why UAPs from Deep neural networks have shown success in solving com- (Khrulkov and Oseledets 2018) bear similarity to the Turing plex problems for different applications ranging from medi- patterns using the formalism of cellular automata (CA): the cal diagnoses to self-driving cars, but recent findings surpris- iterative process for generating UAPs can be approximated ingly show they are not safe and vulnerable to well-designed by such process, and Turing patterns can be easily gener- negligibly perturbed inputs (Szegedy et al. 2013; Goodfel- ated by cellular automata (Young 1984). Besides, this gives low, Shlens, and Szegedy 2015), called adversarial exam- a very simple way to generate new examples by learning the ples, compromising people’s confidence in them. Moreover, parameters of such automata by black-box optimization. We most of modern defenses to adversarial examples are found also experimentally show this formalism can produce exam- to be easily circumvented (Athalye, Carlini, and Wagner ples very close to so-called single Fourier attacks by study- 2018). One reason why adversarial examples are hard to de- ing Fourier harmonics of the obtained examples. fend against is the difficulty of constructing a theory of the The main contributions of the paper are following: crafting process of them. Intriguingly, adversarial perturbations can be transferable • We show that the iterative process to generate Universal across inputs. Universal Adversarial Perturbations (UAPs), Adversarial Perturbations from (Khrulkov and Oseledets single disturbances of small magnitude that lead to misclas- 2018) can be reformulated as a cellular automata that gen- sification of most inputs, were presented in image domain erates Turing patterns. by Moosavi-Dezfooli et. al (Moosavi-Dezfooli et al. 2017), • We experimentally show Turing patterns can be used to Copyright c 2021, Association for the Advancement of Artificial generate UAPs in a black-box scenario with high fooling Intelligence (www.aaai.org). All rights reserved. rates for different networks. 2683 (a) Example of UAPs constructed by (Khrulkov and Oseledets 2018) (b) Turing patterns Figure 1: Visual similarity of Universal Adversarial Perturbations by (Khrulkov and Oseledets 2018) and Turing patterns Background to provide fast convergence: T Universal Adversarial Perturbations p0 (Ji (Xb) q(Ji(Xb)"t)) "t+1 = T ; (4) Adversarial perturbations are small disturbances added to k p0 (Ji (Xb) q(Ji(Xb)"t))kp the inputs that cause machine learning models to make a where 1 + 1 = 1 and (z) = sign(z)jzjr−1, and J (X ) 2 mistake. In (Szegedy et al. 2013) authors discovered these p0 p r i b Rbdi×d noises by solving the optimization problem: for a batch Xb with batch size b is given as a block matrix: 2 3 min k"k2 s:t: C(x + ") 6= C(x); (1) Ji(x1) " 6 . 7 Ji(Xb) = . : (5) x C(·) 4 5 where is an input object and is a neural network clas- J (x ) sifier. It was shown that the solution to the minimization i b model (1) leads to perturbations imperceptible to human eye. For the case of p = 1 (4) takes the form: T Universal adversarial perturbation (Moosavi-Dezfooli "t+1 = sign(Ji (Xb) q(Ji(Xb)"t)): (6) et al. 2017) is a small (k"kp ≤ L) noise that makes clas- sifier to misclassify the fraction (1 − δ) of inputs from the Equation (6) is the first crucial point in our study, and we given dataset µ. The goal is to make δ as small as possible will show, how they connect to Turing patterns. We now and find a perturbation " such that: proceed with describing the background behind these pat- terns and mathematical correspondence between Equation Px∼µ [C(x + ") 6= C(x)] ≥ 1 − δ s:t: k"kp ≤ L; (2) (6) and emergence of Turing patterns as cellular automata is described in next Section. In (Khrulkov and Oseledets 2018) an efficient way of computing such UAPs was proposed, achieving relatively Turing Patterns as Cellular Automata high fooling rates using only small number of inputs. Con- In his seminal work (Turing 1952) Alan Turing studied the Rd Rdi sider an input x 2 , its i-th layer output fi(x) 2 and emergence theory of patterns in nature such as stripes and @f (x) J (x) = i 2 Rdi×d spots that can arise out of a homogeneous uniform state. Jacobian matrix i @x . For a small in- x The proposed model was based on the solution of reaction- Rd put perturbation " 2 , using first-order Taylor expansion diffusion equations of two chemical morphogens (reagents): fi(x + ") ≈ fi(x) + Ji(x)", authors find that to construct a UAP, it is sufficient to maximize the sum of norms of Ja- @n1(i; j) 2 = −µ1r n1(i; j) + a(n1(i; j); n2(i; j)); cobian matrix product with perturbation for a small batch of @t (7) inputs X , constrained with perturbation norm (k"k = L is @n (i; j) b p 2 = −µ r2n (i; j) + b(n (i; j); n (i; j)): obtained by multiplying the solution by L): @t 2 2 1 2 X q Here, n1(i; j) and n2(i; j) are concentrations of two kJi(xj)"kq ! max; s.t. k"kp = 1: (3) morphogens in the point with coordinates (i; j) µ1 and xj 2Xb µ2 are scalar coefficients. a and b are nonlinear func- To solve the optimization problem (3) the Boyd iteration tions, with at least two points (i; j), satisfying (i; j): (Boyd 1974) (generalization of the power method to the a(n1(i; j); n2(i; j)) = 0 problem of computing generalized singular vectors) is found b(n1(i; j); n2(i; j)) = 0: 2684 dj ×dj Turing noted the solution presents alternating patterns where Dj(x) = diag(θ(Mjfj−1(x))) 2 R and with specific size that does not depend on the coordinate 1; if z > 0 θ(z) = @ ReLU(z) = : and describes the stationary solution, which interpolates be- @z 0; if z < 0; tween zeros of a and b. The update matrix from Equation (13) is then: Young et. al (Young 1984) proposed to generate Turing patterns by a discrete cellular automata as following. Let us 2J (x )3 consider a 2D grid of cells. Each cell (i; j) is equipped with i 1 T T T 6 . 7 a number n(i; j) 2 f0; 1g. The sum of cells, neighbouring Ji (Xb)Ji(Xb) = Ji (x1) ··· Ji (xb) 4 . 5 = with the current cell (i; j) within the radius ri is multiplied Ji(xb) by w, while the sum of values of cells, neighbouring with the X = JT (x)J (x): (16) current cell (i; j) between radii rin and rout, is multiplied i i by −1. If the total sum of these two terms is positive, the x2Xb new value of the cell is 1, otherwise 0. This process can be Performance of the UAPs from (Khrulkov and Oseledets written by introducing the convolutional kernel Y (m; l): 2018) increases with the increase of batch size.