Procedural Noise Adversarial Examples for Black-Box A‚acks on Deep Convolutional Networks

Kenneth T. Co Luis Munoz-Gonz˜ alez´ Imperial College London Imperial College London [email protected] [email protected] Sixte de Maupeou Emil C. Lupu Imperial College London Imperial College London [email protected] [email protected]

ABSTRACT robust to malicious adversaries. Yet despite the prevalence of neural Deep Convolutional Networks (DCNs) have been shown to be vul- networks, their vulnerabilities are not yet fully understood. nerable to adversarial examples—perturbed inputs speci€cally de- It has been shown that machine learning systems are vulnerable signed to produce intentional errors in the learning algorithms at to aŠacks performed at test time [2, 23, 41, 47, 53]. In particular, test time. Existing input-agnostic adversarial perturbations exhibit DCNs have been shown to be susceptible to adversarial examples: interesting visual paŠerns that are currently unexplained. In this inputs indistinguishable from genuine data points but designed to paper, we introduce a structured approach for generating Universal be misclassi€ed by the learning algorithm [73]. As the perturbation Adversarial Perturbations (UAPs) with procedural noise functions. required to fool the learning algorithm is usually small, detecting Our approach unveils the systemic vulnerability of popular DCN adversarial examples is a challenging task. Fig. 1 shows an ad- models like Inception v3 and YOLO v3, with single noise paŠerns versarial example generated with the aŠack strategy we propose able to fool a model on up to 90% of the dataset. Procedural noise in this paper; the perturbed image of a tabby cat is misclassi€ed allows us to generate a distribution of UAPs with high universal eva- as a shower curtain. Although we focus on computer vision, this sion rates using only a few parameters. Additionally, we propose phenomenon has been shown in other application domains such Bayesian optimization to eciently learn procedural noise param- as speech processing [9, 11], malware classi€cation [19], and rein- eters to construct inexpensive untargeted black-box aŠacks. We forcement learning [24, 37] among others. demonstrate that it can achieve an average of less than 10 queries per successful aŠack, a 100-fold improvement on existing methods. We further motivate the use of input-agnostic defences to increase the stability of models to adversarial perturbations. Œe universality of our aŠacks suggests that DCN models may be sensitive to ag- gregations of low-level class-agnostic features. Œese €ndings give insight on the nature of some universal adversarial perturbations and how they could be generated in other applications.

CCS CONCEPTS •Computing methodologies → Machine learning; •Security and privacy → Usability in security and privacy; Figure 1: Adversarial example generated with a procedural KEYWORDS noise function. From le to right: original image, adversar- ial example, and procedural noise (magni€ed for visibility). arXiv:1810.00470v4 [cs.CR] 23 Nov 2019 Adversarial examples; Bayesian optimization; black-box aŠacks; computer vision; deep neural networks; procedural noise Below are the classi€er’s top 5 output probabilities.

In this paper, we propose a novel approach for generating adver- 1 INTRODUCTION sarial examples based on the use of procedural noise functions. Such Advances in computation and machine learning have enabled deep functions are commonly used in computer graphics and designed learning methods to become the favoured algorithms for various to be parametrizable, fast, and lightweight [34]. Œeir primary pur- tasks such as computer vision [31], malware detection [64], and pose is to algorithmically generate textures and paŠerns on the ƒy. speech recognition [22]. Deep Convolutional Networks (DCN) Procedurally generated noise paŠerns have interesting structures achieve human-like or beŠer performance in some of these appli- that are visually similar to those in existing universal adversarial cations. Given their increased use in safety-critical and security perturbations [30, 44]. applications such as autonomous vehicles [4, 74, 76], intrusion We empirically demonstrate that DCNs are fragile to procedural detection [28, 29], malicious string detection [65], and facial recog- noise and these act as Universal Adversarial Perturbations (UAPs), nition [39, 70], it is important to ensure that such algorithms are i.e. input-agnostic adversarial perturbations. Our experimental 1 results on the large-scale ImageNet classi€ers show that our pro- generalize are more ecient because they do not need to be re- posed black-box aŠacks can fool classi€ers on up to 98.3% of input computed for new data points or models. Œeir generalizability can examples. Œe aŠack also transfers to the object detection task, be described by their transferability and universality. showing that it has an obfuscating e‚ect on objects against the Input-speci€c adversarial perturbations are designed for a spe- YOLO v3 object detector [60]. Œese results suggest that large- ci€c input against a given model, these are neither transferable scale indiscriminate black-box aŠacks against DCN-based machine or universal. Transferable adversarial perturbations can fool mul- learning services are not only possible but can be realized at low tiple models [51] when applied to the same input. Œis property computational costs. Our contributions are as follows: enhances the strength of the aŠack, as the same adversarial input • We show a novel and intuitive vulnerability of DCNs in can degrade the performance of multiple models, and makes pos- computer vision tasks to procedural noise perturbations. sible black-box aŠacks through surrogate models. Perturbations Œese functions characterize a distribution of noise pat- are universal when the same adversarial perturbation can be ap- terns with high universal evasion, and universal pertur- plied successfully across a large portion of the input dataset to bations optimized on small datasets generalize to datasets fool a classi€er [44]. Cross-model universal perturbations are both that are 10 to 100 times larger. To our knowledge, this is transferable and universal, i.e., they generalize across both a large the €rst model-agnostic black-box generation of universal portion of the inputs and across models. Generating adversarial adversarial perturbations. perturbations that generalize is suitable and more ecient in at- • We propose Bayesian optimization [43, 68] as an e‚ective tacks that target a large number of data points and models, i.e. for tool to augment black-box aŠacks. In particular, we show broad spectrum indiscriminate aŠacks. In contrast, input-speci€c that it can use our procedural noise to cra‰ inexpensive aŠacks may be easier to cra‰ when a few speci€c data points or universal and input-speci€c black-box aŠacks. It improves models are targeted or for targeted aŠacks where the aŠacker aims on the query eciency of random parameter selection by to produce some speci€c types of errors. 5-fold and consistently outperforms the popular L-BFGS optimization algorithm. Against existing query-ecient black-box aŠacks, we achieve a 100 times improvement on the query eciency while maintaining a competitive 2.2 Degree of Knowledge success rate. For evasion aŠacks, we assume that the aŠacker has access to the • We show evidence that our procedural noise UAPs appear test input and output. Beyond this, the adversary’s knowledge to exploit low-level features in DCNs, and that this vul- and capabilities range from no access or knowledge of the targeted nerability may be exploited to create universal adversarial system to complete control of the target model. Accordingly, aŠacks perturbations across applications. We also highlight the can be broadly classi€ed as: white-box, grey-box, or black-box [53]. shortcomings of adversarial training and suggest using In white-box seŠings, the adversary has complete knowledge of more input-agnostic defences to reduce model sensitivity the model architecture, parameters, and training data. Œis is the to adversarial perturbations. seŠing adopted by many existing studies including [10, 18, 32, 40, 45, 73]. In grey-box seŠings, the adversary can build a surrogate Œe rest of the paper is structured as follows. In Sect. 2, we model of similar scale and has access to training data similar to that de€ne a taxonomy to evaluate evasion aŠacks. In Sect. 3, we de- used to train the target system. Œis seŠing is adopted in transfer scribe and motivate the use of procedural noise functions. In Sect. 4, aŠacks where white-box adversarial examples are generated on a we demonstrate how di‚erent DCN architectures used in image surrogate model to aŠack the targeted model [33, 52]. Œis approach classi€cation have vulnerabilities to procedural noise. In Sect. 5, we can also be adapted for a black-box seŠing. For example Papernot show how to leverage this vulnerability to create ecient black-box et al. [52] apply a heuristic to generate synthetic data based on aŠacks. In Sect. 6, we analyze how the aŠack transfers to the object queries to the target classi€er, thus removing the requirement for detection task and discuss how it can generalize to other applica- labelled training data. In a black-box seŠing, the adversary has no tion domains. In Sect. 7, we explore denoising as a preliminary knowledge of the target model and no access to surrogate datasets. countermeasure. Finally, in Sect. 8, we summarize our €ndings and Œe only interaction with the target model is by querying it, this is suggest future research directions. o‰en referred to as an “oracle”. 2 ATTACK TAXONOMY Given the adversary’s lack of knowledge, black-box aŠacks rely heavily on making numerous queries to gain information. Œis Our study focuses on aŠacks at test time, also known as evasion increases the chances of the aŠack being detected. Œus, the most aˆacks. To determine the viability and impact of aŠacks in practical dangerous aŠacks are those that require the least queries and re- seŠings, we categorize them according to three factors: (a) the sources. With fewer queries, adversarial perturbations are gener- generalizability of their perturbations, (b) the access and knowledge ated sooner, costs (when using a paid service) are lower and the the adversary requires, and (c) the desired output. Œese factors volume of suspicious queries is reduced. Existing black box aŠacks also describe the threat model being considered. like [3, 12] have shown some success with zeroth order optimization and gradient estimation. However they require tens to hundreds of 2.1 Generalizability thousands of queries on datasets with a large number of features, Œe generalizability of adversarial perturbations refers to their abil- as in realistic natural-image dataset like ImageNet [13, 25]. Œe ity to apply across a dataset or to other models. Perturbations that most query-ecient method reported so far is a bandit optimization 2 framework that achieves 92.9% success with an average of 1, 000 like natural-image classi€cation where existing aŠacks explore the queries per image on ImageNet [25]. entire input space, which has very high dimensionality (≥ 100,000). Procedural noise functions can be classi€ed into three categories: 2.3 Desired Output laŠice gradient noise, sparse convolution noise, and explicit noise Œe adversary’s goal varies according to the application and the ex- [34]. LaŠice gradient noise is generated by interpolating random pected rewards gained from exploiting the system. Usually, aŠacks values or gradients at the points of an integer laŠice, with Perlin noise as a representative example. Sparse convolution noise is a are considered as either targeted or untargeted (indiscriminate). 1 In targeted aŠacks the adversary aims for a speci€c subset of sum of randomly positioned and weighted kernels, with Gabor inputs to be misclassi€ed as their chosen output. In untargeted noise as a representative example. Explicit noise di‚ers from the aŠacks, the aŠacker aims to cause classi€cation errors on a subset others in that the images are generated in advance and stored of inputs. Both these aŠacks disrupt the machine learning system later for retrieval. Œis induces large memory costs and limits its by forcing errors and undermining the model’s reliability. applicability as an inexpensive aŠack. We therefore do not use it In multi-class classi€cation, targeted aŠacks are more challeng- here and leave its investigation for future work. ing due to their speci€city, but successful ones allow a greater degree of manipulation for the aŠacker. On the other side, untar- 3.2 Perlin Noise geted aŠacks are typically easier, as they just have to evade the We chose to use Perlin noise as a representative example for laŠice “correct” classi€cation, and this characteristic is more suited for gradient noise, because of its ease of use, popularity, and simplic- broad indiscriminate aŠacks. ity. Perlin noise was developed as a technique to produce natural- looking textures for computer graphics, with its initial application 3 PROCEDURAL NOISE in motion pictures where it has remained a staple of the industry We introduce procedural noise functions as an intuitive and com- [34]. Although Perlin noise may not be the most expressive noise putationally ecient approach to generate adversarial examples function, it has a simple implementation which makes it suitable in black-box seŠings. Procedural noise functions are algorithmic for inexpensive black-box aŠacks as it is controlled by only a few techniques used for generating image paŠerns, typically used in the parameters. creation of natural details to enhance images in video and graph- We summarize the formal construction of two-dimensional Per- ics production. Œey are designed to be fast to evaluate, scale to lin nose as described by Perlin [54–56]. Œe value at a point (x,y) large dimensions, and have low memory footprint. Œese aŠributes is derived as follows: let (i, j) de€ne the four laŠice points of the make them desirable for generating computationally inexpensive laŠice square where i = {|x |, |x| + 1} and j = {|y|, |y| + 1}. Œe four perturbations. For a more comprehensive survey on procedural gradients are given by qij = V[Q[Q[i] + j]] where precomputed noise, we refer the reader to Lagae et al. [34]. arrays Q and V contain a pseudo-random permutation and pseudo- random unit gradient vectors respectively. Œe four linear functions 3.1 Motivation qij (x − i,y − j) are then bilinearly interpolated by s(x − |x|) and s(y − |y|), where s(t) = 6t5 − 15t4 + 10t3. Œe result is the Perlin Existing Universal Adversarial Perturbations (UAPs) generated by noise value p(x,y) for coordinates (x,y). white-box aŠacks exhibit interesting visual structures that are [44], Œe Perlin noise function has several parameters that determine as of yet, not fully understood. UAPs are particularly interesting the visual appearance of the noise. In our implementation, the as their universality reveals more generic or class-agnostic fea- wavelengths λx , λy and number of octaves Ω contribute the most tures that machine learning algorithms appear to be sensitive to. to the visual change. Œe noise value at point (x,y) with parameters In contrast, input-speci€c adversarial perturbations, though less δper = {λx , λy , Ω} becomes detectable in many cases, can “over€t” and apply only to the inputs they were designed for [79]. Ω Õ 2n−1 2n−1 Current approaches, like the following, cra‰ UAPs using white- Sper(x,y) = p(x · ,y · ) λx λy box knowledge of the model’s learned parameters. Moosavi-Dezfooli n=1 et al. [44] use the DeepFool algorithm [45] iteratively over a set of images. Mopuri et al. [46] use Generative Adversarial Nets (GANs) To achieve more distinct visual paŠerns, we use a sine colour ϕ to compute UAPs, whilst Khrulkov and Oseledets [30] propose to map with an additional frequency parameter sine. Œis colour map p C(p) = (p · πϕ ) use the singular vector method that maximizes the di‚erence in for the noise value is de€ned by sin 2 sine . Œe activations at a targeted hidden layer between the original and the periodicity of the sine function creates distinct bands in the image adversarial examples. to achieve a high frequency of edges. Œe resulting noise generating G (x,y) We hypothesize that procedural noise, which exhibits paŠerns function per at point is de€ned as the composition of our visually similar to those of UAPs (see Appendix A), can also act as Perlin noise function and the sine colour map, a UAP. Procedural noise is simple to implement, fast to compute, Gper(x,y) = C(Sper(x,y)) = sin((Sper(x,y)· 2πϕ ) and does not require the additional overhead of building, training, sine or accessing a DCN to generate adversarial perturbations. Œe with combined parameters δper = {λx , λy , ϕsine, Ω}. parametrization of procedural noise is simpler and this results in a reduced search space, which can enable query-ecient black- box aŠacks. Œis is particularly useful in large-scale applications 1A kernel in image processing refers to a matrix used for image convolution. 3 3.3 Gabor Noise 4.1 Experiment Setup We use Gabor noise as a representative example for sparse con- Œe data used are 5,000 random images from the validation set of volution noise. It has more accurate spectral control than other the ILSVRC2012 ImageNet classi€cation task [63]. It is a widely procedural noise functions [35], where spectral control refers to used object recognition benchmark with images taken from vari- the ability to control the appearance of noise as measured by its ous search engines and manually labelled to 1,000 distinct object energy along each frequency band. Gabor noise can be quickly categories where each image is assigned one ground truth label. evaluated at any point in space and is characterized by only a few Models. We use four distinct DCN architectures pre-trained parameters such as orientation, frequency, and bandwidth [35]. In on ImageNet: VGG-19 [67], ResNet-50 [21], Inception v3 [72], and essence, Gabor noise is a convolution between sparse white noise Inception ResNet v2 [71]. We abbreviate Inception ResNet v2 to and a Gabor kernel g. Œe Gabor kernel is the product of a circular IRv2. Inception v3 and Inception ResNet v2 take input images with Gaussian and a Harmonic function: dimensions 299 × 299 × 3 while the remaining two networks take 2 2 2 images with dimensions 224 × 224 × 3. g(x,y) = e−π σ (x +y ) cos [ 2π (x cos ω + y sin ω)], λ We also take an ensemble adversarially trained version of the where σ is the width of the Gaussian, λ and ω are the period and Inception ResNet v2 architecture: Tramer et al. [75] €ne-tuned the orientation of the Harmonic function [34]. Our implementation of IRv2 network by applying ensemble adversarial training, making it the Gabor kernel uses the OpenCV library [5]. Œe value Sgab(x,y) more robust to gradient-based aŠacks. We refer to this new model at point (x,y) is the sparse convolution with a Gabor kernel where as IRv2ens. It has the same architecture, but di‚erent weights when {(xi ,yi )} are the random points [34]. Gabor noise is an expressive compared to the €rst IRv2. For complete details of the ensemble noise function and will have a large number of dimensions if each adversarial training process, we refer the reader to [75]. random point is assigned di‚erent kernel parameters. To simplify Perturbations. Œe Perlin noise parameters λx , λy , ϕsine and the implementation, we use the same parameters and weights for Gabor noise parameters σ, λ are positive and bounded above by each random point (xi ,yi ). Œis results in noise paŠerns that have the image’s side length d. Increasing the parameters beyond this a uniform texture. will have no impact on the resulting image as it gets clipped by the We add an additional discrete parameter ξ to control the isotropy image dimension. Œe isotropy ξ ∈ [1, 12] and number of octaves of the Gabor noise. Having ξ = 1 results in anisotropic noise, which Ω ∈ [1, 4] are discrete, and evaluations show negligible change in is oriented in one direction. Progressively larger ξ makes it more the image for ξ > 12 and Ω > 4. Œe range on the angle ω ∈ [0, 2π] uniform in all directions. Implementing these changes gives an covers all directions. updated formulation We €x the kernel size and number of random points for Gabor ξ noise so that the Gabor kernels always populate the entire image. Õ Õ ( ) 1 ( − − nπ ) Œe parameters δgab of the Gabor noise have the greater inƒuence Sgab x,y = ξ g x xi ,y yi ; σ, λ, ω + ξ . n=1 i on the resulting visual appearance of the noise paŠern. To provide a baseline, we also test the models against uniform To achieve high-frequency paŠerns and remove unwanted low- random noise perturbations: sgn(r)· ε where r ∈ U(−1, 1)d×d×3, d contrast oscillations, we normalize the variance spectrum of the is the image’s side length, and ε is the `∞-norm constraint on the Gabor noise using the algorithm described by Neyret and Heitz [48]. perturbation. Œis is an `∞-optimized uniform random noise, and it Œis results in min-max oscillations. Note that we refer speci€cally is reasonable to say that aŠacks that signi€cantly outperform this to the frequency of edges, i.e. how o‰en edges appear one a‰er the baseline are non-trivial. other on an image. Œis is di‚erent from the classical frequency Metrics. To measure the universality of a perturbation, we used in spectral analysis. For simplicity we use “low-frequency” or de€ne the universal evasion rate of a perturbation over the dataset. “high-frequency” when referring to paŠerns with low or high edge Given model output f , input x ∈ X, perturbation s, and small ε > 0, frequency in the image. the universal evasion of s over X is We have tried applying the sine colour map to Gabor noise, but it obscures the Gabor kernel structures. Similarly, we have |{x ∈ X : arg max f (x + s) , τ (x)}| tried normalizing the variance spectrum of Perlin noise, but this , ksk∞ ≤ ε, resulted in ƒat images with few edges and no distinct paŠerns. Œe |X | €nal noise generating function Gper with a normalized variance { } spectrum has parameters δgab = σ, λ, ω, ξ . where τ (x) is the true class label of x. An `∞-norm constraint on s ensures that the perturbation is small and does not drastically 4 VULNERABILITY TO PROCEDURAL NOISE alter the visual appearance of the resulting image; this is a proxy In this section, we empirically demonstrate the vulnerability of €ve for the constraint τ (x) = τ (x + s). We choose the `∞-norm as it di‚erent DCN architectures to procedural noise for the ImageNet is straightforward to impose for procedural noise perturbations classi€cation task. We show that randomly chosen procedural noise and is o‰en used in the adversarial machine learning literature. perturbations act as e‚ective UAPs against these classi€ers. Œe In this case, f is the probability output vector, but note that the procedural noise UAPs greatly exceed the baseline uniform random aŠacker only needs to know the output class label arg max f (x + s) noise, is on average universal on more than half the dataset across to determine if their aŠacks succeed. To measure the model’s di‚erent models, and fools each model on more than 72% of the sensitivity against the perturbations for each input, we de€ne the dataset when considering input-speci€c evasion. model’s average sensitivity on an input x over perturbations s ∈ S 4 as Table 1: Universal evasion (%) by percentile for random and |{s ∈ S : arg max f (x + s) , τ (x)}| procedural noise perturbations on Inception v3. , ksk∞ ≤ ε. |S| Œis will help determine the portion of the dataset on which the Percentile Random Gabor Perlin model is more vulnerable. Finally, the input-speci€c evasion rate is min 26.0 24.7 26.4 47.2 |{x ∈ X : s ∈ S such that arg max f (x + s) , τ (x)}| 25th 26.8 39.9 , ksk ≤ ε. 50th 27.0 49.1 57.7 ∃ | | ∞ X 75th 27.1 53.3 63.7 Œis measures how many inputs in the dataset can be evaded with max 27.9 61.8 73.2 perturbations from S. Experiment. We evaluate our procedural noise perturbations on 5,000 random images from the validation set. Œe perturbations For Gabor noise, the parameters σ, ω, and ξ have low correlation generated are from 1,000 Gabor noise, 1,000 Perlin noise, and 10,000 (≤ 0.16) with universal evasion. Œe remaining parameter λ how- uniform random perturbations. 1,000 queries for the procedural ever has a high correlation (≥ 0.52) with the universal evasion rate noise functions was sucient for our results and its search space across all classi€ers; this peaks at 0.90 for IRv2ens. λ corresponds to only has four bounded parameters. We use an `∞-norm constraint the wavelength of the harmonic kernel. Visually, lower values of λ of ε = 16. Œe pixels of the perturbations are €rst clipped to [−ε, ε] indicate thinner and more frequent bands, which is why λ can also and the resulting adversarial example’s pixels are clipped to the be thought of as an inverse frequency. Œe correlations suggest that image space [0, 255]. low-frequency Gabor noise paŠerns correlate with high universal Œus, this is an untargeted black-box aŠack with exactly 1,000 evasion, and this appears to be more pronounced as we move to queries per image for procedural noise. Note that the adversary has the more robust models. no knowledge of the target model and only requires the top label For Perlin noise, the parameters λx and λy have low correlation from the model’s output. (≤ 0.25) with universal evasion. Œe universal evasion rates have a moderate negative correlation with the number of octaves Ω and 4.2 Universality of Perturbations have a moderately-high positive correlation with ϕsine for the non- Œe procedural noise perturbations appear to be both universal adversarially trained models. Œis suggests that high-frequency and transferable across models. Œe results in Fig. 2 show that the Perlin noise paŠerns correlate with high universal evasion. Œe models in order from least to most robust against the perturbations number of octaves indicate the amount of detail or the number are: VGG-19, ResNet-50, Inception v3, IRv2, and then IRv2ens. Œis of curvatures within the image, so a slight negative correlation is not surprising, as the generalization error for these models appear indicates that some paŠerns that achieve high evasion have fewer in the same order, with larger generalization error indicating a less curvatures and details. Visually, the frequency of the sine function robust model. Œe ensemble adversarially trained model has also ϕsine can be thought of as the thinness of the bands in the image. been hardened against adversarial examples, so it was expected to Œis is the opposite of the λ in Gabor noise, so we have a similar be less a‚ected. However, the ensemble adversarial training did result where IRv2ens is more susceptible to low-frequency paŠerns. not fully mitigate the impact of the procedural noise. Low-frequency paŠerns are also more e‚ective on IRv2ens be- Both Gabor and Perlin noise have signi€cantly higher universal cause this model was adversarially trained on gradient-based at- evasion rate than random noise. In Fig. 2a, we represent random tacks, which we hypothesize generated mostly high-frequency per- noise using the median (which, in this case, is very close to the turbations. Œe frequency appears to be an important factor in mean) of its universal evasion rate over all 10,000 perturbations, as determining the strength of the UAP even when Gabor and Per- the variance is very small (less than 10−5) for each model. lin noise have opposite correlations between their frequency and Procedural noise functions create a distribution whose modes their universal evasion. Œis di‚erence shows that these noise pat- have high universal evasion rates. Table 1 shows an example of this terns have notably di‚erent characteristics. Complete correlation on Inception v3, where more than half the Perlin noise perturbations matrices are available in Appendix C. achieve evasion on more than 57% of the dataset. Cross-model Universality. Œe UAPs with high universal eva- Between the two procedural noises, Perlin noise is a stronger sion on one model o‰en have high universal evasion on the other UAP, though not by a large margin. In Fig. 2a, Gabor noise appears models. For example, a Perlin noise perturbation has 86.7%, 77.4%, bimodal, especially with the IRv2 classi€ers. Œis bimodal aspect 73.2%, 58.2%, and 45.9% universal evasion on VGG-19, ResNet-50, shows that there are two large disjoint subsets of Gabor noise: one Inception v3, IRv2, and IRv2ens respectively. with high universal evasion and another with low universal evasion. Œe correlation of the universal evasion rates across models is Œis was not as prominent for Perlin noise. high (≥ 0.80), which shows the transferability of our UAPs across Best Parameters. Given the distribution of universal evasion models. For Gabor noise, there is a high correlation across all mod- rates in Fig. 2a, it is interesting to observe which parameters of els. For Perlin noise, there appears to be a noticeable di‚erence the noise functions contribute most to the evasion rate. Because between IRv2ens and the remaining four models. Œe strongest Per- there are only four parameters for each procedural noise function, lin noise on the undefended models have high-frequency paŠerns the linear correlations between these parameters and the universal that the adversarial training partially mitigated. evasion rates give a general overview of their inƒuence. Œough Overall, the results show that procedural noise perturbations act this analysis may not fully capture multi-variable relationships. as UAPs across all models. Œis is the €rst black-box generation of 5 (a) Universal Evasion Rate

(b) Average Sensitivity

Figure 2: Histogram of (a) universal evasion rate over all perturbations and (b) average sensitivity over all inputs across all models. For example, VGG-19 in (a) shows that about 12% of the Gabor noise perturbations have a universal evasion rate of approximately 0.8. cross-model UAPs, and the procedural noise allows us to draw per- Œis is less so between ResNet-50 and VGG-19 whose correlations turbations from a distribution with high average universal evasion. with the other models range from 0.56-0.81. We now look at the results from an input-speci€c perspective. Input-speci€c Evasion. We consider the case when our untar- geted black-box aŠack is used as an input-speci€c aŠack, i.e. the 4.3 Model Sensitivity on Inputs adversary only needs to €nd at least one adversarial perturbation that evades each input. Œus, Evasion on input x is achieved when Œe model’s sensitivity could vary across the input dataset, i.e. the s ∈ S such that arg max f (x + s) , τ (x). Œis is in contrast to the model’s predictions are stable for some inputs while others are universal∃ aŠack where the adversary cra‰s a single perturbation to more susceptible to small perturbations. We measure this with the fool the model on as many inputs as possible. average sensitivity of single inputs over all perturbations. Results Note that the random noise is optimized for the `∞-norm. We in Fig. 2b show that the average sensitivity of the dataset is bimodal draw the random noise perturbation from {−ε, ε}d×d×3. Œus, for for all models and both procedural noise functions. Œere are two each pixel the added noise is either −ε or ε, rather than drawing distinct subsets of the data: one on the right that is very sensitive from the continuous domain (−ε, ε). It is reasonable to think that a and the other on the le‰ that is very insensitive to the perturbations. larger perturbation is more likely to cause evasion. Œe remaining data points are somewhat uniformly spread in the Table 2 shows that VGG-19 and ResNet-50 are particularly fragile middle. Œe number of points on the le‰ peak of the histogram in as even random noise greatly degrades their performance. Although Fig. 2b is larger for the most robust models. Similarly to Fig. 2a, the Inception models are more robust, both procedural noise pertur- this progression indicates that the order of most to least sensitive bations still achieve more than 72% evasion on all of them. Although models align with the most to least robust models. We omit results ensemble adversarial training improved the robustness of IRv2, it for random noise since more than 60% of the input dataset are still does not mitigate the impact of the procedural noise aŠack. barely a‚ected by it across all models. Œis manifests as a tall peak Œe idea of ensemble adversarial training was to decouple the gen- on the le‰ and a short peak on the right. eration of the adversarial training set from the original model, but When comparing the average sensitivity of inputs between the this was limited since it was only done for gradient-based aŠacks. two procedural noise functions on the same models, the correlations We argue that defences should be more input-agnostic to avoid range between 0.89-0.92, which shows that both procedural noise having to train against all types of aŠacks. perturbations a‚ect very similar groups of inputs for each model. Label Analysis. Œe procedural noise perturbations were not Œe correlation between the average sensitivities for each input designed to be a targeted aŠack, but intuitively, the same universal across the Inception models is at least 0.79, which suggests that perturbation would cause misclassi€cation towards class labels that these models are sensitive to procedural noise on similar inputs. 6 Table 2: Input-speci€c evasion rate (in %) for random and model. Moreover, we test our aŠack against an object detection procedural noise perturbations. Original refers to the top 1 model in Sect. 6, and show that it also degrades the performance of error on the unaltered original images. Strongest attack on models that have a spatial component in their learning task. each classi€er is highlighted. Generalizability of Procedural Noise. From the label analy- sis of procedural noise UAPs, we observe that most perturbations Classi€er Original Random Gabor Perlin with high universal evasion do not have a strong bias towards VGG-19 29.4 57.1 97.7 98.3 any particular class label—no particular class is targeted more ResNet-50 25.9 55.7 96.2 96.3 than 10% for over 80% of the procedural noise UAPs. Œis suggest Inception v3 22.3 46.8 89.2 93.6 that UAPs leverage more generic low-level features that the model IRv2 20.1 38.7 81.1 87.0 learns, which would explain their universality and indiscriminate IRv2ens 20.1 37.5 72.7 79.4 behaviour in causing misclassi€cation. Amongst white-box UAP aŠacks, our procedural noise has the closest visual appearance to perturbations from the Singular Vector have visually similar textures to the procedural noise paŠern. We AŠack (SVA) by Khrulkov and Oseledets [30]. Œey generate UAPs €nd that this holds true for only a few of the procedural noise UAPs. targeting speci€c layers of DCNs, and found that targeting earlier For a given procedural noise perturbation, we de€ne its top label layers of the network generated more successful UAPs. Œe paŠerns to be the class label it causes the most misclassi€cation towards. obtained for these earlier layers share a visual appearance with When looking at procedural noise UAPs across all models, about 90% procedural noise. Œese layers also correspond to the low-level of Gabor noise perturbations have their top label apply to at most features learned by the network. Other evidence also suggests that 9% of the inputs. For Perlin noise, about 80% of its perturbations procedural noise exploits low-level features, as feature visualization have their top label on at most 10% of the input. Œere are however of earlier layers in neural networks share the same visual appear- a few outliers that have their top label appear above 10% of the ance with some procedural noise paŠerns. Convolutional layers inputs. For example, on Inception v3, the top Gabor noise with induce a prior on DCNs to learn local spatial information [17], and 61.8% universal evasion has “window screen” as its top label and it DCNs trained on natural-image datasets learn convolution €lters applies for 37.8% of the evaded inputs. In contrast, another Gabor that are similar in appearance to Gabor kernels and colour blobs noise perturbation with 58.9% universal evasion has “quilt” as its [49, 78]. Gabor noise appears to be a simple collection of low-level top label, but it only applies for 6.0% of the evaded inputs. As a features whereas Perlin noise seems to be a more complex mixture consequence, it is still possible to use procedural noise to create of low-level features. Œis di‚erence in the complexity of their universal targeted UAPs aimed at speci€c class labels like “window visual appearance may explain why Perlin noise is a stronger aŠack screen” or “brain coral”. However the class labels we can target than Gabor noise. is dependent on the procedural noise and the overall success of Procedural noise aŠacks transfer with high correlation across universal targeted aŠacks may be limited, as it is more dicult to the models most likely because they share the same training set make a perturbation both universal and targeted. (ImageNet), learning algorithms (e.g. backpropagation), and have Perlin noise has a relatively large amount of “brain coral” clas- similar components in their architectures (e.g. convolutional lay- si€cations, with other labels such as “maze” and “shower curtain” ers). Œis increases the likelihood that they share input-agnostic also appearing frequently in the top €ve most classi€ed labels per vulnerabilities. In this way, our results appear to support the idea classi€er. For Gabor noise, there was no label that consistently that DCNs are sensitive to aggregations of low-level features. appeared at the top across all models. Œese results indicate that Security Implications. In transfer learning, a model trained Perlin noise has a larger bias towards certain classes, while Gabor on one task is re-purposed as an initialization or €xed feature noise is more indiscriminate. extractor for another similar or related task. When used as a feature extractor, the initial layers are o‰en frozen to preserve the low-level 4.4 Discussion features learned. Œe subsequent layers, closer to the output, are Adversarial examples exploit the model’s sensitivity, causing large then re-trained for the new task [78]. Hidden layers from models changes in the model’s output by applying speci€c small changes pre-trained on the ImageNet dataset are o‰en re-used for other to the input. More speci€cally, recent work has shown adversarial natural-image classi€cation tasks [50]. Œis makes it a notable examples exploit ”non-robust” features that the model learns but target as vulnerabilities that exploit low-level features encoded in that are incomprehensible to humans [26]. It is likely that UAPs the earlier layers carry over to the new models. may be exploiting ”universal” non-robust features that the DCN Transfer learning has many bene€ts as training entire models has learned. from scratch for large-scale tasks like natural-image classi€cation Textures. Previous results have shown ImageNet-trained DCNs can be costly both computationally in training and in terms of to be more reliant on textures rather than shapes [14]. Œough gathering the required data. However, this creates a systemic threat, not explicitly shown in the later Inception architectures (Inception as subsequent models will also be vulnerable to aŠacks on low-level v3, IRv2), these are still likely to have a strong texture-bias due features like procedural noise. Precisely characterizing the extent to similarities in the training. Œe texture bias however does not of the vulnerability of re-purposed models to the same aŠack is an fully explain why small, and sometimes imperceptible, changes interesting direction for future work. to the texture can drastically alter the classi€cation output. We Procedural noise is an accessible and inexpensive way for gener- aŠribute adversarial perturbations more to the sensitivity of the ating UAPs against existing image classi€ers. In our experiments 7 the Gabor and Perlin noise functions modi€ed only four parameters, aŠacks and show that Bayesian optimization is an ecient method and each parameter was bounded in a closed interval. Drawing from for choosing parameters in such black-box aŠacks. this space of perturbations has generated UAPs with high universal evasion rates. AŠackers can take advantage of the small search 5.1 Bayesian Optimization space by using procedural noise to cra‰ query-ecient untargeted Bayesian optimization is a sequential optimization algorithm used black-box aŠacks as we will show in Sect. 5. to €nd optimal parameters for black-box objective functions [43, 68]. Strengths & Limitations. Procedural noise is one of the €rst Œis technique is o‰en e‚ective in solving various problems with black-box generation of UAPs. Other existing black-box aŠacks expensive cost functions such as hyperparameter tuning, reinforce- o‰en optimize for input-speci€c evasion, and other existing UAP ment learning, and combinatorial optimization [66]. Bayesian op- generation methods are white-box or grey-box. Whilst procedural timization consists of a probabilistic surrogate model, usually a noise is also a generative model, it di‚ers from other generative Gaussian Process (GP), and an acquisition function that guides its models like Bayesian networks, Generative Adversarial Networks queries. GP regression is used to update the belief on the parameters (GANs), or Variational Autoencoders (VAEs) as it does not require with respect to the objective function a‰er each query [66]. the additional overhead of building and training these generative Gaussian Processes. A GP is the generalization of Gaussian models–which o‰en requires more resources and stronger adver- distributions to a distribution over functions and is typically used saries to execute successfully. as the surrogate model for Bayesian optimization [57]. We use GPs Comparing procedural noise aŠacks with existing black-box at- as they induce a posterior distribution over the objective function tacks is not straightforward, as procedural noise naturally has high that is analytically tractable. Œis allows us to update our beliefs universal evasion rates. Œis gives procedural noise an advantage about the objective function a‰er each iteration [68]. in that, despite having no access to the target model, randomly A Gaussian Process GP(m,k) is fully described by a prior mean drawn procedural noise paŠerns are likely to have high universal function m : X → R and positive-de€nite kernel or covariance evasion. Œe search space for our procedural noise has only four function k : X × X → R. We describe GP regression to understand dimensions, whereas most aŠacks are designed for the whole input how our GP is updated when more observations are available. Œe space of hundreds of thousands of dimensions. However, this does following expressions give the GP prior and Gaussian likelihood come at a cost, as procedural noise functions do not capture ad- respectively versarial perturbations outside their codomain. Other aŠacks that ( | ) N( ( ) ) explore the whole input space are able to take full advantage of a p f X = m X , K stronger adversary (i.e. white-box or grey-box seŠing) or larger p(y | f , X) = N(f (X), σ 2I) query limits. We explore this trade-o‚ further with input-speci€c where N denotes a normal distribution. Elements of the mean and black-box aŠacks in Sect. 5.3. covariance matrix are given by mi = m(xi ) and Ki, j = k(xi , xj ). Another limitation of this experiment was the use of an `∞-norm Given observations {X, y} and an arbitrary point x, the updated constraint. Although it is o‰en used in the literature as a proxy posterior mean and covariance on the n-th query are given by for human perception, `p -norms have limitations [16] e.g., low- 2 −1 frequency paŠerns appear to be more visible than high-frequency mn(x) = m(x) − (K + σ I) (y − m(X)) paŠerns for the same ` -norm. Œis frequency could be used as p k (x, x) = k(x, x) − k(x, X)(K + σ 2I)−1k(X, x). an additional constraint and developing a more reliable proxy for n human perception remains an interesting avenue for future work. We take the mean function to be zero m ≡ 0 to simplify evalu- Summary. Procedural noise aŠacks specialize as untargeted ation and since no prior knowledge can be incorporated into the black-box perturbations with naturally high universal evasion. Of- mean function [66] as is the case for black-box seŠings. A GP’s fensively, this has applications as a large-scale indiscriminate aŠack ability to model a rich distribution of functions rests on its covari- and, defensively, as a standard test on machine learning services. ance function which controls important properties of the function We expand on this perspective in the following sections. In Sect. 5, distribution such as di‚erentiability, periodicity, and amplitude we develop query-ecient black-box aŠacks using procedural noise [57, 66]. Any prior knowledge of the target function is encoded and Bayesian optimization. In Sect. 6, we apply our procedural in the hyperparameters of the covariance function. In a black box noise aŠack against a DCN designed for object detection, showing seŠing, we adopt a more general covariance function in the Matern´ that the aŠack can generalize to other tasks. In Sect. 7, we test an 5/2 kernel √ √ input-agnostic defence in median €lter denoising. 2 ! ! 0 5r 5r 5r k / (x, x ) = 1 + + exp − 5 2 l 3l2 l 5 EFFICIENT BLACK-BOX ATTACKS where r = x−x0 andl is the length-scale parameter [68]. Œis results Whilst in previous sections we have shown that procedural noise in twice-di‚erentiable functions, an assumption that corresponds functions are an ecient way to generate adversarial perturbations, to those made in popular black-box optimization algorithms like another signi€cant advantage they bring is their low-dimensional quasi-Newton methods [68]. search space. Œis enables the use of query-ecient black-box Acquisition Functions. Œe second component in Bayesian optimization techniques that otherwise do not scale well to high- optimization is an acquisition function that describes how optimal a dimensional problems. In this section, we compare several black- query is. Intuitively, the acquisition function evaluates the utility of box optimization techniques for both input-speci€c and universal candidate points for the next evaluation [7]. Œe two most popular 8 choices are the Expected Improvement (EI) and Upper Con€dence We focus on Perlin noise as it outperforms Gabor noise, with the Bound (UCB) [66]. First we de€ne µ(x) and σ 2(x) as the predictive laŠer averaging 58% universal evasion on the validation set. ( ) ( ) g(xbest)−µ(x) Table 3 shows that Bayesian optimization (BayesOpt) reliably mean and variance of g x respectively. Let γ x = σ (x) . Œe acquisition functions are outperforms L-BFGS in terms of universal evasion rate on the train- ing sets and the resulting universal evasion rates on the validation αEI(x) = σ(x)(γ (x)Φ(γ (x)) + N(γ (x) | 0, 1)) set. For comparison, random parameters for Perlin noise in Sect. 4 αUCB(x) = µ(x) + κσ(x), κ > 0 had a 98th percentile of 70.2% and a maximum of 73.1% universal evasion. Bayesian optimization consistently reached or passed this where Φ is the normal cumulative distribution function. EI and 98th percentile while L-BFGS did not. It is reasonable for these UCB have both been shown to be e‚ective and data-ecient in optimization algorithms not to beat the maximum since the train- real black-box optimization problems [68]. However, most studies ing sets were signi€cantly smaller. Similar trends appear between have found that EI converges near-optimally and is beŠer-behaved Bayesian optimization, random selection, and L-BFGS for Gabor than UCB in the general case [7, 66, 68]. Œis makes EI the best noise perturbations. We include these results in the Appendix D. candidate for our acquisition function. Table 3: Comparison on Inception v3 between universal Per- 5.2 Universal Black-box Attack lin noise black-box attacks. Universal evasion rates (%) of For a universal black-box aŠack using procedural noise, the goal the optimized perturbations are shown for their respective is to €nd the optimal parameters δ ∗ for the procedural noise gen- training set and the validation set. erating function G so that the universal evasion rate of perturba- ( ∗) tion G δ generalizes to unknown inputs Xval. Œe aŠacker has BayesOptper L-BFGSper a smaller set of inputs X and optimizes their perturbation δ ∗ train Train Size Train Val. Train Val. for this dataset. Œe performance of the aŠack is measured by its 50 78.0 71.4 74.0 69.9 universal evasion rate over the validation set Xval. In a practical seŠing, this is where the aŠacker optimizes their procedural noise 125 77.6 70.2 76.0 71.5 UAP over a small dataset, then injects that optimized perturbation 250 71.6 71.2 71.2 69.7 500 75.0 72.9 73.4 70.8 to other inputs–with the goal of causing as many misclassi€cations as possible. Experiment. We use the Inception v3 model, `∞-norm ε = 16, 5.3 Input-speci€c Black-box Attack and the same 5,000 data points tested in Sect. 4 as Xval. Xtrain are points from the ILSVRC2012 validation set not in Xval. We test for In this section, we use procedural noise for an input-speci€c black- training set sizes of 50, 125, 250, and 500, which corresponds to 1%, box algorithm. Œe goal of the adversary is to evade as many inputs 2.5%, 5%, and 10% of the validation set size. in the dataset, maximizing the input-speci€c evasion rate on inputs We compare Bayesian optimization with Limited-memory BFGS that are not misclassi€ed by the model. An important metric here (L-BFGS) [38], a quasi-Newton optimization algorithm that is o‰en is the query-eciency of these aŠacks, as requiring large volumes used in black-box optimization and machine learning. As the proce- of queries per sample becomes impractical in real-world scenarios. dural noise functions are non-di‚erentiable, we estimate gradients Metrics. We de€ne the success rate of an aŠack to be its input- using €nite di‚erence approximation. Œis gradient approximation speci€c evasion excluding clean inputs that are already misclassi€ed. is similar to what is used for other black-box aŠacks like zeroth- Œe average queries is measured over successful evasions. order optimization [12], but here it is applied to a signi€cantly Experiment. We use the Inception v3 model, `∞-norm ε = 16, smaller search space. When L-BFGS converges, possibly to a local and the same 5,000 data points from Sect. 4. For a given input optima, we restart the optimization with a di‚erent random initial x, the goal is to achieve evasion with arg max f (x + s) , τ (x) by point, stopping when the query limit is reached and choosing the minimizing the probability of the true class label τ (x). In this case best optima value found. we allow the aŠacker to access the model’s output probability vec- We set a maximum query limit of 1,000 universal evasion eval- tor, as the black-box optimization algorithms gain minimal usable uations on the training set Xtrain. In practice, this limit was not information if the output is binary (τ (x) or ¬τ (x)). necessary as both algorithms converged faster to their optimal As in the previous section, we compare Bayesian optimization values: within the €rst 100 queries for Bayesian optimization and and L-BFGS with a limit of 1,000 queries per input and number within the €rst 250 queries for L-BFGS. Œese are untargeted uni- of restarts when it converges. We use sparse GPs to beŠer scale versal black-box aŠacks where the adversary has no knowledge of Bayesian optimization [42], as the standard GP scales cubically the target model and only requires the top label from the model’s with the number of observations. We also compare with uniform outputs. random parameter selection from Sect. 4. Œese are untargeted Results. Œe best procedural noise UAP computed from the input-speci€c black-box aŠacks where the adversary has no knowl- training sets generalized well to the much larger validation set, con- edge of the target model and only requires the output probability sistently reaching 70% or more universal evasion on the validation vector of the model. set for Perlin noise. Œis is a surprising result as the training sets Results. Table 4 shows that Bayesian optimization reached a were 10-100 times smaller than the validation set. Œis may be due high 91.6% success rate with with just 7 queries per successful to the inherent universality of our procedural noise perturbations. evasion on average, under a restrictive 100 query limit. Œis vastly 9 Table 4: Comparison on Inception v3 between our input- Procedural noise perturbations have naturally high evasion rates, speci€c Perlin noise black-box attacks and bandit attack although the expressiveness of our chosen functions can be less [25] for di‚erent query limits. than aŠacks whose larger codomains can capture more kinds of adversarial perturbations. Œese other aŠacks make this trade- AŠack ‹ery Limit Average ‹eries Success Rate (%) o‚ by sacri€cing eciency in black-box scenarios, as the number

BayesOptper 100 7.0 91.6 of queries they need to cra‰ successful adversarial examples is BayesOptper 1,000 8.4 92.8 large. On the other hand, we can increase the expressiveness of our L-BFGSper 100 19.8 71.7 procedural noise functions by introducing more parameters by, for L-BFGSper 1,000 70.1 86.5 example, using di‚erent colour maps. However, this may come at a Randomper 1,000 36.3 91.6 the cost of its naturally high universal evasion.

BanditsTD 100 29.0 36.7 Summary. Black-box optimization techniques like Bayesian BanditsTD 1,000 224 73.6 optimization are able to take advantage of the drastically reduced BanditsTD 10,000 888 96.9 search space of procedural noise. Together with its naturally high universal evasion, we show that procedural noise gives rise to aFor each input, we divide the total number of perturbations by the number of those that evade that input to get an expected number of queries. inexpensive yet potent untargeted black-box aŠacks, whether it be input-speci€c or universal. It is also shown to be competitive with existing aŠacks that o‰en require orders of magnitude more queries or resources. improves on the query eciency over random parameters by 5- fold whilst aŠaining the same accuracy. Œe improvement for 6 OTHER APPLICATIONS Bayesian optimization increasing the query limit from 100 to 1,000 In this section, we show how procedural noise aŠacks extend to was very incremental. Œis suggests that the limited codomain of object detection DCNs by aŠacking YOLO v3 model. We then the procedural noise function is seŠing an upper bound on the discuss how exploiting the sensitivity of DCNs to low-level features, aŠack’s success rate. like in procedural noise aŠacks, can be generalized to cra‰ universal L-BFGS performed the worst as we observe it can get trapped and aŠacks for DCNs in other application domains. waste queries at poor local optima, which requires several restarts from di‚erent initial points to improve its performance. Œere were similar trends for Gabor noise where Bayesian optimization had 6.1 Attacking Object Detection 78.3% success with 9.8 queries on average. For both procedural Object detection requires the identi€cation of locations and classes noise functions, Bayesian optimization improved the average query of objects within an image. “Single Shot Detectors” (SSD) such as eciency over L-BFGS and random parameter selection by up to 7 the di‚erent versions of YOLO [58, 59] generate their predictions times while retaining a success rate greater than 83%. We include a‰er a single pass over the input image. Œis allows SSDs to process the other results in Appendix D. images in real-time speeds while maintaining high accuracy. Other Comparison. We compare our results with Ilyas et al. [25], types of object detectors such as Fast R-CNN [15] and Faster R-CNN where they formalize their aŠack as a gradient estimation problem [61] are based on region proposals. Œey have comparable accuracy (like in white-box gradient-based aŠacks), and use a bandit opti- but do not achieve the same image processing speed as SSDs. We mization framework to solve it. We test the bandits aŠack on the test our aŠack against YOLO v3 [60], the latest iteration of YOLO, same model, dataset, and `∞-norm for maximum query limits of which achieves high accuracy and detects objects in real time. 100, 1,000, and 10,000. Metrics. We use precision, recall, and mean average precision Œe results show that the evasion rate of our input-speci€c proce- (mAP) as our primary metrics; mAP is frequently used when com- dural noise aŠack greatly outperforms the bandits aŠack when the paring the overall performance of object detectors. In object detec- query limit per image is a thousand or less. For signi€cantly larger tion the classi€er predicts a bounding box to identify the location query limits, the Perlin noise Bayesian optimization aŠack has a of an object and assigns that box a class label. Œe Intersection Over competitive success rate at a drastically beŠer query eciency– Union (IOU) is the ratio of the intersection area over the union needing 100 times less queries on average. area between the predicted and ground truth bounding boxes. A Œe procedural noise aŠack is optimized for universal evasion threshold is usually set to determine what IOU constitutes a posi- and a restrictively small amount of queries. Most existing meth- tive classi€cation. True positives occur when the IOU is larger than ods explore the entire image space which has a hundred thousand the threshold and the class is identi€ed correctly. False negatives dimensions, and some aŠacks reduce their search space with tech- occur when the threshold is not met with the correct class for a niques like tiling, but the dimensionality is still much larger than ground truth bounding box. False positives occur when the IOU the four parameters of our procedural noise. Other input-speci€c is less than the threshold, there are no intersecting ground truth black-box aŠacks [3, 6, 12] require tens to hundreds of thousands boxes, or there are duplicate predicted bounding boxes. of queries on realistic natural-image datasets, which makes them Experiment. We use the MS COCO dataset [36] which contains inecient, but almost certain to €nd an adversarial example given 80 di‚erent classes, with a large proportion of targets being the enough queries. Œe bandits method makes a small sacri€ce in “person” class. Œis has become one of the benchmark datasets for success rate for more query eciency, and our procedural noise object detection. We use standard seŠings as described in [60], with aŠack takes it further for greater query-eciency. input dimensions 416 × 416 × 3 and an IOU threshold of 0.5. 10 We use `∞-norm ε = 16 and apply each of our procedural noise all the metrics decrease – indicating that there are many false perturbations on the 1,000 random images from the validation set. positive and false negatives. However, for three classes, “backpack”, Œe perturbations generated are from 1,000 Gabor noise, 1,000 Perlin “book”, and “toaster”, all three metrics improve. Œese labels did not noise, and 1,000 `∞-optimized uniform random perturbations. Œe have as much representation in the test set which may explain this parameters are chosen uniformly at random so that we can analyze anomalous result. the results, as in Sect. 4. Œis is a universal untargeted black-box Œe frequency of the sine ϕsine for Perlin noise had the largest aŠack. inverse correlation of less than -0.72 with each of the three metrics. Œis means that high-frequency paŠerns decrease the model’s per- formance metrics, similarly to what we have observed for the image classi€ers. Œe thickness λ of Gabor noise is moderately correlated at 0.4 with the precision, but not the recall or mAP. Œis suggests that thicker Gabor noise perturbations decrease the number of false positives relative to the other perturbations. Discussion. Object detection is a more complex task than image classi€cation as it has to identify multiple objects and their locations within an image. Although YOLO v3 has a di‚erent architecture, task, and dataset from the ImageNet classi€ers, we see that the same procedural noise perturbations are still able to greatly degrade its performance. Œis shows that it is more likely caused by the models’ sensitivity towards perturbations rather than a texture bias, as the object detection task has a spatial component to it. Œis suggests that our procedural noise aŠacks may generalize to other DCNs on Figure 3: Histogram of metrics over all perturbations on computer vision tasks with natural-image data. YOLO v3. Results for (top) all classes and (bottom) the “per- In our discussion in Sect. 4.4, we hypothesized that the proce- son” class. dural noise perturbations are an aggregation of low-level features at high frequencies that the DCNs strongly respond to. Œe prior that Results. Œe procedural noise perturbations signi€cantly de- convolutional layers induce and similarities across natural-image crease the mAP of the model, with 75% of Perlin noise perturbations datasets may be why DCNs are sensitive to these noise paŠerns. halving the mAP or worse. Œe precision and recall also decreased in Additionally, like in Sect. 5, we can also create more ecient black- the presence of our perturbations. Again, Perlin noise outperforms box aŠacks by applying black-box optimization techniques such as Gabor noise on average but Gabor noise has a smaller minimum, Bayesian optimization to enhance the aŠack against object detec- with at least one perturbation causing 0.09 mAP. Compared to uni- tors. When using Bayesian optimization, an aŠacker can focus on form random noise, which maintained around 0.34 mAP, both Perlin minimizing a speci€c metric (precision, recall, mAP, or F1 score). and Gabor noise had larger impact on the model’s performance. We represent uniform random noise using the median (which, in this case, is very close to the mean) on each metric as the variance is very small (less than 10−5). 6.2 Beyond Images In Fig. 3, for the metrics across all classes, we can observe that the One of the main strengths of procedural noise is that it allows to precision was maintained or decreased while the recall and mAP describe a distribution of UAPs with only a few parameters. Œis decreased. Some perturbations decreased all three metrics, which compressed representation allows for an ecient generation of indicate that these cause both false positives and false negatives UAPs both for undermining a machine learning service or, defen- consistently. When the perturbation increases the precision while sively, for testing the robustness of a model. In practice, compressed decreasing the recall and mAP, then the noise is likely masking representations can be learned by generative models such as GANs objects rather than introducing new objects to the image. Œis [46, 77] or Variational Autoencoders (VAEs), however these incur masking e‚ect has serious implications for security applications additional training, calibration, and maintenance costs. Training like surveillance and autonomous vehicles. algorithms or defences that incorporate domain-speci€c knowledge We focus on the “person” class as it constitutes a majority of may be needed to mitigate the sensitivity towards aŠacks that make the targets and is semantically meaningful in the context of some use of these compact representations. applications. In Fig. 3, metrics for “person” follow a similar trend DCNs in other applications may also be vulnerable to aggrega- to the metrics across all classes. However, the increase in precision tions of low-level features due to the use of convolutional layers. is very small (¡0.10) compared to the large drops in recall and mAP Future aŠacks can exploit how DCNs rely on combining low-level caused by Perlin noise. Œe higher precision indicates fewer false features rather than understanding the more dicult global fea- positives, while the decrease in recall indicates more false negatives. tures in the input data. A natural next step would be to apply Œis indicates that the classi€er is making fewer predictions overall, these ideas in exploratory aŠacks or sensitivity analysis on sensory which means that the noise is masking persons in the image. applications like speech recognition and natural language process- Whilst for the most relevant “person” class, our procedural noise ing. To expand our procedural noise aŠack framework to other appears to have an obfuscating e‚ect, for other classes like “zebra” applications, it is worth identifying paŠerns in existing adversarial 11 examples for domains like speech recognition [9, 11] and reinforce- the same 1,000 of each procedural noise, 1,000 uniform random ment learning [24, 37] to €nd analogues of procedural noise. As a perturbations, and 5,000 validation set points as in Sect. 4. We starting point, these paŠerns can be found by €nding perturbations test how denoising improves robustness for `∞ aŠack norms ε = that maximize the hidden layer di‚erence as in the Singular Vector 4, 8, 12, and 16. Œe decrease in the number of uniform random AŠack [30] or by using feature visualization on earlier layers of perturbations is due to results in Sect. 4, where we found that its DCNs to infer the low-level features that a model learns. universal evasion rate had very low variance across 10,000 samples, and our results show it is similar for 1,000 samples. 7 PRELIMINARY DEFENCE Results. Œe min-max oscillations present in the procedural Œe DCNs we tested are surprisingly fragile to procedural noise as noise perturbations may have allowed the noise paŠerns to per- UAPs. Improving their robustness to adversarial examples is not a sist despite the denoising, as the universal evasion rates are not straightforward task. A robust defence needs to defend not only completely mitigated by the defence. Across the di‚erent `∞-norm against existing aŠacks but also future aŠacks, and o‰en proposed values, the denoising decreased the median and mean universal eva- defences are shown to fail against new or existing aŠacks [8]. sion rates by a consistent amount when compared with no defence: Among existing defences, adversarial training appears to be 7.2-10.8% for Gabor noise and 13.2-16.9% for Perlin noise. Fig. 4 more robust than others [1]. However, we have shown in Table 2 shows that the decrease in e‚ectiveness of Perlin noise is much that ensemble adversarial training against gradient-based aŠacks greater than that for Gabor noise, suggesting that the denoising did not signi€cantly diminish the input-speci€c evasion rate of appears to slightly mitigate the e‚ectiveness of high-frequency our procedural noise aŠack. Œis suggests that such defences do noise paŠerns. not generalize well as the aŠacks used for adversarial training do Œis denoising defence grants a reasonable increase in robustness not suciently represent the entire space of adversarial examples. despite the simplicity of the algorithm. However, this measure has Training against all types of adversarial aŠacks would become com- not fully mitigated the sensitivity to procedural noise. We hope putationally expensive, especially for high-dimensional tasks like future work explore more input-agnostic defences that minimize ImageNet. Œus, defences that regularize the model or incorporate the sensitivity of large-scale models to small perturbations with domain-speci€c knowledge may be more ecient strategies. techniques like model compression [20], Jacobian regularization DCNs’ weakness to adversarial perturbations across inputs may [27, 30, 62, 69], or other types of denoising. be a result of their sensitivity—small changes to the input cause large changes to the output. Œus, input-agnostic defences that minimize the impact of small perturbations may be e‚ective in reducing the models’ sensitivity without the need to train against 8 CONCLUSION all types of aŠacks. As a preliminary investigation, we brieƒy We highlight the strengths of procedural noise as an indiscrimi- explore here using denoising to defend against the universality of nate and inexpensive black-box aŠack on DCNs. We have shown our procedural noise perturbations. that popular DCN architectures for image classi€cation have sys- temic vulnerabilities to procedural noise perturbations, with Perlin noise UAPs that achieve 58% or larger universal evasion across non- adversarially trained models. Œis weakness to procedural noise can be used to cra‰ ecient universal and input-speci€c black-box aŠacks. Our procedural noise aŠack augmented with Bayesian optimization aŠains competitive input-speci€c success rates whilst improving the query eciency of existing black-box methods over 100 times. Moreover, we show that procedural noise aŠacks also work against the YOLO v3 model for object detection, where it has Figure 4: Histogram of metrics over all perturbations on In- an obfuscating e‚ect on the “person” class. ception v3 with and without median denoising for (le ) Ga- Our results have notable implications. Œe universality of our bor noise and (right) Perlin noise for `∞-norm ε = 16. black-box method introduces the possibility for large-scale aŠacks on DCN-based machine learning services, and the use of Bayesian optimization makes untargeted black-box aŠacks signi€cantly more 7.1 Denoising ecient. We hypothesize that our procedural noise aŠacks exploit Denoising with spatial €lters is a common pre-processing step low-level features that DCNs are sensitive towards. If true, this has in signal processing and is thus an aŠractive defence due to its worrying implications on the safety of transfer learning, as it is simplicity and pervasiveness across signal processing applications. o‰en these low-level features that are preserved when retraining Median €ltering is a smoothing operation that replaces each entry models for new tasks. It may be the case that universal aŠacks with the median of its neighbouring entries, and it o‰en preserves can be extended to other application domains by using compact edges while removing noise. Œe idea is to smooth out the high- representations of UAPs that exploit the sensitivity of models to frequency noise, which we know to be highly correlated with the low-level features. universal evasion rates for Perlin noise on Inception v3. We have shown the diculty of defending against such novel Experiment. We retrain Inception v3 with a median denoising approaches. In particular, ensemble adversarial training on gradient- €lter applied as a pre-processing step. For this evaluation, we use based methods was unable to signi€cantly diminish our procedural 12 noise aŠack. We suggest that future defences take more input- vision and paˆern recognition. 770–778. agnostic approaches to avoid the costs in defending and retraining [22] Geo‚rey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N against all possible aŠacks. Our work prompts the need for further Sainath, et al. 2012. Deep Neural Networks for Acoustic Modeling in Speech research of more intuitive and novel aŠack frameworks that use Recognition: Œe Shared Views of Four Research Groups. IEEE Signal Processing Magazine 29, 6 (2012), 82–97. analogues of procedural noise in other machine learning applica- [23] Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and JD tion domains such as audio processing and reinforcement learning. Tygar. 2011. Adversarial machine learning. In Workshop on Security and Arti€cial We hope future work will explore in more depth the nature of cross- Intelligence. 43–58. [24] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter model universal adversarial perturbations, as these vulnerabilities Abbeel. 2017. Adversarial aŠacks on neural network policies. arXiv preprint generalize across both inputs and models, and a more formal frame- arXiv:1702.02284 (2017). work could beŠer explain why our procedural noise aŠacks are [25] Andrew Ilyas, Logan Engstrom, and Aleksander Madry. 2018. Prior convic- tions: Black-box adversarial aŠacks with bandits and priors. arXiv preprint e‚ective. arXiv:1807.07978 (2018). [26] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial Examples Are Not Bugs, Œey REFERENCES Are Features. arXiv preprint arXiv:1905.02175 (2019). [1] Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated Gradients [27] Daniel Jakubovitz and Raja Giryes. 2018. Improving DNN robustness to ad- Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. versarial aŠacks using Jacobian regularization. In Proceedings of the European arXiv preprint arXiv:1802.00420 (2018). Conference on Computer Vision (ECCV). 514–529. [2] Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug [28] Ahmad Javaid, ‹amar Niyaz, Weiqing Sun, and Mansoor Alam. 2016. A deep Tygar. 2006. Can Machine Learning Be Secure?. In Symposium on Information, learning approach for network intrusion detection system. In Proceedings of the Computer and Communications Security. 16–25. 9th EAI International Conference on Bio-inspired Information and Communications [3] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. 2018. Black-box AŠacks Technologies (formerly BIONETICS). ICST (Institute for Computer Sciences, Social- on Deep Neural Networks via Gradient Estimation. (2018). Informatics and ?, 21–26. [4] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat [29] Min-Joo Kang and Je-Won Kang. 2016. Intrusion detection system using deep Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai neural network for in-vehicle network security. PloS one 11, 6 (2016), e0155781. Zhang, et al. 2016. End to end learning for self-driving cars. arXiv preprint [30] Valentin Khrulkov and Ivan Oseledets. 2018. Art of Singular Vectors and Uni- arXiv:1604.07316 (2016). versal Adversarial Perturbations. In Procs. Conf. on Computer Vision and Paˆern [5] G. Bradski. 2000. Œe OpenCV Library. Dr. Dobb’s Journal of So‡ware Tools Recognition. 8562–8570. (2000). [31] Alex Krizhevsky, Ilya Sutskever, and Geo‚rey E Hinton. 2012. Imagenet Clas- [6] Wieland Brendel, Jonas Rauber, and MaŠhias Bethge. 2017. Decision-Based si€cation with Deep Convolutional Neural Networks. In Advances in Neural Adversarial AŠacks: Reliable AŠacks Against Black-Box Machine Learning Information Processing Systems. 1097–1105. Models. arXiv preprint arXiv:1712.04248 (2017). [32] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial Examples [7] Eric Brochu, Vlad M Cora, and Nando De Freitas. 2010. A Tutorial on Bayesian in the Physical World. arXiv preprint arXiv:1607.02533 (2016). Optimization of Expensive Cost Functions, with Application to Active User Mod- [33] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial Machine eling and Hierarchical Reinforcement Learning. arXiv preprint arXiv:1012.2599 Learning at Scale. arXiv preprint arXiv:1611.01236 (2016). (2010). [34] Ares Lagae, Sylvain Lefebvre, Rob Cook, Tony DeRose, George DreŠakis, David S [8] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Ebert, John P Lewis, Ken Perlin, and MaŠhias Zwicker. 2010. A Survey of Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. Procedural Noise Functions. In Computer Graphics Forum, Vol. 29. 2579–2600. 2019. On Evaluating Adversarial Robustness. arXiv preprint arXiv:1902.06705 [35] Ares Lagae, Sylvain Lefebvre, George DreŠakis, and Philip Dutre.´ 2009. Pro- (2019). cedural noise using sparse Gabor convolution. ACM Transactions on Graphics [9] Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, (TOG) 28, 3 (2009), 54. Clay Shields, David Wagner, and Wenchao Zhou. 2016. Hidden Voice Commands.. [36] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva In USENIX Security Symposium. 513–530. Ramanan, Piotr Dollar,´ and C Lawrence Zitnick. 2014. Microso‰ coco: Common [10] Nicholas Carlini and David Wagner. 2017. Towards Evaluating the Robustness objects in context. In European conference on computer vision. Springer, 740–755. of Neural Networks. In Symposium on Security and Privacy. 39–57. [37] Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, [11] Nicholas Carlini and David Wagner. 2018. Audio adversarial examples: Targeted and Min Sun. 2017. Tactics of adversarial aŠack on deep reinforcement learning aŠacks on speech-to-text. arXiv preprint arXiv:1801.01944 (2018). agents. arXiv preprint arXiv:1703.06748 (2017). [12] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 2017. Zoo: [38] Dong C Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for Zeroth Order Optimization Based Black-box AŠacks to Deep Neural Networks large scale optimization. Mathematical programming 45, 1-3 (1989), 503–528. without Training Substitute Models. In Workshop on Arti€cial Intelligence and [39] Andre´ Teixeira Lopes, Edilson de Aguiar, Alberto F De Souza, and Œiago Oliveira- Security. 15–26. Santos. 2017. Facial expression recognition with convolutional neural networks: [13] Yali Du, Meng Fang, Jinfeng Yi, Jun Cheng, and Dacheng Tao. 2018. Towards coping with few data and the training sample order. Paˆern Recognition 61 (2017), ‹ery Ecient Black-box AŠacks: An Input-free Perspective. In Proceedings of 610–628. the 11th ACM Workshop on Arti€cial Intelligence and Security. ACM, 13–24. [40] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and [14] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, MaŠhias Bethge, Felix A Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Wichmann, and Wieland Brendel. 2018. ImageNet-trained CNNs are biased AŠacks. arXiv preprint arXiv:1706.06083 (2017). towards texture; increasing shape bias improves accuracy and robustness. arXiv [41] Patrick McDaniel, Nicolas Papernot, and Z Berkay Celik. 2016. Machine Learning preprint arXiv:1811.12231 (2018). in Adversarial SeŠings. IEEE Security & Privacy 14, 3 (2016), 68–72. [15] Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference [42] Mitchell McIntire, Daniel Ratner, and Stefano Ermon. 2016. Sparse Gaussian on computer vision. 1440–1448. Processes for Bayesian Optimization.. In UAI. [16] Ian Goodfellow. 2018. Defense Against the Dark Arts: An Overview of Adver- [43] J Mockus,´ V Tiesis, and A Zilinskas.´ 1978. Œe Application of Bayesian Methods sarial Example Security Research and Future Research Directions. arXiv preprint for Seeking the Extremum. Vol. 2. Towards Global Optimization 2 (1978), 117–129. arXiv:1806.04169 (2018). [44] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal [17] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Frossard. 2017. Universal Adversarial Perturbations. In Conference on Computer Press. Vision and Paˆern Recognition. 86–94. [18] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and [45] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Harnessing Adversarial Examples. arXiv preprint arXiv:1412.6572 (2014). Deepfool: a Simple and Accurate Method to Fool Deep Neural Networks. In [19] Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Conference on Computer Vision and Paˆern Recognition. 2574–2582. Patrick McDaniel. 2016. Adversarial perturbations against deep neural networks [46] Konda Mopuri, Utkarsh Ojha, Utsav Garg, and R. Venkatesh Babu. 2018. NAG: for malware classi€cation. arXiv preprint arXiv:1606.04435 (2016). Network for Adversary Generation. In Procs. Conf. on Computer Vision and Paˆern [20] Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing Recognition. 742–751. deep neural networks with pruning, trained quantization and hu‚man coding. [47] Luis Munoz˜ Gonzalez´ and Emil C Lupu. 2018. Œe Secret of Machine Learning. arXiv preprint arXiv:1510.00149 (2015). ITNOW 60, 1 (2018), 38–39. [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer 13 [48] Fabrice Neyret and Eric Heitz. 2016. Understanding and Controlling Contrast [75] Florian Tramr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Oscillations in Stochastic Texture Algorithms using Spectrum of Variance. Ph.D. and Patrick McDaniel. 2018. Ensemble Adversarial Training: AŠacks and De- Dissertation. LJK/Grenoble University-INRIA. fenses. In International Conference on Learning Representations. [49] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature visual- [76] Bichen Wu, Forrest N Iandola, Peter H Jin, and Kurt Keutzer. 2017. SqueezeDet: ization. Distill 2, 11 (2017), e7. Uni€ed, Small, Low Power Fully Convolutional Neural Networks for Real-Time [50] Maxime Oquab, Leon BoŠou, Ivan Laptev, and Josef Sivic. 2014. Learning Object Detection for Autonomous Driving.. In CVPR Workshops. 446–454. and transferring mid-level image representations using convolutional neural [77] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. networks. In Proceedings of the IEEE conference on computer vision and paˆern 2018. Generating adversarial examples with adversarial networks. arXiv preprint recognition. 1717–1724. arXiv:1801.02610 (2018). [51] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. 2016. Transferability [78] Jason Yosinski, Je‚ Clune, Yoshua Bengio, and Hod Lipson. 2014. How transfer- in Machine Learning: From Phenomena to Black-box AŠacks using Adversarial able are features in deep neural networks?. In Advances in neural information Samples. arXiv preprint arXiv:1605.07277 (2016). processing systems. 3320–3328. [52] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, [79] Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang and Ananthram Swami. 2017. Practical Black-box AŠacks Against Machine Gan, and Yong Yang. 2018. Transferable Adversarial Perturbations. In Computer Learning. In Asia Conference on Computer and Communications Security. 506– Vision–ECCV 2018. Springer, 471–486. 519. [53] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman. 2018. SoK: Security and Privacy in Machine Learning. In European Symposium A EXAMPLES OF PERTURBATIONS on Security and Privacy. 399–414. [54] Ken Perlin. 1985. An Image Synthesizer. ACM Siggraph Computer Graphics 19, 3 Figs. 5 and 6 show examples of perturbations from procedural noise (1985), 287–296. functions and existing white-box UAP aŠacks. Notice the visual [55] Ken Perlin. 2002. Improved noise reference implementation. (2002). hŠps: similarities in structure between the noise paŠerns in Figs. 5 and 6. //mrl.nyu.edu/∼perlin/noise/ Accessed: 2018-06-09. [56] Ken Perlin. 2002. Improving Noise. In ACM Transactions on Graphics, Vol. 21. 681–682. [57] Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Œe MIT Press. [58] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Uni€ed, real-time object detection. In Proceedings of the IEEE conference on computer vision and paˆern recognition. 779–788. [59] Joseph Redmon and Ali Farhadi. 2017. YOLO9000: beŠer, faster, stronger. In Proceedings of the IEEE conference on computer vision and paˆern recognition. 7263–7271. [60] Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018). [61] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91–99. [62] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning. Omnipress, 833–840. Figure 5: Procedural noise patterns with (top) Gabor noise [63] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean and (bottom) Perlin noise, both with decreasing frequency Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of from le to right. Computer Vision 115, 3 (2015), 211–252. [64] Joshua Saxe and Konstantin Berlin. 2015. Deep Neural Network Based Malware Detection using Two Dimensional Binary Program Features. In Intl. Conference on Malicious and Unwanted So‡ware (MALWARE). 11–20. [65] Joshua Saxe and Konstantin Berlin. 2017. eXpose: A character-level convolutional neural network with embeddings for detecting malicious URLs, €le paths and registry keys. arXiv preprint arXiv:1702.08568 (2017). [66] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Fre- itas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimiza- tion. Proc. IEEE 104, 1 (2016), 148–175. [67] Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Net- works for Large-scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014). [68] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural information Figure 6: UAPs generated for VGG-19 using the white-box Processing Systems. 2951–2959. Singular Vector Attack [30]. [69] Jure Sokolic,´ Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. 2017. Robust large margin deep neural networks. IEEE Transactions on Signal Processing 65, 16 (2017), 4265–4280. [70] Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2013. Deep convolutional network cas- B TOP 5 INPUT-SPECIFIC EVASION cade for facial point detection. In Proceedings of the IEEE conference on computer vision and paˆern recognition. 3476–3483. In Table 5, we consider the top 5 input-speci€c evasion rate where [71] Christian Szegedy, Sergey Io‚e, Vincent Vanhoucke, and Alexander A Alemi. the true class label is outside the top 5 class labels in the perturbed 2017. Inception-v4, Inception-Resnet and the Impact of Residual Connections on image x +s. Œe top 5 error is o‰en used alongside the top 1 error to Learning. In AAAI, Vol. 4. 12. [72] Christian Szegedy, Vincent Vanhoucke, Sergey Io‚e, Jon Shlens, and Zbigniew measure the performance of classi€ers on ImageNet. Top 5 evasion Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In is more dicult than top 1 as the con€dence in the true label has to Conference on Computer Vision and Paˆern Recognition. 2818–2826. [73] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru be degraded suciently enough for it to be below €ve other class Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing Properties of Neural labels. Despite that, Perlin noise is able to achieve top 5 evasion Networks. arXiv preprint arXiv:1312.6199 (2013). on more than half the inputs for all tested models. We see that our [74] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Auto- mated testing of deep-neural-network-driven autonomous cars. In Proceedings procedural noise still remains a strong aŠack for the top 5 error of the 40th international conference on so‡ware engineering. ACM, 303–314. metric. 14 Table 5: Top 5 input-speci€c evasion rate (in %) for random D BLACK-BOX ATTACK RESULTS and procedural noise perturbations. Original refers to the For Sect. 5, full results for input-speci€c and universal black-box top 5 error on the unaltered original images. ‡e strongest aŠacks against Inception v3 are in Tables 6 and 7. We include results attack on each classi€er is highlighted. for Gabor noise (gab) and Perlin noise (per). Œe bandit aŠacks [25] include the use of time (T) and time with data (TD) priors. Methods Classi€er Original Random Gabor Perlin are separated according to the base aŠack that was used: Gabor VGG-19 9.5 28.0 90.2 92.6 noise, Perlin noise, and bandits framework. ResNet-50 8.4 29.5 82.5 85.3 Inception v3 6.3 18.0 66.9 79.5 Table 6: Results for input-speci€c black-box attacks on In- IRv2 4.5 13.2 56.3 66.4 ception v3 with `∞-norm ε = 16. Gabor and Perlin noise IRv2 5.1 12.1 44.1 53.6 ens attacks are labeled with “gab” and “per” respectively. Best success rate per method is highlighted. C CORRELATION MATRICES AŠack ‹ery Limit Average ‹eries Success Rate (%) Procedural noise perturbations had their parameters chosen uni- formly at random and with constraint `∞-norm ε = 16. For Sect. 4, BayesOptgab 100 9.8 83.1 Figs. 7 and 8 show the correlation matrices between the proce- BayesOptgab 1,000 10.9 83.6 dural noise parameters and the corresponding universal evasion L-BFGSgab 100 6.0 44.7 L-BFGS 1,000 6.0 44.7 rates on the various ImageNet classi€ers. Œese correlations quan- gab Random 1,000 62.2 86.1 tify the e‚ects of each parameter and the transferability of UAPs gab across models. Œe labels “V19”, “R50”, and “INv3” refer to VGG-19, BayesOptper 100 7.0 91.6 ResNet-50, and Inception v3 respectively. BayesOptper 1,000 8.4 92.8 L-BFGSper 100 19.8 71.7 L-BFGSper 1,000 70.1 86.5 Randomper 1,000 36.3 91.6

BanditsT 100 31.3 14.3 BanditsT 1,000 330 47.6 BanditsT 10,000 1,795 88.3 BanditsTD 100 29.0 36.7 BanditsTD 1,000 224 73.6 BanditsTD 10,000 888 96.9

Figure 7: Correlations between Gabor noise parameters (δgab = {σ, ω, λ, ξ }) and universal evasion rates on each im- age classi€er.

Figure 8: Correlations between Perlin noise parameters (δper = {λx , λy , ϕsine, ω}) and universal evasion rates on each image classi€er.

15 Table 7: Results for universal black-box attacks on Inception v3 with `∞-norm ε = 16. Gabor and Perlin noise attacks are labeled with “gab” and “per” respectively. Universal evasion rates (%) of the optimized perturbations are shown for their respective training set and the validation set.

BayesOptgab L-BFGSgab BayesOptper L-BFGSper Train Size Train Val. Train Val. Train Val. Train Val. 50 64.0 57.6 58.0 51.6 78.0 71.4 74.0 69.9 125 64.6 58.0 58.4 54.6 77.6 70.2 76.0 71.5 250 60.0 58.4 58.8 56.0 71.6 71.2 71.2 69.7 500 64.8 62.4 59.8 58.0 75.0 72.9 73.4 70.8

E ADVERSARIAL EXAMPLES Figs. 9 and 10 contain some samples of adversarial examples from our procedural noise aŠacks against image classi€cation and object detection tasks.

Figure 9: Gabor noise adversarial examples on (top) Incep- tion v3, (middle) VGG-19, and (bottom) ResNet-50. Figure 10: Perlin noise adversarial examples on YOLO v3.

16