On the Impact of the Activation Function on Deep Neural Networks Training
Soufiane Hayou 1 Arnaud Doucet 1 Judith Rousseau 1
Abstract obtained similar results using a topological measure of ex- The weight initialization and the activation func- pressiveness. tion of deep neural networks have a crucial impact Since the training of deep neural networks is a non-convex on the performance of the training procedure. An optimization problem, the weight initialization and the ac- inappropriate selection can lead to the loss of in- tivation function will essentially determine the functional formation of the input during forward propagation subspace that the optimization algorithm will explore. We and the exponential vanishing/exploding of gra- follow here the approach of (Poole et al., 2016) and (Schoen- dients during back-propagation. Understanding holz et al., 2017) by investigating the behaviour of ran- the theoretical properties of untrained random net- dom networks in the infinite-width and finite-variance i.i.d. works is key to identifying which deep networks weights context where they can be approximated by a Gaus- may be trained successfully as recently demon- sian process as established by (Neal, 1995), (Matthews et al., strated by (Schoenholz et al., 2017) who showed 2018) and (Lee et al., 2018). that for deep feedforward neural networks only a specific choice of hyperparameters known as the In this paper, our contribution is three-fold. Firstly, we ‘Edge of Chaos’ can lead to good performance. provide a comprehensive analysis of the so-called Edge of While the work by (Schoenholz et al., 2017) dis- Chaos (EOC) curve and show that initializing a network on cuss trainability issues, we focus here on training this curve leads to a deeper propagation of the information acceleration and overall performance. We give a through the network and accelerates the training. In particu- comprehensive theoretical analysis of the Edge of lar, we show that a feedforward ReLU network initialized Chaos and show that we can indeed tune the ini- on the EOC acts as a simple residual ReLU network in tialization parameters and the activation function terms of information propagation. Secondly, we introduce a in order to accelerate the training and improve class of smooth activation functions which allow for deeper performance. signal propagation (Proposition3) than ReLU. In partic- ular, this analysis sheds light on why smooth versions of ReLU (such as SiLU or ELU) perform better experimentally 1. Introduction for deep neural networks; see, e.g., (Clevert et al., 2016), (Pedamonti, 2018), (Ramachandran et al., 2017) and (Mil- Deep neural networks have become extremely popular as letarí et al., 2018). Lastly, we show the existence of optimal they achieve state-of-the-art performance on a variety of points on the EOC curve and we provide guidelines for the important applications including language processing and choice of such point and we demonstrate numerically the computer vision; see, e.g., (Goodfellow et al., 2016). The consistence of this approach. We also complement previous success of these models has motivated the use of increas- empirical results by illustrating the benefits of an initializa- ingly deep networks and stimulated a large body of work to tion on the EOC in this context. All proofs are given in the arXiv:1902.06853v2 [stat.ML] 26 May 2019 understand their theoretical properties. It is impossible to Supplementary Material. provide here a comprehensive summary of the large num- ber of contributions within this field. To cite a few results relevant to our contributions, (Montufar et al., 2014) have 2. On Gaussian process approximations of shown that neural networks have exponential expressive neural networks and their stability power with respect to the depth while (Poole et al., 2016) 2.1. Setup and notations 1 Department of Statistics, University of Oxford, Oxford, We use similar notations to those of (Poole et al., 2016) United Kingdom. Correspondence to: Soufiane Hayou
(a) ReLU with (σb, σw) = (1, 1) (b) Tanh with (σb, σw) = (1, 1) (c) Tanh with (σb, σw) = (0.3, 2)
Figure 1. Draws of outputs for ReLU and Tanh networks for different parameters (σb, σw). Figures (a) and (b) show the effect of an initialization in the ordered phase, the outputs are nearly constant. Figure (c) shows the effect of an initialization in the chaotic phase.
2 1 1 then for σ < min( , ) and σb ≥ 0, we have close inputs (in terms of correlation) lead to very different w Mφ Cφ (σb, σw) ∈ Dφ,var ∩ Dφ,corr and Kφ,var(σb, σw) = outputs. Therefore, in the chaotic phase, at the limit of Kφ,corr(σb, σw) = ∞. infinite width and depth, the output function of the neural network is non-continuous everywhere. Figure 1(c) shows an example of such behaviour for Tanh. The proof of Lemma1 is straightforward. We prove that 0 2 Definition 2 (Edge of Chaos). For (σb, σw) ∈ Dφ,var, sup F (x) = σ Mφ and then apply the Banach fixed point w let q be the limiting variance2. The Edge of Chaos theorem. Similar ideas are used for Cφ,δ. (EOC) is the set of values of (σb, σw) satisfying χ1 = Example: For ReLU activation function, we have M = 2 0 √ 2 ReLU σwE[φ ( qZ) ] = 1. 1/2 and CReLU,δ ≤ 1 for any δ > 0. To further study the EOC regime, the next lemma introduces In the domain of convergence D ∩ D , for all φ,var φ,corr a function f called the ‘correlation function’ showing that a, b ∈ d, we have y∞(a) = y∞(b) almost surely and R i i that the correlations have the same asymptotic behaviour as the outputs of the network are constant functions. Figures the time-homogeneous dynamical system cl+1 = f(cl ). 1(a) and 1(b) illustrate this behaviour for ReLU and Tanh ab ab with inputs in [0, 1]2 using a network of depth L = 20 with Lemma 2. Let (σb, σw) ∈ Dφ,var ∩ Dφ,corr such d N = 300 neurons per layer. The draws of outputs of these that q > 0, a, b ∈ R and φ a measurable func- l 2 networks are indeed almost constant. tion such that supx∈S E[φ(xZ) ] < ∞ for all com- l+1 l pact sets S. Define fl by c = fl(c ) and f √ √ab √ ab Under the conditions of Lemma1, both the variance and σ2+σ2 [φ( qZ )φ( q(xZ + 1−x2Z )) by f(x) = b wE 1 1 2 . Then the correlations converge exponentially fast (contraction q mapping). To refine this convergence analysis, (Schoenholz liml→∞ supx∈[0,1] |fl(x) − f(x)| = 0. et al., 2017) established the existence of and such q c The condition on φ in Lemma2 is violated only by activa- that |ql − q| ∼ e−l/q and |cl − 1| ∼ e−l/c when fixed a ab tion functions with square exponential growth (which are points exist. The quantities and are called ‘depth scales’ q c not used in practice), so from now onwards, we use this since they represent the range of depth to which the variance approximation in our analysis. Note that being on the EOC and correlation can propagate without being exponentially 0 is equivalent to (σb, σw) satisfying f (1) = 1. In the next close to their limits. More precisely, if we write χ1 = 2 0 √ 2 2 00 √ √ section, we analyze this phase transition carefully for a large σwE[φ ( qZ) ] and α = χ1 + σwE[φ ( qZ)φ( qZ)] −1 class of activation functions. then the depth scales are given by q = − log(α) and −1 c = − log(χ1) . The equation χ1 = 1 corresponds to an infinite depth scale of the correlation. It is called the 3. Edge of Chaos EOC as it separates two phases: an ordered phase where To illustrate the effect of the initialization on the EOC, we the correlation converges to 1 if χ < 1 and a chaotic phase 1 plot in Figure 2(c) the output of a ReLU neural network where χ > 1 and the correlations do not converge to 1. In 1 with 20 layers and 100 neurons per layer with parameters this chaotic regime, it has been observed in (Schoenholz √ (σ2, σ2 ) = (0, 2) (as we will see later EOC = {(0, 2)} et al., 2017) that the correlations converge to some value b w 2 c < 1 when φ(x) = Tanh(x) and that c is independent of The limiting variance is a function of (σb, σw) but we do not the correlation between the inputs. This means that very emphasize it notationally. On the Impact of the Activation Function on Deep Neural Networks Training
(a) Convergence of the correlation to (b) Correlation function f (c) Output of 100x20 ReLU net- 1 with c0 = 0.1 work on the EOC
Figure 2. Impact of the EOC initialization on the correlation and the correlation function. In (a), the correlation converges to 1 at a sub-exponential rate when the network is initialized on the EOC. In (b), the correlation function f satisfies f 0(1) = 1 on the EOC. for ReLU). Unlike the output in Figure 1(a), this output sense of definition2 is due to the fact that the variance is l 1 displays much more variability. However, we prove below unchanged (qa = qa) on the weak EOC, so that the limiting that the correlations still converge to 1 even in the EOC variance q depends on a. However, this does not impact the regime, albeit at a slower rate. analysis of the correlations, therefore, hereafter the weak EOC is also called the EOC. 3.1. ReLU-like activation functions This class of activation functions has the interesting property of preserving the variance across layers when the network ReLU has replaced classical activations (sigmoid, Tanh,...) is initialized on the EOC. We show in Proposition1 below which suffer from gradient vanishing (see e.g. (Glorot et al., that, in the EOC regime, the correlations converge to 1 at a 2011) and (Nair and Hinton, 2010)). Many variants such as slower rate (slower than exponential). We only present the Leaky-ReLU were also shown to enjoy better performance result for ReLU but the generalization to the whole class is in test accuracy (Xu et al., 2015). This motivates the analysis straightforward. of such functions from an initialization point of view. Let us first define this class. Example: ReLU: The EOC is reduced to the singleton Definition 3 (ReLU-like functions). A function φ is ReLU- 2 2 like if it is of the form (σb , σw) = (0, 2), hence we should initialize ReLU net- works using the parameters (σ2, σ2 ) = (0, 2). This result ( b w λx if x > 0 coincides with the recommendation in (He et al., 2015) φ(x) = βx if x ≤ 0 whose objective was to make the variance constant as the input propagates but who did not analyze the propagation where λ, β ∈ R. of the correlations. (Klambauer et al., 2017) performed a similar analysis by using the ‘Scaled Exponential Linear ReLU corresponds to λ = 1 and β = 0. For this class of Unit’ activation (SELU) that makes it possible to center activation functions, the EOC in terms of definition2 is the mean and normalize the variance of the post-activation reduced to the empty set. However, we can define a weak φ(y). The propagation of the correlations was not discussed version of the EOC for this class. From Lemma1, when q 2 therein either. 2 σb σw < 2 2 , the variances converge to q = 2 Figure 2(b) displays the correlation function f for two differ- λ +β 1−σw/2 and the correlations converge to 1 exponentially fast. If ent sets of parameters (σb, σw). The blue graph corresponds q 2 2 2 to the EOC (σ , σw) = (0, 2), and the red one corresponds σw > 2 2 the variances converge to infinity. We then b λ +β to an ordered phase (σ , σ ) = (1, 1). have the following result. b w In the next result, we show that a fully connected feedfor- Lemma 3 (Weak EOC). Let φ be a ReLU-like function with ward ReLU network initialized on the EOC (weak sense) 0 λ, β defined as above. Then fl does not depend on l, and acts as if it has residual connections in terms of correlation 0 l fl (1) = 1 and q bounded holds if and only if (σb, σw) = propagation. This could potentially explain why training q 2 (0, λ2+β2 ). ReLU is faster on the EOC (see experimental results). We q 2 further show that the correlations converge to 1 at a polyno- We call the singleton {(0, )} the weak EOC. 2 λ2+β2 mial rate of 1/l on the EOC instead of an exponential rate in the ordered phase. The non existence of EOC for ReLU-like activation in the On the Impact of the Activation Function on Deep Neural Networks Training
(a) Tanh (b) ReLU (c) ELU
Figure 3. EOC curves for different activation functions (red dashed line). For smooth activation functions (Figures (a) and (c)), the EOC is a curve in the plane (σb, σw), while it is reduced to a single point for ReLU.
Proposition 1 (EOC acts as Residual connections). Con- Algorithm 1 EOC curve 2 2 sider a ReLU network with parameters (σb , σw) = (0, 2) ∈ Input: φ satisfying conditions of Proposition2, σb l EOC and correlations cab. Consider also a ReLU network Initialize q = 0 with simple residual connections given by while q has not converged do 2 V [φ](q) q = σ + 0 Nl−1 b V [φ ](q) l l−1 X l l−1 l end while yi(a) = yi (a) + W ijφ(yj (a)) + Bi 1 return (σb, √ ) j=1 V [φ0](q)
l σ2 l where W ∼ N (0, w ) and B ∼ N (0, σ2). Let cl be ij Nl−1 i b ab the corresponding correlation. Then, for any σw > 0 and whether we can have σmax = ∞. The answer is yes for the σb = 0, there exists a constant γ > 0 such that following large class of ‘Tanh-like’ activation functions. Let φ ∈ 9π2 Definition 4 (Tanh-like activation functions). 1 − cl ∼ γ(1 − cl ) ∼ as l → ∞ D2( , ). φ is Tanh-like if ab ab 2l2 R R 1. φ bounded, φ(0) = 0, and for all x ∈ , φ0(x) ≥ 0, 3.2. Smooth activation functions R xφ00(x) ≤ 0 and xφ(x) ≥ 0. We show that smooth activation functions provide better 2. There exist α > 0 such that |φ0(x)| e−α|x| for large signal propagation through the network. We start by a result & x (in norm). on the existence of the EOC. 1 Lemma 4. Let φ be a Tanh-like activation function, then Proposition 2. Let φ ∈ Dg be non ReLU-like such that φ(0) = 0 and φ0(0) 6= 0. Assume that V [φ] is non- φ satisfies all conditions of Proposition2 and EOC = 0 {(σ , √ 1 ): σ ∈ +}. decreasing and V [φ ] is non-increasing. Let σmax := b 0 √ 2 b R q E[φ ( qZ) ] sup |x − V [φ](x) | and for σ < σ let q be the x≥0 V [φ0](x) b max σb Recall that the convergence rate of the correlation to 1 for 2 V [φ] 2 smallest fixed point of the function σb + V [φ0] . Then we ReLU-like activations on the EOC is O(1/` ). We can 1 have EOC = {(σb, √ √ ): σb < σmax}. improve this rate by taking a sufficiently regular activation [φ0( q Z)2] E σb function. Let us first define a regularity class A. 2 Example : Tanh and ELU (defined by φELU (x) = x for Definition 5. Let φ ∈ Dg. We say that φ is in A if there x x ≥ 0 and φELU (x) = e − 1 for x < 0) satisfy all condi- exists n ≥ 1, a partition (Si)1≤i≤n of R and g1, g2, ..., gn ∈ 2 (2) Pn tions of Proposition2. We prove in the Appendix that SiLU Cg such that φ = i=1 1Si gi. (a.k.a Swish) has an EOC. Using Proposition2, we propose Algorithm1 to determine This class includes activations such as Tanh, SiLU, ELU k the EOC curves. (with α = 1). Note that Dg ⊂ A for all k ≥ 3. Figure3 shows the EOC curves for different activation func- For activation functions in A, the next proposition shows tions. For ReLU, the EOC is reduced to a point while that the correlation converges to 1 at the rate O(1/`) which 2 smooth activation functions have an EOC curve (ELU is a is better than O(1/` ) of ReLU-like activation functions. smooth approximation of ReLU). Proposition 3 (Convergence rate for smooth activations). A natural question which arises from the analysis above is Let φ ∈ A such that φ is non-linear (i.e. φ(2) is non- On the Impact of the Activation Function on Deep Neural Networks Training
• sup |f(x) − x| ≤ 1 x∈[0,1] βq
˜l Tr(Qab) 2 • For l ≥ 1, | l+1 − 1| ≤ ˜ βq Tr(Qab )
Moreover, we have
lim βq = ∞. σb→0 (σb,σw)∈EOC
The result of Proposition4 suggests that by taking small σb, we can achieve two important things. First, it makes Figure 4. Impact of the smoothness of the activation function on the function f close to the identity function, this slows the convergence of the correlations on the EOC. The convergence 2 further the convergence of the correlations to 1, i.e., the rate for ReLU is O(1/` ) and O(1/`) for Tanh and ELU. information propagates deeper inside the network. Note that the only activation functions satisfying f(x) = x for all x ∈ [0, 1] are linear functions which are not useful. β identically zero). Then, on the EOC, we have 1 − cl ∼ q Second, it makes the Trace of the covariance matrix of the √ l 2 [φ0( qZ)2] E √ gradients approximately constant through layers, which where βq = q [φ00( qZ)2] E means, we avoid vanishing of the information during backpropagation (More precisely, we preserve the overall Choosing a smooth activation function is therefore better for spectrum of the covariance matrix since the Trace is the deep neural networks since it provides deeper information sum of the eigenvalues). propagation. This could explain for example why smooth We also have lim q = 0 so that if σ too small then versions of ReLU such as ELU perform better (Clevert σb→0 b yl(a) ≈ 0. Hence, a trade-off has to be taken into account et al., 2016) (see experimental results). Figure 3.2 shows when initializing on the EOC. Using Proposition4, we the evolution of the correlation through the network layers can deduce the maximal depth to which the correlations for different activation functions. For function in A (Tanh can propagate without being within a distance to 1. and ELU), the graph shows a rate of O(1/`) as expected l+1 l 1 2 Indeed, we have for all l, |c − c | ≤ , therefore compared to O(1/` ) for ReLU. βq for L ≥ 1, |cL − c0| ≤ L . Assuming c0 < c < 1 βq So far, we have discussed the impact of the EOC and the for all inputs where c is a constant, the maximal depth smoothness of the activation function on the behaviour of l we can reach without loosing (1 − ) × 100% of the c . We now refine this analysis by studying βq as a function information is Lmax = bβq(1 − c − )c, this satisfies of (σb, σw). We also show that βq plays a more impor- limσ →0 Lmax = ∞. tant role in the information propagation process. Indeed, b we show that βq controls the propagation of the correla- tion and the back-propagation of the Gradients. For the Choice of σb on the Edge of Chaos : Given a network of back-propagation part, we use the approximation that the depth L, it follows that selecting a value of σb on the EOC weights used during forward propagation are independent such that βq ≈ L appears appropriate. of the weights used during backpropagation. This simpli- fies the calculations for the gradient backpropagation; see We verify numerically the benefits of this rule in the next (Schoenholz et al., 2017) for details and (Yang, 2019) for a section. theoretical justification. Note that ReLU-like activation functions do not satisfy con- Proposition 4. Let φ ∈ A be a non-linear activation func- ditions of Proposition4. The next lemma gives easy-to- tion such that φ(0) = 0, φ0(0) 6= 0. Assume that V [φ] verify sufficient conditions for Proposition4. is non-decreasing and V [φ0]) is non-increasing, and let Lemma 5. Let φ ∈ A such that xφ(x)φ0(x) ≥ 0 and 00 σmax > 0 be defined as in Proposition2. Let E be a φ(x)φ (x) ≤ 0 for all x ∈ R. Then, φ satisfies all condi- differentiable loss function and define the gradient with re- tions of Proposition4. spect to the th layer by ∂E ∂E and let ˜l l ∂yl = ( ∂yl )1≤i≤Nl Qab = i Example: Tanh and ELU satisfy all conditions of Lemma5. ∂E T ∂E E[ ∂yl(a) ∂yl(b) ] (Covariance matrix of the gradients dur- This may partly explain why ELU performs experimentally 0 √ 2 2E[φ ( qZ) ] better than ReLU (see next section). Another example is ing backpropagation). Recall that βq = 00 √ 2 . qE[φ ( qZ) ] an activation function of the form λx + βTanh(x) where Then, for any σb < σmax, by taking (σb, σw) ∈ EOC we λ, β ∈ R. We check the performance of these activations in have the next section. On the Impact of the Activation Function on Deep Neural Networks Training
(a) ELU (b) ReLU (c) Tanh
Figure 5. 100 epochs of the training curve (test accuracy) for different activation functions for depth 200 and width 300 using SGD. The red curves correspond to the EOC, the green ones corresponds to an ordered phase, and the blue curves corresponds to an Initialization on the EOC plus a Batch Normalization after each layer. The upper figures show the test accuracies with respect to the epochs while the lower figures show the accuracies with respect to time.
4. Experiments Table 1. Test accuracies for width 300 and depth 200 with different In this section, we demonstrate empirically the theoretical activation function on MNIST and CIFAR10 after 100 epochs results established above. We show that: MNIST EOC EOC + BN ORD PHASE • For deep networks, only an initialization on the EOC RELU 93.57± 0.18 93.11± 0.21 10.09± 0.61 could make the training possible, and the initialization ELU 97.62± 0.21 93.41± 0.3 10.14± 0.51 on the EOC performs better than Batch Normalization. TANH 97.20± 0.3 10.74± 0.1 10.02± 0.13
• Smooth activation functions in the sense of Proposition CIFAR10 EOC EOC + BN ORD PHASE 3 perform better than ReLU-like activation, especially RELU 36.55± 1.15 35.91± 1.52 9.91± 0.93 for very deep networks. ELU 45.76± 0.91 44.12± 0.93 10.11± 0.65 TANH 44.11± 1.02 10.15± 0.85 9.82± 0.88 • Choosing the right point on the EOC further accelerates the training.
We demonstrate empirically our results on the MNIST in the optimization algorithm being stuck eventually at a and CIFAR10 datasets for depths L between 10 and 200 very poor test accuracy of ∼ 0.1 (equivalent to selecting and width 300. We use SGD and RMSProp for training. the output uniformly at random). Figure5 also shows that We performed a grid search between 10−6 and 10−2 with EOC combined to BatchNorm results in a worse learning exponential step of size 10 to find the optimal learning rate. curve and dramatically increases the training time. Note For SGD, a learning rate of ∼ 10−3 is nearly optimal for that it is crucial here to initialize BatchNorm parameters L ≤ 150, for L > 150, the best learning rate is ∼ 10−4. to α = 1 and β = 0 in order to keep our analysis on the For RMSProp,10−5 is nearly optimal for networks with forward propagation on the EOC valid for networks with depth L ≤ 200 (for deeper networks, 10−6 gives better BatchNorm. Table4 presents test accuracy after 100 epochs results). We use a batchsize of 64. for different activation functions and different training meth- ods (EOC, EOC+BatchNorm, Ordered phase) on MNIST Initialization on the Edge of Chaos. We initialize ran- and CIFAR10. For all activation functions but Softplus, l iid 2 EOC initialization leads to the best performance. Adding domly the network by sampling Wij ∼ N (0, σw/Nl−1) BatchNorm to the EOC initialization makes the training l iid 2 and Bi ∼ N (0, σb ). Figure5 shows that the initialization worse, this can be explained the fact that parameters α and on the EOC dramatically accelerates the training for ELU, β are also modified during the first backpropagation. This ReLU and Tanh. The initialization in the ordered phase invalidates the EOC results for gradient backpropagation (here we used (σb, σw) = (1, 1) for all activations) results (see proof of Proposition4). On the Impact of the Activation Function on Deep Neural Networks Training
(a) Epoch 10 (b) Epoch 50 (c) Epoch 100
Figure 6. Test accuracies for Tanh network with depths between 30 and 90 and width 300 using different points on the EOC.
Table 2. Test accuracies for width 300 and depth 200 with different Table 3. Best test accuracy achieved after 100 epochs with Tanh activation function on MNIST and CIFAR10 after 10, 50 and 100 on MNIST epochs DEPTH L = 30 L = 50 L = 200 MNIST EPOCH 10 EPOCH 50 EPOCH 100 2k × 10−2 0.080 0.040 0.020 RELU 66.76± 1.95 88.62± 0.61 93.57± 0.18 WITH RULE βq ≈ L 0.071 0.030 0.022 ELU 96.09± 1.55 97.21± 0.31 97.62± 0.21 TANH 89.75± 1.01 96.51± 0.51 97.20± 0.3
CIFAR10 EPOCH 10 EPOCH 50 EPOCH 100 5. Discussion RELU 26.46± 1.68 33.74± 1.21 36.55± 1.15 ELU 35.95± 1.83 45.55± 0.91 47.76± 0.91 The Gaussian process approximation of Deep Neural Net- TANH 34.12± 1.23 43.47± 1.12 44.11± 1.02 works was used by (Schoenholz et al., 2017) to show that very deep Tanh networks are trainable only on the EOC. We give here a comprehensive analysis of the EOC for a large class of activation functions. We also prove that smoothness plays a major role in terms of signal propagation. Numer- Impact of the smoothness of the activation function on ical results in Table4 confirm this finding. Moreover, we the training. Table4 shows the test accuracy at different introduce a rule to choose the optimal point on the EOC, epochs for ReLU, ELU, Tanh. Smooth activation functions this point is a function of the depth. As the depth goes to perform better than ReLU. More experimental results with infinity (e.g. L = 400), we need smaller σb to achieve the RMSProp and other activation functions of the form x + best signal propagation. However, the limiting variance q αTanh(x) are provided in the supplementary material. also becomes close to zero as σb goes to zero. To avoid this problem, one possible solution is to change the acti- Selection of a point on the EOC. We have showed that a vation function to ensure that the coefficient β becomes sensible choice is to select σ such that L ∼ β on the EOC. q b q large independently of the choice of σ on the EOC (see Figure6 shows test accuracy of a Tanh network for differ- b supplementary material). ent depths using σb ∈ {0.05, 0.2, 0.5}. With σb = 0.05, we have βq ∼ 50. We see for depth 50, the red curve Our results have implications for Bayesian neural networks (σb = 0.05) is the best. For other depths L between 30 and which have received renewed attention lately; see, e.g., 90, σb = 0.05 is the value that makes βq the closest to L (Hernandez-Lobato and Adams, 2015) and (Lee et al., 2018). among {0.05, 0.2, 0.5}, which explains why the red curve They indeed indicate that, if one assigns i.i.d. Gaussian prior is approximately better for all depths between 30 and 90. distributions to the weights and biases, we need to select To further confirm this finding, we search numerically for not only the prior parameters (σb, σw) on the EOC but also −2 the best σb ∈ {2k × 10 : k ∈ [1, 50]} for depths an activation function satisfying Proposition3 to obtain a 30, 100, 200. Table4 shows the results. non-degenerate prior on the induced function space. On the Impact of the Activation Function on Deep Neural Networks Training
References K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rec- tifiers: Surpassing human-level performance on imagenet S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl- classification. ICCV, 2015. Dickstein. Deep information propagation. 5th Inter- national Conference on Learning Representations, 2017. G. Klambauer, T. Unterthiner, and A. Mayr. Self- normalizing neural networks. Advances in Neural In- I. Goodfellow, Y. Bengio, and A. Courville. Deep formation Processing Systems, 30, 2017. Learning. MIT Press, 2016. http://www. deeplearningbook.org. G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient G.F. Montufar, R. Pascanu, K. Cho, and Y. Bengio. On independence, and neural tangent kernel derivation. the number of linear regions of deep neural networks. arXiv:1902.04760, 2019. Advances in Neural Information Processing Systems, 27: 2924–2932, 2014. J. M. Hernandez-Lobato and R.P. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Gan- networks. ICML, 2015. guli. Exponential expressivity in deep neural networks through transient chaos. 30th Conference on Neural In- formation Processing Systems, 2016.
R.M. Neal. Bayesian Learning for Neural Networks, volume 118. Springer Science & Business Media, 1995.
A.G. Matthews, J. Hron, M. Rowland, R.E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. 6th International Conference on Learning Representations, 2018.
J. Lee, Y. Bahri, R. Novak, S.S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as Gaussian processes. 6th International Conference on Learning Representations, 2018.
D.A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and ac- curate deep network learning by exponential linear units (elus). ICLR, 2016.
D. Pedamonti. Comparison of non-linear activation func- tions for deep neural networks on mnist classification task. arXiv 1804.02763, 2018.
P. Ramachandran, B. Zoph, and Q.V. Le. Searching for activation functions. arXiv e-print 1710.05941, 2017.
M. Milletarí, T. Chotibut, and P. Trevisanutto. Expectation propagation: a probabilistic view of deep feed forward networks. arXiv:1805.08786, 2018.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. AISTATS, 2011.
V. Nair and G.E. Hinton. Rectified linear units improve restricted boltzmann machines. ICML, 2010.
B. Xu, N. Wang, T. Chen, and M. Li. Empirical eval- uation of rectified activations in convolution network. arXiv:1505.00853, 2015. On the Impact of the Activation Function on Deep Neural Networks Training A. Proofs We provide in this supplementary material the proofs of theoretical results presented in the main document, and we give additive theoretical and experimental results. For the sake of clarity we recall the results before giving their proofs.
A.1. Convergence to the fixed point: Proposition 1
02 00 2 1 Lemma 1. Let Mφ := sup [|φ (xZ) + φ (xZ)φ(xZ)|]. Suppose Mφ < ∞, then for σ < and any σb, we have x≥0E w Mφ (σb, σw) ∈ Dφ,var and Kφ,var(σb, σw) = ∞ √ 0 0 2 Moreover, let Cφ,δ := supx,y≥0,|x−y|≤δ,c∈[0,1]E[|φ (xZ1)φ (y(cZ1 + 1 − c Z2)|]. Suppose Cφ,δ < ∞ for some positive δ, 2 1 1 then for σ < min( , ) and any σb, we have (σb, σw) ∈ Dφ,var ∩ Dφ,corr and Kφ,var(σb, σw) = Kφ,corr(σb, σw) = w Mφ Cφ ∞.
l l Proof. To abbreviate the notation, we use q := qa for some fixed input a.
l l l l−1 Convergence of the variances: We first consider the asymptotic behaviour of q = qa. Recall that q = F (q ) where
2 2 √ 2 F (x) = σb + σwE[φ( xZ) ]. The first derivative of this function is given by Z √ √ √ √ √ F 0(x) = σ2 [√ φ0( xZ)φ( xZ)] = σ2 [φ0( xZ)2 + φ00( xZ)φ( xZ)], (4) wE x wE where we use Gaussian integration by parts, E[ZG(Z)] = E[G0(Z)], an identity satisfied by any function G such that E[|G0(Z)|] < ∞. Using the condition on φ, we see that the function F is a contraction mapping for σ2 < 1 and the Banach fixed-point w Mφ l theorem guarantees the existence of a unique fixed point q of F , with liml→+∞ q = q. Note that this fixed point depends only on F , therefore this is true for any input a and Kφ,var(σb, σw) = ∞.
q d p l l Convergence of the covariances: Since Mφ < ∞, then for all a, b ∈ R there exists l0 such that | qa − qb| < δ for all l > l0. Let l > l0, using Gaussian integration by parts, we have l+1 q q q dcab 2 0 l 0 l l l 2 l = σwE[|φ ( qaZ1)φ ( qb(cabZ1 + 1 − (cab) Z2)|]. dcab We cannot use the Banach fixed point theorem directly because the integrated function here depends on l through ql. For l l ease of notation, we write c := cab. We have
Z cl l+1 l+1 l dc 2 l l−1 |c − c | = | l (x)dx| ≤ σwCφ|c − c |. cl−1 dc Therefore, for σ2 < min( 1 , 1 ), cl is a Cauchy sequence and it converges to a limit c ∈ [0, 1]. At the limit w Mφ Cφ √ 2 2 √ √ 2 σb + σwE[φ( qz1)φ( q(cz1 + 1 − c z2)))] c = f(c) = . q The derivative of this function is given by √ 0 2 0 √ 0 √ f (x) = σwE[φ ( qZ1)φ ( q(xZ1 + 1 − xZ2)].
0 By assumption on φ and the choice of σw, we have supx|f (x)| < 1 so f is a contraction and has a unique fixed point. Since f(1) = 1 then c = 1. The above result is true for any a, b, therefore Kφ,var(σb, σw) = Kφ,corr(σb, σw) = ∞. On the Impact of the Activation Function on Deep Neural Networks Training
d Lemma 2. Let (σb, σw) ∈ Dφ,var ∩ Dφ,corr such that q > 0, a, b ∈ R and φ an activation function 2 l+1 l such that supx∈K [φ(xZ) ] < ∞ for all compact sets K. Define fl by c = fl(c ) and f by f(x) = E √ a,b a,b 2 2 √ √ 2 σb +σwE[φ( qZ1)φ( q(xZ1+ 1−x Z2)) q . Then liml→∞ supx∈[0,1] |fl(x) − f(x)| = 0.
Proof. For x ∈ [0, 1], we have q q 1 1 2 2 l l fl(x) − f(x) = (q − )(σb + σwE[φ( qaZ1)φ( qbu2(x))]) l l q qaqb σ2 q q √ √ + w ( [φ( ql Z )φ( ql u (x))] − [φ( qZ )φ( qu (x))]), q E a 1 b 2 E 1 2 √ 2 where u2(x) := xZ1 + 1 − x Z2. The first term goes to zero uniformly in x using the condition on φ and Cauchy-Schwartz inequality. As for the second term, it can be written again as q q q l √ l √ l √ E[(φ( qaZ1) − φ( qZ1))φ( qbu2(x))] + E[φ( qZ1)(φ( qbu2(x)) − φ( qu2(x)))]. Using Cauchy-Schwartz and the condition on φ, both terms can be controlled uniformly in x by an integrable upper bound. We conclude using dominated convergence.
0 Lemma 3 (Weak EOC). Let φ be a ReLU-like function with λ, β defined as above. Then fl does not depend on l, and 0 l q 2 having fl (1) = 1 and q bounded is only achieved for the singleton (σb, σw) = (0, λ2+β2 ). The Weak EOC is defined as this singleton.
l l Proof. We write q = qa throughout the proof. Note first that the variance satisfies the recursion: λ2 + β2 ql+1 = σ2 + σ2 [φ(Z)2]ql = σ2 + σ2 ql. (5) b wE b w 2
q 2 2 2 2 2 −1 For all σw < λ2+β2 , q = σb 1 − σw(λ + β )/2 is a fixed point. This is true for any input, therefore Kφ,var(σb, σw) = ∞ and (i) is proved.
2 0 2 2 λ2+β2 2 2 2 Now, the EOC equation is given by χ1 = σwE[φ (Z) ] = σw 2 . Therefore, σw = λ2+β2 . Replacing σw by its critical value in (5) yields
l+1 2 l q = σb + q .
2 l Thus q = σb + q if and only if σb = 0, otherwise q diverges to infinity. So the frontier is reduced to a single point 2 2 0 2 −1 (σb , σw) = (0, E[φ (Z) ] ), and the variance does not depend on l.
2 2 Proposition 1 (EOC acts as Residual connections). Consider a ReLU network with parameters (σb , σw) = (0, 2) ∈ EOC l and let cab be the corresponding correlation. Consider also a ReLU network with simple residual connections given by
Nl−1 l l−1 X l l−1 l yi(a) = yi (a) + W ijφ(yj (a)) + Bi, j=1
l 2 l σw 2 l where W ∼ N (0, ) and B ∼ N (0, σ ). Let c be the corresponding correlation. Then, by taking σw > 0 and ij Nl−1 i b ab σb = 0, there exists a constant γ > 0 such that 9π2 1 − cl ∼ γ(1 − cl ) ∼ ab ab 2l2 as l → ∞. On the Impact of the Activation Function on Deep Neural Networks Training
Proof. Let us first give a closed-form√ formula of the correlation function f of a ReLU network. In this case, we have 2 f(x) = 2E[(Z1)+(xZ1 + 1 − x Z2)+] where (x)+ := x1x>0. Let x ∈ [0, 1], f is differentiable and satisfies
0 √ f (x) = 2 [1 1 2 ], E Z1>0 xZ1+ 1−x Z2>0 which is also differentiable. Simple algebra leads to 1 f ”(x) = √ . π 1 − x2
Since arcsin0(x) = √ 1 and f 0(0) = 1/2, 1−x2 1 1 f 0(x) = arcsin(x) + . π 2 √ Using the fact that R arcsin = x arcsin + 1 − x2 and f(1) = 1, we conclude that for x ∈ [0, 1], f(x) = 1 x arcsin(x) + √ π 1 2 1 π 1 − x + 2 x. q 2 l l−1 2 l−1 2 σw l−1 For the residual network, we have qa = qa + σwE[φ( qa Z) ] = (1 + 2 )qa . 1 Let δ = 2 . We have σw 1+ 2
l l−1 2 l−1 cab = δcab + δσwE[φ(Z1)φ(U2(cab ))] σ2 = cl−1 + δ w (f(cl−1) − cl−1) ab 2 ab ab
Now, we use Taylor expansion near to conclude. However, since f is not differentiable in 1 for all orders, we use a change of variable x = 1 − t2 with t close to 0, then √ π √ 2 arcsin(1 − t2) = − 2t − t3 + O(t5), 2 12 so that √ π √ 2 arcsin(x) = − 2(1 − x)1/2 − (1 − x)3/2 + O((1 − x)5/2), 2 12 and √ π √ 11 2 x arcsin(x) = x − 2(1 − x)1/2 + (1 − x)3/2 + O((1 − x)5/2). 2 12 Since √ p √ 2 1 − x2 = 2(1 − x)1/2 − (1 − x)3/2 + O((1 − x)5/2), 4 we obtain that √ 2 2 f(x) = x + (1 − x)3/2 + O((1 − x)5/2). (6) x→1− 3π 0 1 π l l+1 Since (f(x) − x) = π (arcsin(x) − 2 ) < 0 and f(1) = 1, for all x ∈ [0, 1), f(x) > x. If c < c then by taking the image by f (which is increasing because f 0 ≥ 0) we have that cl+1 < cl+2, and we know that c1 = f(c0) ≥ c0, so by induction the sequence cl is increasing, and therefore it converges to the fixed point of f which is 1.
Using a Taylor expansion of f near 1, we have √ 2 2 cl = cl−1 + δ (1 − cl−1)3/2 + O((1 − cl−1)5/2) ab ab 3π ab ab and √ 2 2 cl = cl−1 + (1 − cl−1)3/2 + O((1 − cl−1)5/2). ab ab 3π ab ab On the Impact of the Activation Function on Deep Neural Networks Training √ l 2 2 3/2 5/2 Now let γl := 1 − cab for a, b fixed. We note s = 3π , from the series expansion we have that γl+1 = γl − sγl + O(γl ) so that s γ−1/2 = γ−1/2(1 − sγ1/2 + O(γ3/2))−1/2 = γ−1/2(1 + γ1/2 + O(γ3/2)) l+1 l l l l 2 l l s = γ−1/2 + + O(γ ). l 2 l
Thus, as l goes to infinity s γ−1/2 − γ−1/2 ∼ l+1 l 2 and by summing and equivalence of positive divergent series
s γ−1/2 ∼ l. l 2
l 9π2 l Therefore, we have 1 − cab ∼ 2l2 . Using the same argument for cal, we conclude.
1 0 Proposition 2. Let φ ∈ Dg be non ReLU-like function. Assume V [φ] is non-decreasing and V [φ ] is non-increasing. Let q V [φ](x) 2 V [φ] σmax := supx≥0 |x − V [φ0](x) | and for σb < σmax let qσb be the smallest fixed point of the function σb + V [φ0] . Then √ 1 we have EOC = {(σb, 0 √ 2 ): σb < σmax}. E[φ ( qZ) ]
To prove Proposition2, we need to introduce some lemmas. The next lemma gives a characterization of ReLU-like activation functions. Lemma 1.1 (A Characterization of ReLU-like activations). Let φ ∈ D1(R, R) such that φ(0) = 0 and φ0 non-identically zero. We define the function e for non-negative real numbers by √ V [φ](x) [φ( xZ)2] e(x) = = E √ V [φ0](x) E[φ0( xZ)2]
Then, for all x ≥ 0, e(x) ≤ x. Moreover, the following statements are equivalent
• There exists x0 > 0 such that e(x0) = x0.
• φ is ReLU-like, i.e. there exists λ, β ∈ R such that φ(x) = λx if x > 0 and φ(x) = βx if x ≤ 0.
√ √ R z 0 √ Proof. Let x > 0. We have for all z ∈ R, φ( xz) = x 0 φ ( xu)du. This yields
Z Z √ 2 0 √ 2 E[φ( xZ) ] = xE[ φ ( xu)du ] 0 Z |Z| 0 √ 2 ≤ xE[|Z| φ ( xu) du] 0 Z Z 0 √ 2 = xE[Z φ ( xu) du] 0 0 √ 2 = xE[φ ( xZ) du] where we have used Cauchy-Schwartz inequality and Gaussian integration by parts. Therefore e(x) ≤ x. On the Impact of the Activation Function on Deep Neural Networks Training
Now assume there exists x0 > 0 such that e(x0) = x0. We have Z Z √ 2 0 √ 2 E[φ( x0Z) ] = x0E[ φ ( x0u)du ] 0 Z Z Z 0 0 √ 2 0 √ 2 = x0E[1Z>0 φ ( x0u)du ] + x0E[1Z≤0 φ ( x0u)du ] 0 Z Z Z Z Z Z 0 Z 0 0 √ 2 0 √ 2 ≤ x0E[1Z>0 1du φ ( x0u) du] + x0E[1Z≤0 1du φ ( x0u) du]. 0 0 Z Z
The equality in Cauchy-Schwartz inequality implies that 0 √ - For almost every z > 0, there exists λz such that φ ( x0u) = λz for all u ∈ [0, z]. 0 √ - For almost every z < 0, there exists βz such that φ ( x0u) = βz for all u ∈ [z, 0]. Therefore, λz, βz are independent of z, and φ is ReLU-like.
It is easy to see that for ReLU-like activations, e(x) = x for all x ≥ 0.
The next trivial lemma provides a sufficient condition for the existence of a fixed point of a shifted function. 0 + + Lemma 1.2. Let g ∈ C (R , R) such that g(0) = 0 and g(x) ≤ x for all x ∈ R . Let tmax := supx≥0 |x − g(x)| (tmax may be infinite). Then, for all t ∈ [0, tmax), the shifted function t + g(.) has a fixed point.
Proof. Let t ∈ [0, tmax). There exists x0 > 0 such that t + g(.) < x0 − g(x0) + g(.). So we have t + g(0) = t and t + g(x0) < x0, which means that t + g(.) crosses the identity line, therefore the fixed point exists.
1 V [φ](x) Corollary 1.1. Let φ ∈ D (R, R) such that φ is non ReLU-like. Let tmax = supx≥0 |x − V [φ0](x) |. Then, For any 2 2 V [φ] σb ∈ [0, tmax), the shifted function σb + V [φ0] has a fixed point q. Moreover, by taking q to be the greatest fixed point, we have limσb→0 q = 0.
V [φ](x) The limit of q is zero because it is a fixed point of the function V [φ0](x) which has only 0 as a fixed point for non ReLU-like functions.
2 V [φ] Corollary 1.1 proves the existence of a fixed point for the shifted function σb + V [φ0] , which is a necessary condition for p 0 (σb, 1/ V [φ ](q)) to be in the EOC where q is the smallest fixed point. It is not a sufficient condition because q may not be 2 1 the smallest fixed point of σb + V [φ0](q) V [φ]. We further analyse this problem hereafter. + + g(x) Definition 6 (Permissible couples). Let g, h ∈ C(R , R ) and c > 0. Define the function k(x) = c + h(x) for x ≥ 0 and let q = inf{x : k(x) = x}. We say that (g, h) is permissible if for any c ≥ 0 such that q < ∞, q is the smallest fixed point g(.) of the function c + h(q) . Lemma 1.3. Let g, h ∈ C(R+, R+). Then the following statements are equivalent
1. (g, h) is permissible. 2. For any c > 0 such that q is finite, we have g(q) − g(x) < (q − x)h(q) for x ∈ [0, q).
g(.) 1 Proof. If q is a fixed point of c + h(.) , then q is clearly a fixed point of I(x) = c + h(q) g(x). Having q is the smallest fixed g(.) g(x) g(q) point of c + h(q) is equivalent to c + h(q) > x for all x ∈ [0, q). Since c = q − h(q) , we conclude.
Corollary 1.2. Let g, h ∈ C(R+, R+). Assume h is non-increasing, then (g, h) is permissible.
h(q) g(x) Proof. Since h is non-increasing, we have for x ∈ [0, q), g(q) − g(x) ≤ h(q)(q − c) − h(x) g(x) = h(q)(q − (c + h(x) )). g(x) We conclude using the fact that c + h(x) > x for x ∈ [0, q). On the Impact of the Activation Function on Deep Neural Networks Training
Corollary 1.3. Let φ be a non ReLU-like function. Assume V [φ] is non-decreasing and (V [φ],V [φ0]) is permissible. Then, 2 2 1 for any σ < tmax := sup |x − e(x)|, by taking σ = 0 √ 2 , we have (σb, σw) ∈ EOC. Moreover, we have b x≥0 w E[φ ( qZ) ] limσb→0 q = 0.
We can omit the condition ’V [φ] is non-decreasing’ by choosing a small tmax. Indeed, by taking a small σb, the limiting variance q is small, and we know that V [φ] is increasing near 0 because V [φ]0(0) = φ0(0)2 > 0.
The proof of Proposition2 is straightforward from corollary A.3.
Lemma 4. Let φ be a Tanh-like activation function, then φ satisfies all conditions of Proposition2 and EOC = √ 1 + {(σb, 0 √ 2 ): σb ∈ R }. E[φ ( qZ) ]
0 1 √ 0 √ √ Proof. For x ≥ 0, we have V [φ] (x) = x E[ xZφ ( xZ)φ( xZ)] ≥ 0, so V [φ] is non-decreasing. Moreover, 0 0 1 √ 00 √ 0 √ 0 V [φ ] (x) = x E[ xZφ ( xZ)φ ( xZ)] ≤ 0, therefore V [φ ] is non-increasing. To conclude, we still have to show that tmax = ∞. Using the second condition on φ, there exists M > 0 such that |φ0(y)|2 ≥ Me−2α|y|. Let x > 0. we have √ 0 √ 2 −2α| xZ| E[φ ( xZ) ] ≥ ME[e ] 2 Z ∞ √ e−z /2 = 2M e−2α xZ √ dz 0 2π 2 √ = 2Me2α xΨ(2α x) 2M ∼ √ 2α x
2 e−x /2 where Ψ is the Gaussian cumulative function and where we used the asymptotic approximation Ψ(x) ∼ x for large x. Using this lower bound and the upper bound on φ, there exists x , k > 0 such that for x > x , we have x − V [φ](x) ≥ √ 0 0 V [φ0](x) x − k x → ∞ which concludes the proof.
Proposition 3 (Convergence rate for smooth activations). Let φ ∈ A such that φ non-linear (i.e. φ(2) is non-identically 0 √ 2 l βq 2E[φ ( qZ) ] zero). Then, on the EOC, we have 1 − c ∼ where βq = 00 √ 2 . l qE[φ ( qZ) ] √ l 2 Proof. We first prove that liml→∞ c = 1 on the EOC. Let x ∈ [0, 1) and u2(x) := xZ1 + 1 − x Z2, we have
0 2 0 √ 0 √ f (x) = σwE[φ ( qZ1)φ ( qu2(x))] 2 0 √ 2 1/2 0 √ 2 1/2 ≤ σw(E[φ ( qZ1) ]) (E[φ ( qu2(x)) ]) = 1
2 1 where we have used Cauchy Schwartz inequality and the fact the σ = 0 √ 2 . Moreover, the equality holds if and √ w E[φ ( qZ) ] 0 √ 2 0 √ only if there exists a constant s such that φ ( q(xz1 + 1 − x z2)) = sφ ( qz1) for almost any z1, z2 ∈ R, which is equivalent to having φ0 equal to a constant almost everywhere on R, hence φ is linear and q does not exists. This proves that for all x ∈ [0, 1), f 0(x) < 1. Integrating both sides between x and 1 yields f(x) > x for all x ∈ [0, 1). Therefore cl is non-decreasing and converges to the fixed point of f which is 1.
3 Now we want to prove that f admits a Taylor expansion near 1. It is easy to do that if φ ∈ Dg. Indeed, using the conditions on φ, we can easily see that f has a third derivative at 1 and we have
0 2 0 √ 2 f (1) = σwE[φ ( qZ) ] 00 2 00 √ 2 f (1) = σwqE[φ ( qZ) ]. On the Impact of the Activation Function on Deep Neural Networks Training
A Taylor expansion near 1 yields
(x − 1)2 f(x) = 1 + f 0(1)(x − 1) + f 00(1) + O((x − 1)3) 2 (x − 1)2 = x + + O((x − 1)3). βq
(2) The proof is a bit more complicated for general φ ∈ A. We prove the result when φ (x) = 1x<0g1(x) + 1x≥0g2(x). The generalization to the whole class is straightforward. Let us first show that there exists g ∈ C1 such that f (3)(x) = √ 1 g(x). 1−x2 We have
00 2 00 √ 00 √ f (x) = σwqE[φ ( qZ1)φ ( qU2(x))] 2 00 √ √ 2 00 √ √ = σwqE[φ ( qZ1)1U2(x)<0g1( qU2(x))] + σwqE[φ ( qZ1)1U2(x)>0g2( qU2(x))].
00 √ √ Let G(x) = E[φ ( qZ1)1U2(x)<0g1( qU2(x))] then
0 00 √ x 1 √ G (x) = E[φ ( qZ1)(Z1 − √ Z2)δU2(x)=0 √ g1( qU2(x))] 1 − x2 1 − x2 00 √ √ x 0 √ + E[φ ( qZ1)1U2(x)<0 q(Z1 − √ Z2)g1( qU2(x))]. 1 − x2
After simplification, it is easy to see that G0(x) = √ 1 G (x) where G ∈ C1. By extending the same analysis to the 1−x2 1 1 second term of f 00, we conclude that there exists g ∈ C1 such that f (3)(x) = √ 1 g(x). 1−x2
Let us now derive a Taylor expansion of f near 1. Since f (3) is potentially non defined at 1, we use the change of variable x = 1 − t2 to compensate this effect. Simple algebra shows that the function t → f(1 − t2) has a Taylor expansion near 0
t4 f(1 − t2) = 1 − t2f 0(1) + f 00(1) + O(t5). 2 Therefore, (x − 1)2 f(x) = 1 + (x − 1)f 0(1) + f 00(1) + O((x − 1)5/2). 2 3 Note that this expansion is weaker than the expansion when φ ∈ Dg. l Denote λl := 1 − c , we have
2 λl 5/2 λl+1 = λl − + O(λl ) βq therefore,
−1 −1 λl 3/2 −1 λl+1 = λl (1 − + O(λl )) βq
−1 λl 3/2 = λl (1 + + O(λl )) βq
−1 1 1/2 = λl + + O(λl ). βq
By summing (divergent series), we conclude that λ−1 ∼ l . l βq On the Impact of the Activation Function on Deep Neural Networks Training
Proposition 4. Let φ ∈ A be a non-linear activation function such that φ(0) = 0, φ0(0) 6= 0. Assume that V [φ] is 0 non-decreasing and V [φ ]) is non-increasing, and let σmax > 0 be defined as in Proposition2. Define the gradient with th ∂E ∂E ˜l ∂E T ∂E respect to the l layer by l = ( l )1≤i≤Nl and let Qab = E[ l l ] denote the covariance matrix of the gradients ∂y ∂yi ∂ya ∂yb 0 √ 2 2E[φ ( qZ) ] during backpropagation. Recall that βq = 00 √ 2 . qE[φ ( qZ) ]
Then, for any σb < σmax, by taking (σb, σw) ∈ EOC we have
• sup |f(x) − x| ≤ 1 x∈[0,1] βq
˜l Tr(Qab) 2 • For l ≥ 1, | l+1 − 1| ≤ ˜ βq Tr(Qab ) Moreover, we have lim βq = ∞. σb→0 (σb,σw)∈EOC
To prove this result, let us first prove a more general result. 2 Proposition 5 (How close is f to the identity function?). Let φ ∈ D (R, R) − {0} and (σb, σw) ∈ Dφ,var with q the corresponding limiting variance. Then, 2 2 0 √ 2 σw 00 √ 2 sup |f(x) − x| ≤ |σwE[φ ( qZ) ] − 1| + qE[φ ( qZ) ] x∈[0,1] 2
Proof. Using a second order Taylor expansion, we have for all s ∈ [0, 1] (1 − x)2 |f(x) − f(1) − f 0(1)(x − 1)| ≤ sup |f 00(θ)|. 2 θ∈[0,1]
2 0 (1−x) 00 We have f(1) = 1. Therefore |f(x) − x| ≤ (1 − x)|f (1) − 1| + 2 supθ∈[0,1] |f (θ)|. For θ ∈ [0, 1], we have 00 2 00 √ 00 √ f (θ) = σwqE[φ ( qZ1)φ ( qU2(θ))] 2 00 √ 2 ≤ σwqE[φ ( qZ) ] σ2 √ = w q [φ00( qZ)2] 2 E using Cauchy-Schwartz inequality.
2 As a result, for φ ∈ D (R, R) − {0} and (σb, σw) ∈ EOC with q the corresponding limiting variance, we have √ qE[φ00( qZ)2] 1 sup |f(x) − x| ≤ 0 √ 2 = x∈[0,1] 2E[φ ( qZ) ] βq which is the first result of Proposition4.
Now let us prove the second result for gradient backpropagation, we show that under some assumptions, our results of forward information propagation generalize to the back-propagation of the gradients. Let us first recall the results in (Schoenholz et al., 2017) (we use similar notations hereafter).
Let E be the loss we want to optimize. The backpropagation process is given by the equations
∂E l l−1 l = δiφ(yj ) ∂Wij
Nl+1 ∂E X δl = = φ0(yl) δl+1W l+1. i ∂yl i j ji i j=1 On the Impact of the Activation Function on Deep Neural Networks Training
l l l l 2 Although δi is non Gaussian (unlike yi), knowing how q˜a = E[(δi) ] changes back through the network will give us an idea about how the norm of the gradient changes. Indeed, following this approach, and using the approximation that the weights used during forward propagation are independent from those used for backpropagation, (Schoenholz et al., 2017) showed that l l+1 Nl+1 q˜a =q ˜a χ1 Nl 2 0 √ 2 where χ1 = σwE[φ ( qZ) ].
Considering a constant width network, authors concluded that χ1 controls also the depth scales of the gradient norm, i.e. l L −(L−l/ξ∆) −1 q˜a =q ˜a e where ξ∆ = − log(χ1). So in the ordered phase, gradients can propagate to a depth of ξ∆ without being exponentially small, while in the chaotic phase, gradient explode exponentially. On the EOC (χ1 = 1), the depth scale is infinite so the gradient information can also propagate deeper without being exponentially small. The following result shows that our previous analysis on the EOC extends to the backpropagation of gradients, and that we can make this propagation better by choosing a suitable activation function and an initialization on the EOC. We use the following approximation to ease the calculations: the weights used in forward propagation are independent from those used in backward propagation.
Proposition 6 (Better propagation for the gradient). Let a and b be two inputs and (σb, σw) ∈ Dφ,var with q the limiting l l l variance. We define the covariance between the gradients with respect to layer l by q˜ab = E[δi(a)δi(b)]. Then, we have
q˜l N √ √ | ab × l − 1| ≤ |σ2 [φ0( qZ)2] − 1| + (1 − cl )σ2 q [φ00( qZ)2] → 0. l+1 N wE ab w E σb→0 q˜ab l+1
Proof. We have
l l l q˜ab = E[δi(a)δi(b)]
Nl+1 Nl+1 0 l 0 l X l+1 l+1 X l+1 l+1 = E[φ (yi(a))φ (yi(b)) δj (a)Wji δj (b)Wji ] j=1 j=1
Nl+1 0 l 0 l l+1 l+1 X l+1 2 = E[φ (yi(a))φ (yi(b))] × E[δj (a)δj (b)] × E[ (Wji ) ] j=1
l+1 Nl+1 2 0 √ 0 √ l ≈ q˜ab σwE[φ ( qZ1)φ ( qU2(cab))] Nl l+1 Nl+1 0 l =q ˜ab f (cab). Nl We conclude using the fact that |f 0(x) − 1| ≤ |f 0(1) − 1| + (1 − x)f 00(1)
The dependence in the width of the layer is natural since it acts as a scale for the covariance. We define the gradient with th ∂E ∂E ˜l ∂E T ∂E respect to the l layer by l = ( l )1≤i≤Nl and let Qab = E[ l l ] denote the covariance matrix of the gradients ∂y ∂yi ∂ya ∂yb during backpropagation. Then, on the EOC, we have √ Tr(Q˜l ) qE[φ00( qZ)2] 2 | ab − 1| ≤ (1 − cl ) √ ≤ . ˜l+1 ab [φ0( qZ)2] β Tr(Qab ) E q
So again, the quantity |φ|EOC controls the vanishing of the covariance of the gradients during backpropagation. This was expected because linear activation functions do not change the covariance of the gradients.
B. Further theoretical results B.1. Results on the Edge of Chaos
The next lemma shows that under some conditions, the EOC does not include couples (σb, σw) with small σb > 0. On the Impact of the Activation Function on Deep Neural Networks Training
Lemma 5 (Trivial EOC). Assume there exists M > 0 such that E[φ00(xZ)φ(xZ)] > 0 for all x ∈]0,M[. Then, there exists + 1 1 σ > 0 such that EOC ∩ ([0, σ) × R ) = {(0, |φ0(0)| )}. Moreover, if M = ∞ then EOC = {(0, |φ0(0)| )}.
Activation functions that satisfy the conditions of Lemma5 cannot be used with small σb > 0 (note that using σb = 0 would lead to q = 0 which is not practical for the training), therefore, the result of Proposition4 do not apply in this case. However, as we will see hereafter, SiLU (a.k.a Swish) has a partial EOC, and still allows better information propagation (Proposition 3) compared to ReLU even if σb not very small.
1 2 V [φ] Proof. It is clear that (0, |φ0(0)| ) ∈ EOC. For σb > 0 we denote by q the smallest fixed point of the function σb + V [φ0]
(which is supposed to be the limiting variance on the EOC). Using the condition on φ and the fact that limσb→0 q = 0, there 00 √ √ exists σ > 0 such that for σb < σ we have E[φ ( qZ)φ( qZ)] > 0. Now let us prove that for σb ∈]0, σ[, the limiting variance does not satisfy the EOC equation. q V [φ] Let tmax = supx>0 |x − V [φ0] | and σb ∈]0, min(tmax, σ)[. Recall that for all x ≥ 0 we have that
0 2 0 √ 2 00 √ √ F (x) = σw(E[φ ( xZ) ] + E[φ ( xZ)φ( xZ)])
2 0 0 2 00 √ √ Using σw = 1/V [φ ](q) (EOC equation) we have that F (q) = 1 + σwE[φ ( qZ)φ( qZ)]) > 1. Therefore, the function 2 1 σb + V [φ0](q) V [φ] crosses the identity in a point qˆ < q, hence (σb, σw) 6∈ Dφ,var. Therefore, for any σb ∈]0, σ[, there is no σw such that (σb, σw) ∈ EOC.
If M = ∞, the previous analysis is true for any σ > 0, by taking the limit σ → ∞, we conclude.
This is true for activations such as Shifted Softplus (a shifted version of Softplus in order to have φ(0) = 0) and SiLU (a.k.a Swish). + Corollary 1. EOCSSoftplus = {(0, 2)} and there exists σ > 0 such that EOCSiLU ∩ ([0, σ[×R ) = {(0, 2)}
1 Proof. let s(x) = 1+e−x for all x ∈ R (sigmoid function).
1. Let sp(x) = log(1 + ex) − log(2) for x ∈ R (Shifted Softplus). We have sp0(x) = s(x) and sp00(x) = s(x)(1 − s(x)). For x > 0 we have
00 E[sp (xZ)sp(xZ)] = E[s(xZ)(1 − s(xZ))sp(xZ)] = E[1Z>0(s(xZ)(1 − s(xZ))sp(xZ))] + E[1Z<0(s(xZ)(1 − s(xZ))sp(xZ))] = E[1Z>0(s(xZ)(1 − s(xZ))sp(xZ))] + E[1Z<0(s(xZ)(1 − s(xZ))sp(−xZ))] = E[1Z>0(s(xZ)(1 − s(xZ))(sp(xZ) + sp(−xZ)))] > 0,
2+ey +e−y where we have used the fact that sp(y) + sp(−y) = log( 4 ) > 0 for all y > 0. We conclude using Lemma5. 2. Let si(x) = xs(x) (SiLU activation function, known also as Swish). We have si0(x) = s(x) + xs(x)(1 − s(x)) and si00(x) = s(x)(1 − s(x))(2 + x(1 − 2s(x))). Using the same technique as for SSoftplus, we have for x > 0
00 2 E[si (xZ)si(xZ)] = E[xZ × s(xZ) × (1 − s(xZ))(2 + xZ(1 − 2))] = E[1Z>0G(xZ)],
where G(y) = ys(y)(1 − s(y))(2 + y(1 − 2s(y)))(2s(y) − 1). The only term that changes sign is (2 + y(1 − 2s(y))). It is positive for small y and negative for large y. We conclude that there M > 0 such that E[si00(xZ)si(xZ)] > 0 for x ∈]0,M[. On the Impact of the Activation Function on Deep Neural Networks Training
B.2. Beyond the Edge of Chaos
Can we make the distance between f and the identity function small independently from the choice of σb? The answer is yes if we select the right activation function. Let us first define a semi-norm on D2(R, R). 00 √ 2 2 yE[φ ( yZ) ] Definition 7 (EOC semi-norm). The semi-norm |.|EOC is defined on D ( , ) by |φ|EOC = sup + 0 √ 2 . R R y∈R E[φ ( yZ) ] 2 |.|EOC is a norm on the quotient space D (R, R)/L(R) where L(R) is the space of linear functions.
V [φ] + When |φ|EOC is small, φ is close to a linear function, which implies that the function V [φ0] defined on R is close to the identity function. Thus, for a fixed σb, we expect q to become arbitrarily big when |φ|EOC goes to zero.
Lemma 2.1. Let (φn)n∈N be a sequence of functions such that limn→∞ |φn|EOC = 0. Let σb > 0 and assume that for all n ∈ N there exists σw,n such that (σw, σw,n) ∈ EOC. Let qn be the limiting variance. Then limn→∞ qn = ∞
2 1 σb 1 Proof. The proof is straightforward knowing that f(0) ≤ 2 |φn|EOC , which implies that q ≤ 2 |φn|EOC . 2 Corollary 2.1. Let φ ∈ D (R, R) − {0} and (σb, σw) ∈ EOC with q the corresponding limiting variance. Then, 1 sup |f(x) − x| ≤ |φ|EOC . x∈[0,1] 2
Corollary 2.1 shows that by taking an activation function φ such that |φ|EOC is small and by initializing the network on the EOC, the correlation function is close to the identity function, i.e., the signal propagates deeper through the network. However, note that there is a trade-off to take in account here: we loose expressiveness by taking |φ|EOC too small, because this would imply that φ is close to a linear function. So there is a trade-off between signal propagation and expressiveness We check this finding with activation functions of the form φα(x) = x + αTanh(x). Indeed, we 2 00 √ 2 have |φ | ≤ α sup + [Tanh ( xZ) ] → 0. So by taking small α, we would theoretically provide deeper α EOC y∈R E α→0 signal propagation. However, note that we loose expressiveness as α goes to zero because φα becomes closer to the identity function. So There is also a trade-off here. The difference with Proposition4 is that here we can compensate the expressiveness issue by adding more layers (see e.g. (Montufar et al., 2014) who showed that expressiveness grows exponentially with depth).
C. Experiments C.1. Training with RMSProp For RMSProp, the learning rate 10−5 is nearly optimal for networks with depth L ≤ 200 (for deeper networks, 10−6 gives better results). This learning rate was found by a grid search with exponential step of size 10. Figure7 shows the training curves of ELU, ReLU and Tanh on MNIST for a network with depth 200 and width 300. Here also, ELU and Tanh perform better than ReLU. This confirms that the result of Proposition3 is independent of the training algorithm. ELU has faster convergence than Tanh. This could be explained by the saturation problem of Tanh.
C.2. Training with activation φα(x) = x + αTanh(x)
As we have already mentioned, φα satisfies all conditions of Proposition3. Therefore, we expect it to perform at least better than ReLU for deep neural networks. Figure8 shows the training curve for width 300 and depth 200 with different activation functions. φ0.5 has approximately similar performance as ELU and better than Tanh and ReLU. Note that φα does not suffer form saturation of the gradient, which could explain why it performs better than Tanh.
C.3. Impact of φ00(0)
00 Since we usually take σb small on the EOC, then having φ (0) = 0 would make the coefficient βq even bigger. We test this result on SiLU (a.k.a Swish) for depth 70. SiLU is defined by
φSiLU (x) = x sigmoid(x) On the Impact of the Activation Function on Deep Neural Networks Training
Figure 7. 100 epochs of the training curves of ELU, ReLU and Tanh networks of depth 200 and width 300 on MNIST with RMSProp
Figure 8. 100 epochs of the training curves of ELU, ReLU, Tanh and φ0.5 networks of depth 200 and width 300 on MNIST with SGD we have φ00(0) = 1/2. consider a modified SiLU (MSiLU) defined by
−x2 φMSiLU (x) = x sigmoid(x) + (e − 1)/4
00 We have φMSiLU (0) = 0. Figure9 shows the the training curves (test accuracy) of SiLU and MSiLU on MNIST with SGD. MSiLU performs better than SiLU, expecially at the beginning of the training. On the Impact of the Activation Function on Deep Neural Networks Training
Figure 9. 50 epochs of the training curves of SiLU and MSiLU on MNIST with SGD