Geometry of Neural Network Loss Surfaces via Random Matrix Theory
Jeffrey Pennington 1 Yasaman Bahri 1
Abstract 2014; Choromanska et al., 2015; Neyshabur et al., 2015), architecture design, and generalization (Keskar et al.). Understanding the geometry of neural network loss surfaces is important for the development of 1.1. Related work improved optimization algorithms and for build- ing a theoretical understanding of why deep There is no shortage of prior work focused on the loss learning works. In this paper, we study the ge- surfaces of neural networks. Choromanska et al.(2015) ometry in terms of the distribution of eigenvalues and Dauphin et al.(2014) highlighted the prevalence of of the Hessian matrix at critical points of varying saddle points as dominant critical points that plague op- energy. We introduce an analytical framework timization, as well as the existence of many local minima and a set of tools from random matrix theory that at low loss values. Dauphin et al.(2014) studied the dis- allow us to compute an approximation of this dis- tribution of critical points as a function of the loss value tribution under a set of simplifying assumptions. empirically and found a trend which is qualitatively sim- The shape of the spectrum depends strongly on ilar to predictions for random Gaussian landscapes (Bray the energy and another key parameter, φ, which & Dean, 2007). Choromanska et al.(2015) argued that the measures the ratio of parameters to data points. loss function is well-approximated by a spin-glass model Our analysis predicts and numerical simulations studied in statistical physics, thereby predicting the exis- support that for critical points of small index, the tence of local minima at low loss values and saddle points number of negative eigenvalues scales like the 3/2 at high loss values as the network increases in size. Good- power of the energy. We leave as an open prob- fellow et al. observed that loss surfaces arising in prac- lem an explanation for our observation that, in tice tend to be smooth and seemingly convex along low- the context of a certain memorization task, the dimensional slices. Subsequent works (Kawaguchi, 2016; energy of minimizers is well-approximated by Safran & Shamir, 2016; Freeman & Bruna, 2016) have fur- 1 2 the function 2 (1 − φ) . thered these and related ideas empirically or analytically, but it remains safe to say that we have a long way to go before a full understanding is achieved. 1. Introduction 1.2. Our contributions Neural networks have witnessed a resurgence in recent years, with a smorgasbord of architectures and configura- One shortcoming of prior theoretical results is that they are tions designed to accomplish ever more impressive tasks. often derived in contexts far removed from practical neu- Yet for all the successes won with these methods, we have ral network settings – for example, some work relies on managed only a rudimentary understanding of why and in results for generic random landscapes unrelated to neural what contexts they work well. One difficulty in extend- networks, and other work draws on rather tenuous connec- ing our understanding stems from the fact that the neural tions to spin-glass models. While there is a lot to be gained network objectives are generically non-convex functions from this type of analysis, it leaves open the possibility that in high-dimensional parameter spaces, and understanding characteristics of loss surfaces specific to neural networks their loss surfaces is a challenging task. Nevertheless, an may be lost in the more general setting. In this paper, we improved understanding of the loss surface could have a focus narrowly on the setting of neural network loss sur- large impact on optimization (Saxe et al.; Dauphin et al., faces and propose an analytical framework for studying the spectral density of the Hessian matrix in this context. 1Google Brain. Correspondence to: Jeffrey Pennington
n2,m 1 X 3. Primer on Random Matrix Theory L = n = e2 . (2) 2 2m iµ i,µ=1 In this section we highlight a selection of results from the theory of random matrices that will be useful for our anal- The Hessian matrix is the matrix of second derivatives of ysis. We only aim to convey the main ideas and do not ∂2L the loss with respect to the parameters, i.e. Hαβ = , ∂θα∂θβ attempt a rigorous exposition. For a more thorough intro- (1) (2) where θα ∈ {W ,W }. It decomposes into two pieces, duction, see, e.g., (Tao, 2012). H = H0 + H1, where H0 is positive semi-definite and where H1 comes from second derivatives and contains all 3.1. Random matrix ensembles of the explicit dependence on the residuals. Specifically, The theory of random matrices is concerned with proper- n2,m ties of matrices M whose entries M are random variables. 1 X ∂yˆiµ ∂yˆiµ 1 ij [H ] ≡ ≡ [JJ T ] , (3) 0 αβ m ∂θ ∂θ m αβ The degree of independence and the manner in which the i,µ=1 α β Mij are distributed determine the type of random matrix ensemble to which M belongs. Here we are interested where we have introduced the Jacobian matrix, J, and, primarily in two ensembles: the real Wigner ensemble for n2,m 2 which M is symmetric but otherwise the M are i.i.d.; and 1 X ∂ yˆiµ ij [H ] ≡ e . (4) T 1 αβ m iµ ∂θ ∂θ the real Wishart ensemble for which M = XX where i,µ=1 α β Xij are i.i.d.. Moreover, we will restrict our attention to studying the limiting spectral density of M. For a random We will almost exclusively consider square networks with n×n matrix Mn ∈ , the empirical spectral density is de- n ≡ n0 = n1 = n2. We are interested in the limit of R large networks and datasets, and in practice they are typ- fined as, n ically of the same order of magnitude. A useful charac- 1 X ρMn (λ) = δ(λ − λj(Mn)) , (5) 1 n Although we additionally assume the random variables are j=1 independent, our framework does not explicitly require this as- 2 sumption, and in principle it could be relaxed in exchange for In our context of squared error, each of the n2 targets may be more technical computations. considered an effective example. Geometry of Neural Networks where the λj(Mn), j = 1, . . . , n, denote the n eigenvalues from which ρ can be recovered using the inversion formula, of Mn, including multiplicity, and δ(z) is the Dirac delta 1 function centered at z. The limiting spectral density is ρ(λ) = − lim Im G(λ + i) . (12) π →0+ defined as the limit of eqn. (5) as n → ∞, if it exists. 3.2.2. THE R TRANSFORM For a matrix M of the real Wigner matrix ensemble G whose entries M ∼ N (0, σ2), Wigner(1955) computed Given the Stieltjes transform of a probability distribu- ij ρ R its limiting spectral density and found the semi-circle law, tion , the transform is defined as the solution to the √ functional equation, ( 1 2 2 2πσ2 4σ − λ if |λ| ≤ 2σ 1 ρSC(λ; σ, φ) = . (6) R G(z) + = z . (13) 0 otherwise G(z) Similarly, if M = XXT is a real Wishart matrix with The benefit of the R transform is that it linearizes free con- n×p 2 X ∈ R and Xij ∼ N (0, σ /p), then the limiting spec- volution, in the sense that, tral density can be shown to be the Marchenko-Pastur dis- RA+B = RA + RB , (14) tribution (Marcenkoˇ & Pastur, 1967), ( if A and B are freely independent. It plays a role in free ρ(λ) if φ < 1 probability analogous to that of the log of the Fourier ρMP(λ; σ, φ) = , (1 − φ−1)δ(λ) + ρ(λ) otherwise transform in commutative probability theory. (7) where φ = n/p and, The prescription for computing the spectrum of A + B is as follows: 1) Compute the Stieltjes transforms of ρ 1 p A ρ(λ) = (λ − λ−)(λ+ − λ) and ρB; 2) From the Stieltjes transforms, deduce the R 2πλσφ (8) transforms RA and RB; 3) From RA+B = RA + RB, p 2 λ± = σ(1 ± φ) . compute the Stieltjes transform GA+B; and 4) Invert the The Wishart matrix M is low rank if φ > 1, which explains Stieltjes transform to obtain ρA+B. the delta function density at 0 in that case. Notice that there is an eigenvalue gap equal to λ−, which depends on φ. 4. Warm Up: Wishart plus Wigner
3.2. Free probability theory Having established some basic tools from random matrix theory, let us now turn to applying them to computing the Suppose A and B are two random matrices. Under what limiting spectral density of the Hessian of a neural network conditions can we use information about their individual at critical points. Recall from above that we can decom- spectra to deduce the spectrum of A + B? One way of pose the Hessian into two parts, H = H0 + H1 and that T analyzing this question is with free probability theory, and H0 = JJ /m. Let us make the secondary assumption the answer turns out to be exactly when A and B are freely that at critical points, the elements of J and H1 are i.i.d. independent. Two matrices are freely independent if normal random variables. In this case, we may take H0 to be a real Wishart matrix and H1 to be a real Wigner matrix. Ef1(A)g1(B) ··· fk(A)gk(B) = 0 (9) We acknowledge that these are strong requirements, and whenever fi and gi are such that we will discuss the validity of these and other assumptions
Efi(A) = 0 , Egi(B) = 0 . (10) in sec.5. Notice that when k = 1, this is equivalent to the defini- 4.1. Spectral distribution tion of classical independence. Intuitively, the eigenspaces of two freely independent matrices are in “generic posi- We now turn our attention to the spectral distribution of H tion” (Speicher, 2009), i.e. they are not aligned in any spe- and how that distribution evolves as the energy changes. cial way. Before we can describe how to compute the spec- For this purpose, only the relative scaling between H0 and trum of A + B, we must first introduce two new concepts, H σ = 1 1 is relevant√ and we may for simplicity take H0 and the Stieltjes transform and the R transform. σH1 = 2. Then we have, √ ρ (λ) = ρ (λ; 1, φ), ρ (λ) = ρ (λ; 2, φ) , 3.2.1. THE STIELTJES TRANSFORM H0 MP H1 SC (15)
For z ∈ C \ R the Stieltjes transform G of a probability which can be integrated using eqn. (11) to obtain GH0 and distribution ρ is defined as, GH1 . Solving eqn. (13) for the R transforms then gives, Z ρ(t) 1 G(z) = dt , (11) R (z) = , R (z) = 2z . (16) z − t H0 1 − zφ H1 R Geometry of Neural Networks
1.0 1.0 1.4 ϵ= 0.0 0.8 0.8 1.2 ϵ= 0.1 1.0 ϵ= 0.5 0.6 0.6 0.8 ϵ= 1.0 ρ(λ) 0.4 0.4 0.6 0.4 0.2 0.2 0.2
-2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 5 λ λ λ (a) φ = 1/4 (b) φ = 1/2 (c) φ = 3/4
Figure 1. Spectral distributions of the Wishart + Wigner approximation of the Hessian for three different ratios of parameters to data points, φ. As the energy of the critical point increases, the spectrum becomes more semicircular and negative eigenvalues emerge.
We proceed by computing the R transform of H, is nontrivial. We discuss a method for doing so in the sup- plementary material. The full result is too long to present 1 R = R + R = + 2z , (17) here, but we find that for small α, H H0 H1 1 − zφ 3/2 so that, using eqn. (13), we find that its Stieltjes transform − c α(, φ) ≈ α0(φ) , (20) GH solves the following cubic equation, c 3 2 2φ GH − (2 + zφ)GH + (z + φ − 1)GH − 1 = 0 . (18) where the critical value of ,
The correct root of this equation is determined by the 1 2 3/2 c = (1 − 20φ − 8φ + (1 + 8φ) ) , (21) asymptotic behavior GH ∼ 1/z as z → ∞ (Tao, 2012). 16 From this root, the spectral density can be derived through is the value of the energy below which all critical points are the Stieltjes inversion formula, eqn. (12). The result is illus- minimizers. We note that c can be found in a simpler way: trated in fig.1. For small , the spectral density resembles it is the value of below which eqn. (18) has only real roots the Marchenko-Pastur distribution, and for small enough at z = 0, i.e. it is the value of for which the discriminant φ there is an eigenvalue gap. As increases past a crit- of the polynomial in eqn. (18) vanishes at z = 0. Also, we ical value , the eigenvalue gap disappears and negative c observe that c vanishes cubically as φ approaches 1, eigenvalues begin to appear. As gets large, the spectrum 2 becomes semicircular. ≈ (1 − φ)3 + O(1 − φ)4 . (22) c 27
4.2. Normalized index The 3/2 scaling in eqn. (20) is the same behavior that was Because it measures the number of descent directions3, found in (Bray & Dean, 2007) in the context of the field one quantity that is of importance in optimization and in theory of Gaussian random functions. As we will see later, the analysis of critical points is the fraction of negative the 3/2 scaling persists in a more refined version of this cal- eigenvalues of the Hessian, or the index of a critical point, culation and in numerical simulations. α. Prior work (Dauphin et al., 2014; Choromanska et al., 2015) has observed that the index of critical points typi- 5. Analysis of Assumptions cally grows rapidly with energy, so that critical points with many descent directions have large loss values. The precise 5.1. Universality form of this relationship is important for characterizing the There is a wealth of literature establishing that many prop- geometry of loss surfaces, and in our framework it can be erties of large random matrices do not depend on the details readily computed from the spectral density via integration, of how their entries are distributed, i.e. many results are Z 0 Z ∞ universal. For instance, Tao & Vu(2012) show that the α(, φ) ≡ ρ(λ; , φ) dλ = 1 − ρ(λ; , φ) dλ . spectrum of Wishart matrices asymptotically approaches −∞ 0 (19) the Marcenko-Pastur law regardless of the distribution of Given that the spectral density is defined implicitly through the individual entries, so long as they are independent, have k k > 2 equation eqn. (18), performing this integration analytically mean zero, unit variance, and finite -th moment for . Analogous results can be found for Wigner matrices (Ag- 3Here we ignore degenerate critical points for simplicity. garwal). Although we are unaware of any existing analy- Geometry of Neural Networks
5.2.2. RESIDUALSAREI.I.D. RANDOMNORMAL 1.5 H=H +H 0 1 First, we note that e ∼ N (0, 2) is consistent with the H=H +QH QT iµ 0 1 definition of the energy in eqn. (2). Furthermore, because
1.0 the gradient of the loss is proportional to the residuals, it vanishes in expectation (i.e. as m → ∞), which specializes our analysis to critical points. So this assumptions seems ρ ( λ ) 0.5 necessary for our analysis. It is also consistent with the pri- ors leading to the choice of the squared error loss function. Altogether we believe this assumption is fairly mild.
2 4 6 8 5.2.3. DATA ARE I.I.D. RANDOMNORMAL λ This assumption is almost never strictly satisfied, but it is Figure 2. Testing the validity of the free independence assump- approximately enforced by common preprocessing meth- tion by comparing the eigenvalue distribution of H0 + H1 (in T ods, such as whitening and random projections. blue) and H0 + QH1Q (in orange) for randomly generated or- thogonal Q. The discrepancies are small, providing support for the assumption. Data is for a trained single-hidden-layer ReLU 5.2.4. WEIGHTSAREI.I.D. RANDOMNORMAL autoencoding network with 20 hidden units and no biases on 150 Although the i.i.d. assumption is clearly violated for a net- 4 × 4 downsampled, grayscaled, whitened CIFAR-10 images. work that has learned any useful information, the weight distributions of trained networks often appear random, and sometimes appear normal (see, e.g., fig. S1 of the supple- ses relevant for the specific matrices studied here, we be- mentary material). lieve that our conclusions are likely to exhibit some univer- sal properties – for example, as in the Wishart and Wigner 5.3. Secondary assumptions cases, normality is probably not necessary. In sec.4 we introduced two assumptions we dubbed sec- On the other hand, most universality results and the tools ondary, and we discuss their validity here. we are using from random matrix theory are only exact in the limit of infinite matrix size. Finite-size corrections 5.3.1. J AND H1 AREI.I.D. RANDOMNORMAL are actually largest for small matrices, which (counter- intuitively) means that the most conservative setting in Given that J and H1 are constructed from the residuals, the which to test our results is for small networks. So we data, and the weights – variables that we assume are i.i.d. will investigate our assumptions in this regime. We expect random normal – it is not unreasonable to suppose that their agreement with theory only to improve for larger systems. entries satisfy the universality conditions mentioned above. In fact, if this were the case, it would go a long way to vali- 5.2. Primary assumptions date the approximations made in sec.4. As it turns out, the situation is more complicated: both J and H1 have sub- In sec.2 we introduced a number of assumptions we structure which violates the independence requirement for dubbed primary and we now discuss their validity. the universality results to apply. We must understand and take into account this substructure if we wish to improve 5.2.1. FREEINDEPENDENCEOF H0 AND H1 our approximation further. We examine one way of doing so in the next section. Our use of free probability relies on the free independence of H0 and H1. Generically we may expect some alignment between the eigenspaces of H0 and H1 so that free inde- 6. Beyond the Simple Case pendence is violated; however, we find that this violation In sec.4 we illustrated our techniques by examining the is often quite small in practice. To perform this analysis, simple case of Wishart plus Wigner matrices. This analysis it suffices to examine the discrepancy between the distribu- shed light on several qualitative properties of the spectrum tion of H + H and that of H + QH QT , where Q is an 0 1 0 1 of the Hessian matrix, but, as discussed in the previous sec- orthogonal random matrix of Haar measure (Chen et al., tion, some of the assumptions were unrealistic. We believe 2016). Fig.2 show minimal discrepancy for a network that it is possible to relax many of these assumptions to pro- trained on autoencoding task; see the supplementary ma- duce results more representative of practical networks. In terial for further details and additional experiments. More this section, we take one step in that direction and focus on precise methods for quantifying partial freeness exist (Chen the specific case of a single-hidden-layer ReLU network. et al., 2016), but we leave this analysis for future work. Geometry of Neural Networks
Probability density H1 over 10 runs, output + noise Probability density H1 over 10 runs, output + noise phi value = 0.0625 phi value = 0.25 6.1. Product Wishart distribution and H1 0.30 0.20
0.25 In the single-hidden-layer ReLU case, we can write H1 as, 0.15 0.20 0 H12 0.15 0.10 H1 = T , H12 0 0.10 0.05 0.05 where its off-diagonal blocks are given by the matrix H12, 0.00 0.00 −15 −10 −5 0 5 10 15 −15 −10 −5 0 5 10 15 Eigenvalue Eigenvalue 1 X ∂ ∂yˆiµ [H ] = e 12 ab;cd m iµ (2) (1) iµ ∂Wcd ∂Wab Figure 3. Spectrum of H1 at initialization, using 4 × 4 downsam- m (23) 1 X pled, grayscaled, whitened CIFAR-10 in a single layer ReLU au- = e δ θ(z(1))x . m cµ ad aµ bµ toencoder with 16 hidden units. Error signals have been replaced µ=1 by noise distributed as N (0, 100), i.e. = 50. Theoretical pre- diction (red) matches well. Left φ = 1/16, right φ = 1/4. Here it is understood that the matrix H12 is obtained from the tensor [H12]ab;cd by independently vectorizing the first two and last two dimensions, δad is the Kronecker delta the cavity method (Dupic & Castillo, 2014). As the specific function, and θ is the Heaviside theta function. As dis- form of the result is not particularly enlightening, we defer cussed above, we take the error to be normally distributed, its presentation to the supplementary material. The result X (2) (1) may also be derived using free probability theory (Muller, ecµ = Wcj [zjµ ]+ − ycµ(x) ∼ N (0, 2) . (24) 2002; Burda et al., 2010). From either perspective it is pos- j sible to derive the the R transform, which reads, We are interested in the situation in which the layer width φz n and the number of examples m are large, in which case RH1 (z) = 2 2 . (28) (1) 2 − φ z θ(zaµ ) can be interpreted as a mask that eliminates half of the terms in the sum over µ. If we reorder the indices so See fig.3 for a comparison of our theoretical prediction for that the surviving ones appear first, we can write, the spectral density to numerical simulations4.
m/2 1 X 1 6.2. H0 and the Wishart assumption H ≈ δ e x = I ⊗ eˆxˆT , 12 m ad cµˆ bµˆ m n µˆ=1 Unlike the situation for H1, to the best of our knowledge the limiting spectral density of H0 cannot easily be m where we have written eˆ and xˆ to denote the n× 2 matrices deduced from known results in the literature. In principle derived from e and x by removing the vanishing half of it can be computed using diagrammatics and the method their columns. of moments (Feinberg & Zee, 1997; Tao, 2012), but this approach is complicated by serious technical difficulties Owing to the block off-diagonal structure of H1, the squares of its eigenvalues are determined by the eigenval- – for example, the matrix has both block structure and ues of the product of the blocks, Kronecker structure, and there are nonlinear functions applied to the elements. Nevertheless, we may make T 1 T T T some progress by focusing our attention on the smallest H12H12 = 2 In ⊗ (ˆexˆ )(ˆexˆ ) . (25) m eigenvalues of H0, which we expect to be the most relevant for computing the smallest eigenvalues of H. The Kronecker product gives an n-fold degeneracy to each eigenvalue, but the spectral density is the same as that of The empirical spectral density of H0 for an autoen- 1 coding network is shown in fig.4. At first glance, this M ≡ (ˆexˆT )(ˆexˆT )T . (26) m2 distribution does not closely resemble the Marchenko- Pastur distribution (see, e.g., the = 0 curve in fig.1) It follows that the spectral density of H1 is related to that owing to its heavy tail and small eigenvalue gap. On the of M, other hand, we are not concerned with the large eigen- 2 values, and even though the gap is small, its scaling with ρH1 (λ) = |λ|ρM (λ ) . (27) φ appears to be well-approximated by λ− from eqn. (8)
Notice that M is a Wishart matrix where each factor is itself 4As the derivation requires that the error signals are random, in a product of real Gaussian random matrices. The spectral the simulations we manually overwrite the network’s error signals density for matrices of this type has been computed using with random noise. Geometry of Neural Networks
0.045 0.016 40 0.040 0.014 35 0.035
0.030 0.012 30
Value 0.025 0.010 25 0.020 0.008 20 0.015
0.010 0.006 Histogram 15 0 5 10 15 20 25 30 35 k kth smallest non-zero eigenvalue Gap in spectrum 0.004 10
0.002 5
0.000 0 0 1 2 3 4 5 6 7 8 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Value of eigenvalue
Figure 5. Evolution of the H0 spectral gap as a function of 1/φ, Figure 4. The spectrum of H0 at initialization of a single layer ReLU autoencoder with 16 hidden units and 256 4 × 4 down- comparing the Marcenko-Pastur (red) against empirical results for sampled, grayscaled, whitened CIFAR-10 images. There are 16 a 16-hidden unit single-layer ReLU autoencoder at initialization null directions, and the corresponding zero eigenvalues have been (black dots, each averaged over 30 independent runs). Dataset removed. The inset shows the values of the smallest 35 nonzero was taken from 4×4 downsampled, grayscaled, whitened CIFAR- eigenvalues. The positive value of the first datapoint reflects the 10 images. A single fit parameter, characterizing the variance of existence of a nonzero gap. the matrix elements in the Wishart ensemble, is used. for appropriate σ. See fig.5. This observation suggests Despite the apparent pole at φ = 1, c actually vanishes that as a first approximation, it is sensible to continue to there, 8 represent the limiting spectral distribution of H with the ≈ σ2(1 − φ)3 + O(1 − φ)4 . (33) 0 c 27 Marchenko-Pastur distribution. Curiously, for σ = 1/2, this is precisely the same behavior 6.3. Improved approximation to the Hessian we found for the behavior of c near 1 in the Wishart plus Wigner approximation in sec.4. This observation, The above analysis motivates us to propose an improved combined with the fact that the 3/2 scaling in eqn. (31) is R (z) approximation to H , also what we found in sec.4, suggest that it is H0, rather than H , that is governing the behavior near . σ φz 1 c R (z) = + , (29) H 1 − σzφ 2 − φ2z2 where the first term is the R transform of the Marchenko- 6.4. Empirical distribution of critical points Pastur distribution with the σ parameter restored. As be- We conduct large-scale experiments to examine the distri- fore, an algebraic equation defining the Stieltjes transform bution of critical points and compare with our theoretical of ρH can be deduced from this expression, predictions. Uniformly sampling critical points of varying energy is a difficult problem. Instead, we take more of a 2 = 2 z − σ(1 − φ)G − φ 2σz + (1 − φ)G2 H (30) brute force approach: for each possible value of the index, 2 3 3 4 − φ z + σ(φ − 2) GH + zσφ GH . we aim to collect many measurements of the energy and compute the mean value. Because we cannot control the Using the same techniques as in sec.4, we can use this index of the obtained critical point, we run a very large equation to obtain the function α(, φ). We again find the number of experiments (∼50k) in order to obtain sufficient same 3/2 scaling, i.e., data for each value of α. This procedure appears to be a fairly robust method for inferring the α() curve. 3/2 − c α(, φ) ≈ α˜0(φ) . (31) c We adopt the following heuristic for finding critical points. First we optimize the network with standard gradi- Compared with eqn. (20), the coefficient function α˜0(φ) ent descent until the loss reaches a random value between differs from α0(φ) and, 0 and the initial loss. From that point on, we switch 2 to minimizing a new objective, Jg = |∇θL| , which, 2 2 3/2 σ 27 − 18χ − χ + 8χ 2 unlike the primary objective, is attracted to saddle points. c = , χ = 1+16φ−8φ . 32φ(1 − φ)3 Gradient descent on Jg only requires the computation of (32) Hessian-vector products and can be executed efficiently. Geometry of Neural Networks
0.5 0.30 0.08 1 (1-ϕ) 0.06 2 1 0.25 0.4 (1-ϕ)2 α 0.04 2
0.02 0.20 ϕ =2/3
0.00 0.3 0.36 0.38 0.40 0.42 0.44 0.46 ϕ =1/2 ϵ 0.15 ϕ =1/3 ϵc α ϕ =1/4 Benefit of nonlinearity ϕ =1/8 0.2 0.10 ϕ =1/16 0.1 0.05
0.0 0.00 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.2 0.4 0.6 0.8 1.0 ϵ ϕ (a) Index of critical points versus energy (b) Energy of minimizers versus parameters/data points
Figure 6. Empirical observations of the distribution of critical points in single-hidden-layer tanh networks with varying ratios of param- eters to data points, φ. (a) Each point represents the mean energy of critical points with index α, averaged over ∼200 training runs. Solid 3/2 lines are best fit curves for small α ≈ α0| − c| . The good agreement (emphasized in the inset, which shows the behavior for small 3 α) provides support for our theoretical prediction of the /2 scaling. (b) The best fit value of c from (a) versus φ. A surprisingly good fit 1 2 1 is obtained with c = 2 (1 − φ) . Linear networks obey c = 2 (1 − φ). The difference between the curves shows the benefit obtained from using a nonlinear activation function.
−6 We discard any run for which the final Jg > 10 ; 7. Conclusions otherwise we record the final energy and index. We introduced a new analytical framework for studying We consider relatively small networks and datasets in the Hessian matrix of neural networks based on free order to run a large number of experiments. We train probability and random matrix theory. By decomposing single-hidden-layer tanh networks of size n = 16, which the Hessian into two pieces H = H0 + H1 one can also equals the input and output dimensionality. For each systematically study the behavior of the spectrum and training run, the data and targets are randomly sampled associated quantities as a function of the energy of a from standard normal distributions, which makes this critical point. The approximations invoked are on H0 and a kind of memorization task. The results are summa- H1 separately, which enables the analysis to move beyond rized in fig.6. We observe that for small α, the scaling the simple representation of the Hessian as a member of 3/2 the Gaussian Orthogonal Ensemble of random matrices. α ≈ | − c| is a good approximation, especially for smaller φ. This agreement with our theoretical predictions provides support for our analytical framework and for the We derived explicit predictions for the spectrum and validity of our assumptions. the index under a set of simplifying assumptions. We found empirical evidence in support of our prediction 3/2 As a byproduct of our experiments, we observe that the that that small α ≈ | − c| , raising the question of energy of minimizers is well described by a simple func- how universal the 3/2 scaling may be, especially given the 1 2 tion, c = 2 (1 − φ) . Curiously, a similar functional form results of (Bray & Dean, 2007). We also showed how was derived for linear networks (Advani & Ganguli, 2016), some of our assumptions can be relaxed at the expense of 1 c = 2 (1 − φ). In both cases, the value at φ = 0 and reduced generality of network architecture and increased φ = 1 is understood simply: at φ = 0, the network has technical calculations. An interesting result of our numeri- zero effective capacity and the variance of the target distri- cal simulations of a memorization task is that the energy of bution determines the loss; at φ = 1, the matrix of hidden minimizers appears to be well-approximated by a simple units is no longer rank constrained and can store the entire function of φ. We leave the explanation of this observation input-output map. For intermediate values of φ, the fact as an open problem for future work. that the exponent of (1 − φ) is larger for tanh networks than for linear networks is the mathematical manifestation Acknowledgements of the nonlinear network’s better performance for the same number of parameters. Inspired by these observations and We thank Dylan J. Foster, Justin Gilmer, Daniel Holtmann- by the analysis of Zhang et al.(2016), we speculate that Rice, Ben Poole, Maithra Raghu, Samuel S. Schoenholz, this result may have a simple information-theoretic expla- Jascha Sohl-Dickstein, and Felix X. Yu for useful com- nation, but we leave a quantitative analysis to future work. ments and discussions. Geometry of Neural Networks
References Keskar, Nitish Shirish, Mudigere, Dheevatsa, Nocedal, Jorge, Smelyanskiy, Mikhail, and Tang, Ping Tak Pe- Advani, Madhu and Ganguli, Surya. Statistical mechanics ter. On large-batch training for deep learning: Gen- of optimal convex inference in high dimensions. Physi- eralization gap and sharp minima. ICLR 2017. URL cal Review X, 6(3):031034, 2016. http://arxiv.org/abs/1609.04836.
Aggarwal, Amol. Bulk universality for generalized Laisant, C-A. Integration´ des fonctions inverses. Nou- wigner matrices with few moments. arXiv preprint velles annales de mathematiques,´ journal des candidats arXiv:1612.00421. aux ecoles´ polytechnique et normale, 5:253–257, 1905. Bray, Alan J. and Dean, David S. Statistics of critical Marcenko,ˇ Vladimir A and Pastur, Leonid Andreevich. points of gaussian fields on large-dimensional spaces. Distribution of eigenvalues for some sets of random ma- Phys. Rev. Lett., 98:150201, Apr 2007. doi: 10.1103/ trices. Mathematics of the USSR-Sbornik, 1(4):457, PhysRevLett.98.150201. URL http://link.aps. 1967. org/doi/10.1103/PhysRevLett.98.150201. Muller, Ralf R. On the asymptotic eigenvalue distribution Burda, Z, Jarosz, A, Livan, G, Nowak, MA, and Swiech, of concatenated vector-valued fading channels. IEEE A. Eigenvalues and singular values of products of rect- Transactions on Information Theory, 48(7):2086–2091, angular gaussian random matrices. Physical Review E, 2002. 82(6):061114, 2010. Neyshabur, Behnam, Salakhutdinov, Ruslan, and Srebro, Nathan. Path-sgd: Path-normalized optimization in deep Chen, Jiahao, Van Voorhis, Troy, and Edelman, Alan. neural networks. pp. 2413–2421, 2015. Partial freeness of random matrices. arXiv preprint arXiv:1204.2257, 2016. Safran, Itay and Shamir, Ohad. On the quality of the initial basin in overspecified neural networks. JMLR, 48, 2016. Choromanska, Anna, Henaff, Mikael, Mathieu, Michael,¨ URL http://arxiv.org/abs/1511.04210. Arous, Gerard´ Ben, and LeCun, Yann. The loss surface of multilayer networks. JMLR, 38, 2015. Saxe, Andrew M., McClelland, James L., and Ganguli, Surya. Exact solutions to the nonlinear dynamics of Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, learning in deep linear neural networks. ICLR. Cho, Kyunghyun, Ganguli, Surya, and Bengio, Yoshua. Identifying and attacking the saddle point problem in Speicher, Roland. Free probability theory. arXiv preprint high-dimensional non-convex optimization. Advances arXiv:0911.0087, 2009. in Neural Information Processing Systems 27, pp. 2933– Tao, T. and Vu, V. Random covariance matrices: universal- 2941, 2014. ity of local statistics of eigenvalues. Annals of Probabil- ity, 40(3):1285–1315, 2012. Dupic, Thomas and Castillo, Isaac Perez.´ Spectral density of products of wishart dilute random matrices. part i: the Tao, Terence. Topics in random matrix theory, volume 132. dense case. arXiv preprint arXiv:1401.7802, 2014. American Mathematical Society Providence, RI, 2012.
Feinberg, Joshua and Zee, Anthony. Renormalizing rectan- Wigner, Eugene P. Characteristic vectors of bordered ma- gles and other topics in random matrix theory. Journal trices with infinite dimensions. Annals of Mathematics, of statistical physics, 87(3-4):473–504, 1997. pp. 548–564, 1955.
Freeman, C. Daniel and Bruna, Joan. Topology and geom- Zhang, Chiyuan, Bengio, Samy, Hardt, Moritz, Recht, Ben- etry of half-rectified network optimization. 2016. URL jamin, and Vinyals, Oriol. Understanding deep learn- http://arxiv.org/abs/1611.01540. ing requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. Goodfellow, Ian J., Vinyals, Oriol, and Saxe, Andrew M. Qualitatively characterizing neural network optimization problems. ICLR 2015. URL http://arxiv.org/ abs/1412.6544.
Kawaguchi, Kenji. Deep learning without poor local min- ima. Advances in Neural Information Processing Sys- tems 29, pp. 586–594, 2016.