Random Matrix Improved Covariance Estimation for a Large Class of Metrics Malik Tiomoko, Florent Bouchard, Guillaume Ginolhac, Romain Couillet
To cite this version:
Malik Tiomoko, Florent Bouchard, Guillaume Ginolhac, Romain Couillet. Random Matrix Improved Covariance Estimation for a Large Class of Metrics. ICML 2019 - 36th International Conference on Machine Learning, Jun 2019, Long Beach, United States. hal-02152121
HAL Id: hal-02152121 https://hal.archives-ouvertes.fr/hal-02152121 Submitted on 19 May 2020
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Random Matrix Improved Covariance Estimation for a Large Class of Metrics
Malik Tiomoko 1 2 Florent Bouchard 2 Guillaume Ginholac 3 Romain Couillet 2 1
Abstract constraints with LASSO and graphical LASSO-based ap- Relying on recent advances in statistical estima- proaches (Friedman et al., 2008) or, more interestingly for tion of covariance distances based on random the present article, on exploiting the statistical indepen- matrix theory, this article proposes an improved dence in the entries of the vectors xi. covariance and precision matrix estimation for a (Ledoit & Wolf, 2004) proposes to linearly “shrink” Cˆ as p 2 wide family of metrics. The method is shown Cˆ(ρ) ≡ ρIp + 1 − ρ Cˆ for ρ > 0 chosen to minimize to largely outperform the sample covariance ma- the expected Frobenius distance E[kC − Cˆ(ρ)kF ] in the trix estimate and to compete with state-of-the- asymptotic p, n → ∞ limit with p/n → c > 0. Basic art methods, while at the same time being com- results from random matrix theory (RMT) are used here putationally simpler. Applications to linear and to estimate ρ consistently. This procedure is simple and quadratic discriminant analyses also demonstrate quite flexible and has been generalized in various directions significant gains, therefore suggesting practical (e.g., in (Couillet & McKay, 2014) with a robust statistics interest to statistical machine learning. approach). However, the method only applies a naive ho- mothetic map to each λi(Cˆ) of Cˆ in order to better esti- mate λ (C). A strong hope to recover a better approxi- 1. Introduction i mation of the λi(C)’s then arose from (Silverstein & Bai, Covariance and precision matrix estimation is a fundamen- 1995; Silverstein & Choi, 1995) that provide a random ma- tal and simply posed, yet still largely considered, key prob- trix result relating directly the limiting eigenvalue distri- ˆ lems of statistical data analysis, with countless applications butions of C and C. Unfortunately, while estimating the ˆ in statistical inference. In machine learning, it is notably λi(C)’s from the λi(C)’s is somewhat immediate, estimat- ˆ at the core of elementary methods as linear (LDA) and ing the λi(C)’s backward from the λi(C)’s is a difficult quadratic discriminant analysis (QDA) (McLachlan, 2004). task. (El Karoui et al., 2008) first proposed an optimiza- tion algorithm to numerically solve this problem, however p×p Estimation of the covariance matrix C ∈ R based on with little success as the method is quite unstable and has p n independent (say zero mean) samples x1, . . . , xn ∈ R rarely been efficiently reproduced. (Mestre, 2008) later of- is conventionally performed using the sample covariance n fered a powerful idea, based on contour integral, to consis- ˆ 1 P T 1 n matrix (SCM) C ≡ xixi (and its inverse using P n i=1 tently estimate linear functionals n i=1 f(λi(C)) from Cˆ−1 n p ). The estimate is however only consistent for the λi(Cˆ)’s. But f is constrained to be very smooth (com- and only invertible for n ≥ p. Treating the important prac- plex analytic) which prevents the estimation of the individ- n ∼ p n p tical cases where and even has recently ual λi(C)’s. Recently, Ledoit and Wolf took over the work spurred a series of parallel lines of research. These direc- of El Karoui, which they engineered to obtain a more ef- tions rely either on structural constraints, such as “toeplitz- ficient numerical method, named QuEST (Ledoit & Wolf, ification” procedures for Toeplitz covariance models (par- 2015). Rather than inverting the Bai–Silverstein equations, ticularly convenient for time series) (Bickel et al., 2008; Wu the authors also proposed, with the same approach, to esti- & Pourahmadi, 2009; Vinogradova et al., 2015), on sparse mate the λi(C)’s by minimizing a Frobenius norm distance 1CentraleSupelec,´ University ParisSaclay, France 2GIPSA- (Ledoit & Wolf, 2015) (named QuEST1 in the present arti- lab, University Grenoble-Alpes, France 3LISTIC, University cle) or a Stein loss (Ledoit et al., 2018) (QuEST2 here). Savoie Mont-Blanc, France. Correspondence to: Malik Tiomoko
Divergences f(z) f(z) G(z) 2 2 2 2 dR log (z) log (z) z log (z) − 2 log(z) + 2 2 1 1 1 dB − 4 log(z) + 2 log(1 + z) − 2 log(2) log(z) −z log(z) + z 1 1 1 s+z δKL 2 z − 2 log(z) − 2 log(1 + sz) s log(s + z) + z log z −1 1 δαR 2(α−1) log(α + (1 − α)z) + 2 log(z) z log(z)
Table 1. Distances d and divergences δ, and their corresponding f(z) F (z) 2 2 f(z) functions. log (z) z log (z) − 2 log(z) + 2 log(z) z log(z) − z 1 log(1 + sz) s + z log(1 + sz) − z for Γ ⊂ C a (positively oriented) contour surrounding the z 1 z2 −1 2 eigenvalues of M C. The notation mνp (z; M) reminds the dependence of mνp (z) in the matrix M. Table 2. Values of G(z) and F (z) for “atomic” f(z) functions used in most distances and divergences under study; here s > 0 Then, by exploiting the relation between the Stieltjes trans- and z ∈ C. forms µνp and mµp , it is shown in (Couillet et al., 2018) that, under Assumption 1, for all deterministic M of bounded operator norm, direction of ξ ∈ Sn (the Riemannian manifold of symmet- ric n × n matrices), is given by (Absil et al., 2009) δ(M,C) − δˆ(M,X) → 0 (3) ++ Sn DhX (M)[ξ] = h∇hX (M), ξiM almost surely, where X = [x1, . . . , xn] and I S++ ˆ 1 where h·, ·i. n is the Riemannian metric defined through δ(M,X) ≡ G −mµ˜p (z; M) dz (4) 2πıc ˆ Γ ++ Sn −1 −1 hη, ξiM = tr M ηM ξ . with G such that G0(z) ≡ g(z) = f(1/z), Γˆ a contour surrounding the support of the almost sure limiting eigen- −1 ˆ p p Differentiating δˆ2(M,X) at M in the direction ξ yields: value distribution of M C and µ˜p = n µp + (1 − n )δ0 p (and thus mµ˜p (z) = cmµp (z) + (1 − n )/z). Note that, by the linearity of G in (4), it is sufficient in practice to eval- DhX (M)[ξ] uate δˆ(M,X) for elementary functions (such as f(z) = z, −δˆ(M,X) I = g(−m (z, M))Dm (z, M)[ξ]dz. f(z) = log(z), etc.) in order to cover most distances and µ˜p µ˜p πic Γ metrics of interest (see again Table 1). Table 2 reports the values of G for such atomic functions f. By using the fact that Our main idea is to estimate C by minimizing the approx- Dmµ˜ (z, M)[ξ] imation δˆ(M,X) of δ(M,C) over M. However, it is im- p −1 c h −1 i portant to note that, as discussed in (Couillet et al., 2018), = Dtr M Cˆ − zIp [ξ] the random quantity δˆ(M,X) may be negative with non- p zero probability. As such, minimizing δˆ(M,X) over M c h i−2 = tr M −1Cˆ M −1Cˆ − zI M −1ξ may lead to negative solutions. Our proposed estimation p p method therefore consists in approximating C by the solu- ++ −2 Sn c h −1 i tion to the optimization problem = sym Cˆ M Cˆ − zIp , ξ p M ˆ 2 argminM 0{hX (M)}, where hX (M) ≡ (δ(M,X)) . 1 T (5) where sym(A) = 2 (A + A ) is the symmetric part of A ∈ p×p R , we retrieve the gradient of hX (M) as 3. Methodology and Main Results ∇h (M) − ıπp X 3.1. Estimation Method δˆ(M,X) I We solve (5) via a gradient descent algorithm in the Rie- ˆ −1 ˆ −2 = g −mµ˜p (z; M) sym C(M C − zIp) dz mannian manifold S++ of positive definite n × n matrices. Γˆ n (6) ++ To evaluate the gradient ∇hX (M) of hX at M ∈ Sn , ++ recall that on Sn the differential DhX (M)[ξ] of the func- (recall that the right-hand side still depends on X implicitly ++ + ++ ˆ tional hX : Sn → R , at position M ∈ Sn and in the through µ˜p and C). Random matrix improved covariance estimation
++ Once ∇hX estimated, every gradient descent step in Sn Remark 1 is in fact not surprising. Indeed, finding the min- corresponds to a small displacement on the geodesic start- imum of δ(M,C) over M 0 would result in finding C, ing at M and towards −∇hX (M), defined as the curve which cannot be achieved for unconstrained matrices C and for non vanishing values of p/n. Figure 1 provides a typ- ++ R+ → Sn ical evolution of the distance δ(Mk,C) versus its approx- 1 1 1 1 ˆ − − imation δ(Mk,X) at the successive steps k = 1, 2,... of t 7→ M 2 exp −tM 2 ∇hX (M)M 2 M 2 Algorithm 1, initialized at M0 = Ip. As expected, the dif- ˆ where, for A = UΛU T ∈ S++ in its spectral decompo- ference |δ(Mk,X) − δ(Mk,C)|, initially small (at k = 1, n ˆ sition, exp(A) ≡ U exp(Λ)U T (with exp understood here δ(Ip,X) ' δ(Ip,C)), increases with k, until the gradient applied entry-wise on the diagonal elements of Λ). vanishes and the divergence δ(Mk,C) converges.
That is, letting M0,M1,... and t0, t1,... be the successive 3.2. Practical Implementation iterates and step sizes of the gradient descent, we have, for ++ some given initialization M0 ∈ Sn , In order to best capture the essence of Algorithm 1, as well as its various directions of simplification and practical fast 1 − 1 − 1 1 2 2 2 2 implementation, a set of important remarks are in order. Mk+1 = Mk exp −tkMk ∇hX (Mk)Mk Mk . (7) First note that the generic computation of Mk+1 in Equa- tion (7) may be numerically expensive, unless M and Our proposed method is summarized as Algorithm 1. k ∇hX (M) share the same eigenvectors. In this case, let- ting M = UΩ U T and ∇ h(M ) = U∆ U T, we have Algorithm 1 Proposed estimation algorithm. k k X k k ++ the recursion Require M0 ∈ Cn . 1 1 1 1 − − [δ ] Repeat M ← M 2 exp −tM 2 ∇hX (M)M 2 M 2 k i [ωk+1]i = [ωk]i exp −t (8) with t either fixed or optimized by backtracking line search. [ωk]i Until Convergence. where δk = diag(∆k) and ωk = diag(Ωk). Return M. √ 2 In particular, if M0 = αIn + 1 − α Cˆ is a linear shrink- We conclude this section by an important remark on the age for some α ∈ [0, 1], we immediately find that, for all fundamental limitations of the proposed algorithm. k ≥ 0, Remark 1 (Approximation of δ(M ,C) by δˆ(M ,X)). It k k ˆ ˆ T is fundamental to understand the result from (Couillet et al., Mk = UΩkU 2018) at the heart of the proposed method. There, it is pre- cisely shown that, for every deterministic sequence of ma- where Uˆ ∈ Rp×p are the eigenvectors of Cˆ (i.e., in its spec- (p) (p) T trices {M , p = 1, 2,...} and {C , p = 1, 2,...}, tral decomposition, Cˆ = UˆΛˆUˆ ) and Ωk is recursively de- with M (p),C(p) ∈ Rp×p and max(kC(p)k, kM (p)k) < K fined through (8). for some constant K independent of p, we have that, for (p) (p) (p) (p) (p) 1 (p) (p) This shows that, initialized as such, the ultimate limiting X = [x , . . . , xn ] with x = C 2 z and z 1 i i i estimate M∞ (i.e., the limit of Mk) of C shares the same i.i.d. vectors of i.i.d. zero mean and unit variance entries, eigenvectors as Cˆ, and thus reduces to a “non-linear shrink- age” procedure, similar to that of (Ledoit & Wolf, 2015). δ(M (p),C(p)) − δˆ(M (p),X(p)) → 0 Extensive simulations in fact suggest that, if initialized ran- almost surely as n, p → ∞ and p/n → c ∈ (0, 1). This domly (say with M0 a random Wishart matrix), after a few ˆ ˆ iterates, the eigenvectors of Mk do converge to those of C result seems to suggest that δ(Mk,X) in our algorithm is (see further discussions in Section 4). As such, for com- a good approximation for the sought for δ(Mk,C). This however only holds true so long that M is independent of putational ease, we suggest to initialize the algorithm with k ˆ X which clearly does not stand when proceeding to suc- M0 = Ip or with M0 a linear shrinkage of C. cessive gradient descent steps in the direction of ∇hX (M) which depends explicitly on X. As such, while initializa- A further direction of simplification of Algorithm 1 relates tions with, say, M0 = Ip, allow for a close approximation to the fact that, for generic values of M0 (notably hav- of δ(Mk,C) in the very first steps of the descent, for larger ing eigenvectors different from those of Cˆ), Equation (7) values of k, the descent is likely to drive the optimization is computationally expensive to evaluate. A second-order in less accurate directions. simplification for small t is often used in practice (Jeuris Random matrix improved covariance estimation
0.15 2 0.4 ˆ ˆ ˆ δ(Mk,X) δ(Mk,C) − δ(Mk,X) able definitions of the function hX (M) = δ(M,X) and δ(Mk,C) its resulting gradient. Yet, because of the integral form as- 0.3 0.1 sumed by the gradient (Equation (6)), a possibly computa-
0.2 tionally involved complex integration needs to be numeri-
0.05 cally performed at each gradient descent step. 0.1 In this section, we specify closed-form expressions for the 0 0 5 10 15 5 10 15 gradient for the atomic f functions of Table 2 (which is algorithm step k algorithm step k enough to cover the list of divergences in Table 1).
Figure 1. (left) Evolution of the Fisher distance δ(Mk,C) versus 3.4.1. ESTIMATION OF C δˆ(Mk,X) for k = 1, 2,..., initialized to M0 = Ip,. (right) Evolution of δ(Mk,C) − δˆ(Mk,X). Let us denote ˆ −1 ∇hX (M) ≡ 2δ(M,X) · sym Cˆ · V Λ∇V et al., 2012), as follows −1 ˆ Mk+1 = Mk − t∇hX (Mk) where V are the eigenvectors of M C and we will de- t2 termine Λ∇ for each function f. Again, we recall from the + ∇h (M )M −1∇h (M ) + O(t3). discussion in Section 3.2, that V = Uˆ the eigenvectors of Cˆ 2 X k k X k if M shares the same eigenvectors as Cˆ (which thus avoids Simulations with this approximation suggest almost no dif- −1 ˆ evaluating the eigenvectors V of Mk C at each step k of ference in either the number of steps until convergence or the algorithm). accuracy of the solution. For readability in the following, let us denote λi ≡ −1 −1 λi(M Cˆ), i ∈ {1, . . . , p}, the eigenvalues of the matrix 3.3. Estimation of C −1 M Cˆ and ξ1, . . . , ξp the eigenvalues of In our framework, estimating C−1 rather than C can be −1 1 √ √ T performed by minimizing δ(M,C ) instead of δ(M,C). Λ − λ λ In this case, under Assumption 1, (3) now becomes n
−1 inv T δ(M,C ) − δˆ (M,X) → 0 with Λ = diag(λ1, . . . , λp) and λ = (λ1, . . . , λp) . Fi- nally, for s > 0, let κ ∈ (−1/(s(1 − p/n)), 0) be the almost surely, for every deterministic M of bounded oper- s unique negative number t solution of the equation (see ator norm and X = [x , . . . , x ], where 1 n (Couillet et al., 2018) for details) I inv 1 δˆ (M,X) ≡ F −m inv (z; M) dz µ˜p mµ˜ (t) = −s. 2πıc Γˆ p for F such that F 0(z) ≡ f(z), Γˆ a contour surrounding the With these notations at hand, following the derivations in support of the almost sure limiting eigenvalue distribution (Couillet et al., 2018) (detailed in supplementary material), of MCˆ and µ˜inv = p µinv + (1 − p )δ , where µinv ≡ p n p n 0 p we have the following determinations for Λ . 1 Pp ∇ δ ˆ . The cost function to minimize under this p i=1 λi(MC) Proposition 1 (Case f(t) = t). For f(t) = t, setting is now given by hinv(M) ≡ (δˆinv(M,X))2 with gradient ∇hinv(M) satisfying p X 1 1 X 1 [Λ ] = − + inv ∇ kk 0 2 ∇hX (M) c p m (ξi)(λk − ξi) ıπp i=1 µ˜p δˆinv(M,X) 0 I with mµ˜ the derivative of mµ˜p . −2 p = f −m inv (z; M) sym MCˆ(MCˆ − zI ) M dz. µ˜p p Γˆ Proposition 2 (Case f(t) = log(t)). For f(t) = log(t), With these amendments, Algorithm 1 can be adapted to the −1 estimation of C−1. Table 2 provides the values of F for the [Λ∇]kk = . pλk atomic functions f of interest. Proposition 3 (Case f(t) = log(1 + st)). For s > 0 and 3.4. Application to Explicit Metrics f(t) = log(1 + st), Algorithm 1 is very versatile as it merely consists in a gra- −1 [Λ∇]kk = . dient descent method for various metrics f through adapt- p(λk − κs) Random matrix improved covariance estimation
Proposition 4 (Case f(t) = log2(t)). For f(t) = log2(t), Proposition 8 (Case f(t) = log2(t)). For f(t) = log2(t),
p p p p 2 X λk X λk 2 X 1 X 1 1 [Λinv] = − log (λ ) − − 1 [Λ∇]kk = log (λk) − − ∇ kk k p λ − ξ λ − λ λ p λk − ξi λk − λi i=1 k i i=1 k i k i=1 i=1 i6=k i6=k i6=k p p p p 2 λ log(ξ ) 2 λ log(λ ) 2 2 2 X log(ξi) 2 X log(λi) 2 − 2 log(1 − c) X k i X k i − + − .+ − + − log(1 − c). p λ − ξ p λ − λ pλ p λk − ξi p λk − λi p p i=1 k i i=1 k i k i=1 i=1 i6=k i6=k
These results are mostly achieved by residue calculus 4. Experimental Results for entire analytic functions f or by exploiting more ad- This section introduces experimental results on the direct vanced complex integration methods (in particular branch- application of our proposed method to the estimation of cut methods) for more challenging functions (involving C and C−1 as well as on its use as a plug-in estimator in logarithms in particular). more advanced procedures, here in the scope of linear and Combining these formulas provides an analytical expres- quadratic discriminant analyses (LDA/QDA). sion for the gradient of all aforementioned divergences and square distances, for the estimation of C. 4.1. Validation on synthetic data In this first section, we provide a series of simulations on 3.4.2. ESTIMATION OF C−1 the estimation of C and C−1 based on the Fisher distance Similarly, for the problem of estimating C−1, recalling Re- and for several examples of genuine matrices C. Similar mark 3.3, we may denote results and conclusions were obtained for the other metrics discussed above (the KL divergence and the square Bat- inv ˆinv inv −1 tacharrya distance especially) which are thus not explicitly ∇hX (M) ≡ 2δ (M,X) · sym M · VinvΛ∇ Vinv reported here. The interested reader can refer to the code provided by the authors for self experimentation as supple- with V the eigenvectors of MCˆ. inv mentary material. We redefine in this section λ ≡ λ (MCˆ), and i √ √ i Our preference for the Fisher distance for fair comparisons 1 T ξ1, . . . , ξp the eigenvalues of Λ − n λ λ with Λ = lies in the fact that it is the “natural” Riemannian distance T diag(λ1, . . . , λp) and λ = (λ1, . . . , λp) . Again, for ++ to compare covariance matrices in Sn , therefore in entire s > 0, let κs < 0 be the only negative real number t solu- agreement with the proposed estimation strategy through tion of ++ gradient descents in Sn . Besides, for this specific case, Theorem 4 in (Smith, 2005) establishes an exact and very 1 m inv (t) = − . straightforward formula for the Cramer-Rao bound (CRB) µ˜p s on unbiased estimators of C. Although the compared esti- mators of C are likely all biased and that M initializations With the same approach as in the previous section, we here 0 may by chance bring additional information disrupting a obtain the following values for Λinv. ∇ formally fair CRB comparison, the CRB at least provides Proposition 5 (Case f(t) = t). For f(t) = t, an indicator of relevance of the estimators.
inv 1 − c The examples of covariance matrix C under consideration [Λ∇ ]kk = − . pλk in the following are: (i) [Wishart] a random (p-dimensional) standard Wishart Proposition 6 (Case f(t) = log(t)). For f(t) = log(t), matrix with 2p degrees of freedom, (ii) [Toeplitz a] the Toeplitz matrix defined by Cij = inv |i−j| [Λ∇ ]kk = −1 a , (iii) [Discrete] a matrix C with uniform eigenvector distri- Proposition 7 (Case f(t) = log(1 + st)). For s > 0 and bution and eigenvalues equal to .1, 1, 3, 4 each with multi- f(t) = log(1 + st), plicity p/4.
Figure 2 (for the estimation of C) and Figure 3 (for C−1) inv λk [Λ∇ ]kk = − 1 report comparative performances on the aforementioned C λk − κs Random matrix improved covariance estimation
100.6 100.5 100.4 100 100
100.2 100 10−1 100 10−1
10−0.2
10−0.5 1 1.5 2 2.5 1 1.5 2 2.5 1 1.5 2 2.5 1 1.5 2 2.5
SCM SCM 100.5 100.5 SCM th 100.5 100.5 SCM th Proposed Proposed CRB QuEST1 QuEST1 QuEST2 QuEST2 100 100 100 100
−0.5 10 10−0.5 10−0.5 10−0.5
1 1.5 2 2.5 1 1.5 2 2.5 1 1.5 2 2.5 1 1.5 2 2.5 n n n n p p p p Figure 2. Fisher distance of estimates of C, initialized at linear- Figure 3. Fisher distance of estimates of C−1, initialized at linear- shrinkage. From top-left to bottom-right: Wishart, Toeplitz 0.1, shrinkage. From top-left to bottom-right: Wishart, Toeplitz 0.1, Toeplitz 0.9, Discrete. “SCM th” defined in Remark 2. Averaged Toeplitz 0.9, Discrete. Averaged over 100 random realizations of over 100 random realizations of X, p = 200. X, p = 200. matrices for the SCM, QuEST1, QuEST2, and our pro- tive example, we focus here on linear discriminant analysis posed estimator, the latter three being initialized at M0 the (LDA) and quadratic discriminant analysis (QDA). Both shrinkage estimation from (Ledoit & Wolf, 2004) (consis- exploit estimates covariance matrices of the data or their tent with the choice made for QuEST1, QuEST2 in (Ledoit inverse in order to perform the classification. & Wolf, 2015; Ledoit et al., 2018)). In the figures, “SCM (1) (1) (2) (2) th” refers to the asymptotic analytical approximation of Suppose x1 , . . . , xn1 ∼ N(µ1,C1) and x1 , . . . , xn2 ∼ δ(C, Cˆ) as defined in Remark 2. It is observed that, while N(µ2,C2) are two sets of random independent p- the SCM never reaches the unbiased CRB, in many cases dimensional training vectors forming two classes of a the QuESTx estimators and our proposed method overtake Gaussian mixture. The objective of LDA and QDA is to the CRB, sometimes significantly so. The Wishart ma- estimate the probability for an arbitrary random vector x trix case seems more challenging from this perspective. to belong to either class by replacing the genuine means µa In terms of performances, both our proposed method and and covariances Ca by sample estimates, with in the case of QuESTx perform competitively and systematically better LDA the underlying (possibly erroneous) assumption that n1 n2 than the sample covariance matrix. C1 = C2. Defining C as C ≡ C1 + C2, the n1+n2 n1+n2 Remark 2 (Consistent estimator for δ(C, Cˆ)). With the classification rules for LDA and QDA for data point x de- same technical tools from (Couillet et al., 2018), it is pend on the signs of the respective quantities: straightforward to estimate the distance δ(C, Cˆ). Indeed, ˆ 1 H LDA T ˇ−1 1 T ˇ−1 1 T ˇ−1 δ(C, C) = 2πı Γ f(z)mγ (z) for mγ (z) the Stieljes trans- δx = (ˆµ1 − µˆ2) C x + µ2 C µ2 − µˆ1 C µˆ1 −1 ˆ 2 2 form of the eigenvalue distribution of C C; the limiting 1 δQDA = xT Cˇ−1 − Cˇ−1 x + µˆTCˇ−1 − µˆTCˇ−1 x distribution of the latter is the popular Marcenko-Pastur law x 2 2 1 1 1 2 2 (Marcenko˘ & Pastur, 1967), the expression of which is well 1 1 1 Cˇ−1 n known. The estimate is denoted “SCM th” in Figures 2–3. + µˆTCˇ−1µˆ − µˆTCˇ−1µˆ + log det 1 − log 2 2 2 2 2 2 1 1 1 2 ˇ−1 n The observed perfect match between limiting theory and C2 1 practice confirms the consistency of the random matrix ap- 1 Pna (a) where µˆa ≡ x is the sample estimate of µa and proach even for not too large p, n. na i=1 i ˇ−1 −1 ˇ n1 ˇ Ca are some estimate of Ca , while C ≡ C1 + 4.2. Application to LDA/QDA n1+n2 n2 Cˇ2 with Cˇa the estimation of Ca. As such, in the n1+n2 As pointed out in the introduction, the estimation of the co- following simulations, LDA will exclusively exploit esti- variance and inverse covariance matrices of random vectors mations of C1 and C2 (before inverting their estimated av- are at the core of a wide range of applications in statistics, erage), while QDA will focus on estimating directly the −1 −1 machine learning and signal processing. As a basic illustra- inverses C1 and C2 . Random matrix improved covariance estimation
1 1 1 1
0.95 0.95 0.9 0.9
0.9 0.8 0.8 0.9 Accuracy Accuracy
0.85 0.7 0.7 0.85 0.6 0.8 0.6 2 3 4 5 6 2 3 4 5 6 2 3 4 5 6 2 3 4 5 6
n1+n2 n1+n2 n1+n2 n1+n2 p p p p
1 1 1 1
0.9 0.95 0.95 0.95 0.8 0.9 0.9
0.7 0.9 Accuracy Accuracy 0.85 0.85 SCM 0.6 SCM 0.8 QuEST1 0.85 QuEST1 0.8 QuEST2 QuEST2 Proposed 0.5 Proposed 0.75 0.8 2 3 4 5 6 B/E A/E B/D A/D B/C A/C 2 3 4 5 6 B/C A/E B/E A/C B/D A/D n1+n2 n1+n2 p Combinations (Healthy/Epileptic) p Combinations (Healthy/Epileptic)
Figure 4. Mean accuracy obtained over 10 realizations of LDA Figure 5. Mean accuracy obtained over 10 realizations of QDA classification. From left to right and top to bottom: C1 and C2 are classification. From left to right and top to bottom: C1 and C2 are respectively Wishart/Wishart (independent), Wishart/Toeplitz- respectively Wishart/Wishart (independent), Wishart/Toeplitz- 0.2, Toeplitz-0.2/Toeplitz-0.4, and real application to EEG data. 0.2, Toeplitz-0.2/Toeplitz-0.4, and real application to EEG data.
The first three displays in Figures 4 and 5 compare the ac- products of covariance matrices, our approach may be flex- curacies of the LDA/QDA algorithms for C1 and C2 chosen among Wishart and Toeplitz matrices, and for µ = µ + 80 ibly adapted to further matrix divergences solely depending 2 1 p on eigenvalue relations. The same framework can notably for the LDA and µ = µ + 1 for the QDA settings (in 2 1 p be applied to the Wasserstein distance between zero-mean order to avoid trivial classification). The bottom right dis- Gaussian laws. A reservation nonetheless remains on the plays are applications to real EEG data extracted from the need for n > p in settings involving inverse matrices that, dataset in (Andrzejak et al., 2001). The dataset contains when not met, does not allow to relate the sought-for eigen- five subsets (denoted A-E). Sets A and B were collected values to the (undefined) inverse sample covariance matri- from healthy volunteers while C, D, E were collected from ces. (Couillet et al., 2018) shows that this problem can be epileptic patients. The graph presents all combinations partially avoided for some divergences (not for the Fisher of binary classes between healthy volunteers and epilec- distance though). A systematic treatment of the n < p case tic subjects (e.g., A/E for subsets A and E). There we ob- is however lacking. serve that, for most considered settings, our proposed algo- rithm almost systematically outperforms competing meth- ods, with QuEST1 and QuEST2 exhibiting a much less sta- Our approach also suffers from profound theoretical lim- ble behavior and particularly weak performances in all syn- itations that need be properly addressed: the fact that thetic scenarios. δˆ(M,X) only estimates the sought-for δ(M,C) for M in- dependent of X poses a formal problem when implemented 5. Discussion and Concluding Remarks in the gradient descent approach. This needs be tackled: (i) either by estimating the introduced bias so to estimate Based on elementary yet powerful contour integration tech- the loss incurred or, better, (ii) by accounting for the de- niques and random matrix theory, we have proposed in this ˆ pendence to provide a further estimator δˆ(M(X),X) of work a systematic framework for the estimation of covari- δ(M(X),C) for all X-dependent matrices M(X) follow- ance and precision matrices. Unlike alternative state-of- ing a specific form. Notably, given that, along the gradient the-art techniques that attempt to invert the fundamental descent initialized at M = UDˆ Uˆ T, with Uˆ the eigen- Bai–Silverstein equations (Silverstein & Bai, 1995), our 0 0 vectors of Cˆ and D some diagonal matrix, all subsequent proposed method relies on a basic gradient descent ap- 0 M matrices share the same profile (i.e., M = UDˆ Uˆ T proach in S++ that, in addition to performing competi- k k k n for some diagonal D ), a first improvement would consist tively (if not better), is computationally simpler. k in estimating consistently δ(UDˆ Uˆ T,C) for deterministic While restricted to metrics depending on the eigenvalues of diagonal matrices D. Random matrix improved covariance estimation
The stability of the eigenvectors when initialized at M0 = Couillet, R. and McKay, M. Large dimensional analysis T UDˆ 0Uˆ , with Uˆ the eigenvectors of Cˆ, turns our algorithm and optimization of robust shrinkage covariance matrix into a “non-linear shrinkage” method, as called by (Ledoit estimators. Journal of Multivariate Analysis, 131:99– & Wolf, 2015). Parallel simulations also suggest that arbi- 120, 2014. trary initializations M0 which do not follow this structure tend to still lead to solutions converging to the eigenspace Couillet, R., Tiomoko, M., Zozor, S., and Moisan, E. Ran- of Cˆ. However, this might be a consequence of the incon- dom matrix-improved estimation of covariance matrix sistency of δˆ(M,X) for X-dependent matrices M: it is distances. arXiv preprint arXiv:1810.04534, 2018. expected that more consistent estimators might avoid this El Karoui, N. et al. Spectrum estimation for large dimen- ˆ problem of “eigenvector of C” attraction, thereby likely sional covariance matrices using random matrix theory. leading to improved estimations of the eigenvectors of C. The Annals of Statistics, 36(6):2757–2790, 2008.
We conclude by emphasizing that modern large dimen- Friedman, J., Hastie, T., and Tibshirani, R. Sparse inverse sional statistics have lately realized that substituting large covariance estimation with the graphical lasso. Biostatis- covariance matrices by their sample estimators (or even tics, 9(3):432–441, 2008. by improved covariance estimators) is in general a weak Jeuris, B., Vandebril, R., and Vandereycken, B. A survey approach, and that one should rather focus on estimating and comparison of contemporary algorithms for comput- some ultimate functional (e.g., the result of a statistical test) ing the matrix geometric mean. Electronic Transactions involving the covariance (see, e.g., (Mestre & Lagunas, on Numerical Analysis, 39(ARTICLE), 2012. 2008) in array processing or (Yang et al., 2015) in statis- tical finance). It is to be noted that our proposed approach Ledoit, O. and Wolf, M. A well-conditioned estimator for is consistent with these considerations as various function- large-dimensional covariance matrices. Journal of mul- als of C can be obtained from Equation (2), from which tivariate analysis, 88(2):365–411, 2004. similar derivations can be performed. Ledoit, O. and Wolf, M. Spectrum estimation: A unified framework for covariance matrix estimation and pca in Acknowledgement large dimensions. Journal of Multivariate Analysis, 139: This work is supported by the ANR Project RMT4GRAPH 360–384, 2015. (ANR-14-CE28-0006) and by the IDEX GSTATS Chair at Ledoit, O., Wolf, M., et al. Optimal estimation of a University Grenoble Alpes. large-dimensional covariance matrix under steins loss. Bernoulli, 24(4B):3791–3832, 2018. References Marcenko,˘ V. A. and Pastur, L. A. Distributions of eigen- Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization values for some sets of random matrices. Math USSR- algorithms on matrix manifolds. Princeton University Sbornik, 1(4):457–483, April 1967. Press, 2009. McLachlan, G. Discriminant analysis and statistical pat- Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., tern recognition, volume 544. John Wiley & Sons, 2004. David, P., and Elger, C. E. Indications of nonlinear deter- ministic and finite-dimensional structures in time series Mestre, X. On the asymptotic behavior of the sample esti- of brain electrical activity: Dependence on recording re- mates of eigenvalues and eigenvectors of covariance ma- gion and brain state. Physical Review E, 64(6):061907, trices. 56(11):5353–5368, November 2008. 2001. Mestre, X. and Lagunas, M. Modified Subspace Algo- rithms for DoA Estimation With Large Arrays. 56(2): Positive definite matrices Bhatia, R. . Princeton University 598–614, February 2008. Press, 2009. Moakher, M. Divergence measures and means of symmet- Bickel, P. J., Levina, E., et al. Regularized estimation of ric positive-definite matrices. In New Developments in large covariance matrices. The Annals of Statistics, 36 the Visualization and Processing of Tensor Fields, pp. (1):199–227, 2008. 307–321. Springer, 2012. Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R. Silverstein, J. W. and Bai, Z. D. On the empirical distribu- Manopt, a Matlab toolbox for optimization on mani- tion of eigenvalues of a class of large dimensional ran- folds. Journal of Machine Learning Research, 15:1455– dom matrices. Journal of Multivariate Analysis, 54(2): 1459, 2014. URL http://www.manopt.org. 175–192, 1995. Random matrix improved covariance estimation
Silverstein, J. W. and Choi, S. Analysis of the limiting spectral distribution of large dimensional random matri- ces. Journal of Multivariate Analysis, 54(2):295–309, 1995. Skovgaard, L. T. A Riemannian geometry of the multivari- ate normal model. Scandinavian Journal of Statistics, pp. 211–223, 1984. Smith, S. T. Covariance, subspace, and intrinsic crame/spl acute/r-rao bounds. IEEE Transactions on Signal Pro- cessing, 53(5):1610–1630, 2005. Sra, S. Positive definite matrices and the s-divergence. arXiv preprint arXiv:1110.1773, 2013. Van Erven, T. and Harremos, P. Renyi´ divergence and kullback-leibler divergence. IEEE Transactions on In- formation Theory, 60(7):3797–3820, 2014.
Vinogradova, J., Couillet, R., and Hachem, W. Estima- tion of toeplitz covariance matrices in large dimensional regime with application to source detection. IEEE Trans. Signal Processing, 63(18):4903–4913, 2015. Wu, W. B. and Pourahmadi, M. Banding sample auto- covariance matrices of stationary processes. Statistica Sinica, pp. 1755–1768, 2009. Yang, L., Couillet, R., and McKay, M. R. Minimum variance portfolio optimization in the spiked covariance model. In Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2015 IEEE 6th Inter- national Workshop on, pp. 13–16. IEEE, 2015.