Arxiv:1804.01620V1 [Stat.ML]

arXiv:1804.01620v1 [stat.ML] 4 Apr 2018 rmosrain fteform the of observations from esuyetmto ftecvrac arxo admvect random a of matrix covariance the of estimation study We matrix covariance models, graphical data, missing ust fvco oriae.Ec bevto steprod the is a with observation interest Each dif of variable contain the coordinates. samples vector different of partia where subsets of case vectors, the random for served estimation matrix covariance study We iiisada ciecvrac arxetmto algori estimation matrix sub-samplin covariance optimal active of an design and a bilities t propose of and dimension interest, the of to tor compared small is variables expecte observed the a of where We framework, learning matrix. active covariance an in sub-s the analysis of the entries between the relations and reveals probabilities that bound model, error this an under rive estimator covariance unbiased an analyze otc uhremi:[email protected],[email protected] [email protected], e-mail: author Contact ahcomponent each able needn fec te,adof and other, each of independent aee.Terslsfo hspprcnb sflfranalys for useful be can whi paper this data, from from results a estimated The are be rameter. probabilities to sampling the has problems, or learning given ar active is probabilities that sub-sampling the feature case, lem observati data the missing over the control In the is scenarios estimation ance h oto bevto ed ob eue yslcigwhat learning). selecting o active by data) reduced (i.e., missing be ple to is needs there observation limita of thus cost physical (and the some process are observation there the when to arise may model this with a ecpue yascaigdfeetsbsmln prob sub-sampling and different acquired, associating p be by can captured that hour be per can diffe samples of e.g. sensors number capabilities, or different different bility example, have For have might [2]. network networks sensor and transportatio [1] include data missing with observations from hr sane o pia eoreallocation. resource optimal for need constr a is communication there or processing acquisition, network are sensor u there like be environments can computation estimation covariance distributed observation for of approaches number learning minimal t tive with manner model adaptive graphical and the sequential tify a in consider observed been are have variables approaches learning active estimation, i oec sensor. each to hswr a upre npr yNFudrgatCCF-141000 grant under NSF by part in supported was work This ne Terms Index plctosweepoaiitcmdl edt econstru be to need models probabilistic where Applications o oaineetmto,o oegnrlygahclmod graphical generally more or estimation, covariance For n rtcldfeec ewe isn aaadatv cov active and data missing between difference critical One CIECVRAC SIAINB ADMSBSMLN FVAR OF SUB-SAMPLING RANDOM BY ESTIMATION COVARIANCE ACTIVE δ i easm h probabilities the assume We . — x i smlile ya by multiplied is ciecvrac siain admsampling, random estimation, covariance active .INTRODUCTION 1. ABSTRACT y 0 − [ = x 1 δ ata bevtosconsistent observations Partial . enul admvral.We variable. random Bernoulli 1 x 0 P 1 ( − δ , δ i 2 1 x )= 1) = enul admvari- random Bernoulli 2 dad ae n noi Ortega Antonio and Pavez Eduardo , · · · nvriyo otenCalifornia Southern of University c.edu. δ , p n i x thm. r known, are n networks n einpa- design nmodel. on etrelia- rent ] it,and aints, number d dwhere ed prob- a e ⊤ ,where s, pyour pply proba- g ampling abilities si the in is eu in seful n de- and osam- to where , evec- he iden- o when r l ob- lly .Ac- s. c of uct ferent nthe in these tions ein le or cted ari- el 9. x onscnte eapidt eintesmln distributi sampling the design manner. to adaptive applied probabilities be sampling then can the bounds and matrix u en covariance the allowing true between norm, the dependences precise Frobenius with in bounds errors error derive consider in we errors estimation while consider norm, papers Moreo aforementioned estimator. the covariance of know sample and unbiased different the be to study probabilities only all allow Comp we 4], matrices. covariance 5, [3, sparse and bandable for norm tral ainemti siao,smlrt usbtuigteemp sample the probabilities using modified but of a timation ours to that similar show estimator, unknown matrix They and variance arbitrary probabilities. assumes data compl which missing missing (MCAR), the study i model [5] be random Zhang to at and shown Cai is recently data Most [4] missing nite. In under algorithm. covar lasso matrix rank the covariance low of pirical version approximately matrix of a probability estimation with same on matrices the is with [3] observed of be focus can miss variables from all estimation when matrix covariance studies [3] Lounici 7. Sections in conclusion probabil and P 6 sampling algorithm. Section app of estimation in an covariance presented design active show based optimal we batch for 5 a probl for bounds Section the our In introduce of 4 respectively. and cation results 3 main Sections the work. and related review i follows, we as 2 organized is estimat paper an The as cova when observations. well active complete as that almost an perform is rank, can work approach low estimation this approximately from is matrix conclusion covariance interesting d An the esti to covariance nario. active bound an error in this probabilities apply sub-sampling We of en matrix. and covariance probabilities true b the sub-sampling relation the observations, reveals of that norm number Frobenius the on bound error sw ildsusi oedti nScin5. Section in detail more in a discuss learning active will designing we for as as well as case, data missing vntog h ein r adm hyhv xddistribu fixed a have they observations. random, all are for designs varia few the an a though of mea memory combination each even for linear 7], sparse) [8] 8, (possibly [6, a in works is used these ment is all In [7] PCA. from reduced complexity approach same proje sparse The to [7] trices. in generalized projection are subspace which matrices, random dense from estimation covariance ies o ne u-asinasmtoson assumptions sub-Gaussian under tor iain te neetn cielann prahsfo approaches learning input active m an interesting covariance Other as inverse for estimator [10] timation. matrix algorithm lasso covariance graphical unbiased the an uses and o itiue optn plctos h oko 6 stu [6] of work the applications, computing distributed For nti okw nlz nubae oainemti estima matrix covariance unbiased an analyze we work this In h oko 9 osdr h aewe all when case the considers [9] of work The .RLTDWORK RELATED 2. p i civsotmlmnmxrtsi spec- in rates minimax optimal achieves , x u anrsl san is result main Our . p IABLES i r unknown, are ainsce- mation p lgorithms ,adwe and n, to ma- ction Section n graphi- r of are roofs ls and bles, rcles- irical i ti es- atrix n data ing ni an in on spectral These . h em- the re of tries using s etween rwith or re of tries e,all ver, rdto ared riance ndefi- The . iance esign sure- etely ities tion to s co- em the li- d- to d - cal model selection are [11], [12] and [13]. Vats et. al. [12] uses a random variables include Gaussian, Bernoulli and Bounded random method that combines sampling marginals with conditional indepen- variables. For more information see [18] and references therein. dence testing to learn the graphical model structure. [11, 13] con- Definition 1 ([18]). If E[exp(z2/K2)] 2 holds for some K > 0, sider a more general family of algorithms based on sampling high we say z is sub-Gaussian. If E[exp( z≤/K)] 2 holds for some degree vertices, and prove upper and lower performance bounds. K > 0, we say z is sub-exponential. | | ≤ Random sub-sampling and reconstruction of signals has been studied within graph signal processing [14, 15] and statistics [16]. Definition 2 ([18]). The sub-Gaussian and sub-exponential norms The Bernoulli observation model we use has been studied by [14, 15] are defined as as a sampling strategy for graph signals, where sampling probabil- α α z ψ = inf u> 0 : E[exp( z /u )] 2 . ity designs are proposed for reconstruction of deterministic band- k k α { | | ≤ } limited signals. Also, Romero et. al. [17] derives covariance matrix for α = 2 and α = 1 respectively. estimators assuming the target covariance matrix is a linear combi- Sub-Gaussian and sub-exponential random variables, and their nation of known covariance matrices, which leads to algorithms and norms, are related as follows. theoretical analysis that are fundamentally different from ours. Proposition 1 ([18]). If z and w are sub-Gaussian, then z2 and zw 2 2 are sub-exponential with norms satisfying z ψ1 = z ψ2 , and 3. PROBLEM FORMULATION k k k k zw ψ1 z ψ2 w ψ2 . k k ≤ k k k k 3.1. Notation We have that the following characterization of the product of sub-Gaussian and Bernoulli random variables. We denote scalars using regular font, while we use bold for vectors Lemma 1. Let y1 = δ1x1 and y2 = δ2x2 be a product of Bernoulli and matrices, e.g., a =(ai), and A =(aij ). The Hadamard product δ1, δ2 and sub-Gaussian x1,x2 random variables with Bernoulli between matrices is defined as (A B)ij = aij bij . We use q for entry-wise matrix norms, with q =⊙ 2 corresponding to the Frobeniusk·k probabilities p1 and p2 respectively, the only dependent variables are x1 and x2, then norm. denotes ℓ2 norm or spectral norm when applied to vectors k·k or matrices respectively. 1) yi is sub-Gaussian and yi ψ2 xi ψ2 . 2 2 k k ≤ k k 2) y1 , y2 , and y1y2 are sub-exponential with norms satisfying yiyj ψ1 xixj ψ1 for i, j = 1, 2. 3.2. Unbiased estimation k k ≤ k k n The proof of Lemma 1 can be easily obtained from the definition Consider a random vector x taking values in R . We observe of sub-Gaussian and sub-exponential norms. We omit it for space y = δ x, (1) considerations. We also define the matrix H = (hij ), with entries ⊙ given by kx x k where δ =(δi) is a vector of Bernoulli 0 1 random variables. The i j ψ1 i = j − pipj probability of observing the i-th variable is given by P(δi = 1) = hij = 2 6 ⊤  kxi kψ1 pi. The vector of probabilities is denoted by p = [p1, ,pn] , i = j.  pi and P = diag(p) is a diagonal matrix. The average· number · · of n n Now we state our main result, whose proof appears in Section 6. samples corresponds to E( δi) = pi = m. Given  i=1 i=1 Theorem 1. Let x be zero mean random vector in n with sub- y(1), , y(T ), i.i.d. realizations of y, the i-th variable will be sam- R P P Gaussian entries and norm x . Let y = δ x, where each δ pled in· · average· p T times, if all p = 1, then y = x, and we have i ψ2 i i i is a Bernoulli random variablek k with parameter ⊙ , inde- perfect observation of x. We are interested in studying covariance 0 < pi 1 pendent of each other and of x. Given i.i.d. realizations ≤y(k) T , estimation for x when m < n and 0

2 ⊤ 2 log(n) + log(η) 2 log(n) + log(η) E(y)= Pµ, and Cov(y)= Σ Ξ +(P P ) diag(µµ ), Σˆ Σ q H q γ γ ⊙ − k − k ≤ k k (r T ∨ T ) where Ξ =(ξij ) is defined as ξii = pi and ξij = pipj when i = j. 6 with probability at least 1 2 , where γ is an universal constant. For the rest of the paper we will assume µ = 0. Given a set of − η (k) T of i.i.d. samples y of the random vector y, define Moreover if xi ψ2 = σ√Σii and q 2, then { }k=1 k k ≥ 2 T 2σ 1 ⊤ H q r(Σ) Σ , (3) Σ = y(k)y(k) Ξ†, (2) k k ≤ pˆ2 k k T ⊙ k=1 where pˆ = min pi, and r(Σ) = tr(Σ)/ Σ is the effective rank. X k k † b where Ξ is the Hadamard (entry-wise) inverse of Ξ. A simple Theorem 1 shows that the estimation error Σˆ Σ q 0 in k − k → calculation shows that Σ is an unbiased estimator for Σ. Indeed, probability as the number of samples increases. More importantly, ⊤ E(Σ) = 1 T E(y(k)y(k) ) Ξ† = Σ Ξ Ξ† = Σ. our result reveals that the sampling probabilities pi are closely re- T k=1 ⊙ ⊙ ⊙ lated to the sub-exponential norms of the variables x x through Because Ξ† , the matrixb Σ might not be positive semi-definite i j 0 the matrix H . The bound from (3 ) suggests that distributions b P q (conditions for Σ to be positive semi-definite are given in [4]). with smallerk effectivek rank [3, 18] can tolerate a more aggressive b sub-sampling factor (smaller m). This is not surprising since the b 4. ESTIMATION ERROR effective rank r(Σ) is upper bounded by the actual rank, and can be significantly smaller for distributions whose energy concentrates In this section we present an error analysis of the covariance matrix in few principal components. We also note that the ratio r(Σ)/pˆ2 estimator from (2) when x has sub-Gaussian entries. Sub-Gaussian appears in the bounds of [3] as well. Algorithm 1 Covariance estimation with adaptive sampling 5.3. Numerical evaluation Require: initial distribution p(0), covariance Σ(0) = 0, budget In this section we evaluate our proposed method using the MNIST 1⊤p(0) = m, and batch size B. [19] dataset, which consists of 28 28 images of scanned digits × 1: for t = 0 to N 1 do b from 0 to 9. In our experiments we consider N = 5851 images of − (t) 2: Sample B i.i.d. realizations of y = δ(p ) x the digit 8. The vectorized, and mean removed images are denoted ⊙ N 3: Estimate Σ by zi i=1 with covariance matrix C. We consider estimation of the (t+1) 1 t (t) { } 4: Update Σ Σ + Σ the covariance matrix of xi = zi + θ C e, where e is zero mean ← t+1 t+1 k k (bt+1) (t+1) Gaussian noise with unit variance, therefore Σ = C + θ C I. 5: Update p by solving (5) with Σ and budget m. p k k 6: end for b b b To draw i.i.d. realizations of x, we sample images zi without C b replacement and add Gaussian noise with variance θ . The effective rank of Σ is controlled by the parameter θ andk satisfiesk r(C)+ nθ 5. ACTIVE COVARIANCE ESTIMATION r(Σ)= . 1+ θ In this section we consider a scenario where we cannot observe (on We first compare uniform sub-sampling with non uniform sub- average) more than m variables at a time, but we have the freedom sampling for estimation of Σ with θ = 1/n. We designed the non to choose the sub-sampling probability distribution. uniform sub-sampling distribution using (4) with the true covariance matrix. We report relative errors in Frobenius norm as a function of T/n in Figure 1a. Each point in the plot is an average over 5.1. Sub-sampling distribution for true covariance matrix 50 independent trials. We observe that when m = 0.75n the non Based on the error bound from Theorem 1, we propose designing uniform sampling distribution matches closely the performance of the estimator with full data. It is clear that when m = 0.50n and the sub-sampling distribution by approximately minimizing H q . k k m = 0.25n performance decreases (for uniform and non uniform Theorem 1 suggests that pi should be larger whenever the sub- sampling) as m decreases, and non uniform sampling always out- Gaussian norm of xi is large, but also the product pipj should be performs uniform sampling. In Figure 1b we evaluate our active large when the sub-exponential norm of xixj is large. We assume method from Algorithm 1 with batch size B = 300. We consider that xi ψ2 = σ√Σii. The bounds for hii and hij from (6) and (7) k k 2 the same scenario as in Figure 1a for m = 0.5n and m = 0.25n. respectively suggest the approximation pi Σii. Given a sampling budget m, we estimate the sub-sampling∼ probability vector p by The proposed active covariance estimation method quickly learns solving the following scaled projection problem the optimal sub-sampling distribution, is always better than uniform sampling, and matches the performance of the non uniform sub- 1 sampling method obtained from the true covariance matrix. Finally 1 2 2 ⊤ min p ρ diag(Σ) 2, s.t. 1 p = m, 0 p 1. (4) we show in Figure 1c the same experiment shown in Figure 1b, p,ρ 2 k − k ≤ ≤ but now with covariance matrix with parameter θ = 10/n, which changes the effective rank from r(Σ) = 9.08 with θ = 1/n to 5.2. Sub-sampling distribution for empirical covariance matrix r(Σ) = 17.86. We observe that the problem becomes more difficult since the estimation errors are larger. There is no major difference Since the true covariance matrix is unknown, (4) does not leadtoa between uniform, and non uniform sampling, thus the advantages practical estimator. Instead, we consider a batch based algorithm that of the proposed active covariance estimation method are limited. for a given budget , and a starting sub-sampling distribution, it iter- m This might be due to various effects including, relative magnitudes atively refines the sub-sampling probability distribution as a function of diagonal entries of Σ, sampling budget m, effective rank, and of previous observations. We show the pseudo code for such pro- probability update algorithm. Moreover, since the effective rank did cedure in Algorithm 1, which can be summarized in the following not change much (compared with n), this experiment suggests the steps: observation with variable sub-sampling, covariance estima- effective rank does not quantify effectively the problem difficulty. tion, and sub-sampling distribution update. At the -th iteration, t B Also, a more precise method to update the probabilities p might i.i.d. realizations are observed according to (1) with sub-sampling help improving the performance of the active covariance estimation probabilities p(t). The covariance estimator is a convex combina- algorithm. tion of the estimator at the previous iteration, and the estimator for the current batch. Finally, the new covariance matrix estimator is used to update the sub-sampling probabilities as 6. PROOF OF THEOREM 1

1 We first need the following concentration bounds (t) 1 (t) 2 2 p = arg min p ρ diag(Σ ) 2, (5) p,ρ 2 k − k Lemma 2. Under the same assumptions of Theorem 1, for any ν > ⊤ 0 we have s.t. 1 p = m, 0 p b1. ≤ ≤ 2 2 P( Σij Σij >ν) 2exp c1T min ν /h , ν/hij , | − | ≤ − ij Proposition 2. Algorithm 1 produces an unbiased estimator for Σ. 2 2 P( Σii Σii >ν) 2exp c2T min ν /hii, ν/hii , |b − | ≤ − for off-diagonal and diagonal entries respectively. Proof. We will proceed by induction. For the first iteration we have b used uniform sampling, thus we have that E(Σ(1)) = Σ. Now as- Proof. The error events for off-diagonal entries satisfy (t) sume E(Σ ) = Σ, then at the (t + 1)-th iteration, the covariance T (t+1) (k) (k) of the new data satisfies E(Σ) = Σ, which impliesb E(Σ ) = Σij Σij >ν (y y pipj Σij ) > νTpipj . | − | ⇔ | i j − | 1 (Σ)+b t (Σ(t))= Σ. k=1 t+1 E t+1 E X b b b b b

1.2 1.2 U 25% U 25% 2 U 25% 1 U 50% 1 U 50% U 50% U 75% P 25% P 25% P 25% P 50% P 50% 0.8 0.8 1.5 P 50% A 25% A 25% P 75% A 50% A 50% 0.6 full 0.6 full full 1 Relative error Relative error Relative error 0.4 0.4

0.5 0.2 0.2

0 0 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 T/n T/n T/n (a) Uniform vs non-uniform sub-sampling (b) Active covariance estimation, θ = 1/n (c) Active covariance estimation, θ = 10/n

Fig. 1: Performance of different covariance estimators in relative Frobenius norm. (U) corresponds to uniform sub-sampling pi = m/n, (P) denotes non-uniform sub-sampling found using (4), (A) denotes active covariance estimation using Algorithm 1 .

We apply Bernstein’s inequality [18] for sums of independent zero where 1/γ = min(c1, c2). The proof can be ﬁnished by equating mean sub-exponential random variables, which combined with the to 2/η, solving for ǫ, and doing some min / max manipulations. To bound from Lemma 1 leads to the desired bound. The proof for derive the bound from (3) we bound the entries of H obtaining diagonal terms follows almost the same procedure. 2 2 σ Σii σ Σii We also need the following geometric result which we state hii = , (6) without proof. pi ≤ pˆ 2 2 σ ΣiiΣjj σ ΣiiΣjj n (7) Lemma 3. Let = x R : x 1 > ǫ , and i = x hij 2 . n A { ∈ k k } B { ∈ ≤ pipj ≤ pˆ R : xi > αiǫ , then for all ǫ > 0, and αi (0, 1] that satisfy p p n | | } n ∈ i=1 αi = 1, we have that i=1 i. A⊂ B Then, applying (6) and (7) followed by triangle inequality of the ℓq P The proof of Theorem 1 startsS by bounding the probability of norm we have the event Σˆ Σ q > ǫ Σ q . Pick a set of αij (0, 1] such that n 1 k − k k k ∈ 2 i,j αij = 1, and apply Lemma 3 and the union bound to get 2 n n q σ 1 q 1 q/2 H q 1 Σii + Σii P ˆ ˆ q q q k k ≤ pˆ − pˆq pˆq P( Σ Σ q > ǫ Σ q )= P( Σij Σij > ǫ Σ q ) " i=1 i=1 ! # k − k k k | − | k k 1X X i,j X σ2 1 q 1 q q q 1 diag(Σ) + diag(Σ) q ˆ q q 2 P( Σij Σij > αij ǫ Σ q ) ≤ pˆ pˆ − k k pˆk k ≤ | − | k k " # ij X σ2 tr(Σ) 1 2σ2 1/q q q Σ Σ = P( Σˆ ij Σij > α ǫ Σ q ) 2 (1 pˆ ) + 1 2 r( ) . | − | ij k k ≤ pˆ − ≤ pˆ k k ij X h i n 2/q 2 2 1/q αii ǫ Σ q αii ǫ Σ q The last step uses the fact that a q a 1 for all q 1, and the 2exp c2T min 2k k , k k + k k ≤ k k ≥ ≤ − h hii deﬁnition of effective rank. i=1 ( ii !) X n 2/q 2 2 1/q αij ǫ Σ q αij ǫ Σ q 2exp c1T min 2k k , k k . (− hij hij !) 7. CONCLUSION i6=Xj=1

The last inequality follows from Lemma 2 with constants c1, c2 ap- We studied covariance matrix estimation when the variables are sub- propiately chosen so the inequalities hold for all pairs i, j. We can sampled by a product with Bernoulli 0 1 variables. Variations of q q − further simplify by choosing αij = h / H this model have been traditionally considered in the analysis of miss- ij k kq ing data. We study an unbiased estimator for the covariance matrix P( Σˆ Σ q > ǫ Σ q ) and derive a novel estimation error bound in entry-wise ℓq norm. k − k k k 2 Our bound illustrates the subtle relations between covariance ma- Σ Σ 2 q q trix parameters and sub-sampling distribution. Using this bound, we 2n exp c2T min ǫ k k2 , ǫ k k ≤ − H H q k kq k k propose an active covariance matrix estimation algorithm that also 2 2 2 Σ q Σ q produces an unbiased estimator. We show with numerical experi- + 2(n n)exp c1T min ǫ k k2 , ǫ k k − − H q H q ments that the proposed active covariance estimation algorithm out- k k k k performs uniform sub-sampling, and closely matches non-uniform Σ 2 2 T 2 q Σ q sub-sampling with complete knowledge of the true covariance ma- 2n exp min ǫ k k2 , ǫ k k , ≤ − γ H H q k kq k k trix. 8. REFERENCES [16] Po-Ling Loh and Martin J Wainwright, “High dimensional regression with noisy and missing data: provable guarantees [1] Muhammad Tayyab Asif, Nikola Mitrovic, Justin Dauwels, with non-convexity,” The Annals of Statistics, vol. 40, no. 3, and Patrick Jaillet, “Matrix and tensor based methods for miss- pp. 1637–1664, 2012. ing data estimation in large traffic networks,” IEEE Transac- [17] Daniel Romero, Dyonisius Dony Ariananda, Zhi Tian, and tions on Intelligent Transportation Systems, vol. 17, no. 7, pp. Geert Leus, “Compressive covariance sensing: Structure- 1816–1825, 2016. based compressive sensing beyond sparsity,” IEEE signal pro- [2] Amol Deshpande, Carlos Guestrin, Samuel R Madden, cessing magazine, vol. 33, no. 1, pp. 78–93, 2016. Joseph M Hellerstein, and Wei Hong, “Model-driven data acquisition in sensor networks,” in Proceedings of the Thirtieth [18] Roman Vershynin, High Dimensional Probability: An Intro- international conference on Very large data bases-Volume 30. duction with Applications in Data Science, Cambridge Uni- VLDB Endowment, 2004, pp. 588–599. versity Press, 2018. [3] Karim Lounici, “High-dimensional covariance matrix estima- [19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick tion with missing observations,” Bernoulli, vol. 20, no. 3, pp. Haffner, “Gradient-based learning applied to document recog- 1029–1058, 2014. nition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278 – 2324, 1998. [4] Kamil Jurczak and Angelika Rohde, “Spectral analysis of high-dimensional sample covariance matrices with missing observations,” Bernoulli, vol. 23, no. 4A, pp. 2466–2532, 2017. [5] T Tony Cai and Anru Zhang, “Minimax rate-optimal estimation of high-dimensional covariance matrices with incomplete data,” Journal of multivariate analysis, vol. 150, pp. 55–74, 2016. [6] Martin Azizyan, Akshay Krishnamurthy, and Aarti Singh, “Extreme compressive sampling for covariance estimation,” arXiv preprint arXiv:1506.00898, 2015. [7] Farhad Pourkamali-Anaraki, “Estimation of the sample covariance matrix from compressive measurements,” IET Signal Processing, vol. 10, no. 9, pp. 1089–1095, 2016. [8] Farhad P Anaraki and Shannon Hughes, “Memory and computation efficient pca via very sparse random projections,” in Proceedings of the 31st International Conference on Machine Learning (ICML), 2014, pp. 1341–1349. [9] Mladen Kolar and Eric P Xing, “Consistent covariance selection from data with missing values,” in Proceedings of the 29th International Conference on Machine Learning (ICML), 2012, pp. 551–558. [10] Jerome Friedman, Trevor Hastie, and Robert Tibshirani, “Sparse inverse covariance estimation with the graphical lasso,” Biostatistics, vol. 9, no. 3, pp. 432–441, 2008. [11] Gautamd Dasarathy, Aarti Singh, Maria-Florina Balcan, and Jong H Park, “Active learning algorithms for graphical model selection,” in Artificial Intelligence and Statistics (AISTATS), 2016, pp. 1356–1364. [12] Divyanshu Vats, Robert Nowak, and Richard Baraniuk, “Ac- tive learning for undirected graphical model selection,” in Arti- ficial Intelligence and Statistics (AISTATS), 2014, pp. 958–967. [13] Jonathan Scarlett and Volkan Cevher, “Lower bounds on active learning for graphical model selection,” in The 20th In- ternational Conference on Artificial Intelligence and Statistics (AISTATS), 2017. [14] Siheng Chen, Rohan Varma, Aarti Singh, and Jelena Kovaˇcević, “Signal recovery on graphs: Fundamental limits of sampling strategies,” IEEE Transactions on Signal and Infor- mation Processing over Networks, vol. 2, no. 4, pp. 539–554, 2016. [15] Gilles Puy, Nicolas Tremblay, Rémi Gribonval, and Pierre Vandergheynst, “Random sampling of bandlimited signals on graphs,” Applied and Computational Harmonic Analysis, 2016.