Average (E)BIC-Like Criteria for Bayesian Model Selection

Average (E)BIC-like Criteria for Bayesian Model Selection

Jingnan Xue, Ye Luo and Faming Liang∗

June 1, 2017

Abstract

Markov chain Monte Carlo (MCMC) has been an indispensable tool for Bayesian analysis of complex statistical models even for high-dimensional problems. However, there still lacks a consistent criterion for selecting models based on the outputs of MCMC. The existing deviance information criterion (DIC) is known to be inconsistent and non-invariant for reparameterization. This paper proposes an Average BIC-like (ABIC) model selection criterion and an Average EBIC-like (AEBIC) model selection criterion for low and high-dimensional problems, respectively; establishes their consistency under mild conditions; and illustrates their applications using generalized linear models. The proposed criteria overcome short- comings of DIC. The numerical results indicate that the proposed criteria can signiﬁcantly outperform DIC as well as the MLE-based criteria, such as AIC, BIC and EBIC, in terms of model selection accuracy.

Keywords: Asymptotic Consistency; Deviance Information Criterion; High-Dimensional Data; Gener- alized Linear Model; Markov Chain Monte Carlo.

∗To whom correspondence should be addressed: F. Liang. Liang is Professor, Department of Biostatistics, University of Florida, Gainesville, FL 32611, email: faliang@uﬂ.edu; J. Xue is Graduate Student, Department of Statistics, Texas A&M University, College station, TX 77843; email: [email protected]; Y. Luo is Assistant Professor, Department of Economics, University of Florida, Gainesville, FL 32611, email:[email protected].

1 1 Introduction

Model selection is a fundamental part of the statistical modeling process, and it has been an active research area since 1970s. The most famous model selection criteria are perhaps AIC(Akaike, 1974) and BIC(Schwarz, 1978). The former is commonly called Akaike Information Criterion after Hirotogu Akaike; and the latter is called Bayesian Information Criterion or Schwarz Information Criterion after Gideon Schwarz. Both criteria are boiled down to a trade-off between goodness-of-fit and model complexity: A good model fits the data well with a minimum number of variables/parameters. To be more precise, AIC is defined by

AIC(S) = −2Ln(Xn|βbS) + 2|S|/n,

(1) (2) (n) where Xn = {X ,X ,...,X } denote a set of n iid observations, S denotes a speciﬁc model, |S| denotes the number of parameters of the model, βs denotes the vector of parameters of the model, βbS denotes the maximum likelihood estimate (MLE) of βS, and Ln(·) denotes the averaged log-likelihood function, i.e., n 1 X L (X |β ) = log f(x(i)|β ), (1) n n S n S i=1 where f(·) denote the probability mass/density function of the model. BIC is deﬁned by

BIC(S) = −2Ln(Xn|βbS) + |S| log(n)/n, which penalizes model complexity more than AIC and so will favor simpler models. AIC and BIC represent two very different approaches for model selection. AIC is essentially a data- driven criterion which is to select a model that explains the data well and will make good short-term predictions. The penalty term 2|S| has been shown to be asymptotically equivalent to leave-one-out cross- validation. Based on this observation, different cross-validation-based criteria have been proposed in the literature, see e.g., generalized cross validation (Wahba and Craven, 1979) and WAIC (Watanabe, 2010). AIC and the cross-validation-based criteria are not asymptotically consistent. BIC is derived under the Bayesian framework, for which the differences provide an approximation to the log(Bayes factor) under specific “unit information” prior assumptions (Kass and Raftery, 1995). Under specific conditions, BIC has been shown to be asymptotically consistent, see e.g. Nishii (1984) and Haughton (1988). Since all these AIC, BIC and cross-validation criteria require an exhaustive evaluation of all possible models for the problem, they are difficult to be applied when the model space is large. When the time entered into 1990s, the use of Markov chain Monte Carlo (MCMC) made it possible to fit arbitrarily complex Bayesian models with an arbitrarily large model space. Note that the AIC, BIC and cross-validation criteria essentially fail for these problems. Hence, there was a pressing need for a model selection criterion that is able to select an appropriate model based on the outputs of MCMC. This need was partially addressed by

2 Deviance information criterion (Spiegelhalter et al., 2002), which is deﬁned by

¯ ¯ DIC(S) = −2EβS |Xn,SLn(Xn|βS) + [−2L(Xn|βS) + 2Ln(Xn|βS)],

where EβS |Xn,S denotes the expectation with respect to the posterior distribution of βS for a given model ¯ S, and βS = EβS |Xn,S(βS) denotes the posterior mean of βS. When the sample size n is suﬃciently ¯ large, it can be shown that [−2L¯(Xn|βS) + 2Ln(Xn|βS)] ≈ |S|/n, which measures the model complexity and is called “eﬀective number of parameters” in Spiegelhalter et al. (2002), and EβS |Xn,SLn(Xn|βS) ≈

Ln(Xn|βbS). Therefore, DIC can be viewed as an approximation to AIC with posterior samples. Twelve years later, Spiegelhalter et al. (2014) acknowledged that DIC has some problems, e.g., it is not invariant to reparameterization, it is not asymptotically consistency, it is not based on a proper predictive criterion, and it has a weak theoretical justification. To alleviate the difficulties suffered by DIC, we proposed a new model selection criterion, the so-called average BIC-like Criterion,

ABIC(S) = −2nEβS |Xn,SLn(Xn|βS) + |S| log(n), (2) which shares the same penalty for the model complexity with BIC and is abbreviated as “average-BIC” or “ABIC”. Under appropriate conditions, we can show that ABIC is asymptotically consistent. During the past two decades, the research in model selection has been reflourished due to the emergence of high-dimensional data where the sample size n is typically smaller than the dimension p. To address the ill-posed issue in modeling high-dimensional data, a variety of variable selection methods have been proposed, in particular, for linear and generalized linear models. To name a few, they include Lasso(Tibshirani, 1994), SCAD(Fan and Li, 2001), elastic net (Zou and Hastie, 2005), Dantzig selector (Candes and Tao, 2007), MCP (Zhang, 2010), rLasso (Song and Liang, 2015a), and EBIC (Chen and Chen, 2008, 2012; Chen and Luo, 2013). Like AIC and BIC, all these methods require to find the global optimizer for a penalized log-likelihood function. However, as shown by Ge et al. (2016) and Huo and Chen (2010), except for Lasso, finding the global optimizer for the other methods is strongly NP-hard due to the concaveness of their penalty functions. Therefore, the models selected by these methods are often sub-optimal. Although Lasso is computationally attractive, it is often inconsistent as the required representative condition (Zhao and Yu, 2006) is hard to be satisfied. Practically, Lasso tends to select a model including many false variables. On the other hand, MCMC continue to work as an indispensable tool in Bayesian modeling for high dimensional problems. As indicated by recent publications, see e.g., Johnson and Rossell (2012), Liang et al. (2013) and Song and Liang (2015b), Bayesian methods can often outperform frequentist methods in identifying important variables for high-dimensional problems. Hence, there is now also a need to develop an appropriate criterion for selecting models based on the outputs of MCMC for high-dimensional

3 problems. To address this issue, we extend EBIC to AEBIC, which is deﬁned by

AEBIC(S) = −2nEβS |Xn,SLn(Xn|βS) + |S| log(n) + 2γ|S| log(p). (3)

The asymptotic consistency of this criterion can be established under some sparsity and regularity conditions. Here we would like to point out that our proof for the asymptotic consistency is general, which is not restricted to generalized linear models. The remainder of this article is organized as follows. Section 2 provides an informal derivation of ABIC and AEBIC. Section 3 establishes the consistency properties of ABIC and AEBIC with the proofs deferred to the Appendix. Section 4 evaluates the ﬁnite sample performance of ABIC and AEBIC through some simulation studies and a real data example. Section 5 concludes the article with a brief discussion.

2 Intuitions on ABIC and AEBIC

This section gives an informal derivation and intuitive explanation for ABIC and AEBIC. Let Xn = {X(i) : i = 1, . . . , n} denote a dataset of n iid observations. For example, for high-dimensional regression, X(i) = (y(i), z(i)), where y is the response variable, and z is a high-dimensional random vector containing pn predictors. Under the high dimensional setting, pn can increase with the sample size n. Let S ⊂ ∗ {1, 2, . . . , pn} denote a generic model, let S denote the true model, and let S := {S : S ⊂ {1, 2, ..., pn}} denote the space of all possible models. For high-dimensional problems, the true model can be sparse, i.e., |S∗| = o(n) holds. The problem of model selection is to identify S∗ from S based on the observed data. Let’s consider the problem under the Bayesian framework, which is to choose a model with the maximum posterior probability. Let π(S) denote the prior distribution imposed on the model space S and let

π(βS) denote the prior distribution of βS. Then the posterior probability of the model S is given by

∗ By choosing appropriate priors, it can be shown that P (S |Xn) converges to 1 in probability as n goes to inﬁnity (Liang et al., 2013; Shin et al., 2015), which is the so-called global model consistency in Bayesian ∗ variable selection (Johnson and Rossell, 2012). This further implies that arg maxS P (S|Xn) = S holds in probability as n → ∞.

However, in most cases, it is impossible to calculate P (S|Xn) exactly, so we need to approximate it. The approximation can be done using MCMC or the Laplace method. We note that the latter has been

4 used in deriving BIC and EBIC. Since

arg max P (S|Xn) = arg max m(Xn|S)π(S) = arg max log{m(Xn|S)π(S)}, S S S we only need to care about log{m(Xn|s)π(S)}, which can be re-expressed as

Z log{m(Xn|S)π(S)} = log exp (nLn(Xn|βS)) π(βS)dβS + log π(S). (4)

To deal with the log-integral term in (4), we expand Ln(Xn|βS) at βbS, the MLE of βS. That is,

2 1 0 ∂ Ln(Xn|βbS) Ln(Xn|βS) ≈ Ln(Xn|βbS) + (βS − βbS) (βS − βbS) 2 ∂β ∂βT S S (5) 1 0 = Ln(Xn|βbS) − (βS − βbS) Ib(Xn, βbS) (βS − βbS), 2

where the approximation is valid provided that the prior π(βS) is ”ﬂat” over the neighborhood of βbS and

Ln(Xn|βS) is ignorable outside the neighborhood of βbS. Finally,

Z log{m(Xn|S)π(S)} = log exp (nLn(Xn|βS)) π(βS)dβS + log π(S) |S| |S| 1 ≈ nLn(Xn|βbS) + log π(βbS) + log(2π) − log(n) − log |Ib(Xn, βbS)| + log π(S) 2 2 2 |S| ≈ nLn(Xn|βbS) − log(n) + log π(S), 2 where the second approximation is justiﬁed by the fact that when n is suﬃciently large, log(2π) is compa- rably ignorable than log(n), |Ib(Xn, βbS)| converges to a constant, and log π(βbS) can be controlled at a level of O(1). For the prior π(S), if the setting of Liang et al. (2013) is adopted, then we have

|S| pn−|S| π(S) = λn (1 − λn) , (6)

5 where λn denotes the prior probability of each variable to be included in S and takes a value in the form

1 λn = γ √ , (7) 1 + pn 2π for some parameter γ > 0. If the setting of Chen and Chen (2008) is adopted, then we have

p −γ π(S) ∝ n . (8) |S|

For the case that |S| is far smaller than Pn, it is easy to verify that log π(S) = −γ|S| log(pn) + O(1) holds for both prior settings. Therefore,

|S| log{m(Xn|S)π(S)} ≈ nLn(Xn|βbS) − log(n) − γ|S| log(pn), (9) 2

1 which exactly equals to − 2 EBIC. In this article, we propose a new method to approximate the log-integral term in (4). Starting from (9), |S| we further approximate Ln(Xn|βbS) by E{βS |Xn,S}Ln(Xn|βS)+ 2n , where E{βS |Xn,S} denotes the expectation with respect to the posterior distribution of βS. This approximation has been discussed in Spiegelhalter et al. (2002) and can be veriﬁed as follows. It follows from (5), for suﬃciently large n, we have

exp (nLn(Xn|βS)) π(βS) P (βS|Xn,S) = ∝ exp (nLn(Xn|βS)) π(βS) m(Xn|S) 1 0 ≈ exp nLn(Xn|βbS) π(βbS) exp − (βS − βbS) nIb(Xn, βbS) (βS − βbS) , 2 in the neighborhood of βbS where Ln(Xn|βS) is dominant; that is, the posterior of βs is asymptotically −1 normal with mean βbS and covariance matrix nIb(Xn, βbS) . Therefore,

E{βS |Xn,S}Ln(Xn|βS) 1 0 ≈ E{β |X ,S}{Ln(Xn|βbS) − (βS − βbS) Ib(Xn, βbS) (βS − βbS)} S n 2 1 0 = Ln(Xn|βbS) − E{β |X ,S}(βS − βbS) Ib(Xn, βbS) (βS − βbS) 2 S n 1 = Ln(Xn|βbS) − |S|/n, 2

6 and log{m(Xn|S)π(S)} can be approximated as follows

|S| log{m(Xn|S)π(S)} ≈ nLn(Xn|βbS) − log(n) − γ|S| log(pn) 2 |S| |S| ≈ nE L (X |β ) + − log(n) − γ|S| log(p ) {βS |Xn,S} n n S 2 2 n |S| ≈ nE L (X |β ) − log(n) − γ|S| log(p ). {βS |Xn,S} n n S 2 n

|S| |S| Under the large sample setting, 2 is negligible because it is of lower order of 2 log(n).

As with EBIC, we deﬁne AEBIC as -2 times the approximated log{m(Xn|S)π(S)}, i.e.,

AEBIC(s) = −2nE{βS |Xn,S}Ln(Xn|βS) + |S| log(n) + 2γ|S| log(pn). (10)

For many problems, E{βS |Xn,S}Ln(Xn|βS) doesn’t have a closed form, so we use MCMC samples to approximate it. This leads to the following model selection procedure based on the outputs of MCMC:

(t) (a) Run a MCMC algorithm to simulate T samples {(βS(t) ,S ): t = 1, 2,...,T } from the posterior

distribution π(βS,S|Xn).

(t) (b) Let ST denote the set of distinct models appeared in {S : t = 1, 2,...,T }. For each S ∈ ST , calculate

−2n X (t) AEBIC\ (S) = L (X |β (t) ,S ) + |S| log(n) + 2γ|S| log(p ). #{t : S(t) = S} n n S n {t:S(t)=S}

∗ (c) Set Sb = arg minS∈ST AEBIC\ (S) as the estimator for the true model S .

Remark:

• Compared to AIC, BIC and EBIC, AEBIC has at least two merits: First, it avoids to calculate MLE, which can be very diﬃcult for some problems. Second, it conducts a systematic search over the model space via MCMC; however, AIC, BIC and EBIC often employ a heuristic method to search for candidate models, and the search is often limited and incomplete.

• Compared to DIC, AEBIC has also at least two merits: First, AEBIC is invariant to reparameterization, for which the penalty term involves only the model size instead of model parameters; while this is not for DIC. Second, AEBIC is asymptotically consistent, while DIC is not. The asymptotic consistency of AEBIC for high-dimensional model selection will be rigorously proved in the next section.

• For low-dimensional problems, the penalty term 2γ|S| log(pn) can be ignored, and AEBIC is reduced to ABIC.

7 3 Asymptotic Consistency of AEBIC

For any model S ⊂ {1, 2, ..., pn}, we let βS denote the associated vector of parameters, and let f(x|βS) ∗ denote the likelihood function for a single data point x. Let S denote the true model, and let βS∗ denote the (1) (n) parameters of the true model. Assume |supp(βS∗ )| = s0 = o(n) holds. Let Xn = {X ,...,X } denote (i) 1 Pn (i) a set of n iid observations, and let Ln(Xn|βS) := En[log(f(x |βS))] = n i=1 log(f(x |βS)) denote the empirical log-likelihood function, which is an analog of L(βS) := E[log(f(X|βS))]. Denote the maximizer ˆ of Ln(Xn|βS) and L(βS) as βS and βS,0, respectively. For any ﬁxed S with size s := |S| such that s/n → 0, √ ˆ we know that under some general consistency conditions of M-estimator, n(βS −βS,0) −→ N(0,V ), where 2 −1 ∂ log(f(X|βS )) V := E[ T |βS =βS,0 ] is the inverse of the information matrix for model S. Let q denote the ∂βS ∂βS maximum size of the models under consideration, which is allowed to increase with n and p. Also, we assume that the size of the true model is no greater than q, i.e., s0 ≤ q. Let P(q) := {S|S ⊂ {1, 2, ..., pn}, |S| ≤ q} denote the collection of all models with size smaller than or equal to q. To establish consistency of AEBIC, we assume the following set of conditions:

(A2) log(f(x|βS) is three times diﬀerentiable and locally L1-Lipschitz. There exists a function Jf (x) such 2 that Jf (x) ≥ maxS∈P(q) | log(f(x|βS))| and E[Jf (x) ] ≤ C for some generic constant C > 0. The

third order derivative of log(f(x|βS)) is uniformly continuous and bounded on βS and for all S ⊂ P(q).

∂ log(f(x|βS )) Deﬁne f1(x|βS) := . There exists a function J1(x) > |f1(x|βS,0)| for all x and S ∈ P(q) ∂βS 2 such that (i) E[|J1(x)| ] < C1 for some generic constant C1 > 0, and (ii) supx∈X |J1(x)| < C2q,

where X is the domain of observations Xn, and C2 is some generic constant. The domain X satisﬁes (i) (i) supi=1,2,...,n;j=1,2,...,pn |xj | < M for some generic constant M, where xj denotes the jth component of x(i).

(A3) For any S ∈ P(q), the prior density function π(βS|S) is uniformly Lipschitz and bounded from the 0 0 above, i.e., there exists a generic constant L > 0 such that |π(βS|S) − π(βS|S)| ≤ L||βS − βS|| for 0 S any βS, βS ∈ R . Also, there exists a constant c > 0 such that minS∈P(q) π(βS,0|S) > c.

(A4) There exists an envelope function F (x) > 0 such that f(x|βS) < F (x) for all βS with S ∈ P(q), and E[F (x)] ≤ ∞.

(A5) The domain of βS, denoted by N (S), is compact, i.e., there exists a constant M > 0 such that

|βS|∞ ≤ M for any model S ∈ P(q). There exists an > 0 such that B(βS,0) ⊂ N (S), where B(x) denotes the -ball of x.

8 (A6) For any model S ∈ P(q), there exists a unique maximizer βS,0 of L(βS). Furthermore, for any η > 0, sup L(β ) ≤ L(β ) + (cη2 ∧ 1) for some constant c > 0, where a ∧ b = min(a, b). βS ∈/Bη(βS,0) S S,0

q log(p) 1 4 (A7) mini |βi|/( n ) → ∞ as n → ∞.

Condition (A1) is primarily an extension of the sparse eigenvalue condition stated in Bickel et al. (2009). Condition (A2) is required for uniform convergence of the sample average of the loglikelihood function over diﬀerent models S ∈ P(q). Here we assume that all the observations in Xn can be included in a compact set. This assumption is more or less a technical assumption, which has often been used in theoretical studies of Bayesian methods, see e.g. Jiang (2007). The conditions (A3)-(A5) are regularity conditions for f, π and the domain of βS. Condition (A6) imposes a restriction on the behavior of L(βS) which enforces consistency of the maximum likelihood estimator uniformly over all S ∈ P(q). Condition (A7) is for model selection property.

First, we establish a uniform law of large numbers (ULLN) for Ln(Xn|βS), which is modiﬁed from Lemma 1 of Foygel and Drton (2011). The proofs of all lemmas and theorems are deferred to the Appendix.

q2 log(p) Lemma 1. (ULLN) If conditions (A1)-(A6) hold and n → 0 as n → ∞, then with probability going |S| to 1, uniformly for all S ∈ P(q) and βS ∈ R ,

(i) Ln(Xn|βS) →p L(βS), where →p denotes convergence in probability.

ˆ (ii) βS →p βS,0. √ (iii) For any kβS −βS,0k2 ≤ ln/ q, where ln is a decreasing sequence that converges to 0, IˆS(βS) converges

to IS(βS) in Frobenius norm. Consequently, the maximum and minimum eigenvalue of IˆS(βS) is bounded away from 0 and bounded from the above uniformly for all S ∈ P(q).

1 ˆ q log(p) 4 q3 log(p) ˆ ˆ (iv) kβS − βS,0k2 = O n . If n → 0, then IS(βS)’s eigenvalues are bounded away from 0 and bounded from the above uniformly for all S ∈ P(q).

ˆ Together with condition (A1), statement (iv) of Lemma 1 implies that IˆS(βS) has uniformly bounded minimum and maximum eigenvalues for all S ∈ P(q). ˆ ˆ By Lemma 1, uniformly for any S and for any βS ∈/ Bn (βS), Ln(Xn|βS) − Ln(Xn|βS) = L(βS) − 2 ˆ ˆ ∂ f(xi|βS ) L(βS,0) + op(1). Deﬁne IS(βS) := En[ ˆ2 ]kβS . ∂βS

q3 log(p) Lemma 2 (Approximation). Suppose the conditions (A1)-(A6) hold. If n → 0 as n → ∞, then for any suﬃciently large n and a decreasing sequence ln that converges to 0, with probability going to 1, uniformly over all S ∈ P(q), we have q ˆ q log(p) (i) Ln(Xn|βS) − Ln(βS) ≤p L(βS) − L(βS,0) + O( n ).

9 ˆ √ln (ii) For any βS such that ||βS − βS|| ≤ q , then

r ! 1 √ q2 log(p) L (X |β )−L (X |βˆ ) = − (β −βˆ )T Iˆ (βˆ )(β −βˆ )+O qkβ − βˆ k3 ∨ kβ − βˆ k2 . n n S n n S 2 S S S S S S p S S n S S

Recall that the posterior density function of βS, given model S, is given by

P (βS|Xn,S) = exp{nLn(Xn|βS)}π(βS)/m(Xn|S),

R where m(Xn|S) := exp{nLn(Xn|βS)}π(βS)dβS. Without loss of generality, we assume that π(S) = 0 βS for any s 6∈ P(q). Based on the previous two lemmas, we can now establish our main results. Theorem ˆ ˆ 1 shows that the posterior mean of log P (βS|Xn,S) can be approximated by log P (βS|Xn,S), where βS denotes the MLE of βS. This theorem forms a theoretical foundation for the consistency of AEBIC.

Theorem 1 (Posterior mean approximation of the log-likelihood function). Suppose the conditions (A1)- q4 log2(n) log(p) (A6) hold and n → 0 as n → ∞. Then for suﬃciently large n, with probability going to 1, uniformly over all S ∈ P(q), we have

 s  |S| |S| q4 log2 n log(p) E [L (X |β )] = L (X |βˆ ) − + O , βS |Xn,S n n S n n S 2n  n n 

ˆ where βS denotes the MLE of βS.

Lemma 3 establishes an upper bound for the maximum likelihood values of overﬁtted models.

Lemma 3. Assume the conditions (A1)-(A6) hold. For any model S such that |S| ≤ q, S∗ ⊂ S and q4 log2(p) n → 0, the following statement holds uniformly for such models with probability going to 1,

ˆ ˆ ∗ ζ 2n(Ln(Xn|βS) − Ln(Xn|βS∗ )) ≤ (1 + η)|S\S | log(n p),

1 ˆ where η > 0 is an arbitrarily small constant, ζ > 5 is any ﬁxed constant, and βS∗ denotes the MLE of βS∗ .

Finally, we have the following theorem for the consistency of AEBIC in model selection.

Theorem 2 (Model Selection Consistency). Assume that the conditions (A1)-(A7) are satisﬁed. If γ > 1, q6 log2(n) q4 log2(p) n log(p) → 0, and n → 0, then for suﬃciently large n, with probability going to 1, it holds that for all models S with |S| ≤ q and S 6= S∗,

AEBIC(S∗) < AEBIC(S), where AEBIC(S) is as deﬁned in (10).

10 4 Applications of ABIC and AEBIC to Generalized Linear Models

In this section we consider the applications of ABIC and AEBIC to generalized linear models (GLMs) (Mc-

Cullagh and Nelder, 1989). Let X = (y, z), where y denotes the response variable and z = (z1, z2, . . . , zp) denote a vector of predictors. Suppose that the distribution function of X is given by

f(y|z, β) = exp {a(θ)y + b(θ) + c(y)} , (11) where a(·) and b(·) are continuously diﬀerentiable functions of θ, c(y) is a constant function of y, θ is the T so-called natural parameter that relates y to the predictors via a linear function θ = z β = β1z1+···+βpzp, where β1, . . . , βp are regression coeﬃcients. Here, the intercept term has been treated as a special predictor included in z. The mean function u = E(y|z) = −b0(θ)/a0(θ) := ψ(θ), where ψ(θ) is the inverse of a chosen link function. This class of GLMs includes regression models with the responses that are binary, Poisson, and Gaussian (with known variance), whose respective link functions are logit, log and linear. For this class of GLMs, we follow Liang et al. (2013) to assume that each model S is subject to the following prior distribution

|S| pn−|S| π(S) ∝ λn (1 − λn) I(|S| ≤ q), (12) where λn is chosen according to (7). Further, we let βS be subject to a Gaussian prior distribution 2 N(0, σSI|S|), where I|S| denotes a |S| × |S|-matrix and

1 σ2 = eC0/|S|, (13) S 2π for some positive constant C0. As argued in Liang et al. (2013), under such prior setting, we have log π(βS) = O(1), which satisﬁes the requirement of unit prior information in derivation of BIC (Ando, 2010). The unit prior information is also implicitly required in condition (A3). Then, with an appropriate choice of q, which works as a sparsity constraint for the true model, we can verify that the GLM (11) satisﬁes the conditions (A1)-(A7). We note that under a set of slightly weaker but less general conditions, we have also proved the consistency of AEBIC for GLMs. Refer to the supplementary material of the paper for the detail.

4.1 Low-Dimensional Case

We ﬁrst work on some low-dimensional problems. In this case, ABIC, which is deﬁned in (2), is used as the model selection criterion.

Logistic Regression Consider a logistic regression with three active predictors

logit P (y = 1|z) = 2z1 + z2 + 3z3, (14)

11 for which the total number of candidate predictors is p = 50 and the sample size is n = 500. All candidate predictors are generated from the standard Gaussian distribution. For the prior π(S), we set γ = 0 in (7) and ignored the upper bound q; and for the prior π(βS), we set C0 = 10 in (13).

To simulate from the posterior distribution of (S, βS), we implemented the reversible jump MCMC algorithm (Green, 1995), which includes three types of moves, birth, death and parameter updating. In the birth step, we randomly choose a predictor excluded from the current model S, and include it into S to produce a larger model S0. The regression coeﬃcient of the new predictor is generated from the Gaussian distribution N(0, 102). In the death step, we randomly choose a predictor included in the current model S, and remove it from S to produce a smaller model S0. In the parameter updating step, we keep

S unchanged, but randomly choose one coefficient βi, i ∈ S, to update with a Gaussian random walk proposal for which the variance is set to 0.52. To accelerate the convergence of the simulation, an advanced MCMC algorithm, such as stochastic approximation Monte Carlo (Liang et al., 2007), can be used. The algorithm was run for 106 iterations, where the first 105 iterations were discarded for the burn-in process and the posterior samples were collected at every 10th iteration from the remaining part of the run. Based on the collected samples, ABIC was calculated for different models according to the procedure given in Section 2. Based on the same set of posterior samples, we also calculated AIC,BIC and DIC for different models. For simplicity, we only considered the models with the sampling frequency exceeding 100. The simulation was repeated for 100 times, where a different dataset was generated at each time. The results were summarized in Table 1. The performance of the methods was evaluated using three metrics:

(i) mean (µ ) and standard deviation (σ ) of |Si|, where Si denotes the model selected in the ith run; |Sbi| |Sbi| b b P100 ∗ ∗ P100 ∗ P100 (ii) recall= [ i=1 |S ∩ Sbi|]/[|S | · 100]; and (iii) precision= [ i=1 |S ∩ Sbi|]/[ i=1 |Sbi|].

Table 1: Comparison of AIC, BIC, DIC and ABIC for low dimensional logistic regression.

AIC BIC DIC ABIC µ (σ ) 8.86(1.99) 3.72(0.96) 9.04(2.15) 3.33(0.57) |Sbi| |Sbi| Recall 1.000 1.000 1.000 1.000 Precision 0.339 0.806 0.332 0.901

Table 1 indicates that the simulation results agree well with the theoretical results: BIC and ABIC are consistent for model selection, while AIC and DIC are not and they tend to select larger models. It is remarkable that ABIC also signiﬁcantly outperforms BIC in terms of model selection accuracy, i.e. µ |Sbi| and precision.

Linear Regression Next we consider a linear regression with ﬁve active predictors

y = z1 + 2.2z2 − 1.6z3 + 2z4 − 1.4z5 + , (15)

12 where follows the standard normal distribution. For this example, we also set p = 50 and n = 500 and generated all predictors from the standard normal distribution. In addition, we speciﬁed the same prior distributions and employed the same MCMC algorithm as for the logistic regression example. Table 2 summarizes the results for 100 independent runs. For this example, similar conclusions can be made as for the logistic example: AIC and DIC tend to select larger models, BIC and ABIC tend to select correct models, and ABIC outperforms all the other criteria.

Table 2: Comparison of AIC, BIC, DIC and ABIC for low dimensional linear regression.

AIC BIC DIC ABIC µ (σ ) 8.70(1.15) 5.59(0.82) 9.05(1.08) 5.35(0.74) |Sbi| |Sbi| Recall 1.000 1.000 1.000 1.000 Precision 0.575 0.894 0.552 0.934

4.2 High Dimensional Case

For the high-dimensional case, AEBIC is used as the model selection criterion.

Logistic Regression Consider the logistic regression (14) again. Here we increased p from 50 to 2000, while keeping the sample size n = 500 unchanged. For the prior π(S), we tried four diﬀerent γ values

0.5, 0.6, 0.7, and 0.8, and ﬁxed the upper bound q to 50. For the prior π(βS), we still set C0 = 10. The simulations were done as for the previous examples. Based on the collected samples, we calculated both EBIC and AEBIC for the models with the sampling frequency greater than 20. AIC, BIC and DIC are not applicable to high-dimensional problems. Table 3 summarizes the results for 100 independent runs.

Table 3: Comparison of EBIC and AEBIC for high-dimensional logistic regression.

γ 0.5 0.6 0.7 0.8 µ (σ ) 3.60(0.96) 3.21(0.46) 3.15(0.43) 3.07(0.30) |Sbi| |Sbi| EBIC Recall 1.000 1.000 1.000 1.000 Precision 0.833 0.935 0.949 0.977 µ (σ ) 3.39(0.89) 3.11(0.31) 3.06(0.28) 3.02(0.20) |Sbi| |Sbi| AEBIC Recall 1.000 1.000 1.000 0.997 Precision 0.885 0.965 0.977 0.990

The comparison indicates that as γ increases, both EBIC and AEBIC tend to select more sparse models. This is due to that a larger value of γ imposes a heavier penalty on the model size. It is remarkable that AEBIC is uniformly better than EBIC for all values of γ in terms of model size and true discovery rate.

Linear Regression We reconsidered the linear regression (15) with an enlarged value of p = 2000, while keeping other settings unchanged. We adopted the same prior setting as for the high-dimensional logistic regression example. For simulation, the MH algorithm was slightly modiﬁed for this example. The

13 modification is in the birth step, where we no longer choose new predictors uniformly, instead, we choose each predictor with a probability proportional to |corr(y, xk)|. Here corr(·) denotes the Pearson correlation of two random variables. This modification can significantly accelerate the convergence of the simulation. Table 4 summarizes the results for 100 independent runs. The comparison indicates again that AEBIC outperforms EBIC in terms of model size and true discovery rate.

Table 4: Comparison of EBIC and AEBIC for high-dimensional linear regression.

γ 0.5 0.6 0.7 0.8 µ (σ ) 5.30(0.54) 5.15(0.39) 5.07(0.26) 5.03(0.17) |Sbi| |Sbi| EBIC Recall 1.000 1.000 1.000 1.000 Precision 0.943 0.971 0.986 0.994 µ (σ ) 5.16(0.42) 5.06(0.24) 5.05(0.22) 5.01(0.10) |Sbi| |Sbi| AEBIC Recall 1.000 1.000 1.000 1.000 Precision 0.969 0.988 0.990 0.998

4.3 A Real Data Example

We considered a dataset collected by Singh et al. (2002), which consists of the expression values of 6033 genes on 102 samples, 52 prostate cancer patients and 50 controls. The dataset is downloadable at http: //statweb.stanford.edu/~ckirby/brad/LSI/datasets-and-programs/datasets.html, and has been analyzed in quite a few papers, see e.g., Chen and Chen (2012), Efron (2009), and Liang et al. (2013). Our aim is to identify important genes that are associated with the prostate cancer via high-dimensional logistic regression.

For the prior π(S), we set γ = 0.7 and ﬁx the upper bound q to 50. For the prior π(βS), we set

C0 = 10. The MH algorithm was slightly modiﬁed from the one used before. The modiﬁcation is in the birth step, where we no longer choose new predictors uniformly, instead, we choose each predictor with a probability proportional to the weight wk = Null Deviance − Deviancek + 0.1, where k indexes the genes, “Null Deviance” denotes the deviance of the null model which includes the intercept term only, and

Deviancek denotes the deviance of the model which includes the intercept term and gene k only. The algorithm was run for 107 iterations, where the ﬁrst 106 iterations were discarded for the burn-in process and the samples were collected at every 10th iterations from the remaining part of the run. Based on the collected samples, AEBIC were calculated for each model with the sampling frequency greater than 10. For this example, we observed multiple models with very similar AEBIC values. As in Chen and Chen (2012), we ranked genes based on their appearance frequencies in top 100 models. It is interesting to point out that our rank has much overlap with the rank given in Chen and Chen (2012), which was established based on EBIC but with a diﬀerent model searching procedure. Table 5 gives top 10 genes reported in Chen and Chen (2012) and their corresponding ranks by our method.

14 Table 5: Top 10 genes ranked by EBIC in Chen and Chen (2012) and their corresponding ranks by AEBIC, where the genes are labeled by their column numbers in the dataset.

Gene v610 v1720 v332 v364 v1068 v914 v3940 v1077 v4331 v579 EBIC 1 2 3 4 5 6 7 8 9 10 AEBIC 1 15 8 6 10 5 3 7 4 19

5 Conclusion

In this paper, we have proposed ABIC and AEBIC as general Bayesian model selection criteria for low and high-dimensional problems, respectively; established their consistency under mild conditions; and illustrated their applications using generalized linear models. The numerical results indicate that the proposed criteria can signiﬁcantly outperform the existing ones, such as AIC, BIC, DIC and EBIC, in terms model selection accuracy. Compared to AIC, BIC and EBIC, the proposed criteria avoid to calculate MLEs, which can be very diﬃcult for some problems. Also, the MCMC simulations required by the proposed criteria provide a systematic way to search for the best ABIC or AEBIC models. While the AIC, BIC or EBIC methods often employ a heuristic method to search over the model space, and the search is often limited and incomplete. ABIC and AEBIC have also overcome many limitations of DIC. As explained previously, they both are invariant to reparameterization and asymptotically consistent, while DIC is not. Given these attractive features, we expect that ABIC and AEBIC will be widely adopted by researchers for Bayesian model selection in the near future.

Acknowledgement

Liang’s research was partially supported by grants DMS-1612924 and R01-GM117597.

Appendix A Technical Details

(1) (n) In the appendix, we use Xn = {X ,...,X } to denote a set of n observations, use Pn to denote the empirical measure of the observations, use Ln(Xn|βS) to denote the averaged log-likelihood function as deﬁned in (1), use S(Xn, βS) to denote the score function ∂Ln(Xn|βS)/∂βS, and use IˆS(βS)(or IˆS) to denote 2 T the observed information matrix −∂ Ln(Xn|βS)/∂βS∂βS . For simplicity, we also depress the subscript of pn and replace it by p.

A.1 Proof of Lemma 1

|S| Part (i) Define FS(M) := {log f(x|βS)1(F ≤ M): βS ∈ R }, where F is defined in (A4). Define

Fq(M) := ∪|S|≤q,S⊂{1,2,...,p}FS(M). It is easy to see that for any s ≤ s0, by the Lipschitz condition

15 of log f(x|β) and condition (A6), the covering number of FS(M) relative to the L1(Pn)-norm, which is denoted by N(, FS(M),L1(Pn)), satisﬁes

1 N (, F (M),L (P )) ≤ C , S 1 n 0 |S| for some generic constant C0 > 0, where Pn denotes the empirical measure of n observations.

Therefore, the covering number of Fq(M) satisﬁes

q X 1 N (, F (M),L (P )) ≤ CsC , q 1 n p 0 s s=1

s where Cp is the combinatorial number of choosing s from p. Thus,

q X s log N (, Fs(M),L1(Pn)) ≤ log Cp + log(C0) + q log(1/) - q log p + q log(1/) = o(n). s=1

Therefore, by Uniform Law of Large Numbers (Theorem 2.4.3 of Van der Vaart and Wellner (1997)),

|Ln(Xn|βS) − L(βS)| →p 0, holds uniformly for all S ∈ P(q).

Part (ii) It follows directly from the conclusion of part (i) and condition (A7).

Part (iii) First, by the continuity condition stated in condition (A2), it is easy to see that uniformly over all S ∈ P(q), kIS(βS) − IS(βS,0)k∞ ≤ LkβS − βS,0k holds for some constant L. Hence,

√ √ kIˆS(βS) − IˆS(βS,0)k2 ≤ qkIˆS(βS) − IˆS(βS,0)k∞ ≤ L q||βS − βS,0||. (16)

kβS −βS,0k ˆ For any βS such that q → 0, the eigenvalue of IS(βS) is bounded away from 0 and bounded from the above uniformly if IˆS(βS,0)’s eigenvalues have the same property. Therefore, it suffices to show a ULLN that IˆS(βS,0) converges in probability to IS(βS,0) uniformly in Frobenius norm, for any S ∈ P(q). ∞ Again, we show it by first examining the convergence of IˆS(βS,0) to IS(βS,0) under L norm. By con- 2 2 dition (A1), J(x) performs as an envelop function of ∂ ∂ log f(x|βS,0) . Therefore, by the Lipschitz condition ∂βi∂βj 2 2 of ∂ log(f(x|βS,0)) and the existence of function J, the covering number of F (q) := { ∂ log(f(x|βS,0)) |i, j ∈ ∂βi∂βj I ∂βi∂βj S, S ∈ P(q)} satisfies q ! X 2 s log N(, FI (q),L2(Pn)) - log s Cp + q log(1/), s=1 for any > 0.

16 √ ∗ ˆ By Theorem 2.5.2 of Van der Vaart and Wellner (1997), P ( n maxS∈P(q) kIS(βS,0 − IS(βS,0)k∞) > 2 R 1 p 2 p ∗ x) ≤ C x =0 log N(, FI (q),L2(Pn)) ≤ C1 x q log(p), where P (A) denotes the outer probability of an p event A and C,C1 are some constant that only depends on the function J. Setting x = K q log(p) for any q ∗ ˆ q log(p) 2C1 ˆ K > 0, we have P (maxS∈P(q) ||IS(βS,0) − IS(βS,0)||∞) > K n ) < K , i.e., maxS∈P(q) ||IS(βS,0) − q q log(p) IS(βS,0)||∞) = Op( n ).

Since the Frobenius norm of IˆS(βS,0) − IS(βS,0) satisﬁes

r ! √ q2 log(p) kIˆ (β ) − I (β )k ≤ qkIˆ (β ) − I (β )k = O , (17) S S,0 S S,0 2 S S,0 S S,0 ∞ p n uniformly for all S ∈ P(q), we have

r √ q2 log(p) kIˆ (β ) − I (β )k qkβ − β k + , S S S S,0 2 - S S,0 n by combining inequality (16) and inequality (17). By the assumption that q2 log(p)/n → 0, the statement (iii) holds.

ˆ ˆ 2 ˆ Part (iv) By condition (A6), L(βS) ≥ L(βS,0) + c||βS − βS,0|| . On the other hand, Ln(βS) ≤ Ln(βS,0).

By the Lipschitz condition of log f(x|βS) and the existence of function Jf , the covering number of

F(q) := {log f(x|βS): |S| ≤ q} satisﬁes

q X s log N(, F(q),L2(Pn)) - log( Cp) + q log(1/), s=1 for any > 0. ∗ √ By Theorem 2.5.2 of Van der Vaart and Wellner (1997), P (maxS∈P(q) n(Ln(Xn|βS) − L(βS)| > x) ≤ 2 R 1 p 2 p ∗ C2 x =0 log N(, F(q),L2(Pn)) ≤ C3 x q log(p), where P (A) denotes the outer probability of an event p A and C2,C3 are some constant that only depends on the function Jf . Setting x = K q log(p) for any K > 0, we have r ∗ q log(p) 2C3 P ( max |Ln(Xn|βS) − L(βS)| > K ) ≤ , S∈P(q) n K q q q log(p) ˆ ˆ q log(p) i.e., maxS∈P(q)(Ln(Xn|βS) − L(βS) = O( n ). Consequently, L(βS) ≤ Ln(βS) + Op( n ) ≤ q q log(p) q q log(p) Ln(βS,0) + Op( n ) ≤ L(βS,0) + O( n ). This implies that

r q log(p) ckβˆ − β k2 = O ( ), (18) S S,0 p n

1 ˆ q log(p) 4 i.e., ||βS − βS,0|| ≤ n .

17 1 q3 log(p) q log(p) 4 √1 ˆ ˆ If n → 0, then n = o( q ), i.e., the condition for statement (iii) holds. Hence, IS(βS)’s eigenvalues are bounded away from 0 and bounded from the above uniformly over all S ∈ P(q).

A.2 Proof of Lemma 2

ˆ ˆ Part (i) By the optimality of βS, we have Ln(Xn|βS) − Ln(βS) ≤ Ln(Xn|βS) − Ln(βS,0). By Uniform

Central Limit Theorem obtained in the proof of Statement (iv) of Lemma 1, Ln(Xn|βS) − Ln(βS,0) ≤ q q log(p) L(βS) − L(βS,0) + O( n ). Therefore,

r q log(p) L (X |β ) − L (βˆ ) ≤ L(β ) − L(β ) + O( ). n n S n S S S,0 n

ˆ Part (ii) Consider the local Taylor expansion of Ln(Xn|βS) around βS,

1 L (X |β ) = L (X |βˆ ) − (β − βˆ )T Iˆ (β∗ )(β − βˆ ), n n S n n S 2 S S S S S S

∗ ˆ ˆ ∗ ˆ ˆ where βS is a point between βS and βS. By statement (iii) of Lemma 1, IS(βS) = IS(βS) + op(1) uniformly ˆ √1 with respect to the Frobenius norm when ||βS − βS|| = o( q ). ˆ ∗ The proof of statement (iii) of Lemma 1 in fact provides a bound for the Frobenius norm of IS(βS) − ˆ ˆ ˆ 1 ˆ T ˆ ˆ ˆ 2 √ ˆ IS(βS). Therefore, Ln(Xn|βS) − Ln(Xn|βS) = − 2 (βS − βS) IS(βS − βS) + ||βS − βS|| O( q||βS − βS|| ∨ q q2 log(p) n ).

A.3 Proof of Theorem 1

For suﬃciently large n, with probability going to 1, all statements in Lemma 1 and Lemma 2 hold and the following proof is based on these statements. First, we note that

R where P (βS|Xn,S) = exp(nLn(Xn|βS))π(βS)/m(S) and m(S) = exp(nLn(Xn|βS))π(βS)dβS. βS ˆ We split the integral domain in (19) into three regions: a small neighborhood around βS, denoted q ˆ Kq log(n) ˆ by BK (βS), where K = n ; the area between the small neighborhood BK (βS) and a larger 1 q3 log(p) 8 ˆ ˆ ˆ √ln neighborhood Bn (βS), denoted by BK (βS) − Bn (βS), where n = q and ln = n → 0; and the c ˆ rest area Bn (βS). ˆ 3 √1 For the neighborhood BK (βS), since q log(p)/n → 0, we have K = o( q ). Applying Lemma 2, we ˆ obtain that, with probability going to 1, for any βS ∈ BK (βS),

1 L (X |β ) − L (X |βˆ ) = − (β − βˆ )T Iˆ (β − βˆ ) + r (β )||β − βˆ ||2, n n S n n S 2 S S S S S n S S S 2

18 q q √ ˆ q2 log(p) q2 log(p) where |rn(βS)| ≤ C( qkβS − βSk2 ∨ n ) = C n uniformly for some constant C > 0. Similarly, we have

1 exp(nL (X |β ))π(β ) = exp(nL (X |βˆ )) exp(−n (β −βˆ )T Iˆ (β −βˆ )) exp(nr (β )||β −βˆ ||2)π(β ), n n S S n n S 2 S S S S S n S S S 2 S

q q 2 ˆ 2 q2 log(p) Kq log(n) q4 log (n) log(p) ˆ 2 where nrn(βS)||βS − βS||2 ≤ n n n - n and thus exp(nrn||βS − βS||2) = (1 + q q4 log2(n) log(p) O( n )).

Also, for any βS ∈ BK , by condition (A3), we have

r q log(n) π(β ) = π(βˆ ) + O(||β − βˆ || ) = π(βˆ )(1 + O( ). S S S S 2 S n

q q4 log2(n) log(p) q q log(n) Since n dominates n , we ﬁnally have

n exp(nL (X |β ))π(β ) = π(βˆ ) exp nL (X |βˆ ) − (β − βˆ )T Iˆ (β − βˆ ) n n S S S n n S 2 S S S S S  s  q4 log2(n) log(p) (20) × 1 + O( ) .  n 

ˆ ˆ √ln √1 For any βS ∈ BK (βS) − Bn (βS), since n = q = o( q ), we can apply Lemma 2 again and get √ ˆ |rn(βS)| ≤ q||βS − βS||2 → 0. Therefore

1 L (X |β ) − L (X |βˆ ) = − (β − βˆ )T Iˆ (β − βˆ ) + r (β )||β − βˆ ||2 n n S n n S 2 S S S S S n S S S 2 1 c = − (β − βˆ )0Iˆ (β − βˆ )(1 + o(1)) ≤ − 1 ||β − βˆ ||2, 2 S S S S S 4 S S where c1 > 0 is the lower bound of the minimum eigenvalue of IˆS for all S ∈ P(q). Hence, for suﬃciently large n, we have

nc exp(nL (X |β ))π(β ) ≤ exp(nL (X |βˆ )) exp(− 1 ||β − βˆ ||2)π(β ). (21) n n S S n n S 4 S S S

c ˆ For any βS ∈ Bn (βS), by statement (i) of Lemma 2, we have

r q log(p) L (X |β ) − L (X |βˆ ) ≤ L(β ) − L(β ) + O( ). n n S n n S S S,0 n

By condition (A6), we also have

2 ˆ 2 cln L(βS) − L(βS,0) ≤ −c min( inf kβS − βSk , 1) ≤ − . c βS ∈Bn (βS ) q

19 q 1 2 q log(p) q3 log(p) 8 Note that the quantity ln/q dominates n because ln = n . c ˆ Hence, for any βS ∈ B (βS), we have

cl2 exp(nL (X |β ))π(β ) ≤ exp(nL (X |βˆ ))π(β ) exp{−n( n )(1 + o(1)))} n n S S n n S S q (22) c0nl2 ≤ π(β ) exp(nL (X |βˆ )) exp(− n ), S n n S q for some generic constant c0 > 0. Now let’s calculate m(S) by dividing it into three parts according to the previous partition of the integral region, that is,

Z m(S) := exp(nLn(Xn|βS))π(βS)dβS Z Z = exp(nLn(Xn|βS))π(βS)dβS + exp(nLn(Xn|βS))π(βS)dβS β ∈B (βˆ ) β ∈B (βˆ )−B (βˆ ) S K S S n S K S (23) Z + exp(nLn(Xn|βS))π(βS)dβS c ˆ βS ∈B (βS ) = (Int1) + (Int2) + (Int3).

The ﬁrst term of equation (23) satisﬁes:

Z Int1 : = exp(nLn(Xn|βS))π(βS)dβS ˆ βS ∈BK (βS ) s q4 log2(n) log(p) ≤ exp(nL (X |βˆ ))π(βˆ )(1 + O( )) n n S S n Z n ˆ T ˆ exp(− (βS − βS) IˆS(βS − βS))dβS, ˆ T ˆ ˆ Kq log(n) 2 (βS −βS ) IS (βS −βS )≤c2 n √ 1/2 ˆ where c2 is the upper bound of the maximum eigenvalue of IˆS. Let ξ = nIˆ (βS − βS), then we have

Z n ˆ T ˆ exp(− (βS − βS) IˆS(βS − βS))dβS ˆ T ˆ ˆ Kq log(n) 2 (βS −βS ) IS (βS −βS )≤c2 n Z 2π |S| − 1 − |S| 1 T 2π |S| − 1 = ( ) 2 |IˆS| 2 (2π) 2 exp(− ξ ξ)dξ ≤ ( ) 2 |IˆS| 2 , T n ξ ξ≤c2Kq log(n) 2 n

|S| − 2 1 T where the last inequality is derived from the fact that (2π) exp(− 2 ξ ξ) exactly equals the density function of N(0,I|S|). Now we obtain the upper bound of Int1

20 Similarly, we have

s q4 log2(n) log(p) Int1 : ≥ exp(nL (X |βˆ ))π(βˆ )(1 − O( )) n n S S n Z n ˆ T ˆ exp(− (βS − βS) IˆS(βS − βS))dβS, ˆ T ˆ ˆ Kq log(n) 2 (βS −βS ) IS (βS −βS )≤c1 n √ 1/2 ˆ where c1 is the lower bound of the minimum eigenvalue of IˆS. Let ξ = nIˆ (βS − βS), this time we have

Z n ˆ T ˆ exp(− (βS − βS) IˆS(βS − βS))dβS ˆ T ˆ ˆ Kq log(n) 2 (βS −βS ) IS (βS −βS )≤c1 n Z 2π |S| − 1 − |S| 1 T = ( ) 2 |IˆS| 2 (2π) 2 exp(− ξ ξ)dξ T n ξ ξ≤c1Kq log(n) 2 Z 2π |S| − 1 − |S| 1 T 2π |S| − 1 1 ≥ ( ) 2 |Iˆ | 2 (2π) 2 exp(− ξ ξ)dξ ≥ ( ) 2 |Iˆ | 2 (1 − ), S S |S| n ξT ξ≤5|S| log(n) 2 n n where the ﬁrst inequality holds if we choose K > 5c1, and the second inequality is based on the Lemma 1 of Foygel Barber et al. (2015). Now we obtain the lower bound of Int1

s 4 2 q log (n) log(p) 2π |S| − 1 1 Int1 ≥ exp(nL (X |βˆ ))π(βˆ )(1 − O( ))( ) 2 |Iˆ | 2 (1 − ). n n S S n n S n|S|

q 4 2 Since q log (n) log(p) dominates 1 , therefore we have n n|S|

s 4 2 2π |S| − 1 q log (n) log(p) Int1 = exp(nL (X |βˆ ))π(βˆ )( ) 2 |Iˆ | 2 (1 + O( )). n n S S n S n

The second term of equation (23) satisﬁes

Z Int2 : = exp(nLn(Xn|βS))π(βS)dβS βS ∈Bn −BK Z nc ≤ exp(nL (X |βˆ )) π(β ) exp(− 1 ||β − βˆ ||2)dβ n n S S 4 S S S βS ∈Bn −BK nc Kq log(n) Z ≤ exp(nL (X |βˆ )) exp(− 1 ) π(β )dβ n n S 4 n S S βS ∈Bn −BK ˆ 1 ≤ C5 exp(nLn(Xn|βS)) . nKqc1/4

21 The third term of equation (23) satisﬁes

Z Int3 : = exp(nLn(Xn|βS))π(βS)dβS c ˆ βS ∈Bn (βS ) c0nl2 Z ≤ exp(nL (X |βˆ )) exp(− n ) π(β )dβ n n S q S S c0nl2 ≤ exp(nL (X |βˆ )) exp(− n ). n n S q

Combining them together, we have

m(S) = Int1 + Int2 + Int3 s 4 2 2π |S| − 1 q log (n) log(p) = exp(nL (X |βˆ ))π(βˆ )( ) 2 |Iˆ | 2 (1 + O( )) n n S S n S n 0 2 ˆ 1 ˆ c nln + exp(nLn(Xn|βS))O( ) + exp(nLn(Xn|βS))O(exp(− )) nKqc1/4 q s 4 2 2π |S| − 1 q log (n) log(p) = exp(nL (X |βˆ ))π(βˆ )( ) 2 |Iˆ | 2 1 + O( ) n n S S n S n 0 2 −1 ˆ n |S| − 1 1 −1 ˆ n |S| − 1 c nln +π (βS)( ) 2 |IˆS| 2 O( ) + π (βS)( ) 2 |IˆS| 2 O(exp(− )) . 2π nKqc1/4 2π q

By choosing K > 2/c1, it can be veriﬁed that

s 4 2 −1 ˆ n |S| − 1 1 q log (n) log(p) π (βS)( ) 2 |IˆS| 2 O( ) = o( ), 2π nKqc1/4 n and s 0 2 4 2 −1 n |S| − 1 c nln q log (n) log(p) π (βˆ )( ) 2 |Iˆ | 2 O(exp(− )) = o( ). S 2π S q n Therefore, we ﬁnally have

s 4 2 2π |S| − 1 q log (n) log(p) m(S) = exp(nL (X |βˆ ))π(βˆ )( ) 2 |Iˆ | 2 1 + O( ) . n n S S n S n

ˆ ˆ Now we return to the calculation of EβS |Xn,S[Ln(Xn|βS)] − Ln(Xn|βS)). That is,

Z Z 1 P (β |X ,S)(L (X |β ) − L (X |βˆ ))dβ = − P (β |X ,S) (β − βˆ )0Iˆ (β − βˆ )dβ S n n n S n n S S S n 2 S S S S S S BK Z Z ˆ 2 ˆ + P (βS|Xn,S)rn(βS)||βS − βS|| dβS + P (βS|Xn,S)(Ln(Xn|βS) − Ln(Xn|βS))dβS B B −B K n K (24) Z ˆ + P (βS|Xn,S)(Ln(Xn|βS) − Ln(Xn|βS))dβS c Bn = (I1) + (I2) + (I3) + (I4).

22 The ﬁrst component of equation (24) is

1 Z −1 (I1) = exp(nL (X |β ))π(β ) (β − βˆ )0Iˆ (β − βˆ )dβ m(S) n n S S 2 S S S S S S BK s 1 q4 log2(n) log(p) = exp(nL (X |βˆ ))π(βˆ )(1 + O( )) m(S) n n S S n Z 1 1 exp(−n (β − βˆ )T Iˆ (β − βˆ ))(−1) (β − βˆ )0Iˆ (β − βˆ )dβ . 2 S S S S S 2 S S S S S S BK √ 1/2 ˆ Let ξ = nIˆ (βS − βS), then we have

Z 1 1 exp(−n (β − βˆ )T Iˆ (β − βˆ )) (β − βˆ )0Iˆ (β − βˆ )dβ 2 S S S S S 2 S S S S S S BK Z 1 ˆ T ˆ 1 ˆ 0 ˆ ≤ exp(−n (βS − βS) IˆS(βS − βS)) (βS − βS) IˆS(βS − βS)dβS ˆ T ˆ ˆ Kq log(n) 2 2 (βS −βS ) IS (βS −βS )≤c2 n Z 2π |S| − 1 − |S| 1 T 1 T 2π |S| − 1 1 = ( ) 2 |IˆS| 2 (2π) 2 exp(− ξ ξ) ξ ξdξ ≤ ( ) 2 |IˆS| 2 |S|. T n ξ ξ≤c2Kq log(n) 2 2n n 2n

We also have

Z 1 1 exp(−n (β − βˆ )T Iˆ (β − βˆ )) (β − βˆ )0Iˆ (β − βˆ )dβ 2 S S S S S 2 S S S S S S BK Z 1 ˆ T ˆ 1 ˆ 0 ˆ ≥ exp(−n (βS − βS) IˆS(βS − βS)) (βS − βS) IˆS(βS − βS)dβS ˆ T ˆ ˆ Kq log(n) 2 2 (βS −βS ) IS (βS −βS )≤c1 n Z 2π |S| − 1 1 − |S| 1 T T = ( ) 2 |IˆS| 2 (2π) 2 exp(− ξ ξ)ξ ξdξ T n 2n ξ ξ≤c1Kq log(n) 2 Z 2π |S| − 1 1 − |S| 1 T T = ( ) 2 |IˆS| 2 |S| − (2π) 2 exp(− ξ ξ)ξ ξdξ . T n 2n ξ ξ≥c1Kq log(n) 2

1 T T For suﬃciently large n, exp( 4 ξ ξ) ≥ ξ ξ, so Z − |S| 1 T T (2π) 2 exp(− ξ ξ)ξ ξdξ T ξ ξ≥c1Kq log(n) 2 Z − |S| 1 T ≤ (2π) 2 exp(− ξ ξ)dξ T ξ ξ≥c1Kq log(n) 4 s |S|/2 |S|−1 4 2 − |S| 4(π) [c1Kq log(n)] 1 q log (n) log(p) ≤ (2π) 2 exp(− c Kq log(n)) = o( ), Γ(|S|/2) 1/4 4 1 n where the second inequality is based on the Lemma 2 of Foygel and Drton (2011). Therefore,

Z 1 1 exp(−n (β − βˆ )T Iˆ (β − βˆ )) (β − βˆ )0Iˆ (β − βˆ )dβ 2 S S S S S 2 S S S S S S BK s 4 2 2π |S| − 1 |S| q log (n) log(p) = ( ) 2 |Iˆ | 2 (1 + o( )), n S 2n n

23 and s 4 2 1 q log (n) log(p) 2π |S| − 1 −|S| (I1) = exp(nL (X |βˆ ))π(βˆ )(1 + O( ))( ) 2 |Iˆ | 2 m(S) n n S S n n S 2n

q q4 log2(n) log(p) |S| 1 −|S| ˆ ˆ 2π 2 ˆ − 2 exp(nLn(Xn|βS))π(βS)(1 + O( n ))( n ) |IS| 2n = (25) |S| 1 q q4 log2(n) log(p) ˆ ˆ 2π 2 ˆ − 2 exp(nLn(Xn|βS))π(βS)( n ) |IS| 1 + O( n ) s |S| q4 log2(n) log(p) = − (1 + O( )). 2n n

For the second component of equation (24), we can show

Z ˆ 2 |I2| = P (βS|Xn,S)rn(βS)kβS − βSk dβS

BK r s q log(n) q2 log(p) Z C q4 log2(n) log(p) ≤ C P (β |X ,S)dβ = 6 (1 + o(1)), 6 n n S n S n n BK for some generic constant C6. ˆ ˆ For any βS ∈ Bn (βS) − BK (βS),

1 L (X |β ) − L (X |βˆ ) = − (β − βˆ )T Iˆ (β − βˆ ) + r (β )||β − βˆ ||2 n n S n n S 2 S S S S S n S S S 2 1 = − (β − βˆ )0Iˆ (β − βˆ )(1 + o(1)) ≥ −c ||β − βˆ ||2, 2 S S S S S 2 S S where c2 > 0 is the upper bound of the maximum eigenvalue of IˆS for all S ∈ P(q). Then for the third component of equation (24), we have

Z ˆ |I3| = P (βS|Xn,S)(Ln(Xn|βS) − Ln(Xn|βS))dβS

By choosing K > 4/c1, it can be veriﬁed that

 s  4 2 −1 ˆ n |S| ˆ − 1 1 1 q log (n) log(p) π (βS)( ) 2 |IS| 2 O( ) = o   . 2π nKqc1/4 n n

c To deal with the last component of equation (24), we should ﬁrst recall for βS ∈ Bn ,

c0nl2 nL (X |β ) − nL (X |βˆ ) ≤ − n . n n S n n S q

Therefore, for suﬃciently large n, we have

ˆ ˆ exp(0.5|nLn(Xn|βS) − nLn(Xn|βS)|) > |nLn(Xn|βS) − nLn(Xn|βS)|.

Now the last component of equation (24) becomes

Recall we have already veriﬁed

s  0 2 4 2 −1 n |S| − 1 c nln q log (n) log(p) π (βˆ )( ) 2 |Iˆ | 2 O(exp(− )) = o . S 2π S q  n 

Therefore we ﬁnally have

 s  Z 4 2 ˆ 1 q log (n) log(p) P (βS|Xn,S)(Ln(Xn|βS) − Ln(Xn|βS))dβS = o   . c n n Bn

25 Combining all the steps above, we conclude

A.4 Proof of Lemma 3

For suﬃciently large n, with probability going to 1, all statements in Lemma 1 and Lemma 2 hold and the following proof are based on those statements. Without loss of generality, for any S∗ ⊂ S, we can extend S ∗ βS∗ to a vector in R by adding zeros in each j ∈ S\S . We denote such a vector by βS∗,S. Applying Lemma 1, for any S∗ ⊂ S and |S| ≤ q, we have

q log(p) 1 1 ||βˆ − β || = O(( ) 4 ) = o(√ ), S S,0 2 n q and q log(p) 1 1 ||βˆ ∗ − β ∗ || = O(( ) 4 ) = o(√ ). S S ,0 2 n q Hence, 1 kβˆ − βˆ ∗ k = o(√ ). S S ,S 2 q Applying Lemma 2, we obtain

ˆ ˆ n(Ln(Xn|βS) − Ln(Xn|βS∗ ))

r 2 1 0 √ 3 q log(p) 2 = (βˆ − βˆ ∗ ) nIˆ (βˆ )(βˆ − βˆ ∗ ) + nO( q||βˆ − βˆ ∗ || ∨ ||βˆ − βˆ ∗ || ). (26) 2 S S ,S S S S S ,S S S ,S n S S ,S

ˆ It follows from Lemma 1 that the smallest eigenvalue of IˆS(βS) is uniformly bounded away from 0. There- fore, it is easy to verify

r 2 √ 3 q log(p) 2 1 0 nO( q||βˆ − βˆ ∗ || ∨ ||βˆ − βˆ ∗ || ) = o( (βˆ − βˆ ∗ ) nIˆ (βˆ )(βˆ − βˆ ∗ )), S S ,S n S S ,S 2 S S ,S S S S S ,S which implies the residual term of (26) does not aﬀect our results asymptotically and can thus be ignored. From the proof of statement (iii) of Lemma 1, we know

r q2 log(p) S (βˆ ) − S (β ) = −Iˆ (βˆ )(βˆ − β ) + O( )||βˆ − β || . (27) n S n S,0 S S S S,0 n S S,0 2

26 ∂Ln(Xn|βS ) ˆ Since the score function Sn(βS) := equals to 0 when βS = βS, ∂βS

r q2 log(p) −S (β ) = −Iˆ (βˆ )(βˆ − β ) + O( )||βˆ − β || . n S,0 S S S S,0 n S S,0 2

ˆ Recall that the minimum eigenvalue of IˆS(βS) is uniformly bounded away from 0, so

r q2 log(p) O( )||βˆ − β || = o(I (βˆ )(βˆ − β )), n S S,0 2 S S S S,0 which implies that the residual term of (27) does not aﬀect our results asymptotically and can be ignored. ˆ In the following, for notational simplicity, we rewrite IˆS(βS) as IˆS and rewrite IS(βS,0) as IS. Then we have √ √ √ ˆ ˆ−1 −1 n(βS − βS,0) = (1 + o(1)) nIS Sn(βS,0) = (1 + o(1)) nIS Sn(βS,0), (28) holds uniformly for all S ⊃ S∗, and S ∈ P(q). The same argument holds for S∗ as well, i.e.,

√ √ √ ˆ ˆ−1 −1 n(βS∗ − βS∗,0) = (1 + o(1)) nIS∗ Sn(βS∗,0) = (1 + o(1)) nIS∗ Sn(βS∗,0). (29)

For a vector x, denote the subvector of x indexed by S as xS. Without loss of generality, denote x = (xS∗ , xS\S∗ ) for any x ∈ RS. By combining equation (28) and equation (29), we have

√ √ √ ˆ ˆ −1 −1 n(βS − βS∗,S) = (1 + o(1)) IS nSn(βS,0) − (IS∗ nSn(βS∗,0), 0) . (30)

√ S∗ S\S∗ Denote v = nSn(βS,0) and v = (v , v ) := (V1,V2). Then v has the following properties

0 S∗ S∗ 0 E(v) = 0,E[vv ] = nIS,E[v (v ) ] = nIS∗ .

Therefore, equation (30) can be re-expressed as

√ ˆ ˆ −1 −1 n(βS − βS∗,S) = (1 + o(1))IS (V1,V2) − IS(IS∗ V1, 0)     0 0 V1 = (1 + o(1))I−1 . S  0 0 −1    −E[V2V1]E[V1V1] I2 V2

Go back to the major component of equation (26): For a matrix M, deﬁne M(S1,S2) as the submatrix of

27 M indexed by (S1,S2). Then,

1 0 (βˆ − βˆ ∗ ) nIˆ (βˆ )(βˆ − βˆ ∗ ) 2 S S ,S S S S S ,S 1 0 = (1 + o(1)) (βˆ − βˆ ∗ ) nI (βˆ − βˆ ∗ ) 2 S S ,S S S S ,S  0     1 + o(1) 0 0 0 0 −1 −1 0 0 V1 = [V1,V2]   I ISI     2 0 0 −1 S S 0 0 −1 −E[V2V1]E[V1V1] I2 −E[V2V1]E[V1V1] I2 V2 1 + o(1) = (V − E[V V 0]E[V V 0]−1V )0Σ(S, S∗)(V − E[V V 0]E[V V 0]−1V ), 2 2 2 1 1 1 1 2 2 1 1 1 1

∗ −1 ∗ ∗ −1 ∗ ∗ where Σ(S, S ) := IS (S\S ,S\S ) is the submatrix of IS indexed by (S\S ,S\S ). Deﬁne ∗ ∗ 1 0 0 −1 H(S, S ) := Σ(S, S ) 2 (V2 − E[V2V1]E[V1V1] V1).

Then 1 0 (1 + o(1)) 0 ∗ ∗ (βˆ − βˆ ∗ ) nIˆ (βˆ )(βˆ − βˆ ∗ ) = H (S, S )H(S, S ). 2 S S ,S S S S S ,S 2 To prove the main results, we only need to show that for suﬃciently large n and any η > 0 and ζ > 1/5,

0 ∗ ∗ H (S, S )H(S, S ) ζ sup ∗ ≤ 2(1 + η)log(n p), S⊃S∗,S∈P(q) |S\S | holds with probability going to 1. p ζ ∗ Deﬁne AS,n := 2(1 + η) log(n p)|S\S |, then

H0(S, S∗)H(S, S∗) P r ≥ 2(1 + η) log(nζ p) = P r (||H(S, S∗)|| ≥ A ) . |S\S∗| 2 S,n

S 0 ∗ p ∗ ζ First, for any u ∈ R , ||u||2 = 1, we consider to bound P r u H(S, S ) ≥ (1 + )2|S\S | log(n p) . Rewrite n 0 ∗ 1 X 0 ∗ 1 1 0 0 −1 1 u H(S, S ) := √ u Σ(S, S ) 2 (√ V − E[V V ]E[V V ] √ V ), n n 2,i 2 1 1 1 n 1,i i=1

1 ∂ log f(xi|βS,0) 1 ∂ log f(xi|βS,0) where √ V1,i = and √ V2,i = . For simplicity, we further deﬁne n ∂βS∗ n ∂βS\S∗

0 ∗ 1 1 0 0 −1 1 Y (S) := u Σ(S, S ) 2 (√ V − E[V V ]E[V V ] √ V ). i n 2,i 2 1 1 1 n 1,i

Then it becomes n 1 X u0H(S, S∗) = √ Y (S). n i i=1 It is easy to see that

E[Yi(S)] = 0.

28 Moreover, since ∗ 0 0 0 −1 0 −1 Σ(S, S ) = n E[V2V2] − E[V2V1]E[V1V1] E[V1V2] , then

1 1 1 1 E[(√ V − E[V V 0]E[V V 0]−1 √ V )(√ V − E[V V 0]E[V V 0]−1 √ V )0] = Σ(S, S∗)−1, n 2,i 2 1 1 1 n 1,i n 2,i 2 1 1 1 n 1,i which implies 2 0 E[Yi(s) ] = u IS\S∗ u = 1.

1 2 p 3 p 3 By condition (A2), |Yi(S)| ≤ c3 C2 q := K q , where c3 is a generic constant which bounds the maximum eigenvalue of Σ(S, S∗) from the above. By condition (A1), such a constant exists. By applying the Bernstein’s Inequality, we have

n X 1 x2 P r(| Yi| > x) ≤ 2 exp(− ), 2 p 3 i=1 n + K q x/3 √ for any x > 0. Plugging in x = np2(1 + )|S\S∗| log(nζ p), we get

q n ∗ ζ 0 ∗ ∗ ζ X −1 2(1 + )n|S\S | log(n p) P r(u H(S, S ) ≥ 2(1 + )|S\S | log(n p)) = P r(| Yi| > x) ≤ 2 exp( ). 2 p 3 i=1 n + K q x

q5 log(p) By the assumption that n → 0, we also have

p √ q K q3x ≤ K n 2(1 + )q4 log(nζ p) = o(n).

Hence, for suﬃciently large n,

q P r(u0H(S, S∗) ≥ 2(1 + )|S\S∗| log(nζ p)) ≤ 2 exp(−(1 + /2)|S\S∗| log(nζ p)).

∗ To link this inequality to our final objective, P r(||H(S, S )||2 ≥ AS,n), we rely on Lemma 2 in Chen and S\S∗ Chen (2012), which states that for a given δn > 0, there exists a finite set of unit vectors US(δn) ⊂ R S\S∗ 0 such that for all v ∈ R , ||v||2 ≤ (1+δn) maxu∈US (δn) u v holds. Since US(δn) is a finite set, following the proof of Lemma 2 of Chen and Chen (2012), the cardinality of U(δ ) is bounded by |U (δ )| ≤ K( 1 )|S\S∗| n S n δn for some generic constant K > 0. ∗ √ Hence, for a fixed model S with S ⊃ S and |S| = d ≤ q, we can pick δn such that (1 + δn) 1 + n =

29 √ 1 + η.

q ∗ X 0 ∗ ∗ ζ P {||H(S, S )||2 ≥ AS,n} ≤ P (u H(S, S ) ≥ 2(1 + )|S\S | log(n p))

u∈U(δn) ∗ ζ ≤ |U(δn)|2 exp(−(1 + /2)|S\S | log(n p)).

Then

∗ ∗ P {∃ S with S ⊃ S and |S| ≤ q, ||H(S, S )||2 ≥ AS,n} X ∗ ≤ P {||H(S, S )||2 ≥ AS,n} S⊃S∗ and |S|≤q X ∗ ζ ≤ |US(δn)|2 exp(−(1 + /2)|S\S | log(n p)) S⊃S∗ and |S|≤q q−|S∗| X p ≤ |U (δ )|2 exp(−(1 + /2)|S\S∗| log(nζ p)) d0 S n d0=1 q−|S∗| X 0 0 ζ ≤ 2 exp(d log(p) + log(|US(δn)|) − (1 + /2)d log(n p))} d0=1 q−|S∗| X 0 0 ζ ≤ 2 exp(−(/2)d log(p) − d log(n ) + log(|US(δn)|)). d0=1 √ |S\S∗| Since log(|U (δ )|) ≤ log(K) + |S\S∗| log( ), for any ﬁxed δ , we have S n δn n

0 0 ζ 0 ζ 0 exp(−(/2)d log(p) − d log(n ) + log(|US(δn)|)) ≤ C exp(−d (/2 log(p) + log(n ) − log(d ))),

q5 log(p) 1 0 5 1 for some constant C > 0. Since n → 0, we have log(d ) = o(n ). So if ζ > 5 , then

0 ζ 0 0 ζ− 1 C exp(−d (/2 log(p) + log(n ) − log(d ))) ≤ C exp(−d log(n 5 )).

Therefore,

∗ ∗ P {∃ S with S ⊃ S and |S| ≤ q, ||H(S, S )||2 ≥ AS,n} q−|S∗| X 0 0 ζ ≤ 2 exp(−(/2)d log(p) − d log(n ) + log(|US(δn)|)) d0=1 ∞ X 0 ζ− 1 −(ζ− 1 ) ≤ 2C exp(−d log(n 5 )) ≤ Cn 5 = o(1). d0=1

30 A.5 Proof of Theorem 2

The proof of Theorem 2 is based on the proof of Theorem 2 of Foygel and Drton (2011). For suﬃciently large n, with probability going to 1, all statements in Lemma 1, Lemma 2 and Lemma 3 hold. ∗ ∗ ∗ Let S be the true model and deﬁne A0 = {S : S ⊂ S, |S| ≤ q} and A1 = {S : S * S, |S| ≤ q}, then we consider two cases.

Case 1: S ∈ A0. In this case, we have

AEBIC(S) − AEBIC(S∗)

= −2nEβS |Xn,S[Ln(Xn|βS)] + |S| log(n) + 2γ|S| log(p) ∗ ∗ − − 2nE ∗ [L (X |β ∗ )] + |S | log(n) + 2γ|S | log(p) βS∗ |Xn,S n n S ∗ ∗ = −2nE [L (X |β )] + 2nE ∗ [L (X |β ∗ )] + |S\S | log(n) + 2γ|S\S | log(p). βS |Xn,S n n S βS∗ |Xn,S n n S

From Theorem 1, we know

s |S| |S| q4 log2(n) log(p) E [L (X |β )] = L (X |βˆ ) − + O( ), βS |Xn,S n n S n n S 2n n n and s |S∗| |S∗| q4 log2(n) log(p) E ∗ [L (X |β ∗ )] = L (X |βˆ ∗ ) − + O( ). βS∗ |Xn,S n n S n n S 2n n n Therefore,

AEBIC(S) − AEBIC(S∗)  s  4 2 ∗ q log (n) log(p) ∗ ∗ = |S\S | + O q + |S\S | log(n) + 2γ|S\S | log(p) − 2n(L (X |βˆ ) − L (X |βˆ ∗ )).  n  n n S n n S

From Lemma 3, we also know

ˆ ˆ ∗ ζ 2n(Ln(Xn|βS) − Ln(Xn|βS∗ )) ≤ 2(1 + η)|S\S | log(n p), for any small η and suﬃciently large n. Hence,

AEBIC(S) − AEBIC(S∗)  s  q4 log2(n) log(p) ≥ |S\S∗| + O q + |S\S∗| log(n) + 2γ|S\S∗| log(p) − 2(1 + η)|S\S∗| log(nζ p).  n 

31 γ−1 ∗ For γ > 1, pick η = min( 2 , ξ), where ξ > 0 is a small constant, then 2γ|S\S | log(p) − 2(1 + q ∗ γ−1 ∗ q4 log2(n) log(p) η)|S\S | log(p) ≥ 2 |S\S | log(p). In addition, it is obvious that log(p) dominates O q n , q6 log2(n) as n log(p) → 0. Hence, for suﬃciently large n, we have

 s  q4 log2(n) log(p) 2γ|S\S∗| log(p) − 2(1 + η)|S\S∗| log(p) + O q > 0.  n 

1 1 Choose ζ ∈ ( 5 , 2+2ξ ), for suﬃciently large n, we also have

|S\S∗|(1 + log(n)) > 2(1 + η)|S\S∗| log(nζ ).

Combining them together, we ﬁnally proved AEBIC(S) − AEBIC(S∗) > 0 for suﬃciently large n.

0 ∗ ˆ 0 Case 2: S ∈ A1. Let S = S ∪S and βS,S0 denote a vector corresponding to model S , which is generated ˆ 0 by augmenting βS with zeros in S \S. Following a similar procedure as in case 1, we have

AEBIC(S) − AEBIC(S∗)

∗ ∗ = − 2nE [L (X |β )] + 2nE ∗ [L (X |β )] + (|S| − |S |) log(n) + 2γ(|S| − |S |) log(p) βS |Xn,S n n S βS∗ |Xn,S n n S ˆ ˆ ∗ ∗ = − 2n(Ln(Xn|βS) − Ln(Xn|βS∗ )) + (|S| − |S |) + (|S| − |S |) log(n)  s  q4 log2(n) log(p) + 2γ(|S| − |S∗|) log(p) + O q  n 

ˆ ˆ ≥ − 2n(Ln(Xn|βS) − Ln(Xn|βS∗ )) − q(log(n) + 2γ log(p) + 1 + o(1)) ˆ ˆ ≥ − 2n(Ln(Xn|βS,S0 ) − Ln(Xn|βS∗ )) − q(log(n) + 2γ log(p) + 2). (31)

ˆ By optimality, Ln(Xn|βS∗ ) ≥ Ln(Xn|βS∗,0) = Ln(Xn|βS0,0), since βS∗,0 equals to βS0,0 up to components of zeros in S0\S∗. Then

ˆ ˆ ˆ −2n(Ln(Xn|βS,S0 ) − Ln(Xn|βS∗ )) ≥ −2n(Ln(Xn|βS,S0 ) − Ln(Xn|βS0,0)).

From the proof of Lemma 1, we have

r q log(p) |L (X |βˆ ) − L(βˆ )| = O( ), n n S S n r q log(p) |L (X |β ∗ ) − L(β ∗ )| = O( ). n n S ,0 S ,0 n

32 By condition (A6) and (A7), we have

ˆ ˆ 2 2 L(βS0,0) − L(βS,S0 ) ≥ min(1, c||βS,S0 − βS0,0||2) ≥ min(1, c[min |βS∗,0|] ) j r q log(p) 1 2 q log(p) min(1, c[( ) 4 ] ) = c . n n

Combining them together, we obtain

Back to inequality (31), for suﬃciently large n, we have

ˆ ˆ −2n(Ln(Xn|βS,S0 ) − Ln(Xn|βS∗ )) − q(log(n) + 2γ log(p) + 2) > 0, √ q log(p) nq log(p) ∗ since the assumption n → 0 leads to q log(p) → ∞. Hence, AEBIC(S) > AEBIC(S ) holds.

References

Akaike, H. (1974), “A new look at the statistical model identiﬁcation,” Automatic Control, IEEE Trans- actions on, 19, 716–723.

Ando, T. (2010), Bayesian Model Selection and Statistical Modeling, New York: Chapman & Hall.

Bickel, P., Ritov, Y., and Tsybakov, A. (2009), “Simultaneous analysis of Lasso and Dantzig selector,” Annals of Statistics, 37, 1705–1732.

Candes, E. and Tao, T. (2007), “The Dantzig selector: Statistical estimation when p is much larger than n,” Ann. Statist., 35, 2313–2351.

Chen, J. and Chen, Z. (2008), “Extended Bayesian information criteria for model selection with large model spaces,” Biometrika, 95, 759–771.

— (2012), “Extended BIC for small-n-large-p sparse GLM,” Statistica Sinica, 22, 555–574.

Chen, Z. and Luo, S. (2013), “Selection consistency of EBIC for GLIM with non-canonical links and diverging number of parameters,” Statistics and Its Interface, 6, 275–284.

Efron, B. (2009), “Empirical Bayes Estimates for Large-Scale Prediction Problems,” Journal of the Amer- ican Statistical Association, 104, 1015–1028.

33 Fan, J. and Li, R. (2001), “Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Prop- erties,” Journal of the American Statistical Association, 96, 1348–1360.

Foygel, R. and Drton, M. (2011), “Bayesian model choice and information criteria in sparse generalized linear models,” ArXiv e-prints.

Foygel Barber, R., Drton, M., and Tan, K. M. (2015), “Laplace Approximation in High-dimensional Bayesian Regression,” ArXiv e-prints.

Ge, D., Wang, Z., Ye, Y., and Yin, H. (2016), “Strong NP-hardness result for regularized Lq-minimization problems with concave penalty functions,” arXiv:1501.00622v3.

Green, P. (1995), “Reversible jump Markov chain Monte Carlo computation and Bayesian model determi- nation,” Biometrika, 82, 711–732.

Haughton, D. M. A. (1988), “On the Choice of a Model to Fit Data from an Exponential Family,” Ann. Statist., 16, 342–355.

Huo, X. and Chen, J. (2010), “Complexity of penalized likelihood estimation,” Journal of Statistical Computation and Simulation, 80, 747–759.

Jiang, W. (2007), “Bayesian variable selection for high dimensional generalized linear models: Convergence rates of the ﬁtted densities,” Ann. Statist., 35, 1487–1511.

Johnson, V. E. and Rossell, D. (2012), “Bayesian Model Selection in High-Dimensional Settings,” Journal of the American Statistical Association, 107, 649–660.

Kass, R. and Raftery, A. (1995), “Bayes factors and model uncertainty,” Journal of the American Statistical Association, 90, 773–795.

Liang, F., Liu, C., and Carroll, R. (2007), “Stochastic Approximation in Monte Carlo Computation,” Journal of the American Statistical Association, 102, 305–320.

Liang, F., Song, Q., and Yu, K. (2013), “Bayesian Subset Modeling for High-Dimensional Generalized Linear Models,” Journal of the American Statistical Association, 2013.

McCullagh, P. and Nelder, J. A. (1989), Generalized linear models (Second edition), London: Chapman & Hall.

Nishii, R. (1984), “Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression,” Ann. Statist., 12, 758–765.

Schwarz, G. (1978), “Estimating the Dimension of a Model,” Ann. Statist., 6, 461–464.

34 Shin, M., Bhattacharya, A., and Johnson, V. E. (2015), “Scalable Bayesian Variable Selection Using Nonlocal Prior Densities in Ultrahigh-Dimensional Settings,” ArXiv e-prints.

Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo, P., Renshaw, A. A., D’Amico, A. V., Richie, J. P., Lander, E. S., Loda, M., Kantoﬀ, P. W., Golub, T. R., and Sellers, W. R. (2002), “Gene expression correlates of clinical prostate cancer behavior,” Cancer Cell, 1, 203 – 209.

Song, Q. and Liang, F. (2015a), “High-Dimensional Variable Selection With Reciprocal L1-Regularization,” Journal of the American Statistical Association, 110, 1607–1620.

— (2015b), “A Split-and-Merge Bayesian Variable Selection Approach for Ultra-high dimensional Regres- sion,” Journal of the Royal Statistical Society, Series B, 77, 947–972.

Spiegelhalter, D., Best, N., Carlin, B., and Germany, B. (2014), “The deviance information criterion: 12 years on,” Journal of the Royal Statistical Society, Series B, 76, 485–493.

Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and van der Linde, A. (2002), “Bayesian measures of model complexity and ﬁt,” Journal of the Royal Statistical Society Series B, 64, 583–639.

Tibshirani, R. (1994), “Regression Shrinkage and Selection Via the Lasso,” Journal of the Royal Statistical Society, Series B, 58, 267–288.

Wahba, G. and Craven, P. (1979), “Smoothing Noisy Data with Spline Functions. Estimating the Correct Degree of Smoothing by the Method of Generalized Cross-Validation.” Numerische Mathematik, 31, 377–404.

Watanabe, S. (2010), “Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory,” Journal of Machine Learning Research, 11, 3571–3594.

Zhang, C.-H. (2010), “Nearly unbiased variable selection under minimax concave penalty,” Ann. Statist., 38, 894–942.

Zhao, P. and Yu, B. (2006), “On model selection consistency of Lasso,” Journal of Machine Learning Research, 7, 2541–2563.

Zou, H. and Hastie, T. (2005), “Regularization and Variable Selection via the Elastic Net,” Journal of the Royal Statistical Society. Series B (Statistical Methodology), 67, 301–320.