438 RUSAKOV & GEIGER UAI2002
Asymptotic Model Selection for Naive Bayesian Networks
Dmitry Rusakov and Dan Geiger Computer Science Department Technion - Israel Institute of Technology Haifa, Israel 32000 {rusakov ,dang}@cs.technion.ac.il
Abstract The critical computational part in the evaluation of this criterion is the marginal likelihood integral li = P(DIM) =In P(DIM,w)P(wiM)dw. We write We develop a closed form asymptotic for mula to compute the marginal likelihood of [N, v, = elogl;ke[;hood(Y0,Niw,M) data given a naive Bayesian network model H Y MJ i p(wiM)dw (l) with two hidden states and binary features. This formula deviates from the standard BIC where YD is the averaged sufficient statistics of the score. Our work provides a concrete example data D, N is a number of examples in D, and JL(wiM) that the BIC score is generally not valid for is the prior parameter density for model M. Recall statistical models that belong to a stratified that the average sufficient statistics for multinomial exponential family. This stands in contrast to samples of n binary variables (X1, ... , Xn) is simply linear and curved exponential families, where the counts for each of the possible 2n joint states. Of the BIC score has been proven to provide a ten the prior P(M) is assumed to be equal for all correct approximation for the marginal like models, in which case Bayesian model selection is per lihood. formed by maximizing JI[N,Yv, M]. The quantity rep resented by S(Yv,N,M) = ln ll [N, Yv, M] is called the Bayesian Information Criterion (BIG) for choos ing model M. 1 INTRODUC TION For many types of models the asymptotic evaluation Statisticians are often faced with the problem of choos of integral 1 (as N -+ oo) is a classical Laplace proce ing the appropriate model that best fits a given set dure. This evaluation was first performed for Linear of observations. One example of such problem is the Exponential (LE) models (Schwarz, 1978) and then choice of structure in learning of Bayesian networks for Curved Exponential (CE) models under some ad ditional technical assumptions (Haughton, It (Heckerman, Geiger & Chickering, 1995; Cooper & 1988). was shown that Herskovits, 1992). In such cases the maximum likeli hood principle would tend to select the model of high d est possible dimension, contrary to the intuitive notion S(Yv,N,M)=N·InP(YvlwML)-21nN+R, (2) of choosing the right model. Penalized likelihood ap proaches such as AIC have been proposed to remedy where In P(YvlwML) is the log-likelihood of Yv given this deficiency (Akaike, 1974). the maximum likelihood parameters of the model and dis the model dimension, i.e., the number of indepen We focus on the Bayesian approach to model selection, dent parameters. The error term R = R(Yv,N, by which a model is chosen according to M) M the maxi was shown to be bounded for a fixed Yv (Schwarz, mum posteriori probability given the observed data D: 1978) and uniformly bounded for all Yv -+ Y in CE models (Haughton, 1988). This approximation is re P(MID) ex: P(M, D) = P(M)P(DIM) ferred as a (standard) BIG score. = P(M) In P(DIM,w)P(wiM)dw The use of BIC score for Bayesian model selection for where w denotes the model parameters and n denotes Graphical Models is valid for Undirected Graphical the domain of the model parameters. In particular we Models without hidden variables because these are LE focus on large sample approximation for P(MID). models (Lauritzen, 1996). The justification of BIC for UAI2002 RUSAKOV & GEIGER 439
Directed Graphical Models(called Bayesian Networks) result deferring the proof to an Appendix. Finally, is somewhat more complicated. On one hand discrete Section 6 outlines future research directions. and Gaussian DAG models are CE models (Geiger, Heckerman, King & Meek, 2001; Spirtes, Richardson 2 ASYMPTOTIC & Meek, 1997). On the other hand, the theoretical APPROXIMATIONS justification of the BIC score for CE models has been established under the assumption that the model con Exact analytical formulas are not available for many tains the true distribution - the one that. has generated integrals arising in practice. In such cases some sort the observed data. This assumption limits the applica of approximate or asymptotic solutions are of inter bility of the proof of BIC score's validity for Bayesian est. Asymptotic analysis is a branch of analysis that networks in practical setups. is concerned with obtaining the approximate analyti The eva!uat.ion of the marginal likelihood ITN,[ Y] for cal solutions to problems where a parameter or some Bayesian networks with hidden variables is a wide variable in an equation or integral becomes either very open problem because the class of distributions rep large or very small. resented by Bayesian networks with hidden variables Let z represent such a large parameter. We say that is significantly richer than curved exponential mod f( z) is asymptotically equal to 2:::;' an9n(z), denoted els and it falls into the class of Stratified Exponential =1 by the symbol "�", if (SE) models (Geiger et al., 2001). For such models the effective dimensionality d (Eq. 2) of the model is m no longer the number of network parameters (Geiger, f(z)= L Dn9n(z) + O(gm+i (z)), as z---+ oo, n Heckerman & Meek, 1996; Settimi & Smith, 1998). =1 Moreover, the central problem in the evaluation of the where the big 0 symbol states that the error term is marginal likelihood for this class is that the set of max bounded by a constant multiply of 9m+i (z) and {gn} is
imum likelihood points is sometimes a complex self an asymptotic sequence, i.e., limz--+cc9n+J/9n = 0. A crossing surface. Recently, major progress has been good introduction to asymptotic analysis can be found achieved in analyzing and evaluating this type of inte in (Murray, 1984). grals (Watanabe, 2001). Herein, we apply these tech niques to model selection among Bayesian networks The main objective of this paper is asymptotic approx with hidden variables. imation of marginal likelihood integrals as represented by Eq. 1, which are of the form The focus of this paper is the asymptotic evaluation of IT[N,Y, M] for a binary naive Bayesian model M with IT[N, Y]= e-Nf(w,Y)J.t(w)ck,; (3) binary features. The results are derived under similar l assumptions to the ones made by Schwarz (1978) and where f(w,Y) = -loglikelihood(Y iw). We shall as Haughton (1988). In this sense, our paper general sume that we are dealing with exponential models, so izes the mentioned works, providing valid asymptotic the log-likelihood of sampled data is equal to N times formulas for a new type of marginal likelihood inte the log-likelihood of the averaged sufficient statistics. grals. The resulting asymptotic approximations, pre This assumption holds for the models discussed in this sented in Theorem 3, deviate from the standard BIC paper. score. Hence the standard BIC score is not justified for Bayesian model selection among Bayesian networks Consider Eq. 3 for some fixedY. For large N, the main with hidden variables. Our adjusted BIC score changes contribution to the integral comes from the neigh depending on the different types of singularities of the borhood of the minimum of j, i.e., the maximum of sufficient statistics, namely, the coefficient of the InN -N f(w,Y). Thus, intuitively, the approximation of term is no longer -� but rather a function of the suf IT[N,Y] is determined by the form off near its mini ficient statistics. Moreover, an additional in InN term mum on !1. In the simplest case f(w) achieves a sin appears in some of the 0(1) approximations, which is gle minimum at WML in the interior of !1 and this unaccounted for by the classical score. maximum is non-degenerate, i.e., the Hessian matrix Hf(wMd of f at WML is of full rank. In this case The rest of this paper is organized as follows. Sec the approximation of IT[N, Y] for N ---+ oo is the clas tion 2 introduces the concept of asymptotic expansions sical Laplace procedure (e.g., Wong, 1989, page 495), and presents some methods of asymptotic approxima summarized as follows tion. Section 3 discusses an application of these meth ods. Section 4 reviews naive Bayesian models and ex Lemma 1 (Laplace Approximation) Let plicates the relevant marginal likelihood integrals for these models. Section 5 states and explains our main I(N ) = i e-Nf(u)J.t(u)du 440 RUSAKOV & GEIGER UA12002
where U C JRd. Suppose that f is twice differentiable In general it is not easy to find the largest pole and and convex (1/. f (u) > 0), the minimum of f on U is multiplicity of J(>.). Here, another fundamental math achieved on a single internal point u0, Jl is continuous ematical theory comes to rescue. The resolution of and J-L(u0) # 0. If I (N) absolutely converges, then singularities in algebraic geometry transforms the in tegral J(>.) into a direct product of integrals of a sin � Ce-Nf(uo) -d/2 l(N) N (4) gle variable (Atiyah, 1970, Resolution Theorem; Hi 2 ronaka, 1964). We demonstrate this technique in the where C = (27r)df J-L(uo)[det11. f (u0)]-� is a constant. next section. Note that the logarithm of Eq. 4 yields the BIC score APPLICATION OF WATANABE'S as presented by Eq. 2. 3 ME THOD However, in many cases, and, in particular, in the case of naive Bayesian networks, the minimum of f is achieved not at a single point in !1 but rather on a We now apply the method of Watanabe (2001) to ap variety Wo C !1, called the zero set. Sometimes, this proximate the integral variety may be d'-dimensional surface (smooth mani uiu� [N = e-NL.,9#$• du fold) in !1 in which case the calculation of the integral ] ] ( -•.+,)" is locally equivalent to the d- d' dimensional classical 1 as N tends to infinity. This evaluation is part of the case. The hardest cases to evaluate happen when the proof of our main result (Theorem 3). It is presented variety W0 contains self-crossings. here as a self contained example which can be skipped Fortunately, an advanced mathematical method for without loss of continuity. approximating this type of integrals was introduced to Watanabe's method calls for the analysis of the poles the machine learning community by Watanabe (2001). of the following function Below we introduce the main theorem that enables us to compute the asymptotic form of IT[N, Y] integrated [ ] >. in a neighborhood of a maximum likelihood point. J(>.) = L ufu� du. ( _,,+,)" !::;l�k:O;n Theorem 2 (based on (Watanabe, 2001)) Let 1 To find the poles we transform the integrand function 1/J(u) = LJ
We start with n = 3 and then generalize. Rescaling J(>.) = f J(w)>.J-L(w)dw (Re(>.) > 0) Jf(w)<• the integration range to ( - 1 , 1) and then taking only the positive quadrant, which does not change the poles where E > 0 is a sufficiently small constant. of J(>.), yields 2 2 2 2 2 2 J(>.) = f l)3 (u u u u + u u ) du Applying Theorem 2 to the classical, single maximum, J(O, 1 2 + 1 3 2 3 >. strictly convex case (Lemma 1) gives the largest pole (Jo
and the marginal likelihood integral (1) becomes
ll[N, YJ = { eNL,_Y,ln6,(w)f.J.(w)dw (7 J(o,J)2•+' )
where w = (a I, ..., an, b1, . .., bn, t) are the model pa rameters. Figure 1. A naive Bayesian model. 5 MAIN RESULT 3 We now divide the range (0, 1) according to u2 < u3 or u3 < u2 • Again these cases are symmetric and so This section presents an asymptotic approximation of we continue to evaluate only one of them the integralll[ N, YJ (Eq. 7) for naive Bayesian networks consisting of binary variables X1, ..., Xn and two hid JJl (A) = den states. It is based on two results. First, the clas sification of singular points for these types of models 2001). 3 (Geiger et al. , Second, Watanabe's approach Since the function (1 + u� + u� u§) is bounded on (0, 1) , as explained in Section 2, which provides a method it follows that J(A) is within a constant multiply of to obtain the correct asymptotic formula of ll[N,YJ for 2 the singular points not covered by the classical Laplace J(A) "" r ut>-+ u�A+ldu. approximation scheme. 1(0,1)2 Let Y = {(yJ, ..., y2• )[y; :0:: O,L:y;= 1} be the set of Thus J(A) has poles at A = -3/4 and A = -1 with possible values of sufficient statisticsY = (Y1, .. . , Y2.) multiplicity m = 1. The largest pole is A = -3/4 for data D = { (x;,1 , ..., x;,n}�1. In our asymptotic with multiplicity m = 1. Generalizing the above ap analysis we let the sample size N grow to infinity. proach to n :0:: 3 we get that the largest pole of J (A) is (y1, AJ = -n/4 with multiplicity m = 1, so ][N] is asymp Let Yo C Y be the points ... , y2.) that corre totically equal to cN-'t. spond to the distributions that can be represented by binary naive Bayesian models with n binary variables. y; 4 NAIVE BAYESIAN MO DELS I.e., assuming the indices of are written as vectors n (
In this work we focus on naive Bayesian networks that The set S' C S is the set of points represented by have two hidden states (r = 2) and n binary feature a naive Bayesian model, just as the set S does, but variables X1, ... , Xn. We denote the parameters defin with all links removed; namely, a distribution where all p(x;[ci) p(x;h) ing by a;, the parameters defining by variables are mutually independent and independent of b;, and the parameters defining p( c1) by t. These pa the class node as well. rameters are called the model parameters. We denote S' 0 the joint space parameters P(X = x) by Bx. The fol Clearly C S C Y C Y. We now present our main lowing mapping relates these parameters. result.
n n Theorem 3 Let IT[N,YJ be the marginal likelihood = t (1- 1 + t) (1- 1 B n a�' a;) -x' (1- n b�' b;) -x', (6) of data with sufficient statisticsY given the naive x i=l i=l 442 RUSAKOV & GEIGER UAI2002
Bayesian model with binary variables and two hidden corresponds to selecting .X1= - � and m =3. These states, as represented by Eqs. 6 and 7. Let Y and J1 formulas are different from the standard BIC score, satisfy following assumptions: given by Eq. 9, which only applies to regular points in non-degenerate models, namely, the points in Yo\ S. Al Jl(w) Bounded density. The density is bounded In contrast to the standard BIC score, which is uniform 2 1 and bounded away from zero on n = (0, 1) n+ . for all points Y, the asymptotic approximation given A2 Positive statistics. The statistics Y are such that by our adjusted BIG score depends on the value of Y; > 0 for i = 1, . . . , 2n . Y through the coefficient of In N. This coefficient in the singular cases is not the effective dimensionality A3 Statistics stability. There exists sample size No of the parameter space, because the parametric space such that the sufficient statistics is Y for all sam is singular at these points, namely, not isomorphic to ple sizes N > No. any hypersurface. Instead, the coefficients of InN and In InN terms describe the geometric structure of the
Then, for n 2': 3 as N --+ oo: log-likelihood function near singular points. E.g., for n =2 and Y E S', we get the 2ln lnN term in 0(1) approximation, which is missed by the standard BIC 1. IfY E Y0 \ S (regular point) score formula, Eq. 2. 2n + 1 lniT[N, Y] =N f y - --lnN +0(1), (9) One may argue that evaluating the marginal 2 likelihood on singular points is not needed because one could ex clude from the model all singular points which only 2. If Y E S \ S' (type 1 singularity) have measure zero (with respect to the volume element 2n -1 of the highest dimension). The remaining set would lniT[N,Y] =Njy - - -lnN + 0(1), (10) 2 be a smooth manifold defining a curved exponential model, and so BIC would be a correct asymptotic ex 8. If Y E S' (type 2 singularity) pansion as long as the point Y hasnot been excluded, i.e., it will be correct for regular Y E Yo points. How n+1 lniT[N,Y] = Njy--- lnN + 0(1), (11) ever, this proposal fails in the case of selecting amongst 2 naive Bayesian models.
where jy = ln P(Y!wML ) and WML is the maximum Consider the problem of selecting between two naive likelihood parameters. Bayesian models with n binary features and a binary
Moreover, for n = 1, 2 (degenerate models), hidden class variable. One model M F with all links present between the class variable and each and ev
• If n=2, andYrf.S', or n=1, then ery feature variable, and the other being a degenerate model MD for which all the feature variables are mu 2n -1 lniT[N, Y]=Nfy-- -lnN +0(1), (12) tually independent and independent also of the class 2 variable, namely, there are no links present. Assum ing the two states h1 and h2 of the hidden variable • If n = 2 and Y E S', are interpreted as representing two classes, model MD 3 tells us that the n features in the model do not distin lniT[N,Y] =Njy- lnN 2ln lnN+0(1), (13) 2 + guish between the two classes, and so if model MD is correct, the two classes are not distinguishable using
as N--+ oo. the prescribed n features. If model Mp is correct, it provides some support for the existence of two classes, The firstassumption (bounded density) has been made the strength of which is determined by the parameters by all earlier works; in some applications they hold and of the model. Assume a prior probability of p > 0 and in some they do not. The proof and the results, how 1- p > 0 for the two models, respectively. ever, can be easily modified to apply to any particular Now, if the true data comes from model then kind of singularity of Jl, as long as we know the form MD, of this singularity. The second and third assumptions its large sample statistics falls very close to the set S' are made to ease the proof; the third assumption was of singular points of the full model MF. Even if the also made by Schwarz (1978). statistics are regular due to small perturbations in the 2 -1 sample, evaluation of IT[N,Y] according to the regular Note that Eq. 10 corresponds to selecting .X1 = - n2 case formula will give very large error terms and result and m1 = 1 in Watanabe's method, Eq. 11 corre in an incorrect model selection. Hence, in this case, sponds to selecting .X1 =- ntl and m1 =1, and Eq. 13 UAI2002 RUSAKOV & GEIGER 443
one should evaluate the marginal likelihood of M F us Cooper, G. F. & Herskovits, E. (1992). A Bayesian ing uniform asymptotic formulas, which are valid for method for the induction of probabilistic networks the range of Y near singular points, and which, in the from data. Machine Learning, 9(4), 309-347. limit, are equivalent to the formulas derived in this pa per. This careful evaluation should be performed for Geiger, D., Heckerman, D., King, H., & Meek, C. a non negligible fraction p of the possible large sam (2001). Stratified exponential families: Graphical ple dataset,s, at least according to the prior specifica models and model selection. Annals of Statistics, tion. This phenomenon happens whenever comparing 29 (2)' 505-529. a graphical model against one of its submodels, which is a common practice that requires a careful analysis Geiger, D., Heckerman, D., & Meek, C. (1996). that this paper attempts to provide. Asymptotic model selection for directed networks with hidden variables. In Proceedings of the 12 UAI Conference, (pp. 283-290). 6 FUTURE WORK Haughton, D. M. A. (1988). On the choice of a model We now highlight the steps required for obtaining a to fit data from an exponential family. Annals of fully justified asymptotic model selection criterion for Statistics, 16(1), 342 355. naive Bayesian networks. -
Heckerman, D., Geiger, D., & Chickering, D. M. 1. Develop a closed form asymptotic formula for (1995). Learning Bayesian networks: The combi marginal likelihood integrals for all types of statis nation of knowledge and statistical data. Machine tics Y given an arbitrary naive Bayesian model. Learning, 20(3), 197-243. This step has been partially treated by the current paper. Hironaka, H. (1964). Resolution of singularities of an 2. Extend these solutions by developing uniform algebraic variety over a field of characteristic zero. asymptotic formulas valid for converging statis Annals of Mathematics, 7(1,2), 109-326.
tics YD-+ Y as N-+ oo. Lauritzen, S. L. (1996). Graphical Models. Number 17 3. Develop an algorithm that, given a naive Bayesian in Oxford Statistical Science Series. Clarendon. network and a data set with statistics YD, deter mines the possible singularity types of the limit Murray, J. (1984). Asymptotic Analysis. Number 48 in statistics Y and applies the appropriate asymp Applied Mathematical Sciences. Springer-Verlag. totic formula developed in step 2. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Our work provides a first step and a concrete frame Systems: Networks of Plausible Inference. Mor work to resolve these tasks among naive Bayesian net gan Kaufmann. works and perhaps among Bayesian networks with hid den variables in general. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464. Acknowledgements Settimi, R. & Smith, J. Q. (1998). On the geometry of The second author thanks David Heckerman and Chris bayesian graphical models with hidden variables. Meek for years of collaboration on this subject. In Proceedings of 14th UAI Conference,(pp. 472- 479). Morgan Kaufmann. References Spirtes, P., Richardson, T., & Meek, C. (1997). The Abhyankar, S. S. (1990). Algebraic Geometry for Sci dimensionality of mixed ancestral graphs. Tech entists and Engineers. Number 35 in Mathemati nical Report CMU-PHIL-83, Philosophy Depart cal Surveys and Monographs. AMS. ment, Carnegie Mellon University. Akaike, H. (1974). A new look at the statistical model Watanabe, S. (2001). Algebraic analysis for noniden identification. IEEE Transactions on Automatic tifiable learning machines. Neural Computation, Control, Jg(6), 716-723. 13(4), 899-933. Atiyah, M. (1970). Resolution of singularities and di vision of distributions. Communications on Pure Wong, R. (1989). Asymptotic Approximations of Inte and Applied Mathematics, 13, 145-150. grals. Academic Press. 444 RUSAKOV & GEIGER UA12002
APPENDI X: PROOF OU TLINE 2 where p;(s) = (1-s )((1- s)i-l ( -l)i(l + s)i-1 ), � 2 + and, in particular, p2 ( s) = 1 - s . We index the z vari The integral IT[N,YJ converges for all N 2': 1 and for ables by non-empty subsets of {1,... , n}. Note that, all Y because the likelihood function is bounded. The generally, this transformation is not a diffeomorphism. first claim of Theorem 3 follows from the fact that for We have defined three transformations, from the Y E Yo\ S there are only two (symmetric) maximum model parameters to the joint space parameters likelihood points at each of which the log-likelihood !1 0: function is properly convex. Hence, the marginal like lihood integral can be approximated by the classical Laplace method (Lemma 1). Based on these transformations we now present lemma The proof of the second and third claims of Theorem 3 that facilitates the evaluation of the integral IT[N,YJ. requires the advanced techniques of Watanabe (Sec tion 2). First, the integral IT[N,YJ is transformed by a Lemma 4 Let IT[N,YJ be as represented by Eq. 7 series of transformations into a simpler one. Second, and let f define the normalized log-likelihood function, the sets of extremum points of the exponent (maxi namely mum log-likelihood points) are found, and then the new integral is computed in the neighborhoods of ex tremum points. Finally, the asymptotic form of the where jy lnP(YiwML) and B[x,u,s] largest contribution gives the desired asymptotic ap (T3 o T2)[x, u,s]. Also, let the zero set Uo = proximation to the original integral. We focus on one arg min(x,u,s)EU f(x,u,s) be the set of minimum points thread of our proof which demonstrates this method. of f in (x,u,s) coordinates, and let
o USEFUL TRANSFORMATIONS .J'[N,Y] = I:: ( e-N'L.C,I-' i,fl'dxduds, (14) Ju.._,i (x,u,s)iEUo We first introduce a series of three transformations from the model parameters w = (a, b, t) to the joint where U;,, is a small neighborhood of (x, u, s); E Uo, space parameters Bx that facilitates the approximation z(x,u,s) = T2(x,u,s) and zo; = T2[(x,u,s);]. ofiT[N,YJ. The transformations T1, T2 and T3 are such Suppose that J.L is bounded (AJ), andY is positive {A2) that their composition T = T3 o T o T1 : !1 -t 8 is and fixed {A3). Then, for all N > 1, 2 1 defined by Eq. 6, where !1 = (0, 1)2n+ is the domain of model parameters w and e is the domain of joint lniT[N,YJ = Njy + ln.JJ[N, YJ +0(1) space parameters Bx. We call w's - the source variables and O's - the target variables. These transformations where .lJ[N, YJ is represented by finite sum of neighbor are from (Geiger et a!., 2001). hood integrals.
Transformation T1 : Let T1 : !1 -t U be defined via This lemma follows from the assumptions A1-A3 and a;- b; the facts that T1, T3 are diffeomorphisms, Y E Yo, s = 2t- 1, u; = -- -, x; = ta; t)b;, 2 + (1- [J is compact, and the contributions of non-maximum regions of the integrand are exponentially small. i = 1, ... ,n. The mapping T1 is a diffeomorphism, namely, a one-to-one differentiable map with a differ Lemma 4 states that integral 14 determines the asymp -n+l totic form of the original integral IT[N,YJ. entiable inverse. Furthermore, I det Jr,l = 2 .
Transformation T3: The transformation T3 : A-t 8 PROOF FOR TYPE 2 SINGULARITY is defined on the original target variables in such way that the new target variables z E A are expressed in We focus on the proof of the third claim of Theorem 3 terms of the new source variables (x, u,s) by a number that deals with the singular points in S'. This proof of simple formulas. The exact form of this transforma illustrates the methods required for the proofs of all tion is unimportant for our analysis. We note that T3 cases of Theorem 3. is diffeomorphism and I det Jr3l = 1. For details con sult (Geiger et a!., 2001). Let Y E S'. Our starting point is integral 14, which by Lemma 4 is within a constant multiply of the original Transformation T2: This transformation is defined integral (without the e-Nfy term). We evaluate the 2 1 1 by T2 : U C IR n+ -t A C IR2"- via contributions to the integral .JJ[N, YJ from the neigh borhoods of extremum points (x', u',s') E Ua. The maximum contribution gives the asymptotic order of the original integral. UAI2002 RUSAKOV & GEIGER 445
Projedlon ofltonto (s.u;.u).for 1, = 0.2,y1" 0.3 C3. (x',u', s') EUo- u Uo+ \ Ui Uoi.
C4. (x', u', s') EUo- u Uo+ u Uoj \ ui#j Uo;.
C5. (x',u', s') E(Uo- u Uo+) nj Uoj.
Among these cases, C1 and C3 are almost classical, with f being approximated by a quadratic form in 2n- 1 and n + 1 variables, and the cases C2, C4 and C5 are the most complex, since they correspond to the intersection points of hyper-dimensional planes. We illustrate the treatment of such points for case C2.
= Case C2: (x',u',s') niuoj, i.e., u = 0, s # ±L In this case zo,i = x; for all i and Zo,J = 0 for all other z's. Centering (x,u,s) around the minimum point (x',u', s'), we get Figure 2: The set U0 projected on (s,u;,u1), for li = 0.2, li = 0.3. Examples of points of types C1-C5 are }(x,u,s) = :L(zi- ZI,oi)2 marked. Ll[(xl+ x;)-x;F + L!,k [(1- (s + s')2)uluk- 0]2+ ... L1xf Let 1 = (11, .. . , In) be the model parameters' val ues of n independent variables that define the 2n di + L!,k [(1- s'2)uluk- (s + 2s')suluk]2 mensional point Y, as explicated by the definition of +"higher order terms" S'. The zero set U0 is rather complicated because it contains a number of intersecting multidimensional So, the principal part of j, that bounds j within the planes. We have multiplicative constant near zero, is given by n = }(x,u,s) � 'Lxf + 'Lufu%. Uo Uo- U Uo+ U U Uoj j=l I l,k where - The quadratic form in x1's contributes anN n/2 factor [ Uo- {(!,u,- 1) I u; i =1, ... , n } , to the integral ] N, Y]. This can be shown by decom : \�T·�7;! :_ Uo+-{(l,u,1)lu;E( , ),t-1, ...,n , posing the integral and integrating out the x1's. We { 2 2 }} X=I,U;=O,io/j, are left with the evaluation of the integral '' · = ) UjE(-�,�),sE(-1,1), UO J (X,U,S = e-NL.u?u%du. - j { I - j (1 S ) Uj 1 - ][N l < < li 1(-•,+•)" li - 1 < (1 + s)u1 < li and Uo-, Uo+, Uoj denote the closures of .U0_, Uo+ This is precisely the integral evaluated in Section 3 and Uoj- which was found to be asymptotically equal to eN-'f. Thus the contribution of the neighborhood of With the zero set containing n-dimensional surfaces (x',u',s') to][N, Y] is eN-�. Uo- and Uo+ in a 2n + 1 dimensional space (Figure 2), we expect the asymptotic formulae to reflect this fact In summary, we have decomposed the proof of The 3 S' by the appropriate dimensionality drop of n - L This orem for Y E into five possible cases. We indeed happens, but to prove it requires to closely ex have fully analyzed the second case, using Watanabe's -; method, showing that the contribution to .lJ[N,Y] is amine the form off near the different minimumpoints. Sn This evaluation is complicated by the fact that the zero eN-•. The dominating contribution in the cases C3, planes intersect each other, and such cases are not cov C4, and C5, are all equal to eN-"¥ (the proof of ered by a classic Laplace approximation analysis. this claim is omitted due to space limitations). The dominating contribution in case C1 is only eN-¥. The minimum points of f are divided into five sets Also, the various border points of U0 do not con according to their location in U0 (Figure 2). tribute more than the corresponding internal points. Thus, ln.lJ[N, Y] = _ni1 1nN + 0(1). Hence, due to Cl. (x', u',s') EUoj \ Ui# Uo;. i Lemma 4, lnll[N, Y] =Njy -�InN+ 0(1), which C2. Uo 3 ES'. (x',u', s')= n1 i- confirms Theorem for Y •