Comparison Among Akaike Information Criterion, Bayesian Information Criterion and Vuong's Test in Model Selection: a Case Study of Violated Speed Regulation in Taiwan
Total Page:16
File Type:pdf, Size:1020Kb
VOLUME: 3 j ISSUE: 1 j 2019 j March Comparison among Akaike Information Criterion, Bayesian Information Criterion and Vuong's test in Model Selection: A Case Study of Violated Speed Regulation in Taiwan Kim-Hung PHO1;∗, Sel LY1, Sal LY1, T. Martin LUKUSA2 1Faculty of Mathematics and Statistics, Ton Duc Thang University, Ho Chi Minh City, Vietnam 2Institute of Statistical Science, Academia Sinica, Taiwan, R.O.C., Taiwan *Corresponding Author: Kim-Hung PHO (Email: [email protected]) (Received: 4-Dec-2018; accepted: 22-Feb-2019; published: 31-Mar-2019) DOI: http://dx.doi.org/10.25073/jaec.201931.220 Abstract. When doing research scientic is- Keywords sues, it is very signicant if our research issues are closely connected to real applications. In re- Akaike Information Criteria (AIC), ality, when analyzing data in practice, there are Bayesian Information Criterion (BIC), frequently several models that can appropriate to Vuong's test, Poisson regression, Zero- the survey data. Hence, it is necessary to have inated Poisson regression, Negative a standard criteria to choose the most ecient binomial regression. model. In this article, our primary interest is to compare and discuss about the criteria for select- ing model and its applications. The authors pro- 1. Introduction vide approaches and procedures of these methods and apply to the trac violation data where we look for the most appropriate model among Pois- The model selection criteria is a very crucial son regression, Zero-inated Poisson regression eld in statistics, economics and several other ar- and Negative binomial regression to capture be- eas and it has numerous practical applications. tween number of violated speed regulations and This issue is currently researched theoretically some factors including distance covered, motor- and practically by several statisticians and has cycle engine and age of respondents by using gained many attentions in the last two decades, AIC, BIC and Vuong's test. Based on results on especially in regression and econometric mod- the training, validation and test data set, we nd els. There are three most commonly used model that the criteria AIC and BIC are more consis- selection criteria including Akaike information tent and robust performance in model selection criterion (AIC), Bayesian information criterion than the Vuong's test. In the present paper, the (BIC) and Vuong's test, which are compared authors also discuss about advantages and disad- and discussed in this paper. AIC is rst pro- vantages of these methods and provide some of posed by Akaike [1] as a method to compare dif- suggestions with potential directions in the future ferent models on a given outcome. Meanwhile, research. BIC is proposed by Schwarz [20], is a criterion for model selection among a nite set of models. Vuong's test has been proposed by Vuong [24] in the literature aiming at selecting a single model c 2019 Journal of Advanced Engineering and Computation (JAEC) 293 VOLUME: 3 j ISSUE: 1 j 2019 j March regardless of its intended use. All three crite- criteria for choosing model including Akaike in- ria are the most widespread criteria for choosing formation criterion (AIC), Bayesian information model. criterion (BIC) and Vuong's test. In Section 3, these methods are applied to a real data which Until today, these problems have been stud- could help readers to easily assess them. Some ied and utilized in numerous areas. AIC has of suggestions and some potential directions for been researched and applied extensively in lit- the further research are devoted in Section 4. erature such as: Snipes et al. [19] employ AIC Finally, some conclusions and remarks are given and present about an example from wine rat- in Section 5. ings and prices, Taylor et al. [21] introduce in- dicators of hotel protability: Model selection using AIC, Charkhi et al. [4] research about asymptotic post-selection inference for the AIC, 2. Some of Criteria for Chang et al. [3] present about Akaike Informa- Model Selection tion Criterion-based conjunctive belief rule base learning for complex system modeling, etc. In this section, we present approaches and proce- In addition, BIC is also utilized extensively in dures of ubiquitous methods to choose the most literature for example: Neath et al. [16] intro- ecient model consisting of Akaike Information duce about regression and time series model se- Criteria (AIC), Bayesian Information Criterion lection using variants of the Schwarz information (BIC) and Vuong's test. criterion. Cavanaugh et al. [2] present about generalizing the derivation of the BIC. Weakliem [27] introduce about a critique of the Bayesian 2.1. Akaike Information information criterion for model selection. Neath Criteria (AIC) et al. [15] present about a Bayesian approach to the multiple comparisons problem. Neath et al. AIC is rst proposed by Akaike [1] as a method [17] present about the BIC: background, deriva- to compare dierent models on a given outcome. tion, and applications. Nguefack-Tsague et al. The AIC for candidate model is dened as fol- [23] focus on introduce about Bayesian informa- lows: tion criterion, etc. AIC := −2`(θ^jy) + 2K; (1) Similarly to AIC and BIC, Vuong's test [24] is also used largely in literature for instance: where K is the number of estimated parameters ^ Clarke [5] employ Vuong's test to introduce in the model including the intercept and `(θjy) a simple distribution-free test for non-nested is a log-likelihood at its maximum point of the model selection, Theobald [22] utilize Vuong's estimated model. The rule of choice: the smaller test to present a formal test of the theory of the value of AIC is, the better the model is. universal common ancestry, Lukusa et al. [13] use Vuong's test to evaluate whether the zero- 2.2. Bayesian Information inated Poisson (ZIP) regression model is con- Criterion (BIC) sistent with the real data, Dale et al. [6] perform model comparison using Vuong's test to estimate BIC is rst introduced by Schwarz [20], one of nested and zero-inated ordered probit mod- sometimes calls the Bayesian information cri- els, Schneider et al. [18] present about model terion (BIC) or Schwarz criterion (also SBC, selection of nested and non-nested item response SBIC) which is a criterion for model selection models using Vuong's test, etc. among a nite set of models. The BIC for can- Our main objective in this paper is to provide didate model is dened as follows: researchers an overview of the criteria in model selection for the trac violation data. The rest BIC := −2`(θ^jy) + K ln(n); (2) of the paper is organized as follows. In Section where n is a sample size; K is the number of 2, we present approaches and procedures of the estimated parameters in the model including the 294 c 2019 Journal of Advanced Engineering and Computation (JAEC) VOLUME: 3 j ISSUE: 1 j 2019 j March intercept and ^ is the log-likelihood at its f (Y jX ;Z ; α ) `(θjy) • m (α ; α ) = ln 1 i i i b1 ; maximum point of the estimated model. The i b1 b2 f2 (YijXi;Zi; αb2) where is the predicted rule of selection: the smaller the value of BIC fj (YijXi;Zi; αbj) ; is, the better the model is. The procedure for probability of an observed count for case i applying AIC and BIC are given as follows: from the model j, j = 1; 2, respectively. Step 1: Selecting candidate models which • Moreover for the complete case, V can be can be tted to the data set. easily obtained from the package pscl in R language, (Zeileis at el. [28]). Step 2: Estimating unknown parameters of models. At the signicant level α, the decision rule is given as follows: Step 3: Finding values of AIC and BIC by using the formulas (1) and (2), respectively. • If V > Qα/2, choose model 1. Step 4: Basing on the rule of choice, one can decide the most suitable model. • If V < −Qα/2, choose model 2. • If jV j < Qα/2, both models are equivalent. 2.3. Vuong's Test where Q is an upper quantile of standard nor- Vuong's test [24] is one of the ubiquitous cri- α/2 mal distribution at the level α=2. Similar to al- teria for choosing model and it is often used gorithms for AIC and BIC, to perform Vuong's to the data set with no missing values. Let test, we need to do through following steps: f1(Y jX; Z; W ; α1) and f2(Y jX; Z; W ; α2) be two non-nested probability models. Let and αb1 Step 1: Choosing candidate models which be a consistent estimator of and un- αb2 α1 α2 can be tted to the data set. der the model f1 and f2, respectively. Letting hypotheses Step 2: Estimating unknown coecients of models. • H0: The two models are equally closed to the true data. Step 3: Calculating V by using (3) : Model 1 is closer than model 2. • H1 Step 4: Basing on the rule of choice, one can select the most compatible model. The Vuong's test statistics is provided as follows; (see Mouatassim and Ezzahid [14]): Note that: Step 1 is a very important step in n practice, basing on characteristics of the data p 1 P n mi (αb1; αb2) set, one can choose some reasonable models to n i=1 V = V (αb1; αb2) = ; t. For example, if the data set is a binary, then h (αb1; αb2) candidate models are considered such as logis- (3) tic regression model, probit model and so on. If where the data set is class of count data, one can uti- lize some of models such as: Poisson regression 2 model, binomial regression model, negative bi- h (α1; α2) b b nomial regression model and so on.