IR Applications Using Advanced Tools, Techniques, and Methodologies
Association for Volume 5, June 30, 2005 Institutional Research Enhancing knowledge. Expanding networks. Professional Development, Informational Resources & Networking Analyzing Student Learning Outcomes: Usefulness of Logistic and Cox Regression Models Chau-Kuang Chen, Ed.D., Meharry Medical College
Abstract choice of regression models depends largely on the Logistic and Cox regression methods are practical research objectives and the measurement scales of the tools used to model the relationships between certain outcome variable in the study. Logistic regression analysis student learning outcomes and their relevant explanatory is a suitable technique to investigate the effects of the variables. The logistic regression model fits an S-shaped explanatory variables on the binary outcome, either success curve into a binary outcome with data points of zero and or failure. Moreover, it allows investigators to perform one. The Cox regression model allows investigators to predictions based on the resulting model. The Cox study the duration and timeline of the critical events, regression model becomes the appropriate choice for which are also a binary and dichotomous measure. This studying the risk factors in relation to the duration and paper introduces logistic and Cox regression models by timeline until occurrence of the critical event, which is also illustrating examples, implementing step-by-step SPSS a binary measure. However, the model itself does not gear procedures, and further comparing the similarities and up for the purpose of future predictions. differences of the model characteristics. Logistic regression Logistic regression analysis has been widely used to analysis was conducted to investigate the effects of the study student performance, enrollment, graduation, and explanatory variables such as pre-admission variables, post-graduate employment status, which are restricted to college cumulative GPAs, and curriculum tracks on student a binary outcome such as ”success or failure’, ‘enrolled or licensure examination. Moreover, logistic regression not enrolled’, ‘graduated or not graduated’, and ‘primary analysis was employed to quantify the effect (odds or care or non-primary care specialty’ (Case, et al., 1994; odds ratio) of specific explanatory variables on the binary Sadler, et al, 1997; Strayhorn, 2000; and Hojat, 1995). The outcome holding other variables constant. With regards to logistic regression method allows investigators to estimate Cox regression analysis, the outcome variable of interest and interpret the effects of the explanatory variables on the was the timing of experiencing academic difficulty— binary outcome. The maximum likelihood estimation dismissal, withdrawal, and leave of absence. The Cox technique is readily available to estimate the regression regression model was used to detect when students were coefficients for the non-linear model equation. This most likely to experience academic difficulty beyond their technique maximizes the probability of getting the observed matriculation. The model also allowed the investigators to data given the fitted regression coefficients (Hosemer and measure the effect (relative hazard or hazard ratio) of Lemeshow, 1989; Eliason, 1993; and Pampel, 2000). The specific risk factors on the academic difficulty after adjusting model equation renders itself to perform future predictions for other factors. Identifying the occurrence of critical to assess the predictive power. In addition, investigators events along with the explanatory variables, college can transform the model equation into the odds and odds administrators and faculty could implement intervention ratio of the event occurrence. The odds or odds ratio is a strategies to ensure student success. measure of the direction and strength of the relationship between the explanatory variables and the effects of such Introduction variables on the binary outcome. The logistic regression Regression analyses are statistical procedures that model quantifies the association between the explanatory describe the relationship between an outcome (dependent, variables and the outcome variable of interest after adjusting or response) variable and one or more explanatory for other explanatory variables (Matthews and Farewell, (independent, or predictor) variables or risk factors. The 1996). Therefore, it is a useful tool to analyze the
©Copyright 2005, Association for Institutional Research IR Applications, Number 5, Analyzing Student Learning. . . dependence of a binary outcome on a set of explanatory regression model allows investigators to estimate and variables. interpret the effects of the risk factors on the event occurring. Cox regression analysis is a branch of survival analyses, The partial maximum likelihood estimation technique is which is used to analyze the timeline until a critical event used to estimate the regression coefficients of the model occurs (Cox, 1972; Cox and Oakes, 1984; and Miller equation (Hosemer and Lemeshow, 1989; and Kleinbaum, 1981). Survival analyses discussed in this paper appear 1996). The interpretation of the regression coefficients in in diverse fields under a variety of names. In biomedical the Cox regression model is virtually the same as in and health sciences, these are called survival analyses logistic regression analysis. For the positive regression because the event of interest is mortality or death. In coefficient, the hazard of a student experiencing the critical sociological research, these are often referred to as event event increases as the value of the risk factor increases. history studies. In engineering studies, the term reliability Moreover, if the regression coefficient is zero, the value of theory is commonly applied to the lifetime of an object. the relative hazard becomes one, indicating that the hazard Cox regression analysis allows investigators to answer of a student experiencing the critical event is not affected two questions simultaneously: ”Has the critical event of by the risk factor. However, for the negative regression interest occurred?” and ”Which risk factors contribute to coefficient, the hazard of a student experiencing the critical the event occurrence?“ (Singer and Willett, 1991). In Cox event decreases when the value of the risk factor increases. regression analysis, the hazard and survival functions are For example, if student dropout is the critical event of expressed as a function of time and the risk factors, interest, the Cox regression model can be used to study which enable investigators to address research questions the timeline and the risk factors associated with student in education such as: ”How many semesters elapse dropout. before students drop out of school?”, ”At what year are The model assumption of Cox regression is very different students likely to graduate from the college?”, and ”What from that of the logistic regression model. Logistic explanatory variables contribute to students’ dropout or regression assumes that residuals are normally distributed graduation?” Numerous studies related to the timing of with a mean of zero and a constant variance. However, the student departure from college were conducted during the Cox regression model assumes that the hazard ratio for past decade (DesJardins et al., 1997; Han and Ganges, different students with different values of the risk factor is 1995; Huff and Fang, 1999; Ronco, 1994; Singer and independent of the time variable (Belle, et. al., 2004; and Willett, 1993; and Willett and Singer, 1991). Moreover, in Kleinbaum, 1996). Thus, when the two hazard functions the Cox regression model, investigators can determine are proportional, the title ‘proportional hazards model’ is the effectiveness of college programs by comparing the applied to Cox regression. The proportionality of the model patterns of the survival functions. For example, if the assumption implies that the hazard ratio is constant over survival function for Group A consistently lies above that time. Therefore, the verification of the model assumption of Group B, the intervention program would appear more is essential to ensure the model appropriateness. effective for Group A during the study period. In an attempt to demonstrate the usefulness of the two In the Cox regression model, the hazard function is a methods, this paper provides a brief overview and example function of a set of risk factors and baseline hazard. The illustrations. In particular, it offers step-by-step SPSS PC risk factors are similar to independent variables of the commands used for analyzing a binary result or event linear regression model except they appear to be non- occurring related to student-learning outcome. In this linear in the exponential expression. The baseline hazard study, the research objectives and questions are defined is similar to the constant or intercept of the linear regression as the foundation of the investigation. Next, the study model. It describes the overall level of risk and reveals the variables are gathered to form a research-oriented database, main effect of the time variable (Singer and Willett, 1991). which are extracted from the existing student tracking The hazard function displays the timeline fluctuation of a system. In addition, the principle of model strategies student experiencing the critical event. The student with serves as a guideline to build the models. It comprises all the greater risk has a higher value of the hazard function relevant variables at the initial phase of the model fitting as compared to one with the lesser risk at that same and achieves parsimony and consistency upon model particular time. Thus, a high hazard function indicates the completion. The paper focuses mainly on the model critical event is more likely to occur. Conversely, a low equations, the assessment of model fittings, the hazard function shows that the critical event is less likely interpretation of odds ratio and hazard ratio, and the to occur (Kleinbaum, 1996). For instance, investigators importance of the model assumptions. Moreover, the two can detect the ineffectiveness of the support services methods are placed side by side to analyze the similarities program by comparing the hazard functions. If the hazard and differences of their characteristics. function is higher for Program A than Program B, Program A appears to be ineffective during the study period. Similar to the logistic regression analysis, the Cox
2 IR Applications, Number 5, Analyzing Student Learning. . . .
Literature Review university prior to graduation. The results from this study Using logistic regression analysis (Sadler et al., 1997), suggest that students in the risk group participate in the the second semester enrollment was discovered to be intense and sustained intervention program. significantly associated with the explanatory variables The result of survival analysis indicated that some such as tuition benefits, state residency, race, high school students experiencing academic difficulty remained at GPA, and summer orientation. The results of this study risk throughout the first three years of medical school could help the institution implement appropriate (Fang, 2000). The research finding implied that academic interventions that target students at-risk for leaving. To support programs focusing only on the entering year might investigate student retention at Historical Black Colleges not be sufficient to fully address this extended period of and Universities, a logistic regression model was risk. The study results of survival analysis (Huff and Fang, constructed (McDaniel and Graham, 1999). The research 1999) demonstrated that the increase of the relative risks findings showed that students were more likely to return of students experiencing academic difficulty were for the second year if they earned higher ACT test scores. associated with low MCAT scores, low science GPA, low In addition, students aspiring to achieve doctoral and undergraduate institutional selectivity, being a woman, professional degrees were more likely to persist than being a member of a racial-ethnic underrepresented students with lower degree aspirations. These study results minority, or being older. Clearly, investigators in higher could assist college administrators and faculty to select education can use survival analysis to identify students students who are more likely to succeed at their institution. who are most likely to experience the occurrence of By using logistic regression analysis, one study found critical events—dropout, withdrawal, dismissal, and delay that student performance in the college program predicted of graduation. Knowing the time-to-event occurrence and whether or not students would apply to medical school, related risk factors, college administrators and faculty can get accepted by medical school, and graduate from medical effectively implement the intervention strategies to increase school (Strayhorn, 2000). In constructing this logistic the likelihood of student success. regression model, the significant explanatory variables for the probability of passing the United States Medical Logistic Regression Equation Licensure Examination (USMLE) Step 1 were medical The logistic regression model is primarily written as Y college admission test scores, medical school freshman = P(X) + E, where Y is the binary outcome—event occurring GPAs, sophomore course performances, and financial aid coded as 1 or event not occurring coded as 0 (Hosemer support (Chen et al., 2001). These study results could be and Lemeshow, 1989). The probability P(X) of obtaining used to document the effectiveness of academic programs the binary outcome is considered to be the estimated and applicant screening. Logistic regression analysis (Case value given the explanatory variables (X) are known et al., 1994) was also conducted to investigate the observations. The error term (E) also called the residual, relationship between the initial performance of identifying represents the difference between the actual binary examinees and the ultimate pass rates of the USMLE outcome (Y) and the estimated probability P(X). The model Steps 1 and 2. The study results showed that the probability is commonly written as P(X)=eZ/(1+eZ) or the equivalent of ultimately passing both Steps 1 and 2 was significantly form of P(X) = 1/(1+e-Z), where Z stands for a linear related to the initial score achieved. Based on the availability combination of bo + b1X1 + b2X2 +...+ bpXp (Hosemer and of student performance measures, professional activities, Lemeshow, 1989). The ”e” term in the equation is the satisfaction results, and research productivities, a logistic base of the natural logarithm, which is approximately regression model was able to predict primary care and 2.718. The regression coefficients (b) are unknown non-primary care status from the significant predictors, parameters to be estimated. Moreover, the model assumes which include specialty interest, professional plan, and that residuals have a mean of zero and a constant variance interests expressed in medical school (Hojat, 1995). The of P(X)[1-P(X)], which are statistically independent of one study results could be used to document the different another. tracks of physician training and education. In a logistic regression model, the probability of a By constructing hazard models for different college student obtaining a binary outcome is always in the range exiting modes (graduation, withdrawal, transferal, dismissal, of zero to one, regardless of the value of Z. When Z is and leave of absence), investigators could better understand negative infinity for the model equation P(X) = 1/(1+e-Z), the different factors that influence student behavior (Singer the probability of a student obtaining a binary outcome is and Willett, 1991). A survival analysis study indicated that virtually zero (e-Z becomes positive infinity because of the academic resource index significantly influenced minus negative infinity of Z; and one divided by one plus graduation (DesJardins and Moye, 2000). Using survival positive infinity equals zero). Also, when Z increases from analysis, another study (Han and Ganges, 1995) revealed negative infinity to zero for the model equation P(X)=eZ/ the occurrence of crisis for a selected group of students (1+eZ), the probability of a student obtaining a binary who persisted for four or more years and still left their outcome increases from zero to one half (eZ equals one
3 IR Applications, Number 5, Analyzing Student Learning. . . when Z is zero; and the model equation shows that one regression model. Investigators may conclude that at divided by two equals one half). When Z increases from least one of the explanatory variables contributes to the zero to positive infinity for the model equation P(X)=eZ/ probability of the student obtaining a binary outcome if the (1+eZ), the probability of a student obtaining a binary p value is less than the predetermined significance level outcome increases from one half to one (positive infinity (a=.01, .05, or .001). In logistic regression analysis, the of eZ divided by one plus positive infinity equals one). The Wald statistic is commonly used to test the significance logistic regression equation is the steepest when the of the individual logistic regression coefficients (Hosemer probability equals 0.5 and flattens out on both top and and Lemeshow, 1989). Moreover, the logic of hypothesis bottom tails as the probability value approaches zero and testing for the Wald test in logistic regression is similar to one (Dey and Astin, 1993). All of the characteristics the t test in the linear regression model. In the case that mentioned above allow investigators to fit the S-shaped a specific regression coefficient is significantly different curve into a data set of binary outcomes containing values from zero, the corresponding explanatory variable of zero and one. significantly contributes to the probability of a student The logistic regression equation is a non-linear curve obtaining the binary outcome. rather than a straight-line. Thus, the parameter estimation method for the logistic regression model is called the Interpretations of Odds, Log Odds, Odds maximum likelihood estimation, which is different from Ratio, and Delta P the least-square estimation method for the linear regression By means of the mathematical transformation, model. Therefore, it is necessary to use the iterative investigators can transform an estimated probability into process via the computer software to find better odds, log odds, odds ratio, and delta p, respectively. An approximations of the logistic parameters that satisfy the example of a simple logistic regression model containing log likelihood equation. When the log likelihood equation only one explanatory variable is used to illustrate this is satisfied or maximized, the probability of an individual transformation process and interpret the resulting statistics. obtaining the observed data is maximized (Eliason, 1993; The odds can be derived as a ratio of the two probabilities, Pampel, 2000; and Hosemer and Lemeshow, 1989). In which is written as odds = P(event occurring)/P(event not other words, the solution of the log likelihood equation occurring) = P(X)/[1-P(X)] = e(bo + bX), where P(X) = e(bo + bX)/ implies that the effect of the explanatory variables on the [1 + e(bo + bX)] (Hosemer and Lemeshow, 1989). The term probability of event occurrence is also maximized. ebo refers to the value of the odds for the constant and eâ When all regression coefficients are estimated, the represents the value of the odds related to the explanatory values of the explanatory variables can be plugged into variable. The odds (eb) can be interpreted as a stand-alone the logistic equation to perform predictions. For example, statistic without the involvement of the change (increase if the probability of a student obtaining a binary outcome or decrease) of the explanatory variable. For instance, is greater than or equals to a cut-off point (defaulted value when the value of the odds equals three (0.75/0.25), it of one half) that student is placed into the success group. means that the odds of student obtaining a binary outcome On the other hand, if the probability of a student obtaining (event occurring) is three times higher than that of the a binary outcome is less than one half, then that student same student not obtaining a binary outcome. However, if is categorized into the failure group (SPSS, Inc, 2002). the value of the odds equals one (0.5/0.5=1), the odds of Therefore, by comparing the predictive results and actual obtaining a binary outcome is equivalent to the chance of observations, investigators can calculate the prediction obtaining a head when flipping a fair coin. accuracy for the success and failure groups, as well as Because the base of the natural logarithm is applied to for the combined success and failure group. The cut-off both sides of the odds equation mentioned above, the log point for predicting the binary outcome can be adjusted odds known as logit can be written as loge(odds) = either upward or downward in order to increase specificity loge{P(X)/[1-P(X)]} = bo + bX (Hosemer and Lemeshow, (accuracy of the prediction results for the failure group) or 1989). Given the linear relationship exists between the sensitivity (accuracy of the prediction results for the dependent variable (log odds) and the independent variable successful group) of the model equation. (X), the interpretation of the effect of a specific explanatory To assess the overall model fitting and the significance variable is quite similar to linear regression analysis. The of specific explanatory variables, the logistic regression log odds can be interpreted as a unit change in the model allows investigators to perform the likelihood ratio explanatory variable leads to the magnitude change (â and Wald tests. The likelihood ratio test compares the units) of the log odds of a student obtaining a binary likelihood for the intercept only model to the likelihood for outcome. The log odds provides only the direction of the the model with the explanatory variables (Eliason, 1993; relationship, but is limited in providing meaningful Pampel, 2000; and Hosemer and Lemeshow, 1989). The information about the effect of the explanatory variable. As logic of hypothesis testing for the likelihood ratio test in a result, it is difficult to understand the meaning of log logistic regression is similar to the F test in the linear odds. Therefore, the odds ratio should be used to interpret the effect of the explanatory variable.
4 IR Applications, Number 5, Analyzing Student Learning. . . .
The definition of odds ratio is different from that of the interpreted that an average one-point scale of increase in odds. Recall from the previous paragraph, that the odds is anxiety score leads to a decrease in the odds of a a ratio of the two probabilities when one is confined to the student’s success by a factor of 2, which is the reciprocal event occurring and the other refers to the event not of 1/2. occurring. The odds ratio is the ratio of two odds when the Although odds and odds ratio are commonly used for different values of the explanatory variable apply to them. interpreting logistic regression results, the delta-p statistic The odds ratio is known as odds change, which describes can be used to measure the effect of a specific explanatory the proportionate change in the odds for one-unit difference variable on the probability of obtaining the binary outcome in the explanatory variable (Hosmer and Lemeshow, 1989; (Peng et al., 2002). Using financial aid and student success and Menard, 1995). It is a measure of the association as an example, a delta-p of .10 for a student receiving between the explanatory variable and the odds of the financial support is interpreted as increasing the probability individual obtaining a binary outcome. of a student’s success by 10% as compared to student An odds ratio of greater than one shows that the odds who does not receive financial support. The delta-p is of an event occurring increases when the value of the transformed from the regression coefficient using a method explanatory variable increases (Menard, 1995). It originally recommended by a researcher, Peterson, in demonstrates the existence of a positive relationship 1984 (Peng et al., 2002). The odds ratio can be found in between the explanatory variable and the effect of that SPSS printouts when logistic regression analysis is particular variable. The odds ratio is simplified to be an performed, however, the delta-p statistic cannot be exponential expression (e1). One can use the odds of ebo generated by the SPSS PC Version 12.0 commands. + bX, where b > 0 to illustrate the positive relationship and interpret the meaning of the odds ratio. For the odds of Example of Logistic Regression Analysis ebo + b when X=1 versus the odds of ebo when X=0, the A logistic regression model was constructed for a resulting odds ratio is e1, which is a ratio of ebo + b and ebo. sample of 200 matriculated students randomly selected Again, for the odds of ebo + 2b when X=2 versus the odds of from the population in 1993-1997. The study attempted to ebo + b when X=1, the resulting odds ratio is still eß, which answer the research question, ”How well can the USMLE is a ratio of ebo + 2b and ebo + b. For example, this ratio Step 1 pass status be predicted by the explanatory implies that an average one-unit of increase in the variables: demographics, admission test scores, explanatory variable leads to an increase in the odds of undergraduate GPA, medical school cumulative GPA, obtaining a binary outcome by a factor of eb. Also, using course grades, and financial aid support?“ Thus, the tutorial program and student success as an example, if research objectives of this study were: (a) to identify the odds ratio equals 2, it can be interpreted that an explanatory variables that significantly contribute to the average one-month (time frame) of increase in the probability of passing the USMLE Step 1; and (b) to implementation of a tutorial program contributes to an provide insight into the measure (odds or odds ratio) of the increase in the odds of a student‘s success by a factor of effects of the explanatory variables. 2. In this study, the outcome or response variable was However, an odds ratio of less than one indicates that binary, i.e. USMLE Step 1 pass status, which was (labeled the odds of an event occurring decreases when the value as p-f-grp for the SPSS command), coded 1 for the pass of the explanatory variable increases (Menard, 1995). In group and 0 for the fail group. The probability of a student this case, the odds ratio is simplified to be an exponential passing USMLE Step 1 was a continuous scale between expression (e-ß). One can use the odds of ebo - bX, where zero and one. The explanatory variables were a combination b > 0 to illustrate the negative relationship and interpret of the discrete measure (gender, race, and medical school the meaning of the odds ratio. When the explanatory curriculum track) and the continuous measure variable (X) increases by one unit (from X=0 to X=1), the (undergraduate basic sciences average, undergraduate odds change increases by a factor of e-b, which is the ratio GPA, medical college admission test scores—MCAT of ebo - b and ebo. Moreover, when the explanatory variable physical sciences, MCAT biological sciences, MCAT verbal (X) increases by one unit (from X=1 to X=2), the odds reasoning scores, medical school freshman GPA, number change increases by a factor of e-b, which is the ratio of of sophomore courses failed, and financial aid loan ebo - 2b and ebo - b. It is difficult to convey the concept of odds amount). The coding scheme for the discrete measurement change for a fraction between zero and one. Therefore, a was gender (1 for male and 0 for female), ethnicity (1 for better way of interpreting a negative regression coefficient African American and 0 for Non-African American), is to say an average one-unit of increase in the explanatory historical black colleges and universities status (1 for variable leads to a decrease in the odds for obtaining a HBCU graduate and 0 for Non-HBCU graduate), and medical binary outcome by a factor of eb, which is an inverse of school curriculum track (labeled as curr_grp for the SPSS e-b (0 < e-b < 1) Using anxiety score and student success command, 1 for four-year curriculum track and 0 for five- as an example, if the odds ratio equals 1/2, it can be year curriculum track).
5 IR Applications, Number 5, Analyzing Student Learning. . .
The logistic regression model was constructed using research findings. The collinearity exists if one explanatory the forward selection procedure. At each step, the variable is a function of the other explanatory variables. explanatory variable with the smallest significance level There was no evidence of model collinearity because the 2 for the Wald statistic was entered into the model. The tolerances (TOLi = 1 – Ri , where Ri is a multiple correlation default entry criterion for the explanatory variables was a coefficient between one explanatory variable (Xi) and the p value of .05. The Wald statistics for all variables in the other explanatory variables (Xs) in the equation were large model were examined and the explanatory variable with (a range of .571 to .971) based on a linear regression the largest p value for the Wald statistic was removed printouts (Norusis, 1985; and Menard, 1995). Moreover, from the model. The default removal criterion was p=.10. the assumption of the logistic regression model was If no explanatory variables met the removal criterion, the satisfied because the histogram of residuals appeared next eligible variable was entered into the model. The normally distributed with a mean of zero, and the residuals iteration process for selecting explanatory variables on the scatter diagram also appeared to be parallel with continued until no additional variables met the entry or the X-axis, the indication of a constant variance. removal criterion. As shown in the footnote of Table 1, the logistic regression model containing the four explanatory variables SPSS PC Commands for Logistic fits the data well based on the model chi-square test Regression Analysis (X2=83.40, df=4, and p<.001) and the small value of the – The eight steps of the SPSS PC Version 12.0 2 log likelihood (159.23). Also, the improvement chi square commands required to produce logistic regression analysis value (X2=38.01, df = 4, and p<.001) indicated that the are as follows: model fits the data better than it had initially. The model- Step 1 - Click Analyze, click Regression, and click fitting statistic, namely the pseudo R-square, measured Binary Logistic; Step 2 – Click on dependent variable the success of the model by explaining the variations in (p_f_grp), and click
6 IR Applications, Number 5, Analyzing Student Learning. . . .
Table 1 medical school freshman GPA was strongly correlated Logistic Regression Model for Predicting with the Step 1 performance while the number of courses USMLE Step 1 Pass Status failed during sophomore year was negatively associated with the Step 1 performance. It is evident that basic Variables in Logistic Standard Odds Ratio (eß) the Equation Regression Error of ß SE (ß) science disciplines have a predictive power for the medical Coefficient (ß) licensure examination. This may imply that the medical MCAT Physical 0.62*** 0.209 1.87 Sciences Score school has implemented an adequate basic sciences MCAT Biological 0.51*** 0.144 1.66 curriculum, course instructions, and student assessment Sciences Score compared to other medical schools in the nation. Number of Sophomore -2.53*** 0.639 0.08 Courses Failed Medical School Freshman GPA 1.89*** 0.533 6.59 Survival Analysis Constant -12.21*** 2.317 0.00 The Cox regression model can be used to study the *** p < .001 based on the Wald test [X2 = (ß/SE(ß))2 with df=1] risk factors that are significantly associated with the N=200 timing of the critical event. The timeline known as survival -2 Log Likelihood: -2LL=159.23 Model Chi Square Test: X2=83.40, df=4, and p<.001 time refers to the length of the time interval (month, Improvement Chi Square Test: X2=38.01, df=4, and p<.001 Pseudo R-square: Nagelkerke R-square = 0.49 semester, or year) between the onset time of the study Prediction Accuracy: Pass Group (92%), Fail Group (63%), and Combined Group (84%) and the timing of the event occurrence (Steinberg, 1999). The critical events cover not only negative and unpleasant a student advances to the pass group of the USMLE Step experiences, such as dropout and academic difficulty, but 1. However, if the estimate probability is less than 0.5, a also positive and pleasant results, such as passing certain student falls into the fail group of the USMLE Step 1. examinations and graduation. The effect (odds ratio) of each explanatory variable on Survival analysis is a unique statistical approach for the pass status of the USMLE Step 1 is shown in the last analyzing uncensored and censored data in a single study column of Table 1. When an average of the medical school (Singer and Willett, 1991). With regard to uncensored freshman’s GPA increased by one point, the odds of a data, they are often referred to as a complete set of student passing USMLE Step 1 increased by a factor of timelines about the event occurring as opposed to some 6.59. This value indicates that the effect of the medical censored cases for which the event of interest does not school freshman GPA on the USMLE Step 1 performance occur during the study period (Kleinbaum, 1996; and was very high. A change in the MCAT scores definitely led Steinberg, 1999). For instance, if the graduation status to a change in the USMLE Step 1 performances. If an within a two-year master’s program is the event of interest, average of the MCAT physical sciences score increased the timely graduation in two years can be considered as by one point, the odds of a student passing USMLE Step the uncensored data. Similarly, if the academic difficulty 1 increased by 87%. Meanwhile, when an average of the (dismissal, leave of absence, and withdrawal) from a MCAT biological sciences score increased by one point, seven-year doctorate program is the critical event of interest, the odds of a student passing USMLE Step 1 increased the time-to-event for students A, F, and G illustrated in by 66%. Furthermore, the odds of a student passing Table 2 belongs to the uncensored data. The censored USMLE Step 1 increased by a factor of 0.08 when the data provide only partial information of timelines, which are number of courses failed in sophomore year increased by collected under these circumstances: (1) students do not one. In other words, the odds of a student passing USMLE experience the event of interest (academic difficulty) before Step 1 decreased by a factor of 12.5 (an inverse of 0.08) the study ends (student B - graduated, student D - still because of one additional sophomore course failed in the enrolled); (2) student withdraws from the study (student C basic science disciplines. This implied that the effect of - deceased); and (3) student is lost to follow-up during the sophomore course performances on the USMLE Step 1 study period (student E - transferred out). Because the pass rate was extremely high. A noteworthy finding is that occurrence of time-to-event for students (students B, C, the odds of a student passing the licensure examination D, and E) is not evident and hidden from view, the term depended heavily on sophomore course performances ‘censored’ is applied to it. In essence, censoring occurs and medical school freshman GPAs as compared to when investigators have some partial information about those of pre-admission variables—MCAT physical sciences the time-to-event of the individual students, but they do not and biological sciences scores. know the exact time variable. The survival time is measured In summary, the medical school freshman GPA and in years from the time students enter the study. Students sophomore course performance were significant explanatory C and F entered the study at year two and one, respectively, variables for the USMLE Step 1 performance. The two which are different from the rest of the group who entered MCAT scores (physical and biological sciences) were at the beginning of the study. also significant explanatory variables, regardless of other To understand survival analysis, one needs to begin pre-admission variables such as gender, ethnicity, and with the survival function S(t). As indicated by researchers undergraduate BSA and GPA scores. As expected, the
7 IR Applications, Number 5, Analyzing Student Learning. . .
(Braun and Zwick, 1993; Elandt-Johnson and Johnson, Relationship between Hazard and 1980; and Lee, 1992), the survival function can be Survival Functions expressed as S(t) = 1 – F(t) = P(T>t), where T represents The hazard function may be written as the conditional a specific time of the event occurrence. In other words, failure rate = h(t), where the survival function cumulates the proportion of individuals surviving longer than the time variable, which is a h(t)=lim{P[t < T < ( t+Dt) | T> t] / [Dt]} complement of the cumulative distribution function. The Dt ---> 0 survival function gradually decreases when more individuals experience the critical event. It depicts theoretically the =lim{P[t < T < ( t+Dt) and T > t] / [Dt P(T>t)]} Dt ---> 0
Table 2 The definition of conditional probability, P(A|B) = P(A and Example of Uncensored and Censored B) / P(B) (Meyer, 1970), is applied to the hazard function Data for the Doctorate Program above. That is P(A|B)= P [t < T < (t+Dt) | T > t]; P(A and
The Outcome Variable is Academic Difficulty: Dismissed, Withdrew, and Leave of Absence because of B) = P[t < T< ( t+Dt) and T > t]; and P(B) = P(T > t), where Academic Reasons A represents [t < T < (t+Dt)], a specific survival time (T) Years Time Variable Status Variable* 0 1 2 3 4 5 6 7 of the event occurrence in a narrow time interval between ------+------+------+------+------+------+------+ ------t and t + Dt, and B refers to (T > t), a specific survival time A ------X (dismissed) 2 1 (T) of the event occurrence greater than the time variable B ------graduated 6 0 C ------deceased 3 0 (t). D------study end (still enrolled) 7 0 E ------transferred out 5 0 h(t)=lim{P[t < T < (t+Dt) and T > t] / [ Dt P(T>t) ]} F ------X (leave of absence) 2 1 Dt ---> 0 G------X (withdrew) 5 1 ------+------+------+------+------+------+------+ 0 1 2 3 4 5 6 7 = lim{P[t
8 IR Applications, Number 5, Analyzing Student Learning. . . . the hazard function can be reiterated as the change (Kleinbaum, 1996). By comparing the survival curves for (slope) of an individual experiencing the hazard per unit the experimental group (with the intervention strategy) and time given that individual has survived longer than time (t). the control group (without the intervention strategy), There is a clearly defined relationship between the investigators can determine the effectiveness of the hazard and survival functions. Investigators can derive the intervention strategy if the experimental group appears to hazard function from information regarding the survival be superior. function. As indicated by researchers, h(t) = – [d S(t) / dt] Table 3 / S(t) = – d[logeS(t) / dt] (Braun and Zwick, 1993; Elandt- Johnson and Johnson, 1980; and Lee, 1992). Note that Example of Calculating the Estimated Survival Function S(t) for the Doctorate Program* the cumulative hazard function H(t) = – logeS(t) or H(t) = – ln S(t) because the integral (or sum) is applied to both The Outcome (or Event) Variable of Interest is Academic Difficulty: Dismissed, Withdrew, and Leave of Absence because of Academic Reasons sides of this equation—h(t) = – d[loge S(t)/dt]. Investigators Time Risk Number Number Estimated S(t)** can also derive the survival function from information (in years) Set*** of Events of Censored or % of Surviving t R(t) regarding the hazard function. Because h(t) = – [d S(t) / i i dt] / S(t), the mathematical expression of the survival 0 7 students' survival time > 0 year 0 0 1 2 7 students' survival time > 2 years 2 1 1 x 5/7 = .7143 function can be written as S(t)=e[-òh(u)du], where the integral 5 4 students' survival time > 5 years 1 3 .7143 x 3/4 = .5357 (ò) is denoted by the area under the curve between u = zero * Survival times (in years): 2+, 2+, 3, 5+, 5, 6, and 7 come from Table 2, where + stands for the and u = t (Elandt-Johnson and Johnson, 1980; Kleinbaum, occurrence of academic difficulty ** Kaplan Meier formula: P(B) P(A|B) = P(A and B), e.g., S(t=5)=(.7143) (3/4) =.5357 *** Each student in R(t ) has a survival time = t 1996; and Lee, 1992). Therefore, it is clear that the survival i i function S(t) decreases as the hazard function h(t) increases, and vice versa. Cox Regression Equation Unlike the Kaplan-Meier estimator, the Cox regression Kaplan-Meier Survival Analysis method allows investigators to generate the hazard function Before proceeding to the Cox regression model, it is as a function of the time variable, risk factors, and baseline imperative to describe several terms that are frequently hazard. Investigators can calculate the measure (relative used in survival analysis: Kaplan-Meier estimator, survival risk or relative hazard) of the risk factors and interpret the function, conditional probability, and log-rank test. The hazard ratio. The hazard ratio is the ratio of two hazard Kaplan-Meier estimator allows investigators: (1) to use the functions that allows investigators to measure the definition of conditional probability to derive survival functions association between the risk factors and the effects of for distinct groups; and (2) to determine the program such factors on the risk functions. effectiveness by comparing the survival function among The Cox regression model can be expressed as h(t, X) b1X1 + b2X2 +...+ bpXp groups, respectively. = ho(t)e (Hosmer and Lemeshow, 1999; Because students’ survival for the subsequent time Kleinbaum, 1996; and Lee, 1992). The hazard function, interval depends on students’ survival from the previous h(t, X), is a function of the baseline hazard ho(t) and the time interval, the Kaplan-Meier formula is merely the risk factors (X). The baseline hazard function is similar to application of the conditional probability (Belle, et al. the constant of the linear regression model, which 2004; and Kleinbaum, 1996), which can be expressed as represents the value of the hazard function before the risk P(A and B) = P(B) P(A|B) as illustrated in Table 3. The factors are taken into account. The base of the natural formula can be interpreted as the probability of the logarithm is denoted by e that equals approximately 2.718. occurrence of the joint events A and B equals the probability The risk factors may include continuous and categorical of the event B multiplied by the probability of the event A, variables. The continuous variables, e.g., grade-point- given the occurrence of event B. Event A (After) refers to averages and test scores are measured by an interval or the occurrence of the student surviving for the subsequent ratio scale. The categorical variables, e.g., gender and time interval. Event B (Before) represents the occurrence course grades, are measured by a nominal and ordinal of the student surviving for the previous time interval. scale, respectively. The regression coefficients (b) are Applying the visual examination method on the survival unknown parameters to be estimated by the partial curves, investigators can detect the difference between maximum likelihood estimation approach (Hosemer and two survival functions. For example, if two survival functions Lemeshow, 1989; and Kleinbaum, 1996). The term ‘partial’ are separated in the first half of the study period, but likelihood is applied because the likelihood equation thereafter, are somewhat closer to each other, then a calculates probabilities only for cases of the event large gap forms. This suggests that the intervention strategy occurrence rather than all cases. For a specific value of is more effective earlier during the study period. In Kaplan- the time variable, the hazard function depends on the b1X1 + b2X2 +...+ bpXp Meier survival analysis, the log-rank test allows investigators quantities of e and ho(t). It exhibits graphically to test the significant differences of survival functions at the variation in the time-to-event occurrence (Kleinbaum, different follow-up times among the study groups 1996). Because the hazard function is a non-linear curve,
9 IR Applications, Number 5, Analyzing Student Learning. . . it is necessary to use the iterative process to find better student experiencing the critical event decreases as the approximations of the regression coefficients that satisfy value of the risk factor increases. This implies that the the partial likelihood equation. relative hazard is negatively associated with the risk factor. To assess the model goodness of fit, the Cox regression Investigators should focus on the hazard ratio (HR) model allows investigators to address the research when they study the effects of the risk factors on the question, ”Does the model containing the variables in critical event in Cox regression analysis. The value of the question tell us more about the outcome variable than a hazard ratio reflects the strength of the relationship between model that does not include these variables?” (Willett and a specific risk factor and the effect of that factor. The Singer, 1991). In particular, it allows investigators to perform hazard ratio is a ratio of two hazard functions, which can bX the likelihood ratio test, which compares the likelihood for be expressed as HR = [hM(t)] / [hN(t)] = [ho(t).e ] / bX* b(X – X*) the intercept only model to the likelihood for the model [ho(t).e ] = e (Kleinbaum, 1996). The hazard ratio containing the risk factors within each model. The logic of is the ratio of the relative hazard to the risk factor (X) hypothesis testing for the likelihood ratio test is the same changes, i.e., the relative hazard of X = 0 compared to the for Cox regression and logistic regression models. Based relative hazard of X = 1. The hazard ratio decreasing or on the significance of the likelihood ratio test, investigators increasing is based on the positive or negative regression can claim that at least one of the risk factors contributed coefficient. The hazard ratio can also be interpreted as the to the relative hazard. In addition, Cox regression analysis multiplicative change, rather than additive change, in the uses the Wald test to examine whether or not each hazard of a student experiencing the critical event based individual regression coefficient is significantly different on every unit of the change in a specific factor, holding from zero. The logic of hypothesis testing for the Wald other factors as constant. If the hazard ratio of the Mth test is also similar for both the Cox and logistic regression group (without the intervention strategy) versus the Nth models. If an individual regression coefficient is significantly group (with the intervention strategy) equals three, different from zero, the corresponding risk factor significantly investigators may conclude that students in the Mth group contributes to the hazard function of an event occurring. experience three times greater hazard compared to those Cox regression analysis is known as the proportional in the Nth group. Moreover, if the risk factor (X) with a value hazards model because the model assumes that two of four is for student M and the same risk factor with a hazard functions are proportional to each other over time different value (X*) of three is for student N, the hazard ratio (Kleinbaum, 1996). For example, given two students, if equals eb (X – X*) = eb (4 – 3) = eb. Assuming the hazard ratio b one initially has twice as much relative risk than the is two (e =2, where b = loge2), the hazard of a student second student, then the relative risk for the first student experiencing academic difficulty is two times greater for is two times that of the second student at all time points. student M compared to student N. Note that the baseline
Also, given two student groups, if one group initially has hazard function, ho(t), appears in both the numerator and three times the relative risk compared to the second denominator terms of the hazard ratio, which can be group, the relative risk of the first group is three times cancelled out if the proportional assumption of the hazard more than that of the second group during the study function is held true. period. In other words, the model assumes that the relative hazard is constant across time for two different students Example of Cox Regression Analysis and student groups, respectively. A Cox regression model was used to analyze a sample of 200 matriculated students randomly selected from the Interpretations of Relative Hazard population in 1993-1997. The study attempted to answer and Hazard Ratio the following research questions: “How well can the following The exponential expression of each regression risk factors explain students encountering academic coefficient in the Cox regression model is called relative difficulty: demographics, undergraduate GPAs, medical risk or relative hazard. The magnitude of the relative college admission test scores, medical school academic hazard indicates the direction of the association between performances, medical school curriculum tracks, and the outcome variable and the corresponding risk factor. If financial aid support amounts?” and “In which months do the relative hazard is greater than one (i.e. positive students experience the highest risk of academic difficulty regression coefficient), the hazard of a student experiencing based on the risk factors mentioned above?” Thus, the the critical event increases as the value of the risk factor aims of the study were: (a) to identify risk factors that are increases. This implies that the relative hazard is positively significantly associated with the hazard ratio of students associated with the risk factor. Moreover, if the relative experiencing academic difficulty; (b) to provide insight into hazard becomes one (i.e. regression coefficient is zero), the measure (relative risk, relative hazard) of the effects of the risk factor has no effect. On the other hand, if the the risk factors mentioned above; and (c) to detect at what relative hazard is a positive fraction and less than one (i.e. month students are most likely to experience academic negative regression coefficient), the relative hazard of a difficulty. The outcome variable of interest was a continuous
10 IR Applications, Number 5, Analyzing Student Learning. . . . measure—the hazard function of students experiencing hbcu, course2f, fresh_gp, and loan_amt) and click
11 IR Applications, Number 5, Analyzing Student Learning. . . the inclusion of two more risk factors—gender and number academic difficulty starting at the 20th month (first semester of sophomore courses failed. As shown in Table 4, the of the sophomore year), peaked at the 30th month (second regression coefficients for MCAT verbal reasoning score semester of the sophomore year), and maintained the and gender were significantly different from zero at the same level of hazard through the rest of the study period. .001 significance level. Also, the regression coefficient for The implication of this research finding was that the the number of courses failed during sophomore year was medical school should focus on academic support for significantly different from zero at the .05 significance students in the five-year curriculum track to address this level. extended period of academic difficulty. It was evident that these three risk factors—MCAT verbal reasoning score, gender, and number of courses failed in sophomore year—were significantly associated Figure 1 Hazard Curves for Students Experiencing Academic Difficulty Table 4 Cox Regression Model for Students Experiencing Academic Difficulty
Variables in Logistic Standard Odds Ratio (eß) the Equation Regression Error of ß SE (ß) Coefficient (ß) MCAT Physical -0.68*** 0.119 0.51 Reasoning Score Gender -1.05*** 0.377 0.35 (1 for male; 0 for female) Number of Sophomore 0.75* 0.309 2.11 Courses Failed *p < .05 and *** p < .001 based on the Wald test [X 2 = (ß/SE(ß))2 with df=1] Similarities between Logistic and Cox N=200 -2 Log Likelihood: X2=280.24 Regression Models Model Chi Square Test: X2=39.09, df=3, and p<.001 As illustrated in Table 5, logistic and Cox regression Improvement Chi Square Test: X2=44.60, df=3, and p<.001 models share several common characteristics. The model similarities are mainly reflected in the principle of modeling with academic difficulty. The hazard ratio was used to strategies, the objective of regression models, the method express the measure of the effects of these risk factors. of parameter estimations, and the procedure of variable For instance, the number of courses failed in the selections. sophomore curriculum exhibited the hazard ratio of 2.11. In the principle of modeling strategies, both models It indicated that a student who failed an additional course include all relevant explanatory variables at the initial had about a two times greater hazard of experiencing stage of model fitting and achieve parsimony and academic difficulty as opposed to a student who did not. consistency at the completion stage of model fitting. All On one hand, the hazard ratio (0.35) for gender relevant explanatory variables can be included in a single demonstrated that male students had 0.35 times greater model with both methods as well because multiple effects hazard of experiencing academic difficulty compared to can be simultaneously studied, and the effect of individual female students. Because the hazard ratio of 0.35 was a explanatory variables can be examined while others are positive fraction and less than one, it is more meaningful held constant. Moreover, when fewer explanatory variables to interpret that male students had almost three times are sufficient to explain the occurrence of the event, (inverse of 0.35) less of a hazard than female students to investigators do not need elaborate explanations and experience academic difficulty. On the other hand, the unnecessary variables in models. Furthermore, hazard ratio (0.51) concerning MCAT verbal reasoning investigators need to demonstrate the consistency of the indicated that a one-unit increase in the MCAT verbal model structure. It is important that significant explanatory reasoning score was associated with a decrease in relative variables and the effects of these variables are as identical risk of academic difficulty by a factor of two (inverse of as possible when the models are constructed over time. 0.51). Another characteristic that the two models have in The hazard function h(t, X) against time (t) was plotted common is the study objective. Both logistic and Cox graphically based on the stratified Cox regression model. regression models are applicable to the occurrence of the Figure 1 displays the hazard curves showing the four-year binary outcome or critical event. They are designed to curriculum track is lower than the five-year curriculum study the relationship between certain student learning track. This suggested that students in the four-year outcomes and their relevant explanatory variables. The curriculum track were less likely to experience academic logistic regression model allows investigators to identify difficulty. According to the pattern of the hazard function, explanatory variables that significantly contribute to the students in the five-year curriculum track experienced probability of a student obtaining a binary outcome while
12 IR Applications, Number 5, Analyzing Student Learning. . . . the Cox regression model allows investigators to identify well. (Norusis, 1985). Both models allow investigators to risk factors that significantly contribute to the hazard perform the likelihood ratio test that compares the likelihood function of a student experiencing the critical event. for the intercept only model to the likelihood for the model Both methods use the maximum likelihood estimation containing the explanatory variables within each analysis. technique to estimate the regression coefficients and to If the p value is less than the predetermined significance construct the non-linear model equations. The principle of level (á=.05, .01, or .001), investigators may claim that at maximum likelihood is basically the same, although the least one of the explanatory variables or risk factors in the Cox regression model uses the partial likelihood estimation model significantly contribute to the outcome variable. An (considering probabilities for the event occurrence rather additional similarity of the two models includes the use of than censored cases). The maximum likelihood estimation the Wald statistic as criterion to test the association technique is applied to estimate the regression coefficients between individual explanatory variable and the outcome such that the likelihood of observing data is maximized. variable. If the p value for the Wald test is less than the This technique has been discussed in many statistical predetermined significance level, investigators may books (Eliason, 1993; Hosemer and Lemeshow, 1989; conclude that a specific explanatory variable significantly Kleinbaum, 1996; and Pampel, 2000). First, the probability contributes to the probability or the hazard function of distribution (i.e., model equation) of students obtaining the experiencing a critical event. outcome is prepared. Secondly, the joint probability Logistic and Cox regression models allow investigators distribution is derived based on the product of probability to interpret the effect (odds ratio and hazard ratio) of distributions under the assumption of independence. Thirdly, specific explanatory variables on the outcome variable. logarithm transformation is applied to the joint probability The interpretations of the odds ratio and hazard ratio are distribution to yield the log likelihood functions. Lastly, the the same, although the calculations are quite different. iterative procedure is undertaken to estimate the regression Logistic regression analysis uses the odds ratio (eb) to coefficients. The iterative procedure begins with the value indicate that an average one-unit of change in the for individual parameters in the log likelihood functions, explanatory variable leads to a change in the odds of a followed by a cycle of adjusting initial values to improve the student obtaining a binary outcome by a factor of eb. Cox model fitting. This procedure ultimately achieves the maximum likelihood estimates of the regression coefficients. Table 5 Logistic and Cox regression analyses use the same Summary of Similarities between procedures such as enter, forward, and backward elimination Logistic and Cox Regression Models to select significant explanatory variables (SPSS Inc, 2002). Similarities of Two Models Logistic and Cox Regression Analyses These procedures are briefly described as follows: (a) Inclusion of all relevant explanatory variables or risk factors at the initial stage of the model fitting; and Enter procedure—All explanatory variables are forced to Principle of Modeling Strategies achieving parsimony and consistency upon the be included in the model in one step; (b) Forward stepwise completion stage of the model fitting procedure—Explanatory variables are included in the model Logistic Regression: To identify explanatory variables (X) that significantly one at a time based on the highest Wald or the likelihood- contribute to the probability, P(X), of student ratio statistics with the entry criterion (p=.05). As each obtaining a binary outcome Objective of Regression Models new variable is added to the model, all of the existing Cox Regression: explanatory variables in the model are evaluated for removal To identify risk factors (X) that significantly contribute to the hazard function h(t,X) for the based on the Wald or likelihood-ratio test with the removal duration and timeline of a student experiencing the criterion (p=.10) When no more variables meet the entry or critical event Method of Parameter Estimations The principle of the maximum likelihood estimation the removal criteria, the algorithm of selecting significant is applied to estimate the regression coefficients variables stops; and (c) Backward procedure—All of the Enter, forward, or backward procedures are used to explanatory variables are entered into the model during the Procedure of Variable Selections select significant explanatory variables or risk factors to form the regression model first step. The explanatory variables that meet removal Similar to the F test in linear regression, both criterion (p =.10) are removed sequentially. When no more models use minus two log likelihood (-2LL) test for the significance model fitting (All explanatory variables meet the removal criteria, the algorithm of selecting variables do not contribute to the outcome significant variables stops. Test of Significant Predictors occurrence vs. At least one of the explanatory variables contribute to the outcome occurrence) In both models, investigators utilize the –2 log likelihood Similar to the t test in linear regression, both models use the Wald test for the significance of the value as criterion to make a judgment concerning the individual regression coefficients. significance of the explanatory variables. The –2 log Logistic Regression: likelihood (-2LL), a chi square statistic, is important because To provide insight into the measure (odds, odds ratio, e(ß0 + ß1X1+ ß2X2+…+ ßpXp)) of the effects of the it tells the probability of obtaining the binary outcome given explanatory variables Interpretation of Magnitude Effects the established parameter estimates. It is also a measure Cox Regression: of how well the estimated parameters fit the data; a small To provide insight into the measure (relative hazard, hazard ratio, or e( ß1X1+ ß2X2+…+ ßpXp)) of the effects of value of the –2 log likelihood means the model fits the data risk factors
13 IR Applications, Number 5, Analyzing Student Learning. . . regression analysis utilizes the hazard ratio (eb) to indicate regression model, the prediction results can be used as that an average one-unit of change in the risk factor criteria to make a judgment regarding accuracy of the contributes to a change in the hazard of a student classifications. The cross-tabulating method is used to experiencing the critical event. categorize the predicted and the actual responses into a 2 by 2 table that indicates the accuracy of the classification Differences between Logistic and results. However, in the Cox regression model, the Cox Regression Models classification results are not readily available to assess As described in Table 6, the distinct characteristics of the model accuracy. Although the logistic regression logistic and Cox regression models can be distinguished model can be used to perform future predictions based on by the formation of research questions, the expression of the known explanatory variables for prospective students, model equations, the assessment of model fittings, and the Cox regression model is not capable of performing the verification of model assumptions. such predictions. On the right side of the Cox regression Logistic regression analysis focuses on a critical event equation, all risk factors (X) for prospective students are occurring or not occurring. Examples of research questions the observed values that are ready to be placed into the formed include “Will the student experience the occurrence equation. However, the time variable (t) in the baseline of the critical event, ‘yes’ or ‘no’?” and “What explanatory hazard ho(t) for prospective students is unknown, therefore variables contribute to the occurrence of the critical event, it cannot be placed into the equation to perform future ‘success’ or ‘failure’?” However, Cox regression analysis predictions. is concerned with the duration and timing of the occurrence The availability of pseudo R-squares also differs between of the event. The research questions may be framed as the two models. The pseudo R-square measures the “At what time (month, semester, year) does the student success of the model in explaining the variations in the experience the critical event at the highest risk?”, and ”What risk factors contribute to the occurrence of the Table 6 critical event at different time of the study periods?” Summary of Differences between Logistic and A key difference between logistic and Cox regression Cox Regression Models analyses is to use distinct model equations to study the Differences of Logistic Regression Analysis Cox Regression Analysis relationship between the binary outcome and the Two Models Will the critical event occur (yes or no- When (at what time) will the critical event Formulation of explanatory variables. The logistic regression model is the binary outcome)? What occur (yes or no-the binary outcome)? Research Z Z explanatory variables contribute to the What risk factors contribute the timeline Questions written as P(X) = e / (1+e ), where Z = bo + b1X1 + b2X2 occurrence of the critical event? of the occurrence of the critical event?
+...+ b X , and P(X) is the probability of obtaining a binary Z Z z p p P(X)=e /(1+ e ), where P(X) is the h(t,X) = h0(t) e , where h(t,X) is hazard outcome. The Cox regression model may be expressed probability of event occurrence; e is function of the event occurrence given Expression of the base of the natural logarithm; Z = time variable (t); h0(t) is baseline hazard; Z Model ß + ß X + ß X +…+ ß X ; ßs are e is the base of the natural logarithm; Z as h(t,X)=ho(t)e , where Z = b1X1+ b2X2 +...+ bpXp. The 0 1 1 2 2 p p Equations regression coefficients; and Xs are = ß1X1+ ß2X2+…+ ßpXp; ßs are hazard function, h(t,X), is a function of time (t), the risk explanatory variables regression coefficients; and Xs are risk factors (X), and the baseline hazard h (t). The baseline factors o P(X) is a continuous measure h(t, X) is a continuous measure (hazard hazard is dependent on the time variable, acting as a (probability of a student experiencing function for the timeline of a student the critical event such as pass/fail, experiencing the critical event such as constant term and contributing to the hazard function in achiever/nonachiver) dropout, dismissal, and withdrawal) a multiplicative manner. P(X) begins with zero and increases The hazard function h(t,X) begins with as a smooth S-shaped curve. P(X) is any positive value and goes up and A noteworthy difference between the two models is Pattern of Non- the probability value between zero and down. It is a rate between zero and Linear Curves the pattern of non-linear curves. In the logistic regression one depending on the explanatory positive infinity depending on the product variables and the regression of baseline hazard and the function of model, the main focus is to estimate the probability of coefficients risk factors. obtaining the binary outcome. The estimated probability P(X) cannot be used to detect the h(t, X) can be used to detect the timeline is a continuous measure that begins with zero and timeline of the occurrence of the of the occurrence of the critical events. increases as a smooth S-shaped curve. The value of the critical events. Explanatory variables are the Risk factors are the observed values probability is between zero and one depending on the observed values readily to be plugged readily available for the predictions. into the equation to perform However, the time variable (t) in the explanatory variables and the regression coefficients. predictions baseline hazard function ho(t) is unknown value that cannot be plugged Assessment of However, for the Cox regression model, the hazard function The prediction results allow into the equation to perform predictions. Accurate investigators to identify potential at- is a continuous variable dependent on the duration and Predictions risk students to participate in the The classification results are not timeline of experiencing the critical event. The hazard mandatory intervention program. available to assess prediction accuracy. function is a rate, which is ranged from zero to positive The classification results are available infinity. It begins with any positive value and goes up and to assess prediction accuracy. Assessment of Pseudo R-square is readily available Pseudo R-square is not available to down depending on the product of the baseline hazard Model Fittings to assess the model fittings. assess the model fitting. and the risk factors. Residuals are normally distributed with Hazard ratio are constant across time. If a mean of zero and a constant the hazards are proportional, the survival Verification of Logistic regression, unlike the Cox regression model, variance. The histogram and curves generated by log minus log (LML) Model scattergram of residuals are plotted to should be parallel. has the capability of assessing the prediction power of the Assumptions check for the normality and the model and performing future predictions. In the logistic homogeneity of variance.
14 IR Applications, Number 5, Analyzing Student Learning. . . . data, which means it indicates that the proportion of of the sophomore year and maintained the same level of variations in the outcome variable is accounted for by the risk through the rest of the study period. The research explanatory variables. In the logistic regression model, the results were consistent with the literature stating that an pseudo R-squares are used as criteria to assess the increase in the relative risk for a student experiencing model goodness of fit. However, in the Cox regression academic difficulty was significantly associated with a low model, the pseudo R-squares are not readily available to MCAT score (Huff and Fang, 1999), and students at risk assess the model fitting in SPSS PC Version 12.0 for academic difficulty remained at risk throughout the first commands. three years of medical school (Fang, 2000). The implication Finally, another key difference between the two models of this study was that the medical school addressed lies in the model assumptions. For the logistic regression academic difficulty issues through academic development model, residuals are assumed to have a mean of zero and and support services. a variance of P(X) [1-P(X)]. Investigators must check for violation of the assumption by plotting the histogram and Strengths scatter diagram for residuals. The model assumption is Both logistic and Cox regression models are typically satisfied if (1) the histogram of the residuals is normally used for data analysis concerning binary outcomes such distributed with a mean of zero and (2) the residuals on the as admitted/not admitted, enrolled/not enrolled, and scatter diagram appear to be parallel with the X-axis, (i.e., graduated/not graduated. In particular, the logistic the indication of a constant variance). The Cox regression regression method is capable of allowing investigators to model states that there is a multiplicative relationship answer some important questions linked to learning outcomes”Is student performance likely to improve or not between the baseline hazard function ho(t) and the function of the risk factors. As a result, the ratio of the hazard improve after the implementation of tutorial and remedial functions for an event with different values for the risk programs?” ”Are student graduation and attrition status factor does not depend on time (t). Therefore, investigators significantly associated with the explanatory variables need to check for violation of this assumption by using the concerning student characteristics and college learning log-minus log (LML) plot of the survival function. If the environment?” and ”Can the probability of students hazards are proportional, the survival curves generated by expressing overall college satisfaction be estimated by LML should be parallel (Steinberg, 1999). Under the certain explanatory variables concerning academic assumption of proportional hazards, the resulting curves programs and services?” should be parallel and separated only by a constant In Cox regression analysis, the hazard function is a vertical difference. function of the time-to-event and risk factors. This function provides investigators with the valuable information to Summary answer certain learning outcome questions such as ”How In this study, the main objectives for constructing logistic many semesters elapse before students experience and Cox regression models were accomplished. For logistic academic difficulty?” ”What risk factors are significantly regression analysis, the explanatory variables contributing associated with the occurrence of academic difficulty?” to the probability of a student passing the USMLE Step 1 ”At what month are students likely to pass the licensure were identified. It was evident that the MCAT (physical examination?” ”What explanatory variables significantly and biological sciences) scores, number of sophomore contribute to the success of licensure examination?” ”How courses failed, and medical school freshman GPAs were many years pass before students graduate from the significantly associated with the USMLE Step 1 college?” and ”To what extent student’s timely and delayed performances. The study results confirmed that MCAT graduation are accounted for by the explanatory variables scores and medical school course performances were concerning the overall quality of educational program and significant predictors of the USMLE Step 1 (Chen et al., student support?” 2001; and Haught and Walls, 2002). The implication of the Clearly, these two models are proven to be useful tools study results was that the medical school should continue in studying explanatory variables that are significantly its effort to recruit and admit qualifying students with high associated with binary outcomes. Moreover, the effect of MCAT scores, and to strengthen teaching and learning to a specific explanatory variable or risk factor on the event ensure student success on the licensure examination. occurrence can be investigated holding other explanatory With regard to Cox regression analysis, the method variables or risk factors constant. Both methods also indicated that academic difficulty was significantly allow investigators to assess the model fittings by means accounted for by risk factors such as MCAT verbal of the likelihood ratio test. Furthermore, using the logistic reasoning score, gender, and number of sophomore courses regression model, investigators can perform classifications, failed. Moreover, students in the five-year curriculum track and subsequently evaluate the predictive power. experienced academic difficulty during the first semester of their sophomore year, peaked at the second semester
15 IR Applications, Number 5, Analyzing Student Learning. . .
Limitations with the three groups. In addition, the two sets of relative This study is not completely conclusive because of risk rates are calculated when the probability of a student some methodological limitations. For example, given both falling into a specific target group is compared to that of models were built to study the effects of the explanatory a student in the reference group (Walters, Campbell, and variables on the binary outcomes—passing licensure Lall, 2001). The multinomial logistic regression analysis examination (yes or no) or experiencing academic difficulty relaxes the model assumption of the proportional odds. It (yes or no)—the investigator is unable to use the same does not require that investigators verify the assumption of techniques to study multiple outcome categories. These parallel lines because the relationship between the outcomes categories include three pass- or fail-groups of explanatory variables and the effects of these variables licensure examination (e.g., first-time pass, second-time depends on the outcome category (Plank and Jordan, pass, and fail at least two times) and three categories of 1997). academic difficulty (e.g., dismissal, withdrawal, and leave An implicit feature of the Cox regression model is of absence), respectively. reflected in the model assumption of proportional hazards The Cox regression model is also known as the Cox for all time intervals. If the LML plot of survival curves for proportional hazards model and has to satisfy the model assessing the validity of the model assumption is not assumption of proportional hazards. If the model successful, the smoothed plots of the scaled Schoenfeld assumption is seriously violated, the analysis results residuals proposed by Therneau and Grambsch (Belle, et could be inaccurate and misleading. However, it is difficult al, 2004) may be a better approach. The Schoenfeld to make judgments about the proportional hazards on the residual method not only provides the investigator with an LML plot of survival curves partly because the functions easier visual interpretation, it also offers a statistical test are non-linear instead of straight lines. In other words, the for the proportional hazards assumption (Belle, et al., validity of the model assumption may not be properly 2004). An additional alternative for detecting the violation evaluated because of the limitations of the LML of the model assumption is the likelihood ratio test (Palmer, assessment tool. et al., 2003). Departure from the assumption of proportional hazards can be analyzed by the likelihood ratio test Major Alternatives comparing models with and without the stratifying variable To study the outcome variable with multiple categories by the covariate interaction terms (e.g., the curriculum as a function of the explanatory variables, polytomous track by the three explanatory variables—MCAT verbal logistic regression analysis is a possible alternative (Peng, reasoning score, gender, and the number of sophomore et al., 2002). Polytomous logistic regression analysis courses failed, respectively, in the present study). If no includes ordered and multinomial logistic regression statistical significance is found in these interaction terms, models. The ordered logistic regression model is applicable then the model assumption of proportional hazards is not to three or more ordinal outcome categories (e.g., first- violated. time pass, second-time pass, and at least two-time fail In the Cox regression model, if the effects of the groups for licensure examination). The model is called the explanatory variables change during time in a practical cumulative logit model because it is based on the situation (e.g., age and financial aid amount fluctuate over cumulative response probabilities of being in a category or years), it may lead to a violation of the model assumption. lower (Walters, et al., 2001). The model is also called the For this reason, the extended Cox regression model that proportional odds model because it assumes that the utilizes the time-dependent variables should be considered corresponding regression coefficients in the link function as an alternative to analyze data for that do not require the are equal for each explanatory variable. Therefore, the model assumption (Klein and Moeschberger, 1997; and model assumption has to be verified carefully by the Kleinbaum, 1996). The extended Cox regression model parallel lines test (SPSS, Inc., 2002). If the model allows investigators to study the effect of the explanatory assumption is satisfied, investigators can proceed to variables along with the time-dependent variables on the interpret the effects of the explanatory variables. hazard function. If the model assumption is violated in the ordered logistic regression analysis, the multinomial logistic Editor’s Notes regression model should be considered as another Recently there has been an increased interest in using alternative. The outcome variable of multiple nominal groups various methodologies that better fit the situations we includes the reference group and the target groups. For encounter. While Multiple Linear Regression with its instance, in passing licensure examination, the first-time Ordinary Least Squares has been part of our methodology pass group is labeled as the reference group, the second- for many years, it has always had certain limitations. For time pass group is coded as target group 1, and at least example in predicting probabilities it has the unfortunate two-time fail group is considered as target group 2. Two characteristic of going both negative and also going greater model equations are generated for the nominal outcome than one. Logistic Regression deals with this by creating
16 IR Applications, Number 5, Analyzing Student Learning. . . . a dependent variable, the log odds, that can range from elimination of variables The decision to use a specific negative to positive and that can exceed 1.0. While strategy should be selected based on the situation and Logistic Regression has been discussed in the statistical the intent of the researcher. literature for numerous years, it has become more prevalent Another option the researcher has in Logistic Regression in institutional research during the last several years. is to adjust the cut-point for classifying and observation A second use of Linear Regression has been to explain into the two categories. Dr. Chen used the default of .5 but the time between a process starting and completing. Here you may want to use a proportion that’s closer to the again the linearity of relationships tends to be suspect. In actual split in the sample. addition the regression model is not able to handle data This article will give you an extremely good start in where specific completion date is not available. Cox appropriately using two alternatives to the traditional Regression is capable of dealing with both of these issues. regression methodology. As you begin to use one or both As with logistic regression, it uses a function of the of the alternatives, the references he provides contain the exponential and converts the relationship into one that is information that will be essential for you to use the linear in logarithms. methodologies appropriately. While both the Logistic and Cox procedures are part of many standard statistical software, such as SPSS, References correctly using them is frequently not intuitive. This is Belle, Gerald Van., Fisher, Lloyd D., Heagerty, Patrick where the article by Dr. Chen provides an extremely J., and Lumley, Thomas. (2004). Biostatistics: A valuable service. First it provides a good summary of Methodology for the Health Sciences. Wiley-Interscience when and why a researcher would want to use these types (2nd edition) of models. In addition it provides an example of their use. Braun, Henry I. and Zwick, Rebecca (1993). Empirical In fact it gives you the step-by-step procedures needed to Bayes Analysis of Families of Survival Curves: Applications run your own analysis in SPSS. Next it gives a brief to the Analysis of Degree Attainment. Journal of interpretation of the results. Educational Statistics. Vol. 18., No. 4, pp. 285 - 303 What I think you will find most exciting however is Table Case, S.M., Swanson, D.B., Ripkey, D.R., Bowles, 6 where there is a head to head comparison of the two L.T., and Melnick, D.E. (1996). Performance of the Class methodologies. As an exercise to consider when linear of 1994 in the New Era of USMLE. Academic Medicine regression should be used, the reader may well want to 72:31S-33S add an additional column to Table 6, title the column Chen, Chau-Kuang, Campbell, Vickie C., and Suleiman, Linear Regression, and fill in the cells. For example under Ahmad, (2001). Predicting Student Performances at a Pattern of Non-Linear Curves, one might want to list the Minority Professional School, Association for Institutional use of various nonlinear transformations such as quadratic Research 2001 Annual Forum Paper, ERIC No. ED457714 and cubic terms. The reader might also want to add rows Cox, D. R. (1972). Regression Models and Life Tables. to Table 6 such as Interpretation of Regression Weights. Journal of the Royal Statistical Society, Series B, 34, 187- The interpretation of these weights for the Logistic and 202 Cox procedures can be found in the text of the article. Cox, D. R., and Oakes, D. (1984). Analysis of Survival This suggestion also applies to Table 5 where you may Data. London: Chapman & Hall. want to put in the similarities that Multiple Linear Regression DesJardins, Stephen L. and Moye, Melinda J., (2000). has with Logistic and Cox Regression. Studying the Timing of Student Departure from College, Also found in this article is the discussion of the Air 2000 Annual Forum Paper, ERIC No. ED445650 relationships between Hazard and Survival functions. As DesJardins, S. L., Ahlburg, D. A., & McCall, B.P. such this paper represents an excellent first step or primer (1997). Using Event History Methods to Model the Different and explains techniques that greatly extend our analytical Modes of Student Departure from College. Paper presented methodologies. Readers do need to be warned, however, at the Association for Institutional Research 37th Annual that if they are interested in using these techniques they Forum, Lake Buena Vista, Florida must focus on learning more about them. For example Dey, Eric L. and Astin, Alexander W. (1993). Statistical when using Chi Square and testing within stepwise or Alternatives for Studying College Student Retention: A nested models, it is extremely important to select Comparison Analysis of Logit, Probit, and Linear appropriate models that are theoretically relevant. The Regression. Research in Higher Education Vol 34, No. 5 selection of the specific stratification of the model, where Elandt-Johnson, Regina C. and Johnson, Norman L. stratification is desirable, is also an extremely important (1980). Survival Models and Data Analysis, John Wiley element in the methodology. In terms of building the and Sons, New York. models, Dr. Chen selected Forward-Wald to build his Eliason, Scott R. (1993). Maximum Likelihood survival model. There are other options such as Estimation: Logit and Practice, a Sage University paper, simultaneously entering the variables or backwards 07-096, Newbury Park, CA: Sage
17 IR Applications, Number 5, Analyzing Student Learning. . .
Fang D. (2000). 1998 AAMC CAMCAM: Fact Sheet, Dual Effect of Parity on Breast Cancer Risk in African- Vol. 2, No. 12, P.3 AmericanWomen. Journal of the National Cancer Institute, Gill, Jeff (2000). Generalized Linear Model: A Unified Vol. 95, No. 6, March 19, 2003 Approach. Sage Publication, Thousand Oaks, California. Pampel, Fred C., (2000). Logistic Regression: A Primer. Hachen. David S., Jr. (1988). The competing risks Thousand Oaks, California: Sage Publications. model: A method for analyzing processes with multiple Plank, Stephen B. and Jordan, Will J. (1997). Reducing types of events. Sociological Methods & Research, 17 Talent Loss. The Impact of Information, Guidance, and 21-54. Actions on Postsecondary Enrollment, Report No. 9 Eric Han, Tianqi and Ganges, Tendaji W. (1995). A Discrete- No: ED405429 Time Survival Analysis of the Education Path of Specially Peng, Chao-Ying Joanne, So, Tak-Shing Harry, Stage, Admitted Students. ERIC No. ED387033 Frances K., and St. John, Edward P., (2002). Use and Haught, Patricia A. and Walls, Richard T. (2002). Adult Interpretation of Logistic Regression, Research in Higher Learners: Relationships of Reading, MCAT, and USMLE Education v43, (3). Step 1 Test Results for Medical Students. ERIC No. Peterson, T., (1984). A Comment on Presenting Results ED464943 of Logit and Probit Models. American Sociological Review, Hojat, Mohammadreza, and others (1995). Primary 50(1), 130-131. Care and Non-Primary Care Physicians: A Longitudinal Ronco, Sharon L. (1994). Meandering ways: Studying Study of Their Similarities Differences, and Correlates student dropout with survival analysis. Paper presented at before, during, and after Medical School. Academic the annual Forum of the Association for Institutional Medicine. Vol 70, n1 suppl pS17-S28 Research, New Orleans, LA. Hosemer, David W., and Lemeshow, Stanley, (1989). Sadler, William E., Cohen, Frederic L, and Kockesen, Applied Logistic Regression. New York: John Wiley & Levent, (1997). Factors Affecting Retention Behavior: A Sons, Inc. Model To Predict At-Risk Students, AIR 1997 Annual Hosemer, David W., and Lemeshow, Stanley, (1999). Forum Paper, ERIC No. ED410885 Applied Survival Analysis. New York: John Wiley & Sons, Singer, J.D. & Willett, J.B. (1993). It’s about time: Inc. Using discrete time survival analysis to study the duration Huff, K.L. and Fang, D. (1999). When Are Students and the timing of events. The Journal of Educational Most at Risk of Encountering Academic Difficulty? A Statistics, 18, 115-l95. Study of the 1992 Matriculants to U.S. Medical Schools. Singer, J.D. & Willett, J.B. (1991). Modeling the Days Academic Medicine 74:454-460 of Our Lives: Using Survival Analysis When Designing and Klein, John P., and Moeschberger, Melvin L. (1997). Analyzing Longitudinal Studies of Duration and the Timing Survival Analysis: Techniques for Censored and Truncated of Events. Psychological Bulletin, 110(2), 268-290. Data. Springer-Verlag, New York SPSS, Inc. (2002). SPSS Advanced Models 10.0., Kleinbaum, David G. (1996). Survival Analysis: A Self- Chicago, IL. Learning Text, Springer-Verlag, Inc., New York Steinberg, Milton (1999). Cox Regression Example, Lee, Elisa T. (1992). Statistical Methods for Survival SPSS Advanced Models 10.0 SPSS Inc. Chicago, IL Data Analysis, John Wiley & Sons, Inc. Strayhorn, Gregory (2000). Pre-Admission Program for Matthews, D.E. and Farewell, V.T., (1996). Using and Underrepresented Minority and Disadvantaged Students: Understanding Medical Statistics. 3rd Revised Edition. Application, Acceptance, Graduation Rates, and Basel:S Karger, 245 pp. Timeliness of Graduating from Medical School, Academic McDaniel, Cleve and Graham, Steven (1999). Student Medicine, V75 n4 p355-61 Retention in an Historically Black Institution. Paper Therneau, T. M., and Grambsch, P. (2000). Modelling presented at the annual Forum of the Association for Survival Data: Extending the Cox Model. Springer-Verlag, Institutional Research New York Menard, Scott, (1995). Applied Logistic Regression Walters, S.J., Campbell, M.J., and Lall, R (2001). Analysis. Thousand Oaks, California: Sage Publications. Design and Analysis of Trials with Quality of Life as an Meyer, Paul L. (1970). Introductory Probability and Outcome: A Practical Guide. Journal of Biopharmaceutical Statistical Applications, Addison-Wesley Publishing Statistics 11(3), 155-176. Company, 2nd edition Willett, J.B. & Singer, J.D. (1995). It’s Deja Vu all over Miller, R. G. (1981). Survival Analysis. New York: again: Using multiple-spell discrete-time survival analysis. Wiley The Journal of Educational and Behavioral Statistics, 20, Norusis, Marija J. (1985). Advanced Statistics Guide, 41-67. SPSS, Inc. Willett, J.B. & Singer, J.D. (1991). From whether to Palmer, Julie; Wise, Lauren A.; Horton, Nicholas J.; when: New methods for studying student dropout and Adams-Campbell, Lucile L.; and Rosenberg, Lynn (2003). teacher attrition. Review of Educational Research, 61, 407-450.
18 IR Applications, Number 5, Analyzing Student Learning. . . .
IR Applications is an AIR refereed publication that publishes articles focused on the application of advanced and specialized methodologies. The articles address applying qualitative and quantitative techniques to the processes used to support higher education management.
Editor: Managing Editor: Gerald W. McLaughlin Dr. Terrence R. Russell Director of Planning and Institutional Executive Director Research Association for Institutional Research DePaul University 222 Stone Building 1 East Jackson, Suite 1501 Florida State University Chicago, IL 60604-2216 Tallahassee, FL 32306-4462 Phone: 312/362-8403 Phone: 850/644-4470 Fax: 312/362-5918 Fax: 850/644-8824 [email protected] [email protected]
AIR IR Applications Editorial Board
Dr. Trudy H. Bers Dr. Anne Marie Delaney Dr. Jessica S. Korn Senior Director of Director of Associate Director of Institutional Research, Curriculum Institutional Research Research and Planning Babson College Loyola University of Chicago Oakton Community College Babson Park, MA Chicago, IL Des Plaines, IL Dr. Gerald H. Gaither Dr. Anne Machung Ms. Rebecca H. Brodigan Director of Principal Policy Analyst Director of Institutional Research University of California Institutional Research and Analysis Prairie View A&M University Oakland, CA Middlebury College Prairie View, TX Middlebury, VT Dr. Marie Richman Dr. Philip Garcia Assistant Director of Dr. Harriott D. Calhoun Director of Analytical Studies Director of Analytical Studies University of California-Irvine Institutional Research California State University-Long Beach Irvine, CA Jefferson State Community College Long Beach, CA Birmingham, AL Dr. Jeffrey A. Seybert Dr. David Jamieson-Drake Director of Dr. Stephen L. Chambers Director of Institutional Research Director of Institutional Research and Institutional Research Johnson County Community College Assessment and Associate Duke University Overland Park, KS Professor of History Durham, NC University of Colorado Dr. Bruce Szelest at Colorado Springs Associate Director of Colorado Springs, CO Institutional Research SUNY-Albany Albany, NY
Authors can submit contributions from various sources such as a Forum presentation or an individual article. The articles should be 10-15 double-spaced pages, and include an abstract and references. Reviewers will rate the quality of an article as well as indicate the appropriateness for the alternatives. For articles accepted for IR Applications, the author and reviewers may be asked for comments and considerations on the application of the methodologies the articles discuss. Articles accepted for IR Applications will be published on the AIR Web site and will be available for download by AIR members as a PDF document. Because of the characteristics of Web-publishing, articles will be published upon availability providing members timely access to the material. Please send manuscripts and/or inquiries regarding IR Applications to Dr. Gerald McLaughlin.
19