Marshall University Marshall Digital Scholar

Biochemistry and Faculty Research

12-2015 Predicting Medical Student Success on Licensure Exams Charles A. Gullo Marshall University, [email protected]

Michael J. McCarthy Marshall University, [email protected]

Joseph I. Shapiro Marshall University, [email protected]

Bobby L. Miller Marshall University, [email protected]

Follow this and additional works at: http://mds.marshall.edu/sm_bm Part of the Higher Education Commons, and the Medical Education Commons

Recommended Citation Gullo CA, McCarthy MJ, Shapiro JI, Miller BL. Predicting medical student success on licensure exams. Med Sci Educ. (2015) 25:447-453.

This Article is brought to you for free and open access by the Faculty Research at Marshall Digital Scholar. It has been accepted for inclusion in and Microbiology by an authorized administrator of Marshall Digital Scholar. For more information, please contact [email protected]. Final revised Manuscript Click here to download Manuscript: MS_prediction_2015_revision_final_v2.2.docx Click here to view linked References 1 2 3 4 Title: Predicting Medical Student Success on Licensure Exams 5 6 7 Charles A. Gullo, PhD; Michael J. McCarthy, MS; Joseph I. Shapiro, MD and Bobby L. Miller, 8 MD. The Joan C. Edwards School of Medicine, Marshall University, Huntington, WV 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 Acknowledgments: 38 We would like to thank Ms. Carrie Rockel for editorial assistance with this manuscript and Dr. 39 40 Tracey LeGrow, Associate Dean for Academic standards for her insights on how she and her 41 office currently advises ‘at risk’ students. 42 43 44 45 46 47 48 49 50 51 Corresponding author: Charles Gullo, PhD 52 Associate Dean for Medical Education 53 Joan C. Edwards School of Medicine 54 1600 Medical Center Drive, Suite 3408 55 Huntington WV 25701-3655 56 57 Email: [email protected]; phone: 304-691-8828; Fax 304-691-1726 58 59 60 61 62 1 63 64 65 1 2 3 4 Abstract 5 6 7 Many schools seek to predict performance on national exams required for 8 9 graduation using pre-matriculation and medical school performance data. The need for targeted 10 11 12 intervention strategies for at-risk students has led much of this interest. Assumptions that pre- 13 14 admission data and high stakes in-house medical exams correlate strongly with national 15 16 standardized exam performance needs to be examined. Looking at pre-matriculation data for 17 18 19 predicting USMLE Step 1 performance, we found that MCAT exam totals and math-science 20 21 GPA had the best prediction from a set of pre-matriculation values (adjusted R2=11.7%) for Step 22 23 24 1. The addition of scores from the first medical school exam improved our predictive capabilities 25 26 with a linear model to 27.9%. As we added data to the model we increased our predictive values 27 28 29 as expected. However, it was not until we added data from year two exams that we started to get 30 31 Step 1 prediction values that exceeded 50%. Stepwise addition of more exams in year two 32 33 34 resulted in much higher predictive values, but also led to the exclusion of many early variables. 35 36 Therefore, our best Step 1 predictive value of around 76.7% consisted of three variables from a 37 38 total of 37. These data suggest that the pre-admission information is a relatively poor predictor of 39 40 41 licensure exam performance and that including class exam scores allows for much more accurate 42 43 determination of students who ultimately proved to be at risk for performance on their licensure 44 45 46 exams. The continuous use of this data, as it becomes available, for assisting at-risk students is 47 48 discussed. (251) 49 50 51 52 53 Key words: Predication analysis, Pre-admissions requirements, Medical school performance, 54 Linear Regression 55 56 57 58 59 60 61 62 2 63 64 65 1 2 3 4 Introduction 5 6 7 The Joan C. Edwards School of Medicine is a relatively young medical school created 8 9 from the Teague Cranston Act and graduating its first class in 1982. The mission of this school is 10 11 12 to train a workforce for West Virginia and central Appalachia. Our core mission, which includes 13 14 training students from this region who are likely to practice here results in the selection of 15 16 candidates derived from a small population, making the determination of who can scholastically 17 18 19 succeed a more difficult process. Because of this, we are extremely interested in identifying 20 21 students who may need additional academic coaching and other forms of help in order to pass 22 23 24 their courses and ultimately, their licensure examinations. 25 26 Several recent publications challenge the heavy reliance on pre-matriculation scores such 27 28 29 as MCAT and science GPAs as indicative or predictive of successful performance on in-house 30 31 medical school exams, national licensure exams, and successful academic medical careers [1-4]. 32 33 34 However, some studies have suggested that pre-admissions data may indeed be valid positive 35 36 predictors of future clinical performance [5-7, 2, 8]. While pre-matriculation data may certainly 37 38 be useful for admissions committees when deciding upon their entering class, their utility as 39 40 41 predictors for negative performance on national licensing exams is unclear and requires further 42 43 large scale analysis [9]. 44 45 46 Proposed factors that may strongly influence future academic performance for medical 47 48 students range from pre-matriculation benchmarks, undergraduate GPAs, performance on 49 50 51 internal exams to study habits and use of social networking [10-13]. There is no shortage of 52 53 variables that are potential predictors of future success and medical school admissions program 54 55 officers are keenly aware of the limitations of heavy reliance on pre-matriculation data for their 56 57 58 requirements [14]. Although, it is clear that in order for these predictors to be useful, they must 59 60 61 62 3 63 64 65 1 2 3 4 occur early enough in the student’s educational developmental to provide benefit to at-risk 5 6 7 students. How late is too late is difficult to determine, but a continuous assessment process using 8 9 predictive algorithms may be more useful than a ‘one-off’ first or second-year determination. To 10 11 12 better identify such students, we undertook a study to objectively determine the strongest set of 13 14 predictors from a large set of pre-admission and medical school performance variables that could 15 16 be useful in determining the future outcome of the high stakes national exam, Step 1. The main 17 18 19 focus of this work is determining predictors for Step 1 and providing useful interventions with 20 21 students at risk for poor performance on this exam. However, predictions for Step 2 were 22 23 24 calculated and are discussed briefly. We introduce the utility of a continuous student specific 25 26 data-driven process that allows administrators to track student performance any time during their 27 28 29 first two preclinical years. 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 4 63 64 65 1 2 3 4 Methods 5 6 7 Students who matriculated from the Joan C. Edwards School of Medicine between 2008 8 9 and 2012 were de-identified and studied. Pre-admission data was extracted from the American 10 11 12 Medical College Application Service (AMCAS) database for students in the five matriculation 13 14 years who had subsequently taken United States Medical Licensing Examination (USMLE) Step 15 16 1 (n = 344). Exam scores on institutionally-developed multiple-choice exams were reported from 17 18 19 the school’s in-house learning management system. Results on NBME basic sciences subject 20 21 examinations and Comprehensive Basic Science Self-Assessment (CBSSA) were retrieved from 22 23 24 the NBME secured website. Medical College Admission Tests (MCAT) were reported in the 25 26 categories of verbal reasoning (VR), Physical Science (PS), Biological Science (BS) and total 27 28 29 (T). The analysis of MCAT reports used either the best MCAT scores, the first MCAT scores or 30 31 the lowest MCAT scores. Undergraduate grade-point averages were reported in the categories of 32 33 34 total (UGGPAT) and limited to math and science courses (UGMS-GPA). Results from the 35 36 subject-specific shelf clinical sciences examinations were retrieved from the NBME secured 37 38 website. 39 40 41 A total of 22 pre-admissions and 15 medical school variables were considered (see Table 42 43 1) in our analyses. Medical school data was further divided into MS1 and MS2 years in which 44 45 46 MS1 exam data was calculated from 198 students and MS2 data (e.g. exams, NBME subject 47 48 exams, and CBSSA exams) were calculated from 344 students. The difference is number is 49 50 51 attributed to a change in curriculum which took place during this time and which was 52 53 implemented initially in the MS2 year, resulting in two class years for which we have no MS1 54 55 exam data (that inaugural MS2 class and the class who were MS1s during that same year and 56 57 58 promoted to MS2s during the following year). Thus, our student numbers are smaller when our 59 60 61 62 5 63 64 65 1 2 3 4 predictive calculations include MS1 exam scores as a variable. Students who were exposed to 5 6 7 our new integrated curriculum are not included in these analyses as their internal exams were 8 9 dramatically different. It is also important to note that national exams scores were calculated 10 11 12 from students who took the exams for the first time and second time test taking scores were not 13 14 included in the analysis. The focus of our analysis is on the determination of predicted poor 15 16 performance on Step 1 and Step 2 national exams exclusively. 17 18 19 Biomedical Science Students (BMS) included those who strengthened their 20 21 undergraduate studies with one or two years of graduate studies before entering the medical 22 23 24 program. There were a total of 20 students in the BMS program from 2009 to 2012 that were 25 26 used in some of the analysis. Analysis of data from BMS students included the use of a Student’s 27 28 29 t-test to compare the means between BMS students and non BMS students for the following 30 31 variables: Math/Science GPA, lowest total MCAT, Step 1 and Step 2. No linear regression 32 33 34 analysis was performed with biomedical science students as numbers are too low to make 35 36 statistically meaningful results. The observed comparison between the BMS and the non BMS 37 38 cohorts were statistically significant at p <0.05 for both Math/Science GPA and Total MCAT 39 40 41 scores. 42 43 MS1 and MS2 student data were subjected to multivariate linear regression using the 44 45 46 software platform, Matlab® (The Mathworks, Natick Massachusetts, v2014a). Models were 47 48 varied to include different amounts of data corresponding to times before and following 49 50 51 matriculation. The fitting function, “stepwise”, was used to develop predictive models with the 52 53 additional caveats that only positive coefficients were included and the addition of the coefficient 54 55 significantly improved the predictive capability of the model. When models are described, the 56 57 58 intercept is a scalar added (or subtracted) to the sum of the product of beta coefficients and 59 60 61 62 6 63 64 65 1 2 3 4 variable values. Unless otherwise specified, P values are reported were at the p<0.05 and p<0.01 5 6 7 levels. Visualization of the data was performed using GraphPad® Prism v6.05. 8 9 This study (IRB Study #78931-1) has been approved by the Marshall University 10 11 12 Institutional Review Board under the exempt approval status in September 2015. 13 14 15 16 Results 17 18 In order to assess how important and valid the pre-admissions data we had for our 19 20 21 medical students was at predicting future negative performance on USMLE Step 1 exams, we 22 23 looked at a total of 22 variables (Table 1A). Using the pre-admission variables collected, we 24 25 26 found that the best linear predictive model was a combination of the lowest MCAT total score 27 28 and the undergraduate math-science GPA (UGMS-GPA) with an intercept of 158, and beta 29 30 31 coefficients of 9.68 for the UGMS-GPA and 1.58 for the low total MCAT (both p<0.01) and an 32 33 overall adjusted R2 (AR2) of 0.12. When we include the first medical school exam score results 34 35 2 36 (percent correct) in the model, the AR increases to 0.28 where the intercept is 81.44 and the beta 37 38 value is 1.29 for the low total MCAT and 1.26 for exam 1 score (all p<0.01). GPA when 39 40 included in the model had high p-value and was therefore dropped from the prediction analysis. 41 42 2 43 When we include all grades in year one, the predictive model has an AR of 0.38 and includes 44 45 lowest total MCAT as well as performance on all MS1 exams. Thus, our best predictive model 46 47 48 for year one medical students includes two variables from a total of 24 and these variables 49 50 account for about 38% of variance for predicting how wells student will do on their USMLE 51 52 53 Step 1 exam (See Table 1A and 1B for the total numbers and types of variables considered). The 54 55 Step 1 prediction data using pre-admissions and/or first year performance results is summarized 56 57 in Table 2. 58 59 60 61 62 7 63 64 65 1 2 3 4 In addition, we were also very interested to determine which medical school variables 5 6 7 were highly predictive of future negative board performance of students in their second year, 8 9 recognizing a need for possible remediation of at-risk students at this point as well (see Table 1C 10 11 12 for total additional variables considered for MS2 students). If we look at the performance of the 13 14 first MS2 exam in conjunction with lowest total MCAT score as well as the score of all the MS1 15 16 exams, the Step 1 model predicts at an adjusted R2 of 0.46. This improves to an AR2 of 0.53 17 18 19 when we include the scores of all the MS2 exams using our step-wise linear regression model. 20 21 However, when we exclude pre-admissions values and include the clinical sciences subject 22 23 24 (Miniboard) exams given in the second year along with al MS1 and MS2 exams, the prediction 25 26 improves significantly at an adjusted R2 of 0.65. Surprisingly, when looking at all the possible 27 28 29 exams in the first and second year, the best prediction was derived from just three variables: the 30 31 Microbiology basic science subject exam, the basic science exam, and the CBSSA 32 33 34 exam given at the end of the year. These three alone were able to predict Step 1 Performance at 35 36 an adjusted R2 of 0.77. These data suggest that as a student moves along and completes the 37 38 second year that pre-admission data and many of the exams he/she encounters along the way are 39 40 41 not as strong at predicting future Step 1 results as the three predictors mentioned. The data also 42 43 underscore the irrelevance of pre-admission values at predicting future performance on Step 1 44 45 46 when students are in their second year of medical school. These Step 1 prediction data for 47 48 students in their second year are summarized in Table 3. This approach also suggests a utility in 49 50 51 providing assistance or information to administrators or students themselves at various times 52 53 instead of focusing on one specific endpoint (e.g. at the end of the first or second year) but that 54 55 the most robust data comes from exams taken during the second year. 56 57 58 59 60 61 62 8 63 64 65 1 2 3 4 Regarding our ability to predict USMLE Step 2CK performance, we found that the 5 6 7 lowest total MCAT, the % score of the all MS1 exams and the % score of the first MS2 exam 8 9 had a predictive of 0.32 (AR2). However, the statistical reliability of this comparisons were less 10 11 12 relevant (p-value for lowest MCAT score was 0.226). Not surprisingly, the prediction improved 13 14 when we waited until the end of year two and used the same variables above but now replaced 15 16 the first MS2 exam with the total MS2 exams. Using results from all MS2 exams we were able to 17 18 19 predict 39.2% of the variance. Again, the reliability of this comparison was also statistically 20 21 insignificant (p-value for lowest MCAT was 0.117 and all MS1 exams scores was 0.601). 22 23 24 However, when we drop the use of any pre-admissions values and use two variables only- the 25 26 percent score on all MS2 exams and the Step 1 score our predication gives us an adjusted R2 of 27 28 29 0.4939 (with highly significant p values). Most interestingly, our predictive capacity goes up 30 31 significantly when we use a selection of NBME clinical sciences shelf-examination results. Thus, 32 33 34 using Step 1 scores in addition to four clinical sciences exam results (Family Medicine, 35 36 Obstetrics and Gynecology, Pediatrics and Internal Medicine) our adjusted R2 is now 0.62 with a 37 38 highly significant p-value of close to zero. It is important to note that this predictive capacity 39 40 41 excludes the two additional clinical sciences exams that students take in their second year 42 43 (Surgery and Psychiatry). Finally, Step 2 CK prediction using Step 1 alone gives us an adjusted 44 45 2 46 R of 0.49 (N= 267 and p-value of <0.05). In total, these data are consistent with previous data 47 48 which shows that pre-matriculation performance characteristics add very little to the predictive 49 50 51 power of Step 2CK. Taken together, these data are summarized in Table 3. As Step 2 is taken 52 53 towards the end of the third year, the utility of using data obtained in the second year as useful 54 55 information for students at risk is warranted. 56 57 58 59 60 61 62 9 63 64 65 1 2 3 4 Finally, we also looked at students who entered into our biomedical sciences (BMS) 5 6 7 program. These students had a lower mean MCAT and math/science GPA scores (23.5 +/- 0.94 8 9 and 3.17 +/- 0.08) than their non BMS peers (25.8 +/- 0.2 and 3.40 +/- 0.02) who entered into our 10 11 12 program over the same period (p value for Lowest total MCAT = 0.0093and p value for 13 14 Math/Sci. GPA = 0.0088). Despite being weaker students in these categories, these BMS 15 16 students did just as well as their non BMS peers with average scores of 226.1 (+/-2.7) and 229.9 17 18 19 (+/-4.9) for the USMLE Step 1 and Step 2 CK respectively (p values for Step1 comparisons were 20 21 statistically insignificant). The scores for the non BMS students were 218.8 (+/-1.2) and 233.23 22 23 24 (+/-1.04) on Step 1 and Step 2 CK (p value for Step 1 = 0.1453 and Step 2 = 0.4357) (p values 25 26 for Step 2CK comparisons were statistically insignificant). These data, although with a more 27 28 29 limited set of numbers, further suggest the inherent limitations that exist in the sole use of 30 31 undergraduate GPAs and MCAT scores when predicting success in future medical school 32 33 34 performance. BMS student data was not used as a distinct cohort in the multivariate linear 35 36 regression models due to the small numbers. 37 38 39 40 41 Discussion 42 43 We are very interested in identifying students at risk for failure of their licensure 44 45 examination- namely Step 1 and Step 2 CK. Unfortunately, the pre-admission variables we 46 47 48 analyzed are not very good at making such identifications and are consistent with other 49 50 publications [15, 3, 16, 17]. In contrast, adding a number of medical school performance 51 52 53 variables to the model dramatically improves our ability to predict licensure exam performance 54 55 by our students and predictions get stronger as students take more internal exams. To no small 56 57 degree, this justifies our policy of taking some of our class from the pool of students 58 59 60 participating in a master’s program during which they take some medical school courses despite 61 62 10 63 64 65 1 2 3 4 having pre-admission credentials which, on their own, were not competitive for selection (e.g. 5 6 7 Biomedical Science Students). Notably from a total of 37, three variables have the strongest 8 9 prediction for the USMLE Step 1 exam at the end of year 2 and five have the strongest prediction 10 11 12 for USMLE Step 2 CK exam for all students in their first and second year. For a summary of the 13 14 stepwise significant predictive power of various variables see Figure 1. In brief, pre admissions 15 16 adds very little to the prediction of failure of Step 1 or Step 2 USMLE. The best predications for 17 18 19 Step 1 were achieved with data that comes from the second year (basic sciences miniboard plus 20 21 the CBSSA). The best predictions for Step 2 where achieved with data obtained from the Step 1 22 23 24 result and some of the clinical miniboards (again at the end of the second year). As different 25 26 schools administer different tests (many use shelf or custom exams provided by the national 27 28 29 board of medical examiners (NBME) and some use in-house exams), there is unlikely to ever be 30 31 any consensus as to which specific determinants that a specific school should use to identify at 32 33 34 risk students. Rather, our recommendation is that schools perform this kind of analysis with their 35 36 own internal data and that they perform this in an ‘on-going ‘fashion as the data becomes 37 38 available. 39 40 41 By performing this type of analysis, we are able to start looking at our at risk students 42 43 empirically as they step through various milestones and intervene with much more confidence as 44 45 46 students’ progress through to their second year. In fact, we have built an in-house database that 47 48 allows appropriate administrators to analyze student performance and make predictive 49 50 51 assumptions for future performance that utilizes the data presented here. In an attempt to address 52 53 the issue of increasing our confidence in prediction of future failure on Step 1 earlier, we first 54 55 divided our student into quartiles using prediction data at matriculation and using prediction data 56 57 58 at end of MS1 year. None of the quartile analysis improved our confidence in our predictions 59 60 61 62 11 63 64 65 1 2 3 4 when compared to the student cohort as a whole. However, a limitation in this type of analysis 5 6 7 was the small power of the analysis when cohort was separated into quartiles. We will certainly 8 9 try and revisit this issue as we get data from larger datasets. 10 11 12 Identification of these variables which predict strongly for both of these high stakes 13 14 examinations in this training set data allows us to move forward by 1) validating this data with 15 16 current students and 2) starting to implement individualized remediation programs for students 17 18 19 predicted to fail their USMLE exams (see below). It is obvious that early intervention is 20 21 desirable for better student outcomes, but our initial data suggest more confidence in our 22 23 24 predictions after end of the first year or even during the second year. Our experience with 25 26 biomedical sciences (BMS) students also suggest that early determinants of success are not 27 28 29 always very predictive. Quite a few of our students in this program who were ‘on average’ 30 31 weaker than non-BMS entrants, graduated at the top of their class and/or hold leadership 32 33 34 positions in the medical school class. Although anecdotal, this is consistent with the data 35 36 presented in this manuscript which certainly casts doubt on the use of early pre-admissions data 37 38 when predicting future national exam performance. In our stepwise regression model, many of 39 40 41 our early medical schools exams also failed to be very predictive. 42 43 It is perhaps not surprising that we found pre-admission performance does not strongly 44 45 46 predict future medical school national exam performance or even medical school performance in 47 48 general. This is supported by a publication that presented the “academic backbone” model which 49 50 51 elegantly showed that measures obtained prior to medical school were weaker indicators of 52 53 future medical school performance than were measure obtained during medical school [16]. This 54 55 is also consistent with a study from a single school with a large number of medical students (n= 56 57 58 782) which reported that pre-admissions academic backgrounds (e.g. humanities, biology, 59 60 61 62 12 63 64 65 1 2 3 4 physics, etc.) had no bearing on the outcome of medical school graduation [17]. Although the 5 6 7 findings were used to discuss limitations in medical school admissions requirements and 8 9 policies, these reports and others certainly indicate pre-admission student values as having 10 11 12 limited value in either admissions and/or future medical school performance. We do include pre 13 14 admissions data in our pivot tables and databases that we have available for tracking student 15 16 performance. However, we now understand that its data is less reliable than those such as 17 18 19 internal and external exams taken during medical school. 20 21 This data that came from this analysis is now currently being used in the following 22 23 24 manner: students are now being stratified into risk categories. The top risk category is described 25 26 as students who are at risk for very significant failure on their step 1 exam. The second highest 27 28 29 risk is that of students who are at risk for being slightly below or at the passing rate. The less 30 31 significant risk group (denoted as yellow in our database) are students who would be counseled 32 33 34 to delay the taking of Step 1 by at least 1 clinical rotation. These students would be able to take 35 36 advantage of additional study time that may include participation in practice exams and guided 37 38 tutorials. The most significant risk group (denoted as red in our database) are second year 39 40 41 students who would be strongly encouraged to delay taking Step 1 by at least two rotations and 42 43 offered more structured remediation. The aim of this risk stratification and delay in taking third 44 45 46 year rotations assists the students by helping them achieve passing rates for their first attempt 47 48 when sitting for the exam. It also helps the students by attempting to reduce the rate of dropping 49 50 51 out of an entire year due to delays in clinical rotations due to failure of Step 1 in the first sitting. 52 53 Furthermore, it is our policy that students can only be behind by two clinical rotations before 54 55 they are required to sit out for the full year. The other important internal policy is that students 56 57 58 must pass Step 1 before they can be officially accepted as a third year student. Risk stratification 59 60 61 62 13 63 64 65 1 2 3 4 using prediction analysis is a new process for us and it is certainly not foolproof. Students can 5 6 7 only be encouraged to delay taking the exam and/or take advantage of remediation and not all 8 9 will take the recommendations from our administration. Students who are in the ‘red’ risk 10 11 12 category are certainly not guaranteed to pass Step 1. However, all together we feel this is a value 13 14 added academic advising tool that we can now start using more avidly. We have not 15 16 implemented a risk stratification process for Step 2 CK but are currently discussing plans to do 17 18 19 so. 20 21 There are a number of confounding factors that are likely to have an impact on our data. 22 23 24 Our curriculum has undergone extensive revisions, and there have been dramatic changes to the 25 26 curriculum during the period of study [18]. In particular, we have moved to a system-based spiral 27 28 29 curriculum during this study time and we have altered our pedagogy to deemphasize lectures and 30 31 emphasize self-directed and collaborative learning strategies [19]. We strove to control for 32 33 34 changes in our curriculum by not using all of the students who attended medical school from 35 36 2008 to 2012. As the changes in curriculum had large effects on the exams that students took, we 37 38 made sure that we controlled for this by only using student data from those who took exams that 39 40 41 came from traditional topic based curriculum. Thus, we have more students in our MS2 cohort in 42 43 this student that we did in our MS1 cohort. We look forward to comparing predictions from 44 45 46 students exposed to the two different curriculums to see if there is a significant change. The data 47 48 presented here represents a single institution and some of the data may not be as continuous and 49 50 51 normally distributed as assumed. That said, the predictive value of performance on these 52 53 schoolwork based tests were still quite superior to that of the pre-admission data which we 54 55 collected. We feel that it is important for all schools to consider performing this type of analysis 56 57 58 59 60 61 62 14 63 64 65 1 2 3 4 and not rely on values from published studies as specific internal exams are likely to play a 5 6 7 unique and important role for their own predictions. 8 9 Of course, now that we have a tentative way to identify “at risk” students, we need to 10 11 12 prospectively validate our findings and ultimately develop comprehensive intervention programs 13 14 which change the academic trajectory of such students. The work here thus, allows us to make a 15 16 much more informed decision when identifying students at risk. 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 15 63 64 65 1 2 3 4 Data 5 6 7 Table 1A 8 9 Pre-admission Variables Total = 22 variables 10 Student’s age at medical school matriculation 11 12 Number of medical school matriculants from student’s primary undergraduate 13 institution 14 Student’s overall undergraduate GPA at primary undergraduate institution 15 Student’s undergraduate math-science GPA at primary undergraduate institution 16 17 Ratio of student’s overall undergraduate GPA at primary undergraduate institution 18 to mean overall undergraduate GPA of all medical school matriculants from that 19 institution 20 Number of MCAT exams taken prior to medical school matriculation 21 22 Total exam score from first MCAT 23 VR score from first MCAT 24 BS score from first MCAT 25 PS score from first MCAT 26 27 Highest total MCAT exam score 28 VR score from MCAT exam with highest total score 29 BS score from MCAT exam with highest total score 30 31 PS score from MCAT exam with highest total score 32 Lowest total MCAT exam score 33 VR score from MCAT exam with lowest total score 34 BS score from MCAT exam with lowest total score 35 36 PS score from MCAT exam with lowest total score 37 Golden total MCAT exam score (total of highest of each VR, PS and BS) 38 Individually highest VR score from among all MCAT exams 39 Individually highest BS score from among all MCAT exams 40 41 Individually highest PS score from among all MCAT exams 42 Total number of pre-admissions variables considered in prediction analysis from students 43 admitted to the JCESOM from 2008-2012. 44 45 Table 1B 46 47 MS1 Year Variables Total = 2 variables 48 49 Percentage exam score on first institutionally-developed, multiple-choice exam 50 Overall Percentage exam score all institutionally-developed, multiple-choice exams 51 Total number of MS1 variables considered in prediction analysis from students admitted to the 52 JCESOM from 2008-2012. 53 54 55 56 57 58 59 60 61 62 16 63 64 65 1 2 3 4 Table 1C 5 6 MS2 Year Variables Total = 13 variables 7 8 Percentage exam score on first institutionally-developed, multiple-choice exam 9 Overall Percentage exam score all institutionally-developed, multiple-choice exams 10 NBME Introduction to Clinical Diagnosis Subject Exam 11 NBME Microbiology Subject Exam 12 13 NBME Pathology Subject Exam 14 NBME Subject Exam 15 Step 1 equivalent score on NBME CBSSA exam 16 Family Medicine Clinical Sciences examination 17 18 Internal Medicine Clinical Sciences examination 19 Obstetrics and Gynecology Clinical Sciences examination 20 Pediatrics Clinical Sciences examination 21 Psychiatry Clinical Sciences examination 22 23 Surgery Clinical Sciences examination 24 Total number of MS2 variables considered in prediction analysis from students admitted to the 25 JCESOM from 2008-2012. 26 27 28 29 Table 2. USMLE Step 1 Predictions 30 31 [See attached figure] 32 33 34 The table summarizes the predictions for year one and two medical students and their 35 36 performance on the USMLE Step 1 exams. Adjusted R2 values are shown for the predictions at 37 38 39 each milestone as well as the intercept for the appropriate set of predictors. Grayed out cells in 40 41 the table refer to comparisons that were not included as they fell out of the model due to 42 43 44 decreased significance (p-values were not significant). Abbreviations included in the table are as 45 46 follows: CBSSA- comprehensive basic science self-assessment, MS1 – first year medical, and 47 48 MS2 – second year medical. P-values for all data reported in this table exceeded 0.05 and are not 49 50 51 included for brevity. The total number of students used for each comparison is listed at the 52 53 bottom of the table. 54 55 56 57 58 59 60 61 62 17 63 64 65 1 2 3 4 Table 3. USMLE Step 2CK predictions 5 6 7 [See attached figure] 8 9 This table summarizes the predictions for the year two medical students and their performance 10 11 2 12 on the USMLE Step 2 Clinical knowledge exam. Adjusted R values are shown for the 13 14 predictions at each milestone as well as the intercept for the appropriate set of predictors. Grayed 15 16 out cells in the table refer to comparisons that were not included as they fell out of the model due 17 18 19 to decreased significance (p-values were not significant). Step 1 refers to the total score assigned 20 21 to each student that was provided by the National Board of Medical Examiners for first time test 22 23 24 takers. All clinical subject (Miniboards) exams were taken towards the end of second year after 25 26 all clinical rotation were completed. The total number of students used for each comparison is 27 28 29 listed at the bottom of the table. 30 31 32 33 34 Figure 1. Summary data of all predictions used in this study that reached statistical significance. 35 36 [See attached figure] 37 38 Summary data includes medical school milestones on the X-axis and their corresponding AR2 39 40 41 values on the Y-axis. The first group of values on the X-axis refer to the first year while the 42 43 second groupings refer to those in the second year. Step 1 and Step 2 predictions are given when 44 45 46 comparisons were significant. Total Y1 refers to the total exams for year 1 while total Y2 refers 47 48 to the total exams for the second year. The Miniboards on the step 1 curve refer to the basic 49 50 51 science subject exams while those on the Step 2 refer to relevant exams for the clinical sciences. 52 53 54 55 56 57 58 59 60 61 62 18 63 64 65 1 2 3 4 Refs: 5 6 7 1. Richardson PH, Winder B, Briggs K, Tydeman C. Grade predictions for school-leaving examinations: do 8 they predict anything? Med Educ. 1998;32(3):294-7. 9 2. Collins JP, White GR, Kennedy JA. Entry to medical school: an audit of traditional selection 10 requirements. Med Educ. 1995;29(1):22-8. 11 3. Husbands A, Mathieson A, Dowell J, Cleland J, MacKenzie R. Predictive validity of the UK clinical 12 13 aptitude test in the final years of medical school: a prospective cohort study. BMC Med Educ. 14 2014;14:88. doi:10.1186/1472-6920-14-88. 15 4. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance 16 and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 17 2007;82(1):100-6. doi:10.1097/01.ACM.0000249878.25186.b7. 18 19 5. Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam 20 in relation to academic performance through medical school: a national cohort study of 2001-2004 21 matriculants. Acad Med. 2013;88(5):666-71. doi:10.1097/ACM.0b013e3182864299. 22 6. Kreiter CD, Kreiter Y. A Validity Generalization Perspective on the Ability of Undergraduate GPA and 23 the Medical College Admission Test to Predict Important Outcomes. Teaching and Learning in Medicine. 24 2007;19(2):95-100. doi:10.1080/10401330701332094. 25 26 7. Siu E, Reiter HI. Overview: what's worked and what hasn't as a guide towards predictive admissions 27 tool development. Advances in health sciences education : theory and practice. 2009;14(5):759-75. 28 doi:10.1007/s10459-009-9160-8. 29 8. Kleshinski J, Khuder SA, Shapiro JI, Gold JP. Impact of preadmission variables on USMLE step 1 and 30 step 2 performance. Advances in health sciences education : theory and practice. 2009;14(1):69-78. 31 32 doi:10.1007/s10459-007-9087-x. 33 9. Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance 34 outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14(1):34- 35 42. doi:10.1207/s15328015tlm1401_9. 36 10. Al Shawwa L, Abulaban AA, Abulaban AA, Merdad A, Baghlaf S, Algethami A et al. Factors potentially 37 38 influencing academic performance among medical students. Adv Med Educ Pract. 2015;6:65-75. 39 doi:10.2147/AMEP.S69304. 40 11. Adam J, Bore M, Childs R, Dunn J, McKendree J, Munro D et al. Predictors of professional behaviour 41 and academic outcomes in a UK medical school: A longitudinal cohort study. Med Teach. 2015:1-13. 42 doi:10.3109/0142159x.2015.1009023. 43 12. Puddey IB, Mercer A. Predicting academic outcomes in an Australian graduate entry medical 44 45 programme. BMC Med Educ. 2014;14:31. doi:10.1186/1472-6920-14-31. 46 13. Dickman RL, Sarnacki RE, Schimpfhauser FT, Katz LA. Medical students from natural science and 47 nonscience undergraduate backgrounds. Similar academic performance and selection. JAMA. 48 1980;243(24):2506-9. 49 14. Stratton TD, Elam CL. A holistic review of the medical school admission process: examining correlates 50 51 of academic underperformance. Med Educ Online. 2014;19:22919. doi:10.3402/meo.v19.22919. 52 15. DeZee KJ, Magee CD, Rickards G, Artino AR, Jr., Gilliland WR, Dong T et al. What aspects of letters of 53 recommendation predict performance in medical school? Findings from one institution. Acad Med. 54 2014;89(10):1408-15. doi:10.1097/acm.0000000000000425. 55 16. McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. The Academic Backbone: longitudinal 56 continuities in educational achievement from secondary school and medical school to MRCP(UK) and 57 58 the specialist register in UK medical students and doctors. BMC Med. 2013;11:242. doi:10.1186/1741- 59 7015-11-242. 60 61 62 19 63 64 65 1 2 3 4 17. Neame RL, Powis DA, Bristow T. Should medical students be selected only from recent school-leavers 5 who have studied science? Med Educ. 1992;26(6):433-40. 6 7 18. Miller B, Dzwonek B, McGuffin A, Shapiro JI. From LCME probation to compliance: the Marshall 8 University Joan C Edwards School of Medicine experience. Adv Med Educ Pract. 2014;5:377-82. 9 doi:10.2147/amep.s70891. 10 19. Harden RM. What is a spiral curriculum? Med Teach. 1999;21(2):141-3. 11 doi:10.1080/01421599979752. 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 20 63 64 65 Table1A,B,C Click here to download Table: Table1_ABC.pdf

Table 1A Pre-admission Variables Total = 22 variables Student’s age at medical school matriculation Number of medical school matriculants from student’s primary undergraduate institution Student’s overall undergraduate GPA at primary undergraduate institution Student’s undergraduate math-science GPA at primary undergraduate institution Ratio of student’s overall undergraduate GPA at primary undergraduate institution to mean overall undergraduate GPA of all medical school matriculants from that institution Number of MCAT exams taken prior to medical school matriculation Total exam score from first MCAT VR score from first MCAT BS score from first MCAT PS score from first MCAT Highest total MCAT exam score VR score from MCAT exam with highest total score BS score from MCAT exam with highest total score PS score from MCAT exam with highest total score Lowest total MCAT exam score VR score from MCAT exam with lowest total score BS score from MCAT exam with lowest total score PS score from MCAT exam with lowest total score Golden total MCAT exam score (total of highest of each VR, PS and BS) Individually highest VR score from among all MCAT exams Individually highest BS score from among all MCAT exams Individually highest PS score from among all MCAT exams Total number of pre-admissions variables considered in prediction analysis from students admitted to the JCESOM from 2008-2012. Table 1B MS1 Year Variables Total = 2 variables Percentage exam score on first institutionally-developed, multiple-choice exam Overall Percentage exam score all institutionally-developed, multiple-choice exams Total number of MS1 variables considered in prediction analysis from students admitted to the JCESOM from 2008-2012. Table 1C MS2 Year Variables Total = 13 variables Percentage exam score on first institutionally-developed, multiple-choice exam Overall Percentage exam score all institutionally-developed, multiple-choice exams NBME Introduction to Clinical Diagnosis Subject Exam NBME Microbiology Subject Exam NBME Pathology Subject Exam NBME Pharmacology Subject Exam Step 1 equivalent score on NBME CBSSA exam Family Medicine Clinical Sciences examination Internal Medicine Clinical Sciences examination Obstetrics and Gynecology Clinical Sciences examination Pediatrics Clinical Sciences examination Psychiatry Clinical Sciences examination Surgery Clinical Sciences examination Total number of MS2 variables considered in prediction analysis from students admitted to the JCESOM from 2008-2012. Table 2. USMLE Step 1 Predictions

Academic Milestones MS1 Year MS2 Year USMLE Step 1 Prediction Matriculation 1st MS1 Exam All MS1 Exams 1st MS2 Exam All MS2 Exams Miniboards CBSSA Adjusted R2 0.12 0.28 0.38 0.46 0.53 0.65 0.77 Intercept 145.48 81.44 32.88 3.19 -28.19 76.09 80.67 Undergrad Math & Science GPA 9.68 Lowest Total MCAT 1.58 1.29 0.70 0.68 0.82 1st MS1 Exam Score (%) 1.26 All MS1 Exam Scores (%) 2.00 1.73 0.90 0.83 1st MS2 Exam Score (%) 0.61 All MS2 Exam Scores (%) 1.74 Coefficients Microbiology+ Miniboard 0.06 0.04 Pathology Miniboard 0.07 0.04 CBSSA 0.49 N= 344 198 198 344 344 198 198

The table summarizes the predictions for year one and two medical students and their performance on the USMLE Step 1 exams. Adjusted R2 values are shown for the predictions at each milestone as well as the intercept for the appropriate set of predictors. Grayed out cells in the table refer to comparisons that were not included as they fell out of the model due to decreased significance (p-values were not significant). Abbreviations included in the table are as follows: CBSSA- comprehensive basic science self-assessment, MS1 – first year medical, and MS2 – second year medical. P-values for all data reported in this table exceeded 0.05 and are not included for brevity. The total number of students used for each comparison is listed at the bottom of the table. Table 3. USMLE Step 2CK predictions

Academic Milestones MS2 Year USMLE Step 2 CK Prediction Early Late Step 1 Final Adjusted R2 0.32 0.39 0.49 0.62 Intercept 84.45 51.03 48.93 144.75 Lowest Total MCAT 0.36* 0.44* 1st MS1 Exam Score (%) All MS1 Exam Scores (%) 0.91 0.18* 1st MS2 Exam Score (%) 0.73 All MS2 Exam Scores (%) 1.80 0.95 1st Step 1 score 0.46 0.31

Coefficients Family Medicine Minboard 0.09 Internal Medicine Miniboard 0.11 Obsterics/Gynecology Miniboard 0.07 Pediatrics Miniboard 0.15 N= 125 125 128 248 * Indicated p-values were not significant (>0.05)

This table summarizes the predictions for the year two medical students and their performance on the USMLE Step 2 Clinical knowledge exam. Adjusted R2 values are shown for the predictions at each milestone as well as the intercept for the appropriate set of predictors. Grayed out cells in the table refer to comparisons that were not included as they fell out of the model due to decreased significance (p-values were not significant). Step 1 refers to the total score assigned to each student that was provided by the National Board of Medical Examiners for first time test takers. All clinical subject (Miniboards) exams were taken towards the end of second year after all clinical rotation were completed. The total number of students used for each comparison is listed at the bottom of the table. Figure 1

1.0 Step1 Step2 0.8 2 0.6

0.4 Adjusted R

0.2

0.0

n s o m 1 2 2 d A 1 2 s i a Y Y Y r S p Y rd s x l y l a S e l a is E ta rl ta o B t ta o m t o a o ib C S o b d s T E T n + T i A 1 i in - M s + M re rd 1 P a p + o te 1 ib S p in te M S