Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 (published in the March 2020 issue of Academic Medicine)
MCAT® is a program of the Association of American Medical Colleges This is a publication of the Association of American Medical Colleges. The AAMC serves and leads the academic medicine community to improve the health of all. www.aamc.org
© 2020 Association of American Medical Colleges. May not be reproduced or distributed without prior written permission. To request permission to use the Foreword, please visit www.aamc.org/91514/reproductions.html.
Articles are reprinted from the Academic Medicine issue published in March 2020. To request permission for the articles, please visit www.academicmedicine.org. MCAT Validity Committee Members
Catherine R. Lucey, MD (Chair) Francie Cuffney, PhD Executive Vice Dean and Vice Dean for Education President, National Association of Advisors for the University of California, San Francisco, School of Medicine Health Professions Professor of Biological Sciences Aaron Saguil, MD, MPH (Vice Chair) Meredith College Associate Dean, Regional Education - San Antonio Uniformed Services University of the Health Sciences Martha L. Elks, MD, PhD F. Edward Hébert School of Medicine Senior Associate Dean for Educational Affairs Morehouse School of Medicine Leila Amiri, PhD Assistant Dean for Admissions and Recruitment William Gilliland, MD University of Illinois College of Medicine Associate Dean for Medical Education* Uniformed Services University of the Health Sciences Ngozi F. Anachebe, PharmD, MD F. Edward Hébert School of Medicine Associate Dean of Admissions and Student Affairs Morehouse School of Medicine Jorge A. Girotti, PhD, MHA Associate Dean for Admissions and Special Barbara S. Beckman, PhD Curricular Programs* Associate Dean of Admissions* University of Illinois College of Medicine Tulane University School of Medicine Kristen Goodell, MD Ruth Bingham, PhD Associate Dean of Admissions Associate Specialist and Director* Boston University School of Medicine Pre-Health/Pre-Law Advising Center University of Hawaii at Manoa Joshua T. Hanson, MD, MPH Associate Dean for Student Affairs Kevin Busche, MD, FRCPC University of Texas Health Science Center at San Antonio Assistant Dean for Undergraduate Medical Education Joe R. and Teresa Lozano Long School of Medicine University of Calgary Cumming School of Medicine Loretta Jackson-Williams, MD, PhD Deborah Castellano, MS Vice Dean for Medical Education Senior Program Manager University of Mississippi Medical Center School of Medicine Philadelphia College of Osteopathic Medicine David J. Jones, PhD Julie A. Chanatry, PhD Senior Associate Dean for Admissions* Chair, Health Sciences Advisory Committee University of Texas Health Science Center at San Antonio Colgate University Joe R. and Teresa Lozano Long School of Medicine
Hallen Chung, MA Robert Liotta, MD Director of Admissions Associate Dean of Recruitment and Admissions University of California, San Francisco, School of Medicine Uniformed Services University of the Health Sciences F. Edward Hébert School of Medicine Jerry Clark, PhD, MBA Chief Student Affairs Officer R. Stephen Manuel, PhD Associate Dean for Student Affairs Associate Dean for Medical School Admissions* University of Mississippi Medical Center School of Medicine University of Mississippi Medical Center School of Medicine
Daniel M. Clinchot, MD Stephanie C. McClure, MD Vice Dean for Education Senior Associate Dean for Student and Academic Affairs Associate Vice President for Health Sciences Education Meharry Medical College School of Medicine The Ohio State University College of Medicine Janet McHugh Liesel Copeland, PhD Admissions Officer* Assistant Dean for Medical Education and Admissions Memorial University of Newfoundland Faculty of Medicine Rutgers Robert Wood Johnson Medical School
MCAT® is a program of the 3 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Chad Miller, MD, FACP, FHM Doug Taylor Senior Associate Dean for Undergraduate Medical Education Associate Dean for Student Affairs - Saint Louis University School of Medicine Admissions and Records East Tennessee State University James H. Quillen College Cindy A. Morris, PhD of Medicine Assistant Dean for Admissions Tulane University School of Medicine Carol A. Terregino, MD Senior Associate Dean for Education and Academic Affairs Wanda L. Parsons, MD, FCFP Associate Dean for Admissions Assistant Dean for Admissions* Chair of the Admissions Committee Memorial University of Newfoundland Faculty of Medicine Rutgers Robert Wood Johnson Medical School
Tanisha Price-Johnson, PhD Ian W. Walker, MD Executive Director of Admissions Director of Admissions* University of Arizona College of Medicine – Tucson University of Calgary Cumming School of Medicine
Boyd F. Richards, PhD Robert A. Witzburg, MD Director, Center for Education Research and Evaluation* Associate Dean and Director of Admissions* Columbia University Vagelos College of Physicians Boston University School of Medicine and Surgeons David Wofsy, MD Stuart Slavin, MD, MEd Associate Dean for Admissions Associate Dean for Curriculum* University of California, San Francisco, School of Medicine Saint Louis University School of Medicine Mike Woodson Aubrie Swan Sein, PhD, EdM Director of Admissions Assistant Professor of Educational Assessment in Tulane University School of Medicine Pediatrics and Dental Medicine Director, Center for Education Research and Evaluation Columbia University Vagelos College of Physicians and Surgeons
* indicates that title, affiliation, or both have changed since serving on the MCAT Validity Committee
MCAT® is a program of the 4 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges TABLE OF CONTENTS
Foreword 6 Catherine Reinis Lucey, MD; Aaron Saguil, MD, MPH
The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study 9 Kevin Busche, MD, FRCPC; Martha L. Elks, MD, PhD; Joshua T. Hanson, MD, MPH; Loretta Jackson-Williams, MD, PhD; R. Stephen Manuel, PhD; Wanda L. Parsons, MD, FCFP; David Wofsy, MD; Kun Yuan, PhD. Acad Med. 2020;95:387-395.
Investigating Group Differences in Examinees’ Preparation for and Performance on the New MCAT Exam 18 Jorge A. Girotti, PhD, MHA; Julie A. Chanatry, PhD; Daniel M. Clinchot, MD; Steph- anie C. McClure, MD; Aubrie Swan Sein, PhD, EdM; Ian W. Walker, MD; Cynthia A. Searcy, PhD. Acad Med. 2020;95:365-374.
How to Help Students Strategically Prepare for the MCAT Exam and Learn Foundational Knowledge Needed for Medical School 28 Aubrie Swan Sein, PhD, EdM; Francie Cuffney, PhD; Daniel M. Clinchot, MD. Acad Med. 2020;95:484.
The Diversity and Success of Medical School Applicants With Scores in the Middle Third of the MCAT Score Scale 29 Carol A. Terregino, MD; Aaron Saguil, MD, MPH; Tanisha Price-Johnson, PhD; Ngozi F. Anachebe, PharmD, MD; Kristen Goodell, MD. Acad Med. 2020;95:344-350.
The Consequences of Structural Racism on MCAT Scores and Medical School Admissions: The Past Is Prologue 36 Catherine Reinis Lucey, MD; Aaron Saguil, MD, MPH. Acad Med. 2020;95:351-356.
Leveraging the Medical School Admissions Process to Foster a Smart, Humanistic, and Diverse Physician Workforce 42 Richard M. Schwartzstein, MD. Acad Med. 2020;95:333-335.
MCAT® is a program of the 5 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Foreword Today’s physicians must be capable of applying scientific institutional roadblocks on the success of disadvantaged knowledge and reasoning skills to solve complex problems students and students from minorities underrepresented in clinical care and research. Medical knowledge is in medicine. It stresses the importance of using a holistic exploding, health care delivery is changing, and physicians lens in conducting medical school admissions and also are caring for an increasingly diverse and aging patient innovative approaches to addressing inequities. Finally, population. Now more than ever, medical educators must the commentary by Schwartzstein6 emphasizes the need ensure future physicians are prepared to care for patients to admit students with both the scientific knowledge and from all backgrounds and communities, including those interpersonal competencies necessary to excel in medical that have fewer resources and less access to health care study and practice. He, too, recognizes the impact of services. The redesigned MCAT® exam was introduced in structural inequalities in limiting educational opportunity 2015 to reflect this reality and better assess the concepts and suggests ways medical schools can partner with and reasoning skills students need to be ready for the premedical educators to help students from disadvantaged medical school curriculum. backgrounds obtain the necessary preparation to succeed in medical school and beyond. The AAMC formed the MCAT Validity Committee (MVC) to evaluate the fairness, use, and predictive validity of the Predictive Validity of MCAT Scores new exam. We are the co-chairs of this committee, whose members are educators, admissions officers, researchers, The research presented in the first article by Busche and 1 and prehealth advisors from medical schools and his coauthors describes the strong relationship between undergraduate institutions in the United States and MCAT scores and students’ future performance in medical
Canada. Our research includes a 10-year study following school. They found medium to large associations between several cohorts of students through medical school to MCAT scores and students’ grades in their first-year examine the predictive validity of MCAT scores in relation courses. They also examined the predictive value of using to local and national performance outcomes. The research MCAT scores alone and together with undergraduate also examines national trends in the test preparation and GPAs and showed that using both metrics provides better performance of individuals who take the MCAT exam, and prediction of performance than using either one alone. how admissions officers and their committees use MCAT Importantly, their research shows that students from scores in medical student selection. This publication different backgrounds who enter medical school with the includes the first collection of articles from the MVC’s same MCAT score have similar academic outcomes. The research, originally published in the March 2020 issue of authors found that MCAT scores predict performance Academic Medicine. comparably for students from racial and ethnic minority The first article by Busche et al.1 describes how well MCAT and majority groups, for those from lower and higher scores predicted students’ academic success in the first socioeconomic backgrounds, and for females and males. year of medical school both alone and when used with Their research also found that students with a wide undergraduate grade point averages (GPAs). They found range of MCAT scores reach an important milestone in that MCAT scores do a good job of predicting students’ completing the first year of medical school. Overall, 97% of performance in their first-year courses and their on-time students progressed on time to their second year. progression to the second year of medical school and that students with a wide range of scores perform well. A Additional research findings reported since this article’s second article by Girotti et al.2 reports on how examinees publication show that MCAT scores predict students’ from different backgrounds prepared for and performed performance in their second-year courses, their on the MCAT exam. Their analyses include a close performance on the United States Medical Licensing examination of the preparation and performance trends Examination Step 1 exam, and the time it takes them of students who attended undergraduate institutions with to progress to Year 3.7 The prediction continues to be different levels of resources and by their socioeconomic stronger when students’ MCAT scores are considered status. A Last Page by Swan Sein et al.3 follows, offering together with their undergraduate GPAs. Additionally, students guidance on how to strategically prepare for the MCAT scores provide comparable prediction of students’ MCAT exam. A Perspective by Terregino and colleagues4 Year 2 and Step 1 performance and in their progression to draws on the findings from Busche et al. that students clerkship rotations. That MCAT scores provide the same with MCAT scores across the score range perform well predictive value for applicants from different backgrounds in medical school. Their research shows that a more as they make their way through medical school is an flexible approach to using MCAT scores can help build essential source of evidence about the fairness of using more diverse classes. The Perspective by Lucey and MCAT scores in medical student selection. Saguil5 reviews the impact of structural racism and
MCAT® is a program of the 6 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Group Differences in MCAT these differences. The Last Page by Swan Sein and Preparation and Performance her coauthors3 offers guidance, learning strategies, and information on free resources to help students Studying group differences in MCAT scores is central strategically prepare for the MCAT exam. Providing to the MVC’s research given historical findings that all examinees, especially those from disadvantaged examinees from lower socioeconomic backgrounds and backgrounds, with the opportunity and resources to races and ethnicities underrepresented in medicine score prepare for the MCAT exam is an essential component of lower, on average, than their peers from majority groups. the MVC’s research agenda. Research on academic achievement and educational opportunities reveals that differences in students’ home, school, and community circumstances translate into gaps MCAT Scores and Medical School Admissions in students’ achievement in K-12 education and college, The MVC’s research underscores the important on standardized tests, and even in the undergraduate relationship between students’ economic and educational GPAs of medical school applicants.8 backgrounds and their academic achievement. Terregino and her coauthors4 explored the diversity and success of The designers of the new MCAT exam recognized that applicants scoring in the middle of the MCAT score scale it must assess a wider range of concepts and scientific (495-504) compared to applicants in the upper range reasoning skills than the old one did, which in turn might (505-528). They found that medical schools enrolling increase group differences in test scores. Research higher percentages of students with scores in the middle on premedical students’ opportunities to learn these range were able to create more diverse student bodies foundational concepts, including targeted investigations than those schools that admitted few to no students in at minority-serving and underresourced undergraduate this range. institutions, informed the new test blueprints. Additionally, creating easily accessible free and low-cost They also show that students with a wide range of MCAT preparation materials, providing financial assistance scores can succeed in medical school. A key finding is for students in need, and targeting outreach to students that 95% of students scoring in the middle of the MCAT from disadvantaged backgrounds aimed to deliver the score scale progressed to Year 2 on time — the first resources and information students needed to prepare hurdle in completing medical school with unimpeded for the exam. progress. That students with mid-range scores progress at nearly the same rate as those with upper-range scores 2 Girotti and his coauthors studied the performance of reinforces that medical schools support the students they examinees who took the new exam, comparing current admit by leveraging the curricular, academic support, and trends to the past. Their data show that, even though learning environments that have been designed to help the MCAT exam now tests more concepts, differences them meet their educational goals and students’ needs. between the average MCAT scores of examinees from minority and majority groups are no larger or smaller than The authors stress that using MCAT scores in holistic they were on the previous exam. Although examinees review provides admissions officers with important from minority groups score lower, on average, than those context for interpreting applicants’ academic from majority groups, there are examinees from every backgrounds and selecting students who will help them group who obtain scores in the lower, middle, and higher meet their schools’ missions and goals. In Lucey and ranges of the MCAT score scale. Saguil’s5 Perspective on medical school admissions and societal inequality, we pull together evidence to present 8 Prior research on the old exam and the findings ways that, historically and even today, governmental 1 presented in Busche et al. provide evidence that these programs, education, housing, and other facets of society average score differences do not stem from technical provide unequal opportunities to citizens of different deficiencies in the exam. That is, evidence that students racial and ethnic groups, contributing to disparities in from different backgrounds admitted with the same wealth, health, and academic achievement. For students MCAT score showed similar levels of success in medical from racial and ethnic minority groups, years of structural school indicates that these scores do not show and interpersonal racism and bias have deepened the predictive bias. effects of socioeconomic inequality that presents Despite the widespread dissemination of free and low- barriers to academic achievement. Designing admissions cost preparation resources, Girotti and his coauthors policies and procedures holistically to consider found that examinees from lower socioeconomic applicants’ academic achievements in this context is an backgrounds and who attended less-resourced colleges essential lever for mitigating the effects of this inequality. and universities reported lower use of many free and Terregino and her coauthors encourage us to study the low-cost MCAT preparation resources than their more curricula and support programs of medical schools that
advantaged peers. The MVC has been exploring students’ enroll larger percentages of students with MCAT scores preparation strategies and barriers to understand
MCAT® is a program of the 7 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges in the middle of the scale to help all schools enhance effects of structural inequalities so that medical schools their ability to increase the diversity of their classes can better prepare students from disadvantaged and with capable, academically ready students. In Lucey and underrepresented backgrounds to become successful Saguil’s Perspective, we also urge study of the pioneering physicians. Among the efforts, Schwartzstein calls efforts led by medical schools to increase the diversity for medical schools to form more partnerships with of their student bodies and address the deleterious undergraduate institutions serving disadvantaged students effects of structural inequalities. We highlight successful and utilize online course technology in the classroom to medical school pipeline programs around the country supplement their learning. and innovative curricula on the social and behavioral determinants of health being taught throughout the MCAT Validity Committee medical education continuum to counteract inequities. We thank our colleagues for their commitment to studying Finally, an invited commentary by Schwartzstein6, a the validity of the new MCAT exam and its intersection with member of the committee that developed the new MCAT important issues in medical school admissions. There is exam, reminds us of the importance of valuing academic more to learn, and the MVC’s future research will include data appropriately as a signal of applicants’ critical studying the value of MCAT scores and undergraduate thinking skills and foundational science knowledge needed GPAs in predicting students’ clerkship performance, to master the scientific principles taught in medical performance on the Step 2 exams (CK and CS), and school. Schwartzstein argues that academic readiness, graduation within four or five years. The committee’s while necessary, is insufficient for students to succeed in work is essential to developing the tools and resources medical school and as physicians — students must come the admissions community needs to use MCAT scores in with the personal qualities that are the bedrock of the full context of applicants’ academic achievements, good physicians. attributes, and lived experiences. The MVC welcomes the opportunity to contribute research that will advance Finding ways to address the effects of unequal opportunity our medical schools’ shared goal of developing capable, earlier in students’ academic careers is key to expanding compassionate physicians to care for our increasingly the pipeline of applicants from diverse backgrounds diverse nation. and will elevate our ability to identify the applicants with the myriad experiences, backgrounds, academic Catherine R. Lucey, MD competencies, and humanistic qualities needed of our Chair, MCAT Validity Committee medical students and future physicians. Schwartzstein Aaron Saguil, MD, MPH expands on points made by Lucey and Saguil on the Vice Chair, MCAT Validity Committee importance of investing in initiatives that will mitigate the
References 1. Busche K, Elks ML, Hanson JT, et al. The validity of scores from the new MCAT exam in predicting student performance: results from a multisite study. Acad Med. 2020;95:387-395. 2. Girotti JA, Chanatry JA, Clinchot DM, et al. Investigating group differences in examinees’ preparation for and performance on the new MCAT exam. Acad Med. 2020;95:365-374. 3. Swan Sein A, Cuffney F, Clinchot D. How to help students strategically prepare for the MCAT exam and learn foundational knowledge needed for medical school. Acad Med. 2020;95:484. 4. Terregino CA, Saguil A, Price-Johnson T, Anachebe NF, Goodell K. The diversity and success of medical school applicants with scores in the middle third of the MCAT score scale. Acad Med. 2020;95:344-350. 5. Lucey CR, Saguil A. The consequences of structural racism on MCAT scores and medical school admissions: the past is prologue. Acad Med. 2020;95:351-356. 6. Schwartzstein RM. Leveraging the medical school admissions process to foster a smart, humanistic, and diverse physician workforce. Acad Med. 2020;95:333-335. 7. Association of American Medical Colleges. MCAT® Validity Data Report: New Findings About Predicting Performance Across Preclerkship Courses, Progression to the Third Year, and Performance on the USMLE Step 1 Exam. Washington, DC: AAMC; October 2019. 8. Davis D, Dorsey JK, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013;88:593-602.
MCAT® is a program of the 8 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study Kevin Busche, MD, FRCPC, Martha L. Elks, MD, PhD, Joshua T. Hanson, MD, MPH, Loretta Jackson-Williams, MD, PhD, R. Stephen Manuel, PhD, Wanda L. Parsons, MD, FCFP, David Wofsy, MD, and Kun Yuan, PhD
Abstract Purpose scores and total undergraduate similar prediction of performance The new Medical College Admission grade point averages (UGPAs), alone in M1, using both metrics provided Test (MCAT) was introduced in April and together, with their summative better prediction than either alone. 2015. This report presents findings from performance in M1, and the success Additionally, students with a wide the first study of the validity of scores rate of students with different MCAT range of MCAT scores progressed from the new MCAT exam in predicting scores in their on-time progression to to M2 on time. Finally, MCAT scores student performance in the first year of the second year of medical school (M2). provided comparable prediction of medical school (M1). They assessed whether MCAT scores performance in M1 for students provided comparable prediction of from different sociodemographic Method performance in M1 by students’ race/ backgrounds. The authors analyzed data from the ethnicity, socioeconomic background, national population of 2016 matriculants and gender. Conclusions with scores from the new MCAT exam This study provides early evidence that (N = 7,970) and the sample of 2016 Results scores from the new MCAT exam predict matriculants (N = 955) from 16 medical Correlations of MCAT scores with student performance in M1. Future schools who volunteered to participate summative performance in M1 ranged research will examine the validity of in the validity research. They examined from medium to large. Although MCAT scores in predicting performance correlations of students’ MCAT total MCAT scores and UGPAs provided in later years.
Editor’s Note: An Invited Commentary by R.M. gathered during interviews, when In 2014, the AAMC formed the Schwartzstein appears on pages 333–335. making admissions decisions. Because MCAT Validity Committee (MVC) the meaning of UGPAs differs for to evaluate the validity of scores from applicants from different undergraduate the new MCAT exam. The committee The Medical College Admission institutions and with different majors and includes representatives from 16 U.S. Test (MCAT) is a tool for assessing coursework,1 MCAT scores provide an and 2 Canadian medical schools and 2 applicants’ academic readiness for important common measure of academic prehealth advisors serving in current learning in medical school. It is a readiness. or previous leadership roles with the standardized examination that measures National Association of Advisors for the foundational knowledge of scientific In April 2015, the Association of the Health Professions. The MVC is concepts and reasoning skills needed tasked with evaluating evidence about by entering medical students. Medical American Medical Colleges (AAMC) introduced a new version of the MCAT the fairness, impact, use, and predictive school admissions committees use validity of scores from the new MCAT academic metrics, such as MCAT scores exam. The exam was redesigned to reflect changes in medical education, exam, as professional testing standards and undergraduate grade point averages require.4 This research is the foundation medical science, and health care delivery (UGPAs), with other information for evaluating the soundness of using as well as changes in society and the about applicants’ academic preparation MCAT scores in admissions decisions. and experiences as well as insights increasingly diverse and aging patient population in the United States and The predictive validity research Canada, which have occurred since Please see the end of this article for information conducted by the MVC examines the about the authors. the last revision of the MCAT exam value of scores from the new MCAT in 1991.2,3 Correspondence should be addressed to Kun Yuan, The new exam still tests exam in predicting medical student Association of American Medical Colleges, 655 K foundational concepts in biology, performance on a variety of outcomes St. NW Suite 100, Washington, DC 20001-2399; chemistry, and physics, along with telephone: (202) 909-2080; email: [email protected]. throughout medical school, including verbal reasoning skills; now, it also student performance in individual Acad Med. 2020;95:387–395. includes concepts from first-semester preclerkship and clerkship courses, First published online August 13, 2019 biochemistry and introductory doi: 10.1097/ACM.0000000000002942 passing the United States Medical Copyright © 2019 by the Association of American psychology and sociology. The new exam Licensing Examination (USMLE) Medical Colleges also requires applicants to demonstrate Step exams, and progression through Supplemental digital content for this article is more scientific reasoning skills than the and graduation from medical school. available at http://links.lww.com/ACADMED/A731. old exam did. This research also examines whether
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 387 XXX 9 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
MCAT scores provide comparable We refer to the first group as the national Summative performance in M1. The prediction of performance in medical population because it included all 2016 validity schools identified M1 courses school for students from different matriculants to every U.S. MD-granting to include in this evaluation. To be sociodemographic backgrounds. institution who had scores from the new included, courses had to have at least MCAT exam (N = 7,970). These students one medical student performance Research on the old MCAT exam (1991 were a subset of the more than 21,000 outcome that met the following criteria: to January 2015) addressed these same students who matriculated at the 145 measured individual (not team) questions. Findings from previous accredited U.S. MD-granting institutions performance, had adequate variation research showed that scores from the in 2016. in student performance, represented old MCAT exam predicted student students’ first attempts on these performance throughout medical We refer to the second group as the outcomes, and were continuous, ranging school.5–11 Prior research also found validity sample. It includes the students from 0 to 100. (The majority of M1 that scores from the old exam provided who were enrolled in one of the 16 courses at each school were included similar prediction of performance on the medical schools participating in the because they met these criteria.) USMLE Step 1 exam and medical school MCAT validity research who volunteered graduation for students from different to participate in this study. These schools Examples of selected courses included: racial/ethnic backgrounds.8,12 were selected from 65 medical schools that biochemistry, cell and molecular volunteered to participate in the MCAT biology, cardiovascular and pulmonary In this research report, we present the validity research. They represented a wide systems, behavioral medicine and results from the MVC’s first multisite range of institutional missions, geographic health, health care ethics, introduction study on the predictive validity of scores regions, public/private status, applicant to clinical anatomy, and community from the new MCAT exam. We examined pool sizes and characteristics, curricula, engagement. Although the selected the extent to which total scores predict instruction, and grading practices. courses varied widely in the extent to student performance in the first year which they related to the knowledge and About 80% (N = 955) of the students of medical school (M1). Specifically, skills tested on the MCAT exam, most with scores from the new MCAT exam we addressed the following 3 research taught natural sciences subjects. For who matriculated in 2016 at the validity questions: each selected course, we analyzed one schools volunteered for the study. Among primary outcome. Some were weighted 1. Do the total scores from the new these 955 students, 83% (N = 791) came averages of students’ performance on MCAT exam predict students’ from U.S. medical schools and were also a multiple assessments given throughout summative performance across first- subset of the national population of 2016 the course. Others were based on year courses and on-time progression matriculants with scores from the new students’ performance on the final MCAT exam. The remaining students (N to year 2 (M2)? exam. Because the courses selected = 164) came from 2 Canadian schools. 2. Do the total scores from the new by each validity school comprised the The validity sample was representative of MCAT exam add value beyond majority of M1 courses at the school, the total population of 2016 matriculants UGPAs in predicting students’ students’ average performance in these at the validity schools (N = 2,541) based summative performance across first- courses served as a measure of their on demographic characteristics and year courses? summative performance in their first- undergraduate academic performance. 3. Do the total scores from the new year coursework. MCAT exam provide comparable Data from these 2 groups complemented prediction of summative performance On-time progression to M2. We each other. The national population across first-year courses and on-time generated this progression outcome based progression to year 2 for students included more students and represented more schools, and it allowed for the study on student enrollment records submitted from different sociodemographic by the registrars of the 145 accredited backgrounds? of the first milestone in medical school— sufficient mastery of the curriculum to U.S. MD-granting schools. Students progress on time to the next year. The were categorized into 2 groups: those Method validity sample was smaller, but it allowed who experienced on-time progression to M2 and those who did not, according to Participants for our examination of institution- specific outcomes. their school’s curriculum. This outcome We used data from 2 groups of 2016 allowed us to study each student’s medical school matriculants in our We drew our data from deidentified performance defined by her or his analyses: (1) the national population of research tables in the AAMC’s Data sufficient mastery of the curriculum to 2016 matriculants with scores from the Warehouse and from outcome data progress on time to the next year. We new MCAT exam and (2) the sample provided by the validity schools. This analyzed progression data for all students of 2016 matriculants with scores from study was approved by the institutional in regular MD programs. Students in dual the new exam who attended one of the review board of the American Institutes degree programs were excluded from the medical schools conducting MCAT for Research. analysis. validity research (validity schools) and who volunteered to participate in the Criterion outcomes Predictors research on their institution-specific We used 2 types of criterion outcomes in Total scores from the new MCAT exam student outcomes. this study. and UGPAs were used as predictors.
388 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 10 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
MCAT total scores. The new MCAT Correlation coefficients can range from 0 for range restriction in the validity exam has 4 sections: (1) Biological and (no relationship) to ±1 (perfect positive/ sample compared with each school’s Biochemical Foundations of Living negative relationship).17 We present applicant population (see the Figure 3 16 Systems; (2) Chemical and Physical medians and interquartile ranges to legend for more details). We present the Foundations of Biological Systems; (3) summarize validity coefficients across medians and interquartile ranges of the Psychological, Social, and Biological schools.18 validity coefficients of the predictor(s) Foundations of Behavior; and (4) Critical with the outcome,18 along with the Analysis and Reasoning Skills. The first percentage of variance in the outcome We also calculated the percentages of 3 sections test 10 foundational concepts explained by the predictor(s). medical students in the national sample and 4 scientific inquiry and reasoning who achieved on-time progression to skills in the natural, behavioral, and social Research question 3. We used well- M2, using different ranges of MCAT sciences. The fourth section tests how established regression procedures4,12 well test takers comprehend, analyze, and total scores. to address the research question evaluate what they read, draw inferences about whether MCAT scores provided from text, and apply arguments to new Research question 2. To address the comparable prediction of performance for ideas and situations.13 The new MCAT second research question about the value students from different sociodemographic exam reports 4 section scores and a total of MCAT total scores and UGPAs, alone backgrounds, conducting analyses by score. The 4 section scores range from and together, in predicting students’ race/ethnicity, highest parental education, 118 to 132, with a midpoint at 125. The summative performance in M1, we and gender.19–21 total score is the sum of the 4 section conducted 3 sets of regression analyses: scores, with a range from 472 to 528 and Model 1: MCAT total score as the only For each sociodemographic variable, a midpoint at 500. predictor, Model 2: UGPA as the only students were divided into 2 groups. predictor, and Model 3: both MCAT total For race/ethnicity, students who self- Total UGPAs. Total UGPAs came from score and UGPA as predictors. identified as black or African American; either the application service used by the Hispanic, Latino, or Spanish; American validity school (i.e., the American Medical We conducted the regressions by school Indian or Alaska Native; or Native College Application Service, the Texas and corrected the resulting correlations Hawaiian or other Pacific Islander Medical and Dental School Application Service) or from the validity school itself for the 2 Canadian schools that did not use an external application service. In all cases, UGPAs were verified and standardized by the application services or the validity schools to allow medical schools to compare the academic experiences of applicants from undergraduate institutions that used different academic calendars and grading systems.14,15 UGPAs ranged from 0 to 4 for all but one validity school in Canada, which calculated students’ UGPAs on a 0–100 scale.
Data analysis Research question 1. To address the first research question about the extent to which MCAT scores predict students’ Figure 1 Median and interquartile range of the corrected correlations of Medical College M1 performance, we correlated MCAT Admission Test (MCAT) total scores with students’ summative performance in the first year of total scores with validity school students’ medical school (M1). Summative performance is the mean of students’ scores from the M1 summative performance across M1 courses that each validity school selected to include in this study. Summative performance showed courses. These analyses were conducted a strong correlation (above 0.90) with students’ official end-of-year performance, such as their M1 grade point average (GPA) or M1 class rank. The observed correlations were corrected for by school to control for differences across range restriction in MCAT total scores and undergraduate GPAs (UGPAs) due to student selection schools in the course outcomes used to in the admissions process.16 The correlation correction adjusted for the differences in the standard generate the summative performance deviations of MCAT scores and UGPAs in each school’s validity sample compared with the school’s outcome. The validity coefficients applicant pool while accounting for the correlations among MCAT scores, UGPAs, and the presented in this report represent the outcome in the validity sample. Because more 2017 applicants than 2016 applicants to the validity correlations of MCAT total scores with schools had scores from the new exam, the corrections for range restriction were made using data from the 2017 admissions cycle. The corrected correlation coefficients (validity coefficients) summative performance in M1 after are presented. The number next to the solid circle shows the median correlation. The numbers at correcting for range restriction in MCAT the ends of the interquartile range show the correlation at the 25th (lower end) and 75th (higher total scores and UGPAs due to student end) percentiles of the distribution of corrected correlation coefficients, respectively. The horizontal selection in the admissions process (see line with a Y axis value at 0.3 is the reference line for a medium association.17 See Supplemental the Figure 1 legend for more details).16 Digital Appendix 1 at http://links.lww.com/ACADMED/A731 for the observed correlations.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 389 11 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
were categorized as underrepresented the new MCAT exam based on most 0.47 and 0.68, respectively. The median in medicine (URM).22 Students who sociodemographic characteristics. Slightly correlation of UGPAs with summative self-identified as white or Asian were larger percentages of students received performance in M1 was 0.52, and the categorized as non-URM. fee assistance from the AAMC and 25th and 75th percentiles were 0.42 and identified as black or African American in 0.62, respectively. The results for MCAT Highest parental education was used the validity sample than in the national total scores were similar to those for as a proxy for students’ socioeconomic population. The validity sample and the UGPAs, but the median and interquartile backgrounds because of the close national population had similar means range for MCAT total scores were slightly relationship between education level and and standard deviations (SDs) of MCAT higher than those for UGPAs. When using income.23 Students were categorized into both MCAT total scores and UGPAs to total scores (Meansample = 507.87, SD = 2 groups based on parental attainment of predict summative performance in M1, 7.34; Meanpopulation = 508.57, SD = 6.92; a bachelor’s degree. One group included see Table 1). Both the validity sample the median correlation was 0.65, with an students who reported their parents did and the national population had a mean interquartile range of 0.56–0.79. not have a bachelor’s degree. The other UGPA of 3.70 (SD = 0.25). included students who reported that at Using MCAT scores and UGPAs together least one parent had a bachelor’s degree. Summative performance in M1 explained a much greater percentage Figure 1 shows the median and of the variance in students’ summative The third sociodemographic group was interquartile range of the correlations performance in M1 than using UGPAs based on self-reported gender. One group of MCAT total scores with students’ alone. The percentage of variance included female students, and the other summative performance in M1. The explained by MCAT scores and UGPAs included male students. median correlation was 0.57, and the 25th together based on the median validity and 75th percentiles were 0.47 and 0.68, coefficient (42%) was 1.6 times the We compared the degree to which respectively. percentage of variance explained by MCAT total scores predicted summative UGPAs alone (27%). These results show performance in M1 and on-time On-time progression to M2 by MCAT that, together, MCAT total scores and progression to M2 by race/ethnicity, total scores UGPAs provided stronger prediction of parental education, and gender. We students’ summative performance in M1 Figure 2 shows the percentages of conducted 3 sets of linear regression than either predictor alone. students in the national population analyses for summative performance who progressed to M2 on time by in M1 and 3 sets of logistic regression Predicting performance for students different ranges of MCAT total scores. analyses for on-time progression to M2. from different sociodemographic Overall, the vast majority (7,736; 97%) backgrounds of these students progressed on time, We used the estimated regression Table 2 shows the observed and predicted and the progression rate for students parameters from each regression analysis performance outcomes for students by across a wide range of MCAT total to generate the predicted outcome for race/ethnicity, highest parental education, scores was high. each student. We then computed the and gender; the differences between the average observed and predicted outcomes observed and predicted outcomes; and separately for all students included in The percentage of students who the effect sizes associated with those a sociodemographic group. We tested progressed on time was 93% or above differences. whether the mean residual, that is, the for those students with MCAT total difference between the average observed scores from 494 to 528. Less than 2% In the validity sample, the differences and predicted outcomes, differed from (N = 131) of the national population between the observed and predicted zero. We also computed the effect size reported scores below 494. The number outcomes of summative performance associated with each residual to estimate of students with scores below 494 was too in M1 were minor for students by race/ the magnitude of prediction error. These small to interpret meaningful differences ethnicity, parental education, and gender. effect sizes are measures of the magnitude in their progression rate compared with None of the mean differences were of prediction error. An effect size of 0.2 those who scored at or above 494. statistically significant. The magnitudes is considered small.24 Prediction error of these differences were also of no Value of MCAT total scores and UGPAs with an effect size less than 0.2 means the practical importance. For example, the in predicting students’ summative difference between the average observed mean difference between the observed performance in M1 and predicted outcomes is trivial and and predicted performance for students of no practical importance. All analyses Figure 3 shows the medians and identified as URM was− 0.38 (on a were conducted using Stata (version 14; interquartile ranges of the correlations 0–100 scale), with an effect size of −0.07. StataCorp, College Station, Texas). of MCAT total scores (Panel A), UGPAs The mean difference for students whose (Panel B), and MCAT total scores and parents did not have a bachelor’s degree Results UGPAs together (Panel C) with students’ was −0.09, with an effect size of −0.02. summative performance in M1. The mean difference for female students Characteristics of participants and was 0.13, with an effect size of 0.03. outcomes As described previously, the median Students in the validity sample were correlation of MCAT total scores with In the national population, there were similar to the national population of summative performance in M1 was 0.57, either no differences or only minor ones medical students with scores from and the 25th and 75th percentiles were between the observed and predicted
390 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 12 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
progression rates were the same as the Table 1 predicted rates for medical students Demographic Characteristics of the Validity Sample and the National Population whose parents did not have a bachelor’s of 2016 Medical School Matriculants With Scores From the New MCAT Exam Who degree (96%). Similarly, female students Were Included in an Analysis of the Validity of Scores From the New MCAT Exam in had the same observed and predicted Predicting Student Performance progression rates (96%). National population of Characteristic Validity samplea 2016 matriculantsb Together, these results showed that Medical students, total no. 955 7,970 medical students from different Medical students by gender, no. (%)c sociodemographic backgrounds performed, on average, at the levels that Male 417 (44) 3,655 (46) their MCAT scores predicted they would Female 538 (56) 4,313 (54) across their M1 courses and in their on- Medical students who received fee 94 (10) 546 (7) time progression to M2. assistance from the AAMC, no. (%) Medical students by self-reported race/ethnicity, no. (%)d White 497 (53) 4,541 (57) Discussion Black or African American 178 (19) 939 (12) Our study is the first multisite evaluation Hispanic, Latino, or Spanish 80 (9) 967 (12) of the validity of scores from the new Asian 175 (19) 1,288 (16) MCAT exam in predicting student American Indian or Alaska Native, Native 13 (1) 116 (1) performance in medical school, focusing Hawaiian or other Pacific Islander on performance in M1. It is also the first Other 20 (2) 256 (3) study to assess whether scores from the new exam provide comparable prediction MCAT total score, mean (SD)e 507.87 (7.34) 508.57 (6.92) of performance in M1 for students from f g Total undergraduate GPA, mean (SD) 3.70 (0.25) 3.70 (0.25) different sociodemographic backgrounds. Abbreviations: MCAT indicates Medical College Admission Test; AAMC, Association of American Medical Summative performance across selected Colleges; SD, standard deviation; and GPA, grade point average. M1 courses and on-time progression to a The validity sample consisted of 2016 matriculants enrolled in one of the 16 medical schools participating in M2 were used as outcomes. the MCAT validity research who volunteered to participate in this study. The 2016 matriculants at these schools were provided detailed information about the research. Those who volunteered for the study signed consent forms to allow their school to share their course-based and year-end outcomes with the AAMC for research The correlations of MCAT total scores purposes. The total number of students who matriculated at the validity schools in 2016 was 2,541. About with summative performance in M1 half of them (1,199; 47%) applied with scores from the new MCAT exam. The participation rate in this study was at or above 75% at the majority of validity schools. In total, 955 matriculants with scores from the new ranged from medium to large across MCAT exam were included in this study. The 16 validity schools included in this study are a subset of the 18 the validity schools included in this medical schools partnering with the AAMC on the MCAT validity research and include Boston University School study. These findings are important. of Medicine, Columbia University Valegos College of Physicians and Surgeons, East Tennessee State University James H. Quillen College of Medicine, Meharry Medical College, Memorial University of Newfoundland Faculty Knowing that MCAT scores provide valid of Medicine, Morehouse School of Medicine, Philadelphia College of Osteopathic Medicine, Rutgers Robert information about applicants’ readiness Wood Johnson Medical School, Saint Louis University School of Medicine, The Ohio State University College of for medical school can give admissions Medicine, Tulane University School of Medicine, Uniformed Services University of the Health Sciences F. Edward committees the flexibility to select Hébert School of Medicine, University of Arizona College of Medicine–Tucson, University of Calgary Cumming School of Medicine, University of California, San Francisco, School of Medicine, The University of Illinois College applicants who are academically prepared of Medicine–Chicago, University of Mississippi School of Medicine, and University of Texas School of Medicine for medical school while taking into at San Antonio. These 18 schools were carefully selected from 65 medical schools that volunteered to partner account the number of students they have with the AAMC on the MCAT validity research. Validity schools represent schools with a wide range of missions, regions, and other school and student characteristics. the resources to support. bThe national population includes 2016 medical school matriculants who applied with scores from the new MCAT exam and were included in this study. Students enrolled in MD/PhD or other dual degree programs were The predictive validity findings from our excluded from the analysis due to the planned delay in graduation. cDuring the 2016 application cycle, there were only 2 options available for gender. study are consistent with the findings dStudents identified their racial/ethnic backgrounds when they registered for the MCAT exam or applied to from previous large-scale, multicohort medical school through the American Medical College Application Service (AMCAS). Students may select studies on the predictive validity of as many races/ethnicities as they wish. The percentage of individuals reporting race/ethnicity, as well as the scores from the old MCAT exam.5 They combinations reported, have been stable in recent years.30 Percentages were calculated based on the individuals who provided race/ethnicity information. Percentages add up to more than 100% because racial/ethnic minority are also consistent with findings about results include individuals who may have designated more than one race/ethnicity. the validity of scores from admission eThe most recent MCAT scores available were used in this study to capture all the knowledge and skills students exams for other graduate or professional gained before their last attempt at the MCAT exam. fPostbaccalaureate undergraduate course grades were included in the calculation of the total cumulative school programs, such as the Law School 18 undergraduate GPAs for medical schools that use AMCAS. Admission Test and the Graduate gUndergraduate GPAs ranged from 0 to 4 for all except one validity school in Canada, which calculated students’ Management Admission Test,25 which undergraduate GPAs on a 0–100 scale. The actual GPAs for students from this school ranged from 75.47 to show medium to large correlations with 95.00 and had an average of 83.83 (SD = 4.13). first-year performance.
outcomes of on-time progression to practically significant. For instance, Our study also showed that overall, M2 for students by race/ethnicity, 95% of students identified as URM were students with a wide range of MCAT highest parental education, and gender. predicted to progress on time compared total scores progressed to M2 on time. No differences were statistically or with 94% who did. The observed This outcome is important because it
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 391 13 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
Figure 2 Percentage of the 2016 cohort of medical school matriculants with scores from the new Medical College Admission Test (MCAT) who progressed to year 2 of medical school on time, by MCAT total score range. In total, 7,970 students in the national population of 2016 matriculants with scores from the new MCAT exam were included in the analysis. Students enrolled in MD/PhD or other dual degree programs were not included due to the planned delay in graduation. Less than 2% (N = 131) of the national population reported MCAT scores below 494; in this group, the on-time progression rate was 79%. The numbers of students in the score ranges below 494 are too small to interpret meaningful differences in their progression rates compared with those with scores at or above 494. On-time progression rates are based on observed progression for those students who were admitted to medical school and do not reflect the potential performance of those who were not accepted. Additionally, multiple factors may contribute to the lack of on-time progression, including a possible combination of academic and nonacademic reasons.
represents the first milestone toward Our results also showed that MCAT total Our study has several limitations. It is graduating within the typical time frame scores provided comparable prediction based on data from the first cohort of based on each school’s curriculum. of performance in M1 for students from medical students admitted with scores The finding that students with a wide different sociodemographic backgrounds. from the new MCAT exam. Nationally, range of MCAT total scores progressed The differences between observed and these students represent about 40% of to M2 on time suggests that medical predicted performance on both outcomes the entire 2016 cohort of medical school school admissions committees are we studied were of neither statistical matriculants. By comparison, almost 90% admitting applicants with the academic significance nor practical importance. of 2017 matriculants applied with scores qualifications and demonstrated Evidence of comparability in prediction from the new exam. Consequently, the excellence in domains important for addresses professional testing standards number of matriculants in each specific success in medical school and supporting related to the fairness of test scores used URM group (e.g., black or African those students when they enter.26 in admissions decision making.4 American) in the 2016 cohort was smaller than it would be during a typical Also important is learning that using Results from our study suggest that application year, when all applicants take MCAT scores and UGPAs together MCAT scores provide useful information the same version of the MCAT exam. provided better prediction of M1 about student performance in the first Therefore, in this first validity study of performance than using either alone. year of medical school. The results from scores from the new exam, we did not The median correlation of each academic the validity schools are generalizable examine whether MCAT total scores metric with students’ summative M1 to 2016 matriculants at other medical provided comparable prediction for each performance was greater than 0.50. schools in North America because the specific URM group. Future research will When used together, the median students in the validity sample were continue addressing the comparability correlation increased to 0.65. This representative demographically of the of MCAT scores as more data become finding supports the common practice national population of 2016 medical available. of using both metrics for admissions school matriculants with scores from the decisions26 and is consistent with new MCAT exam and their schools were In addition, our study examined MCAT professional testing standards and carefully selected for the validity research scores in relation to student performance guidance on using MCAT scores in to represent a diverse pool of medical in the first year of medical school. This student selection.4,13 schools in North America. outcome is the closest chronologically to
392 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 14 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
Table 2 Comparison of Observed and Predicted Performance for 2016 Medical School Matriculants From Different Sociodemographic Backgrounds Who Took the New MCAT Exam
Observed summative Predicted summative performance in M1 performance in M1 Sociodemographic Student No. Effect Outcome variable groups students Mean SD Mean SD Difference sizee Summative Racial/ethnic identitya,b Non-URM 596 84.54 5.85 84.37 3.77 0.17 0.03 performance URM 261 82.83 6.88 83.21 3.21 −0.38 −0.07 in M1 (validity sample) Highest parental At least one 609 84.99 6.07 84.96 3.64 0.03 0 educationc,d parent with a bachelor’s degree No parent with a 149 83.87 7.00 83.96 3.91 −0.09 −0.02 bachelor’s degree Gender Male 417 84.37 6.63 84.54 3.92 −0.17 −0.04 Female 538 84.17 5.84 84.04 3.56 0.13 0.03 Observed on-time Predicted on-time Sociodemographic Student No. progression to M2 progression to M2 Effect Outcome variable groups students No. % No. % Difference sizeh On-time Racial/ethnic identitya Non-URM 5,774 5,636 98 5,615 97 1 0.06 progression to URM 1,952 1,830 94 1,850 95 −1 −0.04 M2 (national population)f Highest parental At least one 6,325 6,134 97 6,125 97 0 0 educationc,g parent with a bachelor’s degree No parent with a 1,411 1,351 96 1,352 96 0 0 bachelor’s degree Gender Male 3,655 3,543 97 3,541 97 0 0 Female 4,313 4,157 96 4,159 96 0 0
Abbreviations: MCAT indicates Medical College Admission Test; M1, first year of medical school; SD, standard deviation; URM, underrepresented in medicine; M2, second year of medical school; AMCAS, American Medical College Application Service. aStudents were categorized into URM and non-URM groups based on their self-reported race/ethnicity.22 Students who self-identified as black or African American; Hispanic, Latino, or Spanish; American Indian or Alaska Native; or Native Hawaiian or other Pacific Islander were included in the URM group. Students who self-identified as white or Asian were included in the non-URM group. bThe analysis of summative performance in M1 by students’ self-reported racial/ethnic identity was conducted based on data from 14 medical schools. It excluded students from 2 validity schools because these schools had no URM students who volunteered to participate in the study. cHighest parental education was used as a proxy measure of students’ socioeconomic backgrounds because of the close relationship between education level and income.23 Students were categorized into 2 groups based on parental attainment of a bachelor’s degree. One group included students who self-reported that their parents did not have a bachelor’s degree. The other included students who reported that at least one parent had a bachelor’s degree. dAmong the validity sample, 197 students were excluded from the analysis because their parents’ highest education was unavailable (e.g., students did not provide parental education information in their application, students’ parents were deceased, students did not apply to medical schools through AMCAS). xx− eThe effect size for the prediction error of summative performance mean scores was calculated as g = 12, � ∗ 2 2 s * ()ns11−11+−()ns22 where s =� . �x1 represents the mean of observed summative performance in M1 for ()nn12+−2
students in a group (e.g., URM). x2 represents the mean of predicted summative performance in M1 for the same
group. n1 and n2 are the same, representing the number of students in the group (e.g., URM). s1 represents the
SD of observed summative performance in M1 for the group. s2 represents the standard deviation of predicted summative performance in M1 for the same group.31 fFor the analysis of on-time progression to M2, the authors also tested the robustness of the findings through an alternative analysis approach. They randomly split the national population of 2016 matriculants with scores from the new MCAT exam into 2 groups. They used one group to conduct the regression analysis and used regression parameters from the regression group to generate predicted on-time progression to M2 for the other group. Results support the conclusion that MCAT total scores provide comparable prediction of on-time progression to M2 for students from different sociodemographic backgrounds. gAmong the national population, 234 students were excluded from the analysis because their parents’ highest education was unavailable (e.g., students did not provide parental education information in their application, students’ parents were deceased, students did not apply to medical schools through AMCAS). hThe effect size for the prediction error of on-time progression was calculated as
ha= 2� ( rcsine()Pa12− rcsine()P ), where P1 represents the observed percentage of students who progressed
to M2 on time for students in a group (e.g., URM) and P2 represents the predicted percentage of students who progressed to M2 on time for students in the same group.24
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 393 15 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
Figure 3 Medians and interquartile ranges of corrected correlations of new Medical College Admission Test (MCAT) total scores and undergraduate grade point averages (UGPAs), alone (Panels A and B) and together (Panel C), with students’ summative performance in the first year of medical school (M1). Summative performance is the mean of students’ scores from the M1 courses that each validity school selected to include in this study. Summative performance showed a strong correlation (above 0.90) with students’ official end-of-year performance, such as their M1 GPA or M1 class rank. The observed correlations were corrected for range restriction in MCAT total scores and UGPAs due to student selection in the admissions process.16 The correlation correction adjusted for the differences in the standard deviations of MCAT scores and UGPAs in each school’s validity sample compared with the school’s applicant pool, while accounting for the correlations among MCAT scores, UGPAs, and the outcome in the validity sample. Because more 2017 applicants than 2016 applicants to the validity schools had scores from the new exam, the corrections for range restriction were made using data from the 2017 admissions cycle. The corrected correlation coefficients (validity coefficients) are presented. The numbers next to the solid circles show the median correlations. The numbers at the ends of the interquartile ranges show the correlations at the 25th (lower end) and 75th (higher end) percentiles of the distribution of corrected correlation coefficients, respectively. The horizontal line with a Y axis value at 0.3 is the reference line for a medium association.17 See Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A731 for the observed correlations.
MCAT scores. It comes from courses that Acknowledgments: The authors would like of American Medical Colleges’ (AAMC’s) probably have more concepts in common to thank the members of the Association of Medical College Admission Test (MCAT) with those tested on the MCAT exam American Medical Colleges (AAMC) Medical Validity Research Study protocol and (2) the College Admission Test (MCAT) Validity participating medical schools’ MCAT Validity than courses in the later years of medical Committee for their dedication and tireless Study protocol. Course-based outcome data school, as the MCAT exam was designed efforts to evaluate the new MCAT exam: Ngozi for the validity sample used in this report came to measure the foundational natural, Anachebe, Barbara Beckman, Ruth Bingham, from 16 medical schools that partner with the behavioral, and social science concepts Kevin Busche, Deborah Castellano, Francie AAMC on MCAT validity research. All validity upon which the preclerkship years of the Cuffney, Julie Chanatry, Hallen Chung, Daniel schools granted permission to the AAMC to use the data in MCAT validity research and medical school curriculum are built. Clinchot, Liesel Copeland, Martha Elks, William Gilliland, Jorge Girotti, Kristen Goodell, Joshua publications. Representatives from the validity Hanson, Loretta Jackson-Williams, David Jones, schools were also provided the opportunity to Much remains to be learned about how Catherine Lucey, R. Stephen Manuel, Janet review the manuscript before its submission students fare during the rest of their McHugh, Stephanie McClure, Cindy Morris, for publication. preclerkship and clerkship coursework, Wanda Parsons, Tanisha Price-Johnson, Boyd Previous presentations: Some of the data on the USMLE Step exams, and on Richards, Aaron Saguil, Aubrie Swan Sein, Stuart Slavin, Doug Taylor, Carol Terregino, Ian Walker, presented in this report were presented at the other more distant and comprehensive Robert Witzburg, David Wofsy, and Mike 2017 Association of American Medical Colleges outcomes, as well as for entering classes Woodson. The authors would also like to thank Learn Serve Lead annual meeting in November for which all students were admitted the following AAMC personnel for reviewing 2017, in Boston, Massachusetts, and the 2018 with scores from the same version of earlier drafts of this manuscript: Heather Continuum Connections: A Joint Meeting of the MCAT exam. There is also more to Alarcon, Gabrielle Campbell, Karen Fisher, the Group on Student Affairs (GSA), Group on Resident Affairs (GRA), Organization of Student learn about the value of MCAT scores Marc Kroopnick, Karen Mitchell, Norma Poll, Elisa Siegel, and Geoffrey Young. In addition, Representatives (OSR), and Organization of in predicting performance for students they would like to thank Cynthia Searcy, Andrea Resident Representatives (ORR) in April 2018, in at medical schools that vary in their Carpentieri, Melissa Lee, and Rob Santos for Orlando, Florida. missions, admission practices, curricular their contributions to this article. structures (e.g., discipline vs systems K. Busche is assistant dean for undergraduate Funding/Support: None reported. medical education and assistant professor of clinical based), instructional approaches, and the neurosciences, University of Calgary, Cumming types and amounts of support provided School of Medicine, Calgary, Alberta, Canada. Other disclosures: The Medical College Admission to students.27–29 In addition, there is great Test (MCAT) is a program of the Association of M.L. Elks is senior associate dean for educational interest in learning how MCAT section American Medical Colleges (AAMC). Related affairs and professor of medical education and scores predict performance in different trademarks owned by the AAMC include Medical medicine, Morehouse School of Medicine, Atlanta, subject areas, as well as how the old College Admission Test and MCAT. Georgia. and new MCAT exams compare when J.T. Hanson is associate dean for student affairs Ethical approval: This study was approved predicting student performance in medical and associate professor of medicine, University by the institutional review board of the of Texas Health San Antonio, Joe R. and Teresa school. Future studies of the MVC will American Institutes for Research as part of Lozano Long School of Medicine, San Antonio, delve into these and other questions. 2 different protocols: (1) the Association Texas.
394 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 16 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Research Report
L. Jackson-Williams is vice dean for medical across diverse applicant groups. Acad Med. 20 Jolly P. Diversity of U.S. medical students education and professor of emergency medicine, 1998;73:1095–1106. by parental income. AAMC Analysis in School of Medicine, University of Mississippi Medical 9 Huff KL, Fang D. When are students most Brief. January 2008;8. https://www.aamc. Center, Jackson, Mississippi. at risk of encountering academic difficulty? org/download/102338/data/aibvol8no1.pdf. A study of the 1992 matriculants to U.S. Accessed May 23, 2019. R.S. ManuelManual was associate dean for medical medical schools. Acad Med. 1999;74:454–460. 21 Magrane D, Jolly P. The changing school admissions and associate professor of 10 Violato C, Donnon T. Does the Medical representation of men and women in family medicine, University of Mississippi School of College Admission Test predict clinical Medicine, Jackson, Mississippi, at the time this work academic medicine. AAMC Analysis in was completed. reasoning skills? A longitudinal study Brief. July 2005;5. https://www.aamc.org/ employing the Medical Council of Canada download/75776/data/aibvol5no2.pdf. W.L. Parsons is associate professor of family clinical reasoning examination. Acad Med. Accessed May 23, 2019. medicine, and past assistant dean for admissions, 2005;80(10 suppl):S14–S16. 22 Association of American Medical Colleges. Faculty of Medicine, Memorial University of 11 Gauer JL, Wolff JM, Jackson JB. Do MCAT The status of the new AAMC definition of Newfoundland, St. John’s, Newfoundland, Canada. scores predict USMLE scores? An analysis on “underrepresented in medicine” following D. Wofsy is associate dean for admissions and 5 years of medical student data. Med Educ the Supreme Court’s decision in Grutter. professor of medicine, University of California, Online. 2016;21:31795. Adopted March 19, 2004. https://www.aamc. San Francisco, School of Medicine, San Francisco, 12 Davis D, Dorsey JK, Franks RD, Sackett PR, org/download/54278/data/urm.pdf. Accessed California. Searcy CA, Zhao X. Do racial and ethnic May 23, 2019. group differences in performance on the 23 Wolla SA, Sullivan J. Education, income, and K. Yuan is director, Medical College Admission Test MCAT exam reflect test bias? Acad Med. wealth. Page One Economics. January 2017. Research, Association of American Medical Colleges, 2013;88:593–602. https://research.stlouisfed.org/publications/ Washington, DC. 13 Association of American Medical Colleges. page1-econ/2017/01/03/education-income- Using MCAT Data in 2019 Medical Student and-wealth. Accessed May 23, 2019. Selection. Washington, DC: Association of 24 Cohen J. Statistical Power Analysis for the References American Medical Colleges; 2018. https:// Behavioral Sciences. 2nd ed. Hillsdale, NJ: 1 Lei P-W, Bassiri D, Schultz EM; ACT. www.aamc.org/download/462316/data/ Lawrence Erlbaum Associates;1988. Alternatives to grade point average as a mcatguide.pdf. Accessed May 23, 2019. 25 Kuncel NR, Credé M, Thomas LL. A measure of academic achievement in college. 14 Association of American Medical Colleges. meta-analysis of the predictive validity ACT Research Report Series 2001–4. http:// 2018 AMCAS Applicant Guide. Washington, of the Graduate Management Admission www.act.org/content/dam/act/unsecured/ DC: Association of American Medical Test (GMAT) and undergraduate grade documents/ACT_RR2001-4.pdf. Published Colleges; 2017. https://aamc-orange. point average (UGPA) for graduate student December 2001. Accessed May 23, 2019. global.ssl.fastly.net/production/media/ academic performance. Acad Manage 2 Schwartzstein RM, Rosenfeld GC, Hilborn filer_public/33/f0/33f0bd3f-9721-43cb- Learning Educ. 2007;6:51–68. R, Oyewole SH, Mitchell K. Redesigning 82a2-99332bbda78e/2018_amcas_applicant_ 26 Association of American Medical Colleges. the MCAT exam: Balancing multiple guide_web-tags.pdf. Accessed May 23, 2019. Using MCAT Data in 2018 Medical Student perspectives. Acad Med. 2013;88:560–567. 15 Texas Medical and Dental Schools Selection. Washington, DC: Association of 3 Kroopnick M. AM Last Page: The MCAT Application Service. Entry Year 2019 American Medical Colleges; 2017. exam: Comparing the 1991 and 2015 exams. Application Handbook. Austin, TX: Texas 27 Saks NS, Karl S. Academic support services Acad Med. 2013;88:737. Medical and Dental Schools Application in U.S. and Canadian medical schools. Med 4 American Educational Research Association; Service; 2019. https://www.tmdsas.com/ Educ Online. 2004;9:4348. American Psychological Association; National Forms/ApplicationHandbookEY2019.pdf. 28 Shields PH. A survey and analysis of Council on Measurement in Education. Accessed May 23, 2019. student academic support programs in Standards for Educational and Psychological 16 Sackett PR, Yang H. Correction for range medical schools focus: Underrepresented Testing. Washington, DC: American restriction: An expanded typology. J Appl minority students. J Natl Med Assoc. Educational Research Association; 2014. Psychol. 2000;85:112–118. 1994:86:373–377. 5 Julian ER. Validity of the Medical College 17 Cohen J. A power primer. Psychol Bull. 29 Elks ML, Herbert-Carter J, Smith M, Klement Admission Test for predicting medical school 1992;112:155–159. B, Knight BB, Anachebe NF. Shifting the performance. Acad Med. 2005;80:910–917. 18 Anthony LC, Dalessandro SP, Trierweiler TJ; curve: Fostering academic success in a diverse 6 Donnon T, Paolucci EO, Violato C. The Law School Admission Council. Predictive student body. Acad Med. 2018;93:66–70. predictive validity of the MCAT for medical validity of the LSAT: A national summary of school performance and medical board the 2013 and 2014 LSAT correlation Studies. References cited in tables only licensing examinations: A meta-analysis https://www.lsac.org/data-research/research/ of the published research. Acad Med. predictive-validity-lsat-national-summary- 30 Association of American Medical Colleges. 2007;82:100–106. 2013-and-2014-lsat-correlation. Published Table B-3: Total U.S. medical school 7 Dunleavy DM, Kroopnick MH, Dowd KW, March 2016. Accessed August 9, 2018. enrollment by race/ethnicity and sex, Searcy CA, Zhao X. The predictive validity 19 Coleman AL, Lipper KE, Taylor TE, Palmer 2014–2015 through 2018–2019. Published of the MCAT exam in relation to academic SR, Education Counsel LLC. Roadmap to November 14, 2018. https://www.aamc.org/ performance through medical school: Diversity and Educational Excellence: Key download/321534/data/factstableb3.pdf. A national cohort study of 2001-2004 Legal and Educational Policy Foundations for Accessed May 23, 2019. matriculants. Acad Med. 2013;88:666–671. Medical Schools. 2nd ed. Washington, DC: 31 Hedges LV. Distribution theory for Glass’s 8 Koenig JA, Sireci SG, Wiley A. Evaluating Association of American Medical Colleges; estimator of effect size and related estimators. the predictive validity of MCAT scores 2014. J Educ Behav Stat. 1981:6:107–128.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 395 17 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
Investigating Group Differences in Examinees’ Preparation for and Performance on the New MCAT Exam Jorge A. Girotti, PhD, MHA, Julie A. Chanatry, PhD, Daniel M. Clinchot, MD, Stephanie C. McClure, MD, Aubrie Swan Sein, PhD, EdM, Ian W. Walker, MD, and Cynthia A. Searcy, PhD
Abstract In 2015, the Medical College Admission ethnicities underrepresented in medicine higher and lower SES backgrounds and Test (MCAT) was redesigned to better compared with those from higher SES who are enrolled in undergraduate assess the concepts and reasoning backgrounds and races/ethnicities not institutions with more and fewer skills students need to be ready for the underrepresented in medicine. resources, showing that examinees from medical school curriculum. During the lower SES backgrounds and who attend new exam’s design and rollout, careful In this article, the authors describe the institutions with fewer resources use attention was paid to the opportunities characteristics and scores of examinees many free and low-cost test preparation examinees had to learn the new content who took the new MCAT exam in 2017 resources at lower rates than their peers. and their access to free and low-cost and compare those trends with historical The authors conclude with a description preparation resources. The design ones from 2013, presenting evidence of the next phase of this research: to committee aimed to mitigate possible that the diversity and performance of gather qualitative and quantitative unintended effects of the redesign, examinees has remained stable even data about the preparation strategies, specifically increasing historical mean with the exam’s redesign. They also barriers, and needs of all examinees, group differences in MCAT scores for describe the use of free and low-cost but especially those from lower SES examinees from lower socioeconomic MCAT preparation resources and MCAT and underrepresented racial/ethnic status (SES) backgrounds and races/ preparation courses for examinees from backgrounds.
Editor’s Note: An Invited Commentary by R.M. to complex problems in clinical care and backgrounds and races/ethnicities not Schwartzstein appears on pages 333–335. research for an increasingly diverse and underrepresented in medicine.10,11 aging patient population, and medical education is evolving to address these Research during the MCAT redesign Medical schools aim to admit diverse changes. But these changes also mean process explored potential factors classes of students with the right mix of that medical students need a broader contributing to these historical group academic preparation, extracurricular academic foundation to be prepared for differences in MCAT scores. Davis and 10 experiences, and personal attributes today’s medical school curriculum. To colleagues investigated the potential role needed for medical school. Classes with better assess the readiness of potential of test bias, examining evidence that the these characteristics enable medical medical students, a new version of exam meets professional testing standards schools to meet their missions and goals. the MCAT exam was introduced in for fairness in measurement quality and For the past 90 years, the Medical College April 2015. The new exam requires equitable treatment for examinees from 12 Admission Test (MCAT) has been used multidisciplinary problem-solving and different backgrounds. They studied the steps taken to develop and administer the as one measure of academic preparation, tests more subjects and greater scientific old exam (steps that are still taken for the assessing the scientific concepts and reasoning than the old exam did.1 reasoning skills of entering medical new exam) to minimize the influence of irrelevant factors on scores and maximize students. MCAT scores provide a common measure the opportunities for examinees to of academic preparation, in contrast demonstrate their preparation. Davis and Advances in biomedical science and with undergraduate grade point averages health care delivery require that colleagues presented evidence that scores (UGPAs), which can vary by institution, from the old exam did not show predictive physicians be capable of applying their major, and coursework.2,3 Historically, scientific knowledge and reasoning skills bias against black or Latino medical MCAT scores have predicted important students because the success rates of student outcomes from entry through Please see the end of this article for information students from these races/ethnicities were 4–9 about the authors. graduation from medical school. not higher than their scores predicted. Although MCAT scores provide useful Correspondence should be addressed to Cynthia A. Searcy, Association of American Medical Colleges, information about students’ academic Davis and colleagues10 also reviewed 655 K St. NW Suite 100, Washington, DC 20001; readiness for medical school, the extant research on the early connections telephone: (202) 862-6105; email: [email protected]. average scores of applicants from lower between academic achievement and Acad Med. 2020;95:365–374. socioeconomic status (SES) backgrounds educational opportunities, environments, First published online August 13, 2019 and races/ethnicities underrepresented and experiences. For example, from an doi: 10.1097/ACM.0000000000002940 Copyright © 2019 by the Association of American in medicine (URM) are lower than early age, factors like nutrition, day care Medical Colleges those of applicants from higher SES quality, shared book reading with a family
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 365 XXX 18 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
member, and teacher quality in grade their SES and undergraduate institutions To assess whether any of these scenarios school and beyond are associated with also became central to subsequent MCAT occurred, we compared the characteristics academic achievement, enabling some validity research. It was important to and scores of examinees who took the students to maximize their potential, know if examinees from lower SES new MCAT exam in 2017 with results while limiting the potential of others. backgrounds faced obstacles in their from those who took the old exam in The authors saw that these factors varied awareness of or ability to afford test 2013 to look for decreases in diversity systematically by medical students’ racial/ preparation resources. Another potential or increases in average group test score ethnic backgrounds and were correlated roadblock was the undergraduate differences. We analyzed data from 2017 with achievement gaps between institution itself. Some institutions examinees (who took the new exam) students from majority and minority have fewer resources and could face because information about the concepts backgrounds. challenges providing their students with and skills tested on the new exam had the needed coursework and prehealth been publicized for multiple years at that Even for medical school applicants, these advising resources, potentially hindering point, and many preparation resources factors seem salient, and in their research, students’ access to coursework and other were available. We analyzed data from Davis and colleagues10 found comparative preparation materials needed for the new 2013 examinees (who took the old exam) trends in applicants’ backgrounds. For MCAT exam. because they were not likely influenced by example, applicants from URM racial/ upcoming changes to the exam (in 2015), ethnic backgrounds were more likely to In this article, we provide the first and anyone who wished to retake the test encounter factors shown to negatively national review of the characteristics and could have done so in 2014, while the old affect academic achievement and were scores of examinees who took the new exam was still available. less likely to be exposed to positive MCAT exam. We compare those trends factors. This research on the old MCAT with historical ones, presenting evidence We examined trends by gender, race/ exam suggested that group differences about their stability in the face of the ethnicity, and SES as measured by in MCAT scores likely stemmed from exam’s redesign. We also look closely participation in the Association of differences in examinees’ environments, at the use of test preparation resources American Medical Colleges (AAMC) Fee experiences, and opportunities rather by examinees from different SES Assistance Program (the only measure than from technical deficiencies in the backgrounds and the resources of their of SES available for both 2013 and 2017 test itself. undergraduate institutions. Exploration examinees). We also studied trends based of these areas may reveal key points to on the characteristics of examinees’ The designers of the new MCAT exam guide future resource development and undergraduate institutions, using data considered this research on educational outreach strategies. We conclude with a about selectivity in college admissions13 opportunities, socioeconomic description of the research underway to and the residential makeup of the backgrounds, and academic achievement, further understand the preparation needs campus (the percentage of undergraduate taking care not to introduce changes to of all examinees, but especially those students who lived on campus and/or the exam that would exacerbate existing from economically and educationally were enrolled full-time)14 as proxies for group differences in MCAT scores. A disadvantaged backgrounds. institutional resources. (Examinees whose central consideration in test blueprinting institutions were missing information was examinees’ opportunities to learn the about selectivity or residential setting concepts and skills tested on the exam, Characteristics and Scores of were excluded from this analysis.) This and blueprint decisions were informed Examinees Taking the New MCAT school resource variable included 2 levels: by research on course availability and Exam “more resources” and “fewer resources.” completion rates at undergraduate Students taking the new MCAT exam, Institutions with selective admissions institutions, including minority-serving especially those from lower SES or URM practices and/or residential campuses institutions.1 Easy access to free or racial/ethnic backgrounds, had the were classified as having more resources. low-cost test preparation materials was potential to face new barriers, despite Institutions that employed the least a cornerstone of subsequent outreach efforts to ensure equitable opportunities selective admissions practices (they campaigns to examinees and the advisors to learn the new content and equitable accepted students with a wide range of at their undergraduate institutions, access to resources about the new exam. scores from college admissions tests) and with increased emphasis on venues and For example, real and perceived barriers had primarily nonresidential campuses institutions serving examinees from lower in access to the coursework, textbooks, (fewer than 25% of undergraduate SES and URM racial/ethnic backgrounds. or other resources about the new content students lived on campus and/or were All these efforts were aimed at mitigating could have discouraged some groups of enrolled full time) were classified as the potential for different levels of access students from taking the exam, negatively having fewer resources. to accurate and timely information about affecting the diversity composition of the new MCAT exam while expanding the examinee population as a whole. Compared with more selective the exam’s focus to better align it with Furthermore, real differences in access to institutions, less selective institutions current medical school requirements. learning resources could have increased tend to spend less money per student historical differences in average MCAT and have lower full-time to part-time Since research on the MCAT redesign scores for examinees from lower SES faculty ratios and graduation rates.15 addressed examinees’ opportunities or URM racial/ethnic backgrounds Primarily nonresidential schools are less to learn and their ease of access to compared with those from higher SES or likely to have faculty living near students preparation resources, information about non-URM racial/ethnic backgrounds. or centrally located resources compared
366 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 19 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
with highly residential schools.16,17 participation in the AAMC Fee Assistance reveal better ways to increase access to The least selective institutions that Program, and enrollment in institutions affordable resources that will support predominately serve commuting students with more and fewer resources. It then all examinees, but especially those from may lack robust prehealth advising compares these differences with those lower SES backgrounds. Our analysis resources to guide premedical students from 2013. These findings show that focused on the use of test preparation about the courses needed or the full group differences in performance on the resources based on examinees’ SES and range of resources available to help them new exam are no larger than they were on the resources of their undergraduate prepare for the new exam. Students at the old exam. institutions. Differences in use based these institutions also may have other on these factors may signal the need responsibilities, including childcare and The differences in average MCAT total for additional resources and refined work, that interfere with the time they scores from the new exam were small for outreach targeted to the unique needs of have to prepare for the exam. males versus females (d = 0.3), for those examinees of more limited means. who did not versus did participate in the Characteristics of MCAT examinees AAMC Fee Assistance Program (d = 0.4), We examined survey data (see the Table 1 compares the characteristics of and for those who identified as white Table 3 footnotes for a description of examinees who took the MCAT exam in versus Native Hawaiian/other Pacific the surveys) about 2017 examinees’ use 2017 with the characteristics of those who Islander (d = 0.4). The differences were of test preparation resources developed did so in 2013. Overall, these data show medium for examinees who identified by the AAMC and their completion that the diversity of examinees did not as white versus Hispanic (d = 0.7) and of university-based or commercial change. In both years, the percentages of for those who identified as white versus MCAT preparation courses, in relation examinees identifying as black or African American Indian/Alaska Native (d = 0.6). to their SES and the resources of their American, Hispanic, American Indian/ The differences were large for examinees undergraduate institutions. Examinees’ Alaska Native, or Native Hawaiian/other who identified as white versus black or SES was based on a combination of Pacific Islander were similar, although in African American (d = 0.9) as well as for their parents’ highest level of education 2017, there was a small decrease (2.4%) in examinees attending institutions with and occupation (related to, but defined the percentage of examinees identifying more versus fewer resources (d = 0.9). differently than, the Educational- as white or Caucasian. The percentages These differences are very similar to Occupational indicator provided by the of examinees by the other characteristics the differences in scores from the 2013 American Medical College Application were also similar in both years. examinees who took the old exam. Service20). Examinees who had one or more parent with a bachelor’s degree Although many differences were Notably, these mean differences in scores or who worked in a professional or statistically significant, the only do not tell the whole story about the managerial occupation were classified characteristic to show meaningful change performance of examinees from different as “higher SES,” and examinees whose was gender, with the percentage of backgrounds. Figure 1 compares the parents did not have a bachelor’s degree females increasing from 50.9% in 2013 distribution of MCAT total scores in and who worked in a service or clerical to 54.8% in 2017. Females have long 2017 by examinees’ gender, race/ethnicity, occupation were classified as “lower SES.” outranked males in college enrollment, participation in the AAMC Fee Assistance comprising 56% of undergraduate Program, and institutional resources. Parental education and occupation are students in 2016.18 Compared with the There is substantial overlap in the score related to household income,20–23 and 2013 examinees, the gender makeup distributions of examinees from different examinees who are the first in their of the 2017 examinees more closely backgrounds, as well as substantial families to complete college may not resembled the makeup of undergraduate variation within each group. It is clear benefit from their parents’ firsthand students. from Figure 1 that there were examinees experiences on how to navigate or with low, middle, and high MCAT total succeed in college or on graduate school Scores of MCAT examinees scores in each group. or professional admissions tests (e.g., the Mean MCAT total scores were compared MCAT, Law School Admission Test, or by these same examinee characteristics In summary, the average differences in Graduate Management Admission Test).24 to assess group differences in scores MCAT scores from the new exam were from the new and old exams. For a given no larger than the differences in scores We used the same measure of comparison (e.g., males versus females from the old exam. However, they also institutional resources as defined above who took the new exam), the difference were no smaller. The desire to understand (based on the selectivity and residential between the 2 mean scores was computed, and potentially disrupt some of the setting of examinees’ undergraduate and the magnitude of the difference was factors that influenced these persistent institutions), with examinees’ evaluated using the standardized mean differences was the genesis for the work undergraduate institutions being difference (d). By convention, a d = 0.2 described below. classified as having “more resources” or is considered a small difference, a d = 0.5 “fewer resources.” Again, institutions with is considered medium, and a d = 0.8 is fewer resources may be less equipped Finding Points of Leverage in Test considered large.19 to provide guidance about MCAT Preparation preparation tools than those with more Table 2 compares the MCAT total Understanding examinees’ use of test resources. Examinees attending less- scores of examinees who took the new preparation resources and the barriers resourced institutions are also more likely exam in 2017 by gender, race/ethnicity, they face in accessing them may to live at home, care for dependents, and
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 367 20 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
Table 1 Comparison of the Characteristics of Medical College Admission Test (MCAT) Examinees in 2017 (New Exam) Versus 2013 (Old Exam)a
Difference between % of % of % of examinees in 2017 versus 2013 No. of No. of examinees examinees examinees examinees Characteristicb in 2017 in 2013 % difference Cohen’s hc in 2017 in 2013 Gender Male 45.2 49.1 −3.9d −0.08 36,534 40,851 Female 54.8 50.9 3.9d — 44,233 42,347 Race/ethnicitye White 46.0 48.4 −2.4d −0.05 35,262 35,820 Black or African American 10.7 10.2 0.5d 0.02 8,210 7,540 Hispanic, Latino, or Spanish 11.3 11.8 −0.5d −0.02 8,688 8,736 Asian 28.7 27.8 0.9d 0.02 22,029 20,534 American Indian or Alaska Native 1.1 0.9 0.2d 0.02 845 653 Native Hawaiian or other Pacific 0.3 0.4 −0.1 −0.02 253 296 Islander AAMC Fee Assistance Programf Did not receive 92.3 93.0 −0.7d −0.03 63,790 65,891 Received 7.7 7.0 0.7d — 5,299 4,952 School resourcesg More resources 95.6 95.6 0.0 0.00 64,499 66,031 Fewer resources 4.4 4.4 0.0 — 2,941 3,059
Abbreviation: AAMC indicates Association of American Medical Colleges. aPercentages are based on the number of unique examinees who responded to each question. Examinees who took the MCAT exam more than once were counted once in this table, using data from their most recent test administration. The total number of 2017 examinees was 80,997, and the total number of 2013 examinees was 83,276. b In 2017, NGender = 80,767, NRace/Ethnicity = 76,737, NAAMCFeeAssistance = 69,089, NSchoolResources = 19,330. In 2013, NGender = 83,198, NRace/Ethnicity = 73,989, NAAMCFeeAssistance = 70,843,
NSchoolResources = 21,299. cCohen’s h is an effect size representing the standardized difference between 2 proportions. It allows for simple comparisons of the differences in proportions of examinees taking the exam in 2017 versus 2013. By convention, a d = 0.2 is considered a small difference, a d = 0.5 is considered medium, and a d = 0.8 is considered large.19 dThese values are significant at P < .006, the critical P value after using the Bonferroni correction for the 9 χ2 tests we conducted. eUnderrepresented in medicine race/ethnicity results include examinees who may have designated more than one race/ethnicity. Data for examinees who reported their race/ ethnicity as “other” are not shown. fEligibility for the AAMC’s Fee Assistance Program is limited to examinees who are U.S. citizens or permanent residents (or, for 2017 examinees, have Deferred Action for Childhood Arrivals status). Examinees who were neither U.S. citizens nor permanent residents were excluded from this analysis. A total of 11,981 examinees from 2017 and 12,470 from 2013 were excluded from this analysis because they were not U.S. citizens or permanent residents. gUndergraduate institutions were defined as having “more resources” or “fewer resources” using publicly available data about college selectivity13 and the residential makeup of the campus.14 Schools with more resources were those that were selective in their admissions practices (test score data for first-year students were in the middle two-fifths of selectivity among all baccalaureate institutions) or more selective in their admissions practices (test score data for first-year students were between the 80th and 100th percentile of selectivity among all baccalaureate institutions) or were primarily residential campuses (25%–49% of degree-seeking undergraduate students lived on campus and at least 50% attended full-time) or highly residential campuses (at least half of degree-seeking undergraduate students lived on campus and at least 80% attended full-time). Schools with fewer resources met 2 criteria: (1) had the least selective admissions practices (these institutions either did not report test score data or the test score data they did report indicated that they extend educational opportunity to a wide range of students with respect to academic preparation and achievement) and (2) had campuses that were primarily nonresidential (less than 25% of undergraduate students lived in institutionally owned, operated, or affiliated housing or fewer than 50% of degree-seeking undergraduate students attended full-time).
work to support their families.25–27 These low-cost resources at lower rates than The percentage of examinees who used the additional barriers could affect their their counterparts. For example, fewer Khan Academy MCAT collection was lower awareness of resources and their time examinees from lower SES backgrounds for examinees attending schools with fewer for test preparation, above and beyond than higher SES backgrounds used the resources than for examinees attending the challenges of affording preparation AAMC online practice or sample test schools with more resources (47.5% versus resources. (72.3% versus 79.6%) or the AAMC 53.5%) but not for examinees from lower online item banks (44.7% versus 52.3%). versus higher SES backgrounds (52.4% Table 3 compares the preparation Smaller but significant differences were versus 52.8%). The Khan Academy is a free, practices of examinees from higher seen in the use of the AAMC’s Official online resource with videos that teach and and lower SES backgrounds and those Guide to the MCAT Exam. The differences with multiple-choice questions that assess attending schools with more and fewer in the use of these preparation resources the concepts tested on the MCAT exam. resources. It shows that examinees for students at institutions with more The percentages of examinees who used from lower SES backgrounds and versus fewer resources were similar to the AAMC online interactive tool and the those who attended schools with fewer the differences for students from higher MCAT Flashcards were low and did not resources used many of the free and versus lower SES backgrounds. differ by SES or school resources.
368 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 21 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
Table 2 Comparison of Medical College Admission Test (MCAT) Total Scores for 2017 (New Exam) Versus 2013 (Old Exam), by Examinee Characteristicsa
MCAT total MCAT total Standardized mean difference (d)b No. of No. of score in 2017, score in 2013, examinees examinees
Characteristic mean (SD) mean (SD) Comparison dnew dold in 2017 in 2013 Gender Male 503.1 (10.5) 26.7 (6.4) — — — 36,534 40,851 Female 500.2 (10.6) 24.5 (6.6) Male-Female 0.3 0.3 44,233 42,347 Race/ethnicityc White 503.3 (9.8) 26.7 (5.9) — — — 35,262 35,820 Black or African American 494.2 (9.8) 20.5 (6.3) White-black or African 0.9 1.0 8,210 7,540 American Hispanic, Latino, or Spanish 496.7 (10.4) 22.0 (6.9) White-Hispanic, Latino, or 0.7 0.8 8,688 8,736 Spanish Asian 503.1 (10.7) 26.3 (6.6) White-Asian <0.1 0.1 22,029 20,534 American Indian or Alaska 497.8 (9.9) 23.5 (6.1) White-American Indian or 0.6 0.6 845 653 Native Alaska Native Native Hawaiian or other 499.3 (10.1) 22.6 (6.8) White-Native Hawaiian or 0.4 0.7 253 296 Pacific Islander other Pacific Islander AAMC Fee Assistance Programd Did not receive 501.8 (10.6) 25.7 (6.6) — — — 63,790 65,891 Received 497.2 (10.2) 22.9 (6.5) Did not receive-Received 0.4 0.4 5,299 4,952 School resourcese More resources 502.1 (10.4) 25.9 (6.4) — — — 64,499 66,031 Fewer resources 492.3 (10.0) 19.9 (6.9) More resources-Fewer 0.9 0.9 2,941 3,059 resources
Abbreviations: AAMC indicates Association of American Medical Colleges; SD, standard deviation. aExaminees who took the MCAT exam more than once were counted once in this table, using data from their most recent test administration. The total number of 2017 examinees was 80,997, and the total number of 2013 examinees was 83,276. bThe standardized mean difference, or d statistic, allows for simple comparisons of the differences in scores on the new (2017) and old (2013) MCAT exams, which measure different concepts and are reported on different score scales, as well as comparisons to other standardized tests that differ more substantially from the MCAT exam. A d statistic describes the between-group difference relative to the pooled SD for the groups being compared. By convention, a d = 0.2 is considered a small difference, a d = 0.5 is considered medium, and a d = 0.8 is considered large.19 The formula used to generate the standardized mean differences is:
d = (meanNotUnderrepresented−meanUnderrepresented)/SDpooled, where
()NNMajority −1 varNotUnderrepresentedU+ ( nderrepressented −1)varUnderrepresented SDpooled = ()NNNotUnderrepresentedU−+1 ( nnderrepresented −1) cUnderrepresented in medicine race/ethnicity results include examinees who may have designated more than one race/ethnicity. Data for examinees who reported their race/ ethnicity as “other” are not shown. dEligibility for the AAMC’s Fee Assistance Program is limited to examinees who are U.S. citizens or permanent residents (or, for 2017 examinees, have Deferred Action for Childhood Arrivals status). Examinees who were neither U.S. citizens nor permanent residents were excluded from the analysis. A total of 11,981 examinees from 2017 and 12,470 from 2013 were excluded from this analysis because they were not U.S. citizens or permanent residents. eUndergraduate institutions were defined as having “more resources” or “fewer resources” using publicly available data about college selectivity13 and the residential makeup of the campus.14 Schools with more resources were those that were selective in their admissions practices (test score data for first-year students were in the middle two-fifths of selectivity among all baccalaureate institutions) or more selective in their admissions practices (test score data for first-year students were between the 80th and 100th percentile of selectivity among all baccalaureate institutions) or were primarily residential campuses (25%–49% of degree-seeking undergraduate students lived on campus and at least 50% attended full-time) or highly residential campuses (at least half of degree-seeking undergraduate students lived on campus and at least 80% attended full-time). Schools with fewer resources met 2 criteria: (1) had the least selective admissions practices (these institutions either did not report test score data or the test score data they did report indicated that they extend educational opportunity to a wide range of students with respect to academic preparation and achievement) and (2) had campuses that were primarily nonresidential (less than 25% of undergraduate students lived in institutionally owned, operated, or affiliated housing or fewer than 50% of degree-seeking undergraduate students attended full-time).
Table 3 also shows completion rates for some students attending schools with Likewise, the differences in average university-based or commercial MCAT fewer resources found ways to access MCAT scores on the new exam were preparation courses. Examinees from preparation courses. consistent with the differences in average lower SES backgrounds were less likely to scores on the old exam for examinees complete test preparation courses relative from different backgrounds and for to those from higher SES backgrounds Conclusions those attending schools with different (38.1% versus 50.0%). The difference Overall, the diversity of the population resource levels. was smaller for examinees attending who took the new MCAT exam in 2017 schools with fewer versus more resources was similar to that of the population Although the new exam’s broader scope (43.1% versus 48.7%), suggesting that who took the old exam in 2013. had the potential to alter the composition
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 369 22 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
Male (mean = 503.1; N = 36,534) Gender Female (mean = 500.2; N = 44,233)
White (mean = 503.3; N = 35,262)
Race/Ethnicity Black or African American (mean = 494.2; N = 8,210) Hispanic, Latino, or Spanish (mean = 496.7; N = 8,688)
Asian (mean = 503.1; N = 22,029)
American Indian or Alaska Native (mean = 497.8; N = 845)
Native Hawaiian or other Pacific Islander (mean = 499.3; N = 253)
AAMC Fee Assistance Did not receive (mean = 501.8; N = 63,790) Program Received (mean = 497.2; N = 5,299)
More resources (mean = 502.1; N = 64,499) School resources Fewer resources (mean = 492.3; N = 2,941)
472476 480484 488492 496500 504508 512516 520524 528 MCAT total score Figure 1 Distribution of Medical College Admission Test (MCAT) total scores in 2017, by examinee characteristics. Examinees who took the MCAT exam more than once were counted once, using data from their most recent test administration. Underrepresented in medicine race/ethnicity results include examinees who may have designated more than one race/ethnicity. Data for examinees who reported their race/ethnicity as “other” are not shown. Eligibility for the Association of American Medical Colleges (AAMC) Fee Assistance Program is limited to examinees who are U.S. citizens or permanent residents or have Deferred Action for Childhood Arrivals status. Undergraduate institutions were defined as having “more resources” or “fewer resources” using publicly available data about college selectivity13 and the residential makeup of the campus.14 Schools with more resources were those that were selective in their admissions practices (test score data for first-year students were in the middle two-fifths of selectivity among all baccalaureate institutions) or more selective in their admissions practices (test score data for first-year students were between the 80th and 100th percentile of selectivity among all baccalaureate institutions) or primarily residential campuses (25%–49% of degree-seeking undergraduate students lived on campus and at least 50% attended full-time) or highly residential campuses (at least half of degree-seeking undergraduate students lived on campus and at least 80% attended full time). Schools with fewer resources met 2 criteria: (1) had the least selective admissions practices (these institutions either did not report test score data or the test score data they did report indicated that they extend educational opportunity to a wide range of students with respect to academic preparation and achievement) and (2) had campuses that were primarily nonresidential (less than 25% of undergraduate students lived in institutionally owned, operated, or affiliated housing or fewer than 50% of degree-seeking students attended full-time).
of the examinee pool or increase group ethnic backgrounds were not due to schools with fewer resources reported differences in scores among examinees technical deficiencies in the old MCAT using many of the AAMC’s free and low- from different backgrounds, there is no exam but were likely linked to SES- cost resources at lower rates than their evidence that this occurred. Efforts to test related differences in environment more advantaged counterparts. concepts taught widely at undergraduate and experiences that influenced those institutions and to provide equitable students’ academic achievements.10 Early Our work had several limitations, access to preparation resources may have predictive validity analyses of the first- many of which can be addressed in mitigated the risk of increased group year performance of medical students future studies. Despite the apparent differences. admitted with scores from the new stability in examinee diversity and MCAT exam showed findings similar to MCAT score differences across the The group differences on the new MCAT those from Davis and colleagues. That transition from the old to the new exam that we did see paralleled the is, scores from the new exam did not MCAT exam, other factors could differences seen on other standardized show predictive bias because students have confounded the interpretation undergraduate and graduate/professional from different backgrounds admitted of our findings, such as changes in school admissions tests.10 These to medical school with the same MCAT aspiring applicants’ demographics, differences were not limited to scores on score achieved, on average, the same undergraduate enrollment, trends standardized tests. Davis and colleagues success in their first year of medical in undergraduate science education, showed that, even among medical school school.28 medical school prerequisites, or other applicants, the mean differences in unmeasured factors that may have UGPAs by applicants’ race/ethnicity were We also looked at preparation specifically changed from 2013 to 2017. Another consistent with those for MCAT scores. for the MCAT exam, focusing on limitation relates to our measurement examinees’ use of free and low-cost of SES using the parental income-based Davis and colleagues also showed MCAT preparation resources as well as measure of examinees’ participation in predictive validity findings suggesting their completion of MCAT preparation the AAMC Fee Assistance Program, as that group differences in scores courses. We found that examinees from we did for our analysis of 2013 and 2017 for students from different racial/ lower SES backgrounds or who attended examinees’ diversity characteristics and
370 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 23 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article e
0.11 0.12 0.26 0.22 0.11 0.07 − 0.05
Cohen’s h Cohen’s g g g g
5.3 2.6 4.6 6.0 − 1.8 11.0 10.9
Schools with more resources were were resources Schools with more 14 Difference between % of Difference % difference examinees by school resources
d
42.5 69.4 42.8 43.1 15.4 47.5 21.5
resources with fewer from schools from % of examinees
d
47.8 80.4 53.7 48.7 18.0 53.5 19.7 and the residential makeup of the campus. and the residential
resources with more with more 13 from schools from % of examinees c
0.08 0.17 0.15 0.24 0.01 0.01 − 0.07
19 Cohen’s h Cohen’s g g g g
0.2 0.4 tests we conducted. 4.2 7.3 7.6 19 2 − 2.1 11.9
examinees by SES Difference between % of Difference % difference b
17.7 41.7 52.4 21.2 72.3 44.7 38.1 SES % of
examinees from lower from b
17.9 45.9 52.8 19.1 79.6 52.3 50.0 SES % of
examinees from higher from
Cost Free Low cost Free Low cost Low cost Low cost High cost f h f a f f f h centages are based on the number of unique examinees who responded to each question. Examinees who took the MCAT exam more than once were counted once in this table, using data from their most recent test their most recent counted once in this table, using data from than once were exam more to each question. Examinees who took the MCAT based on the number of unique examinees who responded centages are AAMC online interactive tool Khan Academy MCAT collection Khan Academy MCAT AAMC Flashcards AAMC online item banks (question packs or section bank) MCAT University-based or commercial course prep Official Guide to the MCAT Exam Official Guide to the MCAT (paper or electronic) AAMC online practice or sample test = 0.8 is considered large. medium, and a d = 0.8 is considered a d = 0.5 is considered a small difference, By convention, a d = 0.2 is considered administration. The analyses of the use of free and low-cost preparation resources provided by the AAMC excluded data for examinees participating in the AAMC’s Fee Assistance Program because these students receive because these students receive Fee Assistance Program by the AAMC excluded data for examinees participating in AAMC’s provided resources and low-cost preparation administration. The analyses of the use free than other examinees and may cloud any may be greater of and access to these resources the AAMC about their availability; thus, awareness communications from frequent more and receive for free these resources whether they courses included data for all examinees who reported preparation MCAT The analyses of the use university-based or commercial backgrounds. different in use for examinees from of the differences interpretation Fee Assistance Program. course, including participants in the AAMC’s preparation took an MCAT highest parental occupation was a service, clerical, skilled, and highest parental degree education was no bachelor’s reporting the following: highest parental or managerial. The “lower SES” variable was defined as those examinees professional than the levels of SES indicator (Education-Occupation or EO indicator) in American Medical College Application Service (AMCAS) because or unskilled labor occupation. This variable was defined slightly differently the “lower SES” level in this study identifies subset of examinees whose highest occupation), whereas of parental (irrespective degree education is no bachelor’s EO1 level in AMCAS identifies students whose highest parental when less information about their parents examinees provide highest occupation level is a service, clerical, skilled, or unskilled labor occupation. Additionally, and whose parents’ degree education is no bachelor’s parental between the EO indicator in AMCAS and SES variable this analysis. The number of Post-MCAT minor differences exam than they do in their medical school applications, contributing to other more for the MCAT registering for each condition was as follows: for each condition was as follows: higher SES (N = 19,241) and lower 2,038). Number of End-of-Day Survey (see below) respondents (PMQ) (see below) respondents Questionnaire higher SES (N = 55,911) and lower 7,712). were only included for the PMQ administration associated with an examinee’s most recent MCAT score. About 34% of 2017 examinees completed the survey. Responses were to the survey question: What kinds of resources did to the survey question: What kinds of resources Responses were About 34% of 2017 examinees completed the survey. score. MCAT most recent only included for the PMQ administration associated with an examinee’s were exam? PMQ participants may select any combination of responses. for the MCAT you use to prepare those that were selective in their admissions practices (test score data for first-year students were in the middle two-fifths of selectivity among all baccalaureate institutions) or more selective in their admissions practices (test institutions) or more in the middle two-fifths of selectivity among all baccalaureate data for first-year students were selective in their admissions practices (test score those that were undergraduate students residential campuses (25%–49% of degree-seeking primarily institutions) or were of selectivity among all baccalaureate between the 80th and 100th percentile data for first-year students were score undergraduate students lived on campus and at least 80% attended full-time). Schools with fewer resources campuses (at least half of degree-seeking lived on campus and at least 50% attended full-time) or highly residential indicated that they extend educational opportunity to a wide range of data they did report data or the test score test score not report met 2 criteria: (1) had the least selective admissions practices (these institutions either did (less than 25% of undergraduate students lived in institutionally owned, operated, or affiliated housing primarily nonresidential and achievement) (2) had campuses that were to academic preparation students with respect (N = 934). The number (N = 22,234) and fewer resources resources for each condition was as follows: more undergraduate students attended full-time). The number of PMQ respondents or fewer than 50% of degree-seeking (N = 2,883). (N = 63,277) and fewer resources resources for each condition was as follows: more of End-of-Day Survey respondents = 0.8 is considered large. medium, and a d = 0.8 is considered a d = 0.5 is considered a small difference, a d = 0.2 is considered Undergraduate institutions were defined as having “more resources” or “fewer resources” using publicly available data about college selectivity resources” or “fewer resources” defined as having “more Undergraduate institutions were for the 7 χ correction significant at P < .007, the critical value after using Bonferroni These values were for the examination you took today? d to the question: How did you prepare and about 98% of 2017 examinees responde at the end of test day, This survey is administered the End-of-Day Survey. from Data were The “higher SES” variable was defined as those examinees having one or both of the following characteristics: (1) highest parental education was a bachelor’s degree or higher and/or (2) highest parental occupation was or higher and/or (2) highest parental degree education was a bachelor’s The “higher SES” variable was defined as those examinees having one or both of the following characteristics: (1) highest parental
Study the concepts and skills Practice taking multiple-choice items practice Learn, study,
Table 3 Table Examinees in 2017, by Socioeconomic Status (SES) and Undergraduate (MCAT) Used By Medical College Admission Test Resources Preparation Comparison of the Test Institution Resources resource of preparation Type Learn what is tested on the exam is an effect size representing the standardized difference between 2 proportions. It allows for simple comparisons of the differences in proportions of examinees at schools with more resources versus fewer resources. versus fewer resources. resources of examinees at schools with more in proportions It allows for simple comparisons of the differences between 2 proportions. difference the standardized size representing h is an effect Cohen’s Per Abbreviation: AAMC indicates Association of American Medical Colleges. Abbreviation: is an effect size representing the standardized difference between 2 proportions. It allows for simple comparisons of the differences in proportions of examinees with higher SES versus lower SES. By convention, in proportions It allows for simple comparisons of the differences between 2 proportions. difference the standardized size representing h is an effect Cohen’s Data were from the PMQ. This survey is administered after test day and examinees are allowed approximately 3 weeks to provide responses. Examinees may participate in the PMQ each time they take the MCAT exam. PMQ data Examinees may participate in the PMQ each time they take MCAT responses. 3 weeks to provide allowed approximately after test day and examinees are the PMQ. This survey is administered from Data were e f g h a b c d
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 371 24 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
MCAT scores. We think that measuring compared with those at institutions While our efforts are aimed at issues SES using parental education and with more resources. Additionally, at of access to and preparation for the occupation, as we did to compare the schools with formal advising offices, MCAT exam, far more could be done to preparation of 2017 examinees from trained prehealth advisors help expand access to medical school. There lower and higher SES backgrounds, examinees identify free and low-cost is ample evidence that socioeconomic improves upon the income-based preparatory materials and guide them in factors affect academic achievement and measure and should be replicated in how best to use those resources. Less is that those effects emerge early in life future studies. In addition, findings for known about the services and support and are compounded over time.35,36 The examinees attending institutions with received by examinees at schools with solutions to this problem are complex more versus fewer resources should be fewer resources and with part-time and seem elusive. Lucey and Saguil call replicated in future research because (or no) prehealth advisors. Additional for a broader recognition of and attention our use of institutional selectivity research is needed on the association of to the structural inequalities that may and residential setting as indicators of institutional characteristics like school impede access to a career in medicine.37 school resources is novel, and aggregate resources with MCAT exam preparation. Developing solutions will take partners findings for an institution can mask across the educational pipeline; in fact, dissimilarities in programs, enrollment How effectively examinees from different many U.S. medical schools are engaged patterns, and other factors potentially backgrounds employ proven learning in work that reaches students earlier in associated with the preparation patterns strategies in their test preparation their education.38 Although students we examined here.14 is also unexplored to date. Certain encounter the MCAT exam later in their learning strategies, such as completing academic careers, access to more tailored and reviewing practice questions information and resources to prepare Next Steps throughout the learning process, can for the exam has the potential to better In this article, we aimed to expand lead to more durable learning than other support a successful pathway to medical understanding of examinees’ techniques.30–33 The Last Page by Swan school. preparation for and performance on Sein and colleagues describes a model the MCAT exam by analyzing the test for how students can incorporate these Acknowledgments: The authors would like preparation of examinees from different learning strategies when preparing to thank their colleagues on the Research on backgrounds and who attend different for the MCAT exam.34 Also important Diversity, Group Differences, and Academic Preparation working group of the Association types of undergraduate institutions. is studying the association of these of American Medical Colleges (AAMC) Medical Thinking about the needs of students various preparation strategies with College Admission Test (MCAT) Validity whose parents did not have a bachelor’s test performance. Findings from such Committee (MVC) for their contributions to degree or who attended a school research may reveal which preparation this work: Liesel Copeland, Francie Cuffney, with fewer resources may provide strategies, alone or in combination, William Gilliland, Doug Taylor, and Robert opportunities to increase access to appear most beneficial for students from Witzburg. The authors would also like to thank the full MVC for their dedication and tireless information and resources about the lower versus higher SES backgrounds efforts to evaluate the new MCAT exam: Ngozi MCAT exam. or who attend schools with fewer versus Anachebe, Barbara Beckman, Ruth Bingham, more resources. Kevin Busche, Deborah Castellano, Francie Many factors may be relevant to Cuffney, Julie Chanatry, Hallen Chung, Daniel improving access to test preparation For the next phase of our work, Clinchot, Liesel Copeland, Martha Elks, William resources. Examinees from lower SES we plan to collect qualitative and Gilliland, Jorge Girotti, Kristen Goodell, Joshua backgrounds may have trouble affording quantitative data from examinees and Hanson, Loretta Jackson-Williams, David Jones, Catherine Lucey, R. Stephen Manuel, Janet even low-cost resources. Preparing for the prehealth advisors to better understand McHugh, Stephanie McClure, Cindy Morris, MCAT exam takes considerable time, and preparation strategies and barriers. Wanda Parsons, Tanisha Price-Johnson, Boyd examinees from lower SES backgrounds Understanding why, how, and when Richards, Aaron Saguil, Aubrie Swan Sein, may have other commitments (e.g., work examinees use different preparation Stuart Slavin, Doug Taylor, Carol Terregino, or family obligations) competing for their strategies may reveal common challenges Ian Walker, Robert Witzburg, David Wofsy, time compared with examinees from and allow us to suggest additional and Mike Woodson. The authors would like to higher SES backgrounds. Portability, ease resources and outreach that may improve thank Sandy Koch for her contributions to this article. In addition, they would like to thank of use, and access to a reliable internet their ability to prepare for the MCAT the following AAMC personnel for reviewing 29 connection and a mobile device may exam. Qualitative data from interviews earlier drafts of this article: Heather Alarcon, be especially important for examinees with examinees from lower SES Gabrielle Campbell, Karen Fisher, Karen of limited means or who are struggling backgrounds and who attend institutions Mitchell, Norma Poll, Elisa Siegel, and Geoffrey to balance these competing demands on with fewer resources will give us a rich Young. their time. sense of the full range of challenges Funding/Support: None reported. faced by examinees. Interviews with Examinees also may vary in their their advisors will provide a different Other disclosures: The Medical College Admission awareness of the available test vantage point of those needs. With these Test (MCAT) is a program of the Association of preparation resources and in their data, we can conduct national surveys of American Medical Colleges (AAMC). Related access to prehealth advisors. Examinees examinees to learn about the similarities trademarks owned by the AAMC include Medical and prehealth advisors at institutions and differences in preparation for College Admission Test and MCAT. with fewer resources may be less aware those from different backgrounds and Ethical approval: This study was approved by of the AAMC’s preparation materials educational experiences. the institutional review board of the American
372 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 25 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
Institutes for Research as part of the Association 5 Donnon T, Paolucci EO, Violato C. The 18 Snyder TD, de Brey C, Dillow SA. Digest of American Medical Colleges’ Medical College predictive validity of the MCAT for medical of Education Statistics 2017. 53rd ed. Admission Test (MCAT) Validity Research Study school performance and medical board Table 303.70: Total undergraduate fall protocol. licensing examinations: A meta-analysis enrollment in degree-granting postsecondary of the published research. Acad Med. institutions, by attendance status, sex of Previous presentations: Some of the data 2007;82:100–106. student, and control and level of institution: presented in this article were presented at the 6 Julian ER. Validity of the Medical College Selected years, 1970 through 2027. 2019. 2017 Association of American Medical Colleges Admission Test for predicting medical school https://nces.ed.gov/pubs2018/2018070.pdf. (AAMC) Learn Serve Lead annual meeting in performance. Acad Med. 2005;80:910–917. Accessed June 5, 2019. November 2017, Boston, Massachusetts; and 7 Koenig JA, Wiley A. Medical School 19 Cohen J. Statistical Power Analysis for the at the 2018 Continuum Connections: A Joint Admissions Testing. In: Dillon F, ed. Behavioral Sciences. 2nd ed. Hillsdale, NJ: Handbook on Testing. Westport, CT: Meeting of the Group on Student Affairs (GSA), Lawrence Erlbaum Associates; 1988. Greenwood Press; 1997. 20 Grbic D, Jones DJ, Case ST. The role of Group on Resident Affairs (GRA), Organization 8 Kroopnick MH, Dunleavy DM, Dowd KW, socioeconomic status in medical school of Student Representatives (OSR), and Searcy CA, Zhao X. A comprehensive school- admissions: Validation of a socioeconomic Organization of Resident Representatives (ORR) level analysis of the predictive validity of the indicator for use in medical school in April 2018, Orlando, Florida. Medical College Admission Test (MCAT). admissions. Acad Med. 2015;90:953–960. Paper presented at: American Educational 21 Gregorio JD, Lee JW. Education and J.A. Girotti was associate dean for admissions and Research Association 2013 Annual Meeting; income inequality: New evidence from special curricular programs, and director, Hispanic April 28, 2013; San Francisco, California. cross-country data. Rev of Income Wealth. Center of Excellence in Medicine, at the time this 9 Dunleavy DM, Kroopnick MH, Dowd KW, 2002;48:395–416. work was completed. He is research assistant Searcy CA, Zhao X. The predictive validity 22 Valletta R. Federal Reserve Bank of San professor, Department of Medical Education, of the MCAT exam in relation to academic Francisco Economic Letter. Higher University of Illinois College of Medicine, Chicago, performance through medical school: education, wages, and polarization. Illinois, now. A national cohort study of 2001-2004 http://www.frbsf.org/economic-research/ J.A. Chanatry is chair, Health Sciences Advisory matriculants. Acad Med. 2013;88:666–671. publications/economic-letter/2015/january/ Committee, and laboratory instructor, Department of 10 Davis D, Dorsey JK, Franks RD, Sackett PR, wages-education-college-labor-earnings- Chemistry, Colgate University, Hamilton, New York. Searcy CA, Zhao X. Do racial and ethnic income. Published January 12, 2015. Accessed group differences in performance on the June 5, 2019. D.M. Clinchot is vice dean for education MCAT exam reflect test bias? Acad Med. 23 Wolla SA, Sullivan J. Education, income, and associate vice president for health sciences 2013;88:593–602. and wealth. Page One Economics; https:// education, The Ohio State University College of 11 Association of American Medical Colleges. research.stlouisfed.org/publications/page1- Medicine, Columbus, Ohio. Using MCAT Data in 2019 Medical Student econ/2017/01/03/education-income-and- S.C. McClure is senior associate dean for student Selection. Washington, DC: Association of wealth. Published 2017. Accessed June 5, and academic affairs, professor of internal medicine, American Medical Colleges; 2018. https:// 2019. and program director, Center of Excellence, Meharry www.aamc.org/download/462316/data/ 24 Collier PJ, Morgan DL. “Is that paper Medical College, Nashville, Tennessee. mcatguide.pdf. Accessed June 5, 2019. really due today?”: Differences in first- 12 American Educational Research Association; generation and traditional college students’ A. Swan Sein is assistant professor of educational American Psychological Association; understandings of faculty expectations. assessment in pediatrics and dental medicine National Council on Measurement in Higher Educ. 2008;55:425–446. and director, Center for Education Research and Education. Standards for Educational and Evaluation, Columbia Vagelos College of Physicians 25 Jacoby B. Involving commuter students in and Surgeons, New York, New York; ORCID: Psychological Testing. 3rd ed. Washington, learning: Moving from rhetoric to reality. http:orcid.org/ 0000-0002-3139-4626. DC: American Educational Research New Dir High Educ. 2000;(109):3–12. Association; 2014. 26 Jacoby B, Garland J. Strategies for enhancing I.W. Walker is clinical associate professor, 13 The Carnegie Classification of Institutions commuter student success. J Coll Stud Ret. Department of Emergency Medicine, and past of Higher Education. Undergraduate Profile 2004;6:61–79. director of admissions, University of Calgary Classification. http://carnegieclassifications. 27 Kuh GD, Gonyea RM, Palmer M. The Cumming School of Medicine, Calgary, Alberta, iu.edu/classification_descriptions/ disengaged commuter student: Fact or Canada. undergraduate_profile.php. Accessed June fiction? Commuter Perspect. 2001;27:2–5. C.A. Searcy is senior director of MCAT research 22, 2018. 28 Busche K, Elks ML, Hanson JT, et al. The and development, Association of American Medical 14 The Carnegie Classification of Institutions validity of scores from the new MCAT Colleges, Washington, DC. of Higher Education. Size & Setting exam in predicting student performance: Classification description. http:// Results from a multisite study. Acad Med. carnegieclassifications.iu.edu/classification_ 2020;95:387–395. descriptions/size_setting.php. Accessed June 29 Anderson A. Technology Device Ownership: References 22, 2018. 2015. Pew Research Center. http://www. 1 Schwartzstein RM, Rosenfeld GC, Hilborn 15 Carnevale AP, Strohl J. Separate & Unequal: pewinternet.org/2015/10/29/technology- R, Oyewole SH, Mitchell K. Redesigning How Higher Education Reinforces the device-ownership-2015. Published October the MCAT exam: Balancing multiple Intergenerational Reproduction of 29, 2015. Accessed June 5, 2019. perspectives. Acad Med. 2013;88:560–567. White Racial Privilege. Washington, DC: 30 Karpicke JD, Blunt JR. Retrieval practice 2 Young JW. Grade adjustment methods. Rev Georgetown Public Policy Institute; 2013. produces more learning than elaborative studying with concept mapping. Science. Educ Res. 1993;63:151–165. https://1gyhoq479ufd3yna29x7ubjn- 2011;331:772–775. 3 Goldman RD, Widawski MH. A within- wpengine.netdna-ssl.com/wp-content/ 31 Larsen DP, Butler AC, Roediger HL 3rd. subjects technique for comparing college uploads/SeparateUnequal.FR_.pdf. Accessed Test-enhanced learning in medical education. grading standards: Implications in the June 5, 2019. Med Educ. 2008;42:959–966. validity of the evaluation of college 16 Pascarella ET, Terenzini PT, Blimling GS. 32 Gooding HC, Mann K, Armstrong E. Twelve achievement. Educ Psychol Meas. The impact of residential life on students. tips for applying the science of learning to 1976;36:381–390. In: Schroeder CC, Mable P, eds. Realizing the health professions education. Med Teach. 4 Callahan CA, Hojat M, Veloski J, Erdmann Educational Potential of Residence Halls. San 2017;39:26–31. JB, Gonnella JS. The predictive validity of Francisco, CA: Jossey-Bass Publishers; 1994. 33 Dunlosky J, Rawson KA, Marsh EJ, Nathan three versions of the MCAT in relation to 17 Smith T. Integrating living and learning in MJ, Willingham DT. Improving students’ performance in medical school, residency, residential colleges. In: Schroeder CC, Mable learning with effective learning techniques: and licensing examinations: A longitudinal P, eds. Realizing the Educational Potential of Promising directions from cognitive and study of 36 classes of Jefferson Medical Residence Halls. San Francisco, CA: Jossey- educational psychology. Psychol Sci Public College. Acad Med. 2010;85:980–987. Bass Publishers; 1994. Interest. 2013;14:4–58.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 373 26 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Article
34 Swan Sein A, Cuffney F, Clinchot D. How with low socioeconomic status: A systematic and medical school admissions: The past is to help students strategically prepare for review and meta-analysis. Rev Educ Res. prologue. Acad Med. 2020;95:351–356. the MCAT exam and learn foundational 2017;87:243–282. 38 Orlowski J, Dill M, Fisher K. Results of the knowledge needed for medical school. Acad 36 Sirin SR. Socioeconomic status and academic 2017 Medical School Enrollment Survey. Med. 2020;95:484. achievement: A meta-analytic review of Washington, DC: Association of American 35 Dietrichson J, Bøg M, Filges T, Jørgensen research. Rev Educ Res. 2005;75:417–453. Medical Colleges; 2018. https://store.aamc. A-MK. Academic interventions for 37 Lucey CR, Saguil A. The consequences of org/downloadable/download/sample/ elementary and middle school students structural racism on MCAT exam scores sample_id/183. Accessed June 5, 2019.
Cover Art Artist’s Statement: Sign-out
Sign-out is the formal passing of care two residents signing-out their patients to responsibilities to the incoming treatment each other. The resident speaking has just team. It is one of the more sacred times completed his day shift, which is implied of day at the hospital, as it assures the by his fatigued expression, unshaved healing ministry of medicine remains a beard, and two pagers. The resident in the 24-hour operation. It also symbolizes that center of the painting listens closely, as he physicians need rest and therefore the is just beginning his shift and will need space to relinquish their duties to capable to care for the patients overnight. The hands and fresh eyes. somber gray hues I painted represent the seriousness of the occasion. I positioned When I was a medical student, sign- the figures close together, focusing on the out occurred in the early evening and Sign-out character providing the sign-out. lasted several minutes. As part of the day team, at the end of my shift, I Near the end of medical school, I began My piece is inspired by the style of would cram into the workroom with to appreciate the underlying significance famed Mexican muralist Diego Rivera, several other teams to give a verbal of these brief communications. I saw how whose artwork featured uncomplicated sign-out to the incoming cross- the level of trust between two providers characters and rich colors. For this coverage team. On occasion, I would influenced the format of the questions reason, I chose stylized Hispanic figures have the opportunity to provide asked. I observed how providers can sense to reflect Rivera’s technique, as well as the report. The process was initially heightened concern for a patient in the diversity and inclusion in medicine. distressing—I never enjoyed public faces of their teammates. I noticed how a Sign-out is part of a series of paintings speaking, and the language of medicine doctor’s level of familiarity with a patient that focus on the profession of medicine was still unfamiliar to me. I was quite could translate into a surprisingly accurate that I started as a medical student and nervous during my first few attempts prediction of how a patient’s overnight now continue as a fellow. My objective and would practice with the resident care would unfold. I also witnessed the for these works is to demystify the culture on my shift prior to delivering sign-out compassion residents shared in quelling of medicine and provide insight into the to the team that evening. With the help each other’s fear, doubt, or apprehension journey of becoming a physician. of my fellow classmates and residents, by offering to check in on their colleagues the jargon became familiar, allowing throughout the night. me to deliver more fluid sign-outs. The Suliman EL-Amin, MD, MS attending physicians changed from gods In my acrylic painting Sign-out, on the Suliman EL-Amin is a child and adolescent to people in my mind, and the service cover of this issue, the doctor in the white psychiatry fellow, Mayo Clinic, Rochester, Minnesota; eventually became less daunting. coat is the lead physician and supervises [email protected].
374 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 27 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges ACM
XX ACADMED How to Help Students Strategically Aubrie Swan Sein, PhD, EdM, assistant professor of educational assessment, director, Acad Med Center for Education Research and Evaluation, Columbia Vagelos College of Physicians Prepare for the MCAT Exam and Learn and Surgeons; Francie Cuffney, PhD, professor of biological sciences, head of biology department, Meredith College; and Daniel Clinchot, MD, vice dean for education, associate 11January201929March20198April2019 Foundational Knowledge Needed for vice president for health sciences education, The Ohio State University College of Medicine Medical School
The Medical College Admission Test (MCAT) assesses the knowledge and reasoning skills students need to be prepared for medical school. The MCAT exam can also serve as a learning tool that uses authentic problems to help students integrate key concepts. Applying the concepts they are learning in coursework or self-directed study to those covered on the exam not only prepares students for the exam but also helps them build foundational knowledge needed for medical school. Preparing for and taking exams has a demonstrated contribution to learning,1 and using proven learning strategies2 throughout the study period helps AM Last Page develop lifelong study skills and supports learning in medical school. Academic advisors may recommend that students consider the steps below for exam preparation and therefore medical school. |
• Learn about the requirements for taking the MCAT exam. Lippincott Williams & Wilkins • Evaluate financial resources and eligibility for the Association of American Medical Colleges Fee Assistance Program to obtain fee waivers for test preparation products and discounted registration fees. GATHER • Review the MCAT content outline Revisit frequently during coursework. XX Hagerstown, MD INFORMATION • Discuss when to test and how to prepare with an advisor. Students without an advisor can use the Find-an-Advisor service from National Association of Advisors for the Health Professions. • Review the MCAT Testing Calendar to select a goal test date. • Inventory free and low-cost resources: practice materials, flash cards, class notes.
ASSESS • Take a practice test to establish a baseline and identify strengths and weaknesses. KNOWLEDGE • Decide if a class or self-study is needed to fill in gaps or enhance knowledge.
Daniel Clinchot, MD, • Consider academic, professional, and extracurricular obligations when scheduling studying. vice dean for education, associate vice president for health sciences education, The Ohio State University College of Medicine • Answer practice questions and take practice exams throughout the preparation period to become familiar with CREATE A MCAT question types and to learn from feedback and errors via “test-enhanced learning” (which is the idea 1 ACADEMIC MEDICINE STUDY PLAN that preparing for, taking, and reviewing feedback or errors from questions or tests lead to learning).
• Study different topics in each study session to make connections and integrate concepts. • Build in frequent cumulative review of topics already studied.
1. Briefly review a topic area
8. Repeat the cycle with practice 2. Complete a short set of practice questions in a new topic area questions in the topic area
7. Complete a cumulative set 3. Learn from errors by STUDY AND of practice questions on all A Test-Enhanced explaining why answer choices Learning Study Cycle PRACTICE the topics studied so far are correct or incorrect
6. Apply learning by 4. Review concepts and facts answering more questions in areas of weakness by in the same topic area accessing videos or textbooks
5. Summarize content from memory using compare/contrast tables or concept maps or by giving explanations to others
• Take a full-length practice test under test-day timing conditions. Practice exam functionality (highlight, strikethrough). PREPARE FOR • Plan how to comfortably answer questions within time limits. TEST DAY • Review test day requirements: break length, items allowed in the test center, check-in procedures. • Practice getting to the test center; plan to arrive early. • Eat well; find time to rest/relax.
Learn more at www.aamc.org/mcatprep
References: 1. Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008;42:959–966. 2. Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychol Sci Public Interest. 2013;14:4–58.
Acad Med. 2020;95:484. First published online September 24, 2019 doi: 10.1097/ACM.0000000000003000 Author contact: [email protected] 484 Academic Medicine, Vol. 95, No. 3 / March 2020
MCAT® is a program of the 28 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
The Diversity and Success of Medical School Applicants With Scores in the Middle Third of the MCAT Score Scale Carol A. Terregino, MD, Aaron Saguil, MD, MPH, Tanisha Price-Johnson, PhD, Ngozi F. Anachebe, PharmD, MD, and Kristen Goodell, MD
Abstract Admissions officers assemble classes and backgrounds of 2017 applicants, with high MCAT scores to maintain or of medical students with different accepted applicants, and matriculants improve their national rankings. backgrounds and experiences who can to U.S. MD-granting schools to explore contribute to their institutions’ service, avenues for increasing medical school The authors argue that reexamining leadership, and research goals. While class diversity. They found that schools the use of MCAT scores in admissions schools’ local interests vary, they share a that accepted more applicants with provides an opportunity to diversify the common goal: meeting the health needs midrange MCAT scores had more physician workforce. Despite evidence of an increasingly diverse population. diverse matriculating classes. Many that most students with midrange MCAT Despite the well-known benefits of schools admitting the most applicants scores succeed in medical school, there is diversity, the physician workforce does with scores in the middle of the MCAT a tendency to overlook these applicants not yet reflect the nation’s diversity by score scale were public, community- in favor of those with higher scores. To socioeconomic status, race/ethnicity, or based, and primary care-focused improve the health of all, the authors call other background characteristics. institutions; those admitting the for admitting more students with midrange fewest of these applicants tended to MCAT scores and studying the learning The authors reviewed the Medical be research-focused institutions and environments that enable students with a College Admission Test (MCAT) scores to report pressure to accept applicants wide range of MCAT scores to thrive.
Editor’s Note: An Invited Commentary by R.M. diverse nation that includes patients use measures of academic achievement, Schwartzstein appears on pages 333–335. who are disadvantaged, live in medically such as grade point averages (GPAs) underserved communities, and deserve and MCAT scores, to understand our equitable treatments and cures. applicants’ academic readiness for the As medical school admissions officers, medical school curriculum. Evidence we seek broadly qualified applicants At all levels, medical school leaders from previous and new versions of the who will successfully complete medical believe diversity can help us carry out MCAT exam reveals that students with school and demonstrate the humanistic the work of academic medicine both a broad range of scores are capable of qualities that portend well for the practice locally and nationally.1,2 For example, progressing through medical school of medicine. Keeping these 2 objectives an overwhelming number of medical on time, graduating in 4 or 5 years, in mind, along with the missions and school deans who responded to a recent and passing their licensure exams on Copyright © 2019 by the Association of American Medical Colleges goals of our institutions, we strive to survey reported that they are or will the first attempt.3–5 Yet many of us admit applicants who will provide care be pursuing strategies to increase the limit our choices by only considering through a variety of specialties, become breadth and depth of diversity in their applicants with high MCAT scores. In this researchers and educators, and work to entering classes.1 And, almost all of their Perspective, we argue that this practice improve our health care delivery system. admissions officers recently reported leads to missed opportunities in light of Our local interests may vary, but they that selecting students to foster a diverse what we know to be true: Many applicants reflect the common goal of meeting learning environment or students the health needs of an increasingly who obtain midrange scores on the who have an interest in caring for the MCAT exam have the qualifications and
Please see the end of this article for information underserved was important or very competencies to do well in medical school about the authors. important in identifying applicants who and can add tremendously to the diversity fit with their missions and goals.2 We of our classes. Correspondence should be addressed to Carol A. Terregino, Rutgers Robert Wood Johnson Medical value diversifying our medical school School, 675 Hoes Lane West, Piscataway, NJ 08854; classes in our efforts to develop physicians telephone: (732) 235-4577; email: terregca@rwjms. with cultural humility who will improve Why Is Increasing Diversity So rutgers.edu. health outcomes for all patients. Urgent? Written work prepared by employees of the Federal Government as part of their official duties is, under The case for enhancing the diversity of the U.S. Copyright Act, a “work of the United States We in admissions must take this direction the health care workforce is undeniable. Government” for which copyright protection under and ask, “Are we doing enough to meet A physician workforce that better reflects Title 17 of the United States Code is not available. As such, copyright does not extend to the contributions the diversity goals of our schools and the the diversity of patients can support of employees of the Federal Government. nation?” Examining how we use Medical efforts to reduce health disparities6 College Admission Test (MCAT) scores and increase access to care for patients Acad Med. 2020;95:344–350. 7 First published online August 13, 2019 as we look for future medical students who are underserved. Data show that doi: 10.1097/ACM.0000000000002941 provides one answer. In admissions, we black and Hispanic physicians are more
344 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 29 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges XXX Perspective
likely to care for patients of similar The opportunity to learn in a class applicants’ life and work experiences, as racial/ethnic backgrounds as well as composed of students with different well as other attributes described in their for patients who lack health insurance, perspectives and backgrounds applications, personal essays, letters of have Medicaid coverage, or live in enriches the educational environment recommendation, and interviews. MCAT medically underserved communities.8 for all students and gives them a scores are one important component Research also shows that students better appreciation of the issues in our toolbox, but examining how and from medically underserved or rural that contribute to a lack of access to why we use them during the admissions backgrounds are more likely to return to health care.24 Students report that the process is essential to broadening the similar regions to practice medicine.9–11 diversity of their medical school class pool of qualified applicants for selection. From the patient perspective, better “[enhanced their] training and skills to adherence to recommended health care work with individuals from different While we could completely fill our services and medication, more effective backgrounds.”25 Furthermore, when classes with higher-scoring applicants, communication, and greater trust are diverse classes of students learn in doing so would prevent us from seeing likely to occur when patients see a supportive environments, the meaningful the depth of each person revealed in physician of the same race or ethnicity or interactions among students from the application and those qualities that who speaks their native language.12–15 different backgrounds have the potential are not visible from test scores alone. to mitigate the negative effects of Closely examining the attributes and Equally important are the benefits that stereotype threat and improve the mental demonstrated passions of our applicants come from diversity in our research well-being and educational experiences is at the core of understanding how they workforce. Diverse physician–scientists of underrepresented students during will contribute to our schools’ missions. 26,27 will pursue a wide range of research their medical education. Creating a Data already show us that, when an inquiries that better address the health learning environment that enables all admissions committee admits students needs of the nation and the stark students to reach their highest potential with scores in the middle of the MCAT health disparities in underrepresented is fundamental to producing a diverse score scale who have demonstrated minority populations.16–18 Our workforce physician workforce. Importantly, the capacity and competencies needed will further benefit from an increased expanding the diversity of medical to become physicians, those students succeed at high rates, progressing through number of physicians with diverse student classes will also help prepare future physicians to care for the diverse medical school on time,5 graduating in perspectives and backgrounds who patient population28 they will encounter 4 or 5 years, and passing their licensure can take on leadership roles in health as they begin their careers in medicine. exams on the first attempt.3,4 We know care and government.18 Medical school that MCAT scores do a good job of faculty from minority groups are also assessing applicants’ academic readiness more likely to focus on health disparities How MCAT Scores Are Used in for medical school, but success in medical research than their nonminority Holistic Review school and beyond draws on more than colleagues.19 Identifying the right combination of demonstrated academic readiness.30 students can enable medical schools to The performance of students and the Diversifying the physician workforce is effectively carry out their tripartite roles physicians they will become is complex key to serving patients well. However, the of educating competent and humanistic and multidimensional, comprising students currently enrolled in medical physicians, caring for patients who come academic and clinical knowledge and school woefully underrepresent the from all parts of the population, and problem-solving skills; interpersonal populations they will care for based conducting life-saving research that skills, cultural competence, and the ability on race/ethnicity, geography, and benefits all. to communicate and build relationships; 20 socioeconomic status. Consider that and professional integrity.31 The MCAT schools in 2017 enrolled 21% of students The missions of our respective medical exam was not designed to and cannot be who were underrepresented based on schools may call for us to recruit expected to foretell all the contributions race/ethnicity, but roughly 30% of the applicants who are from our state or students will make in their communities general population is either black/African who grew up in rural communities. and to their patients’ lives. MCAT scores American, Hispanic, Alaska Native/ Some of us may look for students fluent then must be viewed in the context American Indian, or Native Hawaiian/ in other languages to care for patients of the experiences and educational Pacific Islander.21 Schools also enrolled who do not speak English. We may seek opportunities applicants have had in life. 9% of students who were born or future physician–scientists with the skills educated in rural communities, but about and passion for conducting biomedical When the new MCAT exam was launched 20% of the general population lives in research. Holistic review helps us in 2015 with a new score scale, it gave us rural areas.22 Schools enrolled 16% of accomplish our goals because it provides in admissions the opportunity to recharge students whose parents did not have a the foundation for an individualized, and and reevaluate our holistic review bachelor’s degree. Yet approximately 70% institution-specific, review of applicants. processes in a couple of ways. First, we of the American population is without a It is adaptable, supported by evidence, were starting over and needed to develop 4-year college degree.23 Bridging the gap and tied to our missions and goals.29 a new understanding of what MCAT between the makeup of society and the scores told us about applicants’ academic backgrounds of our medical students Holistic review helps us in admissions readiness for medical school. In the early should be a priority for leaders in medical evaluate academic measures, such as application cycles, we were reminded of education. GPAs and MCAT scores, in the context of the importance of contextualizing MCAT
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 345 30 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
scores to identify students who could be carefully considered during the applicants also were more often English- meet institutional missions and serve admissions process. These individuals language learning (non-native English our designated populations. Second, have the potential to bring rich speakers) or underrepresented based on the new MCAT score scale was designed sociodemographic diversity to our race/ethnicity (black/African American, to help us better balance scores with medical school classes and the physician Hispanic, Alaska Native/American Indian, the other information we had about workforce. Data from 2017 applicants or Native Hawaiian/Pacific Islander). applicants, largely by drawing attention who scored in the middle of the MCAT Overall, applicants with scores in the to those applicants who scored in the score scale (495–504) show why these middle third of the MCAT score scale middle of the scale who might help us students are so important (see Figure 1). were more diverse than their peers with meet our institutional missions in unique scores in the upper third of the scale. ways. Some of us heeded this advice and Figure 1 compares the diversity of took advantage of this time to look at a applicants who scored in the middle third Data from the old (in place from 1991 wider range of scores.32 Largely however of the MCAT score scale (495–504) with to January 2015) and new (in place from our community continues to look at the diversity of applicants who scored in April 2015 to present day) MCAT exams applicants with higher MCAT scores. the upper third of the scale (505–528). show that medical students with scores (Not shown are applicants with scores in the middle of the score scale perform We must be vigilant in contextualizing in the lower third of the scale [472–494], at similarly high levels as students with all applicants’ MCAT scores during the who typically represent less than 15% of scores in the upper third of the scale.33,34 admissions process. In the following applicants and less than 3% of accepted For example, graduation data from sections, we explain why. applicants.) Compared with applicants students who took the old exam showed with scores in the upper third of the that 84% of students in the middle score scale, applicants with scores in the third and 89% of students in the upper Applicants With Scores in the middle third were more likely to be the third of the old score scale graduated Middle of the MCAT Score Scale first in their family to obtain a bachelor’s from medical school in 4 years. Five- Are More Diverse degree, have parents in clerical or service year graduation rates were even higher Applicants with scores in the middle positions, or have grown up in rural with 93% of students with scores in of the MCAT score scale should or medically underserved areas. These the middle third and 96% of students
30% 28% Middle third (495-504) 27% 27% 26% Upper third (505-528) 25%
20%
15% 14% 15% 15% 12% 11%
% of applicants 10% 7% 8%
5% 2%
0% No parent with a Parents in service, Non-native English Grew up in rural Grew up in Underrepresented bachelor's degree clerical, skilled, speaker area medically by race/ethnicity unskilled underserved area occupations Sociodemographic characteristics of applicantsa Figure 1 Percentage of 2017 applicants to medical school with Medical College Admission Test (MCAT) scores in the middle or upper third of the new (April 2015–present day) MCAT total score scale, by sociodemographic characteristics. Data are from the 2017 medical school application cycle.20 The primary source of diversity data was the file submitted by the applicant during the 2017 application cycle. The secondary source of diversity data for parental education and occupation, English-language proficiency, and race/ethnicity was MCAT registration data. This secondary data source was only used if data were missing from the applicant file. MCAT total scores were divided into thirds using the distribution of total scores from the 2015 testing year. Percentages are based on unique applicants who answered questions about their locations, backgrounds, and demographics when they applied to medical school. a“No parent with a bachelor’s degree” indicates parents’ highest education is less than a bachelor’s degree. “Parents in service, clerical, skilled, unskilled occupation” indicates parents’ highest occupation. “Non-native English speaker” is based on English-language proficiency being self-reported as advanced, good, fair, or basic, rather than native. “Grew up in rural area” is based on either birth county or high school county being classified as rural, as defined by the federal urban–rural classification scheme for counties. “Grew up in medically underserved area” is based on applicants responding affirmatively to a question on their medical school application asking if they grew up in a medically underserved area during childhood. “Underrepresented by race/ethnicity” is based on applicants self-identifying as black or African American, Hispanic, Alaska Native/American Indian, or Native Hawaiian/Pacific Islander. Applicants underrepresented based on race/ethnicity include both those who identified as an underrepresented race alone and in combination with another race. Percentages include U.S. citizens and permanent residents only.
346 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 31 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
with scores in the upper third of the old were community based, and reported When we compared the schools in the score scale graduating in 5 years.33 While that selecting students who intended to first quartile with those in the fourth graduation data are not yet available for practice primary care was an important quartile, we saw differences in the the new exam, early outcome data for part of their mission. Schools in the diversity of their classes. Schools in the 2016 matriculants showed that those who fourth quartile, however, tended to be first quartile, which accepted larger scored in the middle third of the MCAT research focused and to report pressure to percentages of students with MCAT score scale completed their first year of accept students with high MCAT scores to scores between 495 and 504, matriculated medical school and progressed on time maintain or improve their U.S. News and larger percentages of students who to year 2 at similarly high rates as their World Report rankings. These schools also were the first in their family to obtain a classmates with scores in the upper third reported using MCAT scores to identify college degree, whose parents worked in of the scale. That is, 95% of students with the most academically capable applicants a “service, clerical, skilled, or unskilled” scores in the middle third of the score and that academic metrics were more occupation, who grew up in a rural or scale progressed on time, compared with important than other application data in medically underserved area, and who 98% of students with scores in the upper deciding which applicants to interview. were non-native English speakers. Most third of the scale.34 This is an important of these differences are striking. milestone because there is a period of Admitting Students With Scores adjustment for many students during There is one exception to this trend. in the Middle of the MCAT Score The percentages of matriculants who the first year as they settle into a new Scale Increases Student Diversity curriculum and learning environment, were underrepresented based on race/ and these data suggest that matriculants Our analysis also showed that schools ethnicity showed the opposite pattern. with midrange scores are performing well that accepted more students with scores Schools in the fourth quartile had during this time. in the middle of the MCAT score scale larger percentages of students who were created more diverse classes across a underrepresented based on race/ethnicity range of socioeconomic and demographic than schools in the first quartile. This Some Schools Accept More characteristics that are important to finding might be explained by revisiting Students With Scores in the helping schools meet their missions the characteristics of the schools in the 4 Middle Third of the MCAT Score and goals. Table 1 shows the median quartiles. Schools in the fourth quartile Scale Than Other Schools percentages of 2017 medical school were more likely to be private and could Figure 1 shows that 2017 applicants with matriculants from diverse backgrounds, have had access to resources that enabled scores in the middle third of the MCAT by medical school quartile (based on them to offer more enticing financial percentage of accepted applicants with aid packages for students to attend their score scale were more diverse than their scores in the middle third of the MCAT institutions. On the other hand, the peers with higher scores. But what do we score scale; see Figure 2). majority of schools in the first quartile know about the schools that accepted them? We rank-ordered schools from those that accepted the largest percentage 100
of applicants with scores in the middle 90 1st quartile third of the MCAT score scale to those (N = 35) - third e
l 80
that accepted the smallest percentage of d d applicants with middle-third scores. We i
m 70 h t
then divided the schools into 4 equal s i
quartiles (see Figure 2). w 60 s 2nd quartile t
n (N = 36) a 50 c i The first quartile represents those l 3rd quartile p
p 40 (N = 35) a schools with the most accepted MCAT score d e applicants with middle-third scores, t 30 4th quartile p 21.2% e (N = 36) and the fourth quartile includes those c c
a 20
schools with the fewest. The range of f accepted applicants with scores in the o 7.7%
% 10 middle third of the score scale varied 2.17% widely across schools. For example, in 0 the first quartile, 1 school had 72% of 1 142 its accepted applicants with middle- Medical school rank order third scores, and in the fourth quartile, Figure 2 Rank order of medical schools by the percentage of their 2017 accepted applicants with multiple schools had no accepted Medical College Admission Test (MCAT) scores in the middle third of the new (April 2015–present applicants with scores in this range. day) MCAT total score scale (495–504). Data are from the 2017 medical school application cycle. Only mainland U.S. MD-granting schools were included in the analysis. Schools were rank ordered from the school with the largest percentage of accepted applicants with middle-third scores We then analyzed institutional data and on the new MCAT exam to the school with the smallest percentage. Middle-third total scores admissions survey data from the schools (495–504) were identified using the distribution of total scores from the 2015 testing year. After in each quartile to learn more about their schools were rank ordered, they were divided into equal fourths. The ranges of the percentage of characteristics.2,35 Many schools in the accepted applicants with middle-third MCAT scores were: 21.2% to 72.2% (first quartile), 7.7% first quartile were public institutions, to 20.9% (second quartile), 2.17% to 7.5% (third quartile), and 0% to 2.15% (fourth quartile).
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 347 32 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
Table 1 Median Percentages of 2017 Medical School Matriculants From Diverse Backgrounds, by Medical School Quartilea,b
Median % of matriculants by medical school quartile 1st quartile 2nd quartile 3rd quartile 4th quartile Diversity characteristic: Matriculants ... (21.2%–72.2%) (7.7%–20.9%) (2.17%–7.5%) (0%–2.15%) Who have no parents with a bachelor’s degreec 20 17 15 11 For whom the highest parental occupation is a “service, clerical, 20 19 15 12 skilled, or unskilled” occupationd Who are non-native English speakerse 4 2 2 1 Who grew up in a rural communityf 11 8 5 4 Who grew up in a medically underserved areag 27 20 15 14 Who are underrepresented in medicine based on race/ethnicityh 16 17 18 19 With none of the 6 diversity characteristics abovei 44 52 57 61 With 1 of the 6 diversity characteristics abovej 26 23 23 22 With 2 or more of the 6 diversity characteristics abovek 30 25 18 15
aData are from the 2017 medical school application cycle. The primary source of diversity data was the file submitted by the applicant during the 2017 application cycle. The secondary source of diversity data for parental education and occupation, English-language proficiency, and race/ethnicity was Medical College Admission Test (MCAT) registration data. This secondary source was used only if data were missing from the applicant file. Only mainland U.S. MD-granting schools were included in this analysis.20 bMedical schools were rank ordered by the percentage of their 2017 accepted applicants who had scores in the middle third of the MCAT score scale, then the schools were divided into quartiles. Scores in the middle third of the score scale (495–504) were identified using the distribution of new MCAT total scores from the 2015 administration cycle. cHighest parental education is less than a bachelor’s degree. Nationally, 16% of 2017 matriculants had parents without a bachelor’s degree. dHighest parental occupation is a “service, clerical, skilled, or unskilled” occupation. Nationally, 18% of 2017 matriculants had parents who were in service, clerical, skilled, or unskilled occupations. eNon-native English speaker is based on English-language proficiency being self-reported as advanced, good, fair, or basic, rather than native. Nationally, 4% of 2017 matriculants were non-native English speakers. fGrew up in a rural area is based on either birth county or high school county being classified as rural, as defined by the federal urban–rural classification scheme for counties. Nationally, 9% of 2017 matriculants grew up in a rural area. gGrew up in a medically underserved area is based on the applicant responding affirmatively to a question in their medical school application asking if they grew up in a medically underserved area during childhood. Nationally, 20% of 2017 matriculants grew up in a medically underserved area. hUnderrepresented in medicine based on race/ethnicity is based on the applicant self-identifying as black or African American, Hispanic, Alaska Native/American Indian, or Native Hawaiian/Pacific Islander. Applicants underrepresented based on race/ethnicity include both those who identified as an underrepresented race alone and in combination with another race. Percentages include U.S. citizens and permanent residents only. Nationally, 21% of 2017 matriculants were underrepresented based on race/ethnicity. iThose who met none of the 6 diversity characteristics above. Nationally, 53% of 2017 matriculants met none of the 6 diversity characteristics. jThose who met 1 of the 6 diversity characteristics above. Nationally, 24% of 2017 matriculants met 1 of the 6 diversity characteristics. kThose who met 2 or more of the 6 diversity characteristics above. Nationally, 24% of 2017 matriculants met 2 or more of the 6 diversity characteristics.
were public and either rural or regional medical schools seek. For example, Table 1 demonstrates the institutions. Additionally, many were 21% of matriculants in 2017 identified interrelationships and impact of students’ from states that had substantially smaller as underrepresented based on race/ diversity characteristics at the schools underrepresented populations based on ethnicity (see Table 1). More than 10% in each quartile. The data show that, race/ethnicity.36 These factors could have of these students also reported growing on average, schools in the first quartile created unique challenges for admissions up in a medically underserved area, the matriculated more students with 2 officers at schools with fewer resources most common pairing. Similarly, 16% or more diversity characteristics. The and in certain locations looking to of matriculants were the first in their median percentage of students at schools increase diversity based on race/ethnicity. family to obtain a college degree. Almost in the first quartile who met 2 or more 1 in 5 of these students also reported that diversity criteria was 30%, compared with However, we believe diversity is not a their parents’ highest occupation was in a a median percentage of 15% for schools matter of just counting students one service or clerical job.37 in the fourth quartile. These results characteristic at a time. Diversity, broadly suggest that schools in the first quartile defined, should capture the complexities Research on the relative importance, are constructing classes with richer of a sociodemographically diverse society interdependence, and number of diversity across more than one dimension that is made up of numerous cultures, diversity characteristics of practicing than schools in the other quartiles. languages, nationalities, and geographies. physicians suggests that students with Acknowledging and seeking this rich multiple diversity characteristics will diversity is a sine qua non to meet the be more likely to care for patients who Empowering Our Admissions needs of our patient population. In the live in medically underserved areas, are Community to Change 2017 matriculating class, there were medically indigent, or are poor.9,38,39 Patients need a physician workforce that medical students who reflected this Richer diversity also has value across reflects the sociodemographic diversity of mosaic, presenting with different pairs specialties and in our research, academic, our country and that has been educated and combinations of backgrounds that and policy communities. in a learning environment enriched by
348 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 33 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
the diversity of the class. Because of the The evidence that MCAT scores predict Admissions Decision Making working group pressures medical schools face to admit academic performance in medical of the Association of American Medical applicants with high MCAT scores, we school and on licensing examinations Colleges (AAMC) Medical College Admission 3,4,40 Test (MCAT) Validity Committee (MVC) for run the risk of missing many applicants is strong. Yet, in every class we their contributions to this research: Hallen who are academically qualified and admit, we see variability in the academic Chung, Janet McHugh, Cindy Morris, and Mike bring additional lived experiences to the performance of students with similar Woodson. The authors would also like to thank learning environment. Change starts with MCAT scores.41 The truth is that we all the full MVC for their dedication and tireless us, and it begins with a reexamination of know of students with more modest efforts to evaluate the new MCAT exam: Ngozi how and why we consider MCAT scores MCAT scores who outperformed their Anachebe, Barbara Beckman, Ruth Bingham, Kevin Busche, Deborah Castellano, Francie during the admissions process. classmates with higher MCAT scores. Cuffney, Julie Chanatry, Hallen Chung, Daniel We believe that these students were Clinchot, Liesel Copeland, Martha Elks, William With the data we presented in this successful because of the academics, Gilliland, Jorge Girotti, Kristen Goodell, Joshua Perspective, we issue a call to our attributes, and experiences they brought Hanson, Loretta Jackson-Williams, David Jones, admissions colleagues to redouble and also because of our ability to provide Catherine Lucey, R. Stephen Manuel, Janet efforts to diversify our medical schools a learning environment in which students McHugh, Stephanie McClure, Cindy Morris, from different backgrounds and academic Wanda Parsons, Tanisha Price-Johnson, Boyd and the profession. We urge all of us Richards, Aaron Saguil, Aubrie Swan Sein, in admissions to consider, accept, and trajectories could thrive. Broadly Stuart Slavin, Doug Taylor, Carol Terregino, Ian support more applicants with scores speaking, this environment includes Walker, Robert Witzburg, David Wofsy, and Mike in the middle third of the MCAT academic, social, and wellness support Woodson. The authors would also like to thank score scale. We should balance the in addition to the curriculum, which the following AAMC staff: Cynthia Searcy, Karen weight given to applicants’ MCAT students can take advantage of or we can Mitchell, and Lesley Ward. scores with the information in their require during their education. Some Funding/Support: None reported. applications, transcripts, and letters institutions have instilled an inclusive of recommendation to select students culture of success by carefully creating Other disclosures: The Medical College Admission with promise and to use all available a curriculum and learning environment Test (MCAT) is a program of the Association of that supports all students, including American Medical Colleges (AAMC). Related information to support these students trademarks owned by the AAMC include Medical once they matriculate. It is here that we those from diverse backgrounds and College Admission Test and MCAT. may find greater numbers of potential experiences.42 We must ask ourselves what students who reflect the diversity of we can learn from the schools that do Ethical approval: Reported as not applicable. this well and how we can create similar the country based on demographics, Disclaimers: The views expressed in this article backgrounds, and perspectives. learning environments and cultures. do not necessarily reflect the views of the Uniformed Services University, the US Army, or This issue of imbalance is global, but Diversifying the physician workforce the Department of Defense. the solutions are local. We acknowledge is key to serving patients well in our Previous presentations: Some of the data that institutions are doing good work clinics, labs, and the policy arena. As the admissions community, we have presented in this article were presented at the to identify and admit diverse classes of Association of American Medical Colleges Learn medical students who will contribute to an inherent interest in using MCAT Serve Lead annual meeting in November 2017, in their missions and goals. We encourage scores to admit the applicants our Boston, Massachusetts. every institution to do a little more. What patients need. The data presented would happen to the diversity of our in this Perspective provide evidence C.A. Terregino is professor of medicine, senior that drawing more from the middle associate dean for education and academic affairs, medical school classes if we all committed associate dean for admissions, and chair of the to expanding our consideration of third of the MCAT score scale has the admissions committee, Rutgers Robert Wood applicants with scores in the middle potential to bring those individuals with Johnson Medical School, Piscataway, New Jersey. impressive and diverse backgrounds third of the MCAT score scale who have A. Saguil was associate dean for recruitment demonstrated the qualifications and into the house of medicine. A conscious and admissions, Uniformed Services University competencies to be successful students and concerted effort is needed to of the Health Sciences, F. Edward Hébert School identify these “jewels.” We ask all of us of Medicine, Bethesda, Maryland, at the time and physicians? What would it take to this work was completed. He is associate dean, admit only a few more students from this in admissions to dedicate ourselves to regional education, Uniformed Services University group? Our ask is small, but we recognize the work of identifying those applicants of the Health Sciences, F. Edward Hébert School with a wider range of MCAT scores of Medicine, San Antonio, Texas, now. He is also that some institutions may need to find vice chair, Medical College Admission Test Validity ways to manage the pressure to admit who are likely to succeed in medical Committee, Association of American Medical higher-scoring applicants, such as from school and supporting them in ways that Colleges, Washington, DC. promote their academic and professional national rankings of medical schools. T. Price-Johnson is executive director of We recommend sharing strategies that success. admissions, University of Arizona College of would give admissions committees room Medicine–Tucson, Tucson, Arizona. to be more flexible in their use of MCAT The need for increased diversity in the N.F. Anachebe is associate dean of admissions and scores. By considering applicants with a physician workforce is pressing, for our student affairs, and associate professor of clinical communities, our science, our patients, obstetrics and gynecology, Morehouse School of wider range of scores in our admissions Medicine, Atlanta, Georgia. processes, we likely will find more and our future. K. Goodell is associate dean of admissions, students from diverse backgrounds who Acknowledgments: The authors would like Boston University School of Medicine, Boston, bring new perspectives to our schools. to thank their colleagues on the Research on Massachusetts.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 349 34 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
References Hispanic pediatric surgery patients. J Pediatr of diversity in medical school: A survey of Surg. 2015;50:1586–1589. students. Acad Med. 2003;78:460–466. 1 Orlowski J, Dill M, Fisher K. Results of the 15 Parker MM, Fernández A, Moffet HH, Grant 29 Conrad SS, Addams AN, Young GH. Holistic 2017 AAMC Medical School Enrollment RW, Torreblanca A, Karter AJ. Association review in medical school admissions and Survey. Washington, DC: Association of of patient-physician language concordance American Medical Colleges; 2018. https:// selection: A strategic, mission-driven and glycemic control for limited-English response to shifting societal needs. Acad Med. store.aamc.org/downloadable/download/ proficiency Latinos with type 2 diabetes. sample/sample_id/183/. November 27, 2019. 2016;91:1472–1474. JAMA Intern Med. 2017;177:380–387. 30 Association of American Medical Colleges. 2 Association of American Medical Colleges. 16 Smedley BD, Butler AS, Bristow LR. In the The Core Competencies for Entering Medical 2017 Admissions Officer Survey. April 2017. Nation’s Compelling Interest: Ensuring Students. https://students-residents.aamc. Unpublished data. Diversity in the Health Care Workforce. 3 Dunleavy DM, Kroopnick MH, Dowd KW, Washington, DC: National Academies org/applying-medical-school/article/core- Searcy CA, Zhao X. The predictive validity Press; 2004. competencies. Accessed May 21, 2019. of the MCAT exam in relation to academic 17 Stoff DM, Forsyth A, Marquez ED, McClure 31 NEJM Knowledge+ Team. Exploring the performance through medical school: S. Introduction: The case for diversity ACGME Core Competencies: Parts 1–7. A national cohort study of 2001-2004 in research on mental health and HIV/ 2016. https://knowledgeplus.nejm.org/blog/ matriculants. Acad Med. 2013;88:666–671. AIDS. Am J Public Health. 2009;99(suppl acgme-core-competencies-professionalism. 4 Association of American Medical Colleges. 1):S8–15. Accessed May 21, 2019. Using Scores from the Old MCAT Exam 18 Cohen JJ, Gabriel BA, Terrell C. The case for 32 Price-Johnson T, Hall T, Saguil A, Nakae in Medical Student Selection. Washington, diversity in the health care workforce. Health S, Teitz C. Strategies for mission-oriented DC: Association of American Medical Aff (Millwood). 2002;21:90–102. admissions in light of the new MCAT exam. Colleges; 2016. 19 Pololi LH, Evans AT, Gibbs BK, Krupat E, Presentation at Learn Serve Lead 2017: The 5 Busche K, Elks ML, Hanson JT, et al. The Brennan RT, Civian JT. The experience of AAMC Annual Meeting; November 2017; validity of scores from the new MCAT exam in minority faculty who are underrepresented predicting student performance: Results from Boston, MA. in medicine, at 26 representative U.S. medical 33 Association of American Medical Colleges. a multisite study. Acad Med. 2020:95:387–395. schools. Acad Med. 2013;88:1308–1314. 6 Marrast LM, Zallman L, Woolhandler S, Bor 2007–2009 medical school students who 20 Association of American Medical Colleges. graduated in five years by old MCAT total DH, McCormick D. Minority physicians’ AAMC Applicant Matriculant Data File and score. January 2019. Unpublished data. role in the care of underserved patients: MCAT Data File. Retrieved July 30, 2018. 34 Association of American Medical Colleges. Diversifying the physician workforce may be Unpublished data. key in addressing health disparities. JAMA 21 2013–2017 American Community Survey 2016 medical student matriculant Intern Med. 2014;174:289–291. 5-Year Estimates: ACS Demographic progression rates from year 1 to year 2 of 7 Xierali IM, Castillo-Page L, Conrad S, Nivet and Housing Estimates. United States medical school by middle (495–504) and M. Analyzing physician workforce racial and Census Bureau. https://factfinder. upper (505–528) third MCAT score scale ethnic composition associations: Geographic census.gov/faces/tableservices/jsf/pages/ ranges. May 2018. Unpublished data. distribution (part II). AAMC Analysis in productview.xhtml?pid=ACS_17_5YR_ 35 Association of American Medical Colleges. Brief. August 2014;14. https://www.aamc.org/ DP05&prodType=table. Accessed June 18, AAMC database on U.S. and Canadian download/401814/data/aug2014aibpart2.pdf. 2018. medical schools. February 2018. Unpublished Accessed May 21, 2019. 22 2010 Census Urban and Rural Classification data. 8 Komaromy M, Grumbach K, Drake M, and Urban Area Criteria. United States 36 Association of American Medical Colleges. et al. The role of black and Hispanic Census Bureau. https://www.census.gov/ Characteristics of medical schools by physicians in providing health care for programs-surveys/geography/guidance/geo- percentage of applicants with middle underserved populations. N Engl J Med. areas/urban-rural/2010-urban-rural.html. (495–504) and upper (505–528) third MCAT 1996;334:1305–1310. Accessed May 21, 2019. scores. April 2018. Unpublished data. 9 Rabinowitz HK, Diamond JJ, Veloski JJ, Gayle 23 2013–2017 American Community JA. The impact of multiple predictors on 37 Association of American Medical Colleges. Survey 5-Year Estimates: Educational Medical school matriculants with more than generalist physicians’ care of underserved attainment. United States Census one diversity characteristic. October 2018. populations. Am J Public Health. Bureau. https://factfinder.census. Unpublished data. 2000;90:1225–1228. gov/faces/tableservices/jsf/pages/ 10 Rabinowitz HK, Diamond JJ, Markham productview.xhtml?pid=ACS_17_5YR_ 38 McDougle L, Way DP, Rucker YL. Survey FW, Santana AJ. The relationship between S1501&prodType=table. Accessed June 14, of care for the underserved: A control entering medical students’ backgrounds 2018. group study of practicing physicians and career plans and their rural practice 24 Saha S, Guiton G, Wimmers PF, Wilkerson L. who were graduates of The Ohio State outcomes three decades later. Acad Med. Student body racial and ethnic composition University College of Medicine premedical 2012;87:493–497. and diversity-related outcomes in US medical postbaccalaureate training program. Acad 11 Wade ME, Brokaw JJ, Zollinger TW, et al. schools. JAMA. 2008;300:1135–1145. Med. 2010;85:36–40. Influence of hometown on family physicians’ 25 Association of American Medical Colleges. 39 Wayne SJ, Kalishman S, Jerabek RN, Timm choice to practice in rural settings. Fam Med. Medical School Graduation Questionnaire. C, Cosgrove E. Early predictors of physicians’ 2007;39:248–254. 2018 All Schools Summary Report. practice in medically underserved communities: 12 Alsan M, Garrick O, Graziani GC. Does Washington, DC: Association of American A 12-year follow-up study of University of New Diversity Matter for Health? Experimental Medical Colleges; 2018. https://www.aamc. Mexico School of Medicine graduates. Acad Evidence From Oakland. National Bureau of org/download/490454/data/2018gqallsch Med. 2010;85(10 suppl):S13–S16. Economic Research Working Paper 24787. oolssummaryreport.pdf. Accessed January 40 Julian ER. Validity of the Medical College 2018. http://www.nber.org/papers/w24787. 7, 2019. Admission Test for predicting medical school Accessed May 21, 2019. 26 Garces LM, Jayakumar UM. Dynamic performance. Acad Med. 2005;80:910–917. 13 Traylor AH, Schmittdiel JA, Uratsu CS, diversity: Toward a contextual 41 Association of American Medical Colleges. Mangione CM, Subramanian U. Adherence understanding of critical mass. Educ Res. to cardiovascular disease medications: Does 2014;43:115–124. Using MCAT Data in 2019 Medical Student patient-provider race/ethnicity and language 27 Orom H, Semalulu T, Underwood W 3rd. Selection. Washington, DC: Association of concordance matter? J Gen Intern Med. The social and learning environments American Medical Colleges; 2018. 2010;25:1172–1177. experienced by underrepresented minority 42 Elks ML, Herbert-Carter J, Smith M, 14 Dunlap JL, Jaramillo JD, Koppolu R, Wright medical students: A narrative review. Acad Klement B, Knight BB, Anachebe NF. R, Mendoza F, Bruzoni M. The effects Med. 2013;88:1765–1777. Shifting the curve: Fostering academic of language concordant care on patient 28 Whitla DK, Orfield G, Silen W, Teperow C, success in a diverse student body. Acad Med. satisfaction and clinical understanding for Howard C, Reede J. Educational benefits 2018;93:66–70.
350 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 35 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
The Consequences of Structural Racism on MCAT Scores and Medical School Admissions: The Past Is Prologue Catherine Reinis Lucey, MD, and Aaron Saguil, MD, MPH
Abstract Those in medical education have a workforce if medical schools only students. These differences are not responsibility to prepare a physician admit those applicants with the highest due to differences in aptitude but to workforce that can serve increasingly MCAT scores. The authors review the differences in opportunities. The authors diverse communities, encourage healthy psychometric literature, which showed describe the widespread consequences changes in patients, and advocate for no evidence of bias in the exam, and of structural racism on economic the social changes needed to advance note that the differences in mean MCAT success, educational opportunity, and the health of all. The authors of this scores between racial and ethnic groups bias in the educational environment. Perspective discuss many of the likely are similar to those in other measures of They close with 3 recommendations for causes of the observed differences in academic achievement and performance medical schools that may mitigate the mean Medical College Admission Test on high-stakes tests. consequences of structural racism while (MCAT) scores between students from maintaining academic standards and groups well represented in medicine and The authors then describe the ways in admitting students likely to succeed. those from groups underrepresented in which structural racism in the United Adopting these recommendations may medicine. The lower mean MCAT scores States has contributed to differences help the medical profession build the of underrepresented groups can present in achievement for underrepresented diverse physician workforce needed to challenges to diversifying the physician students compared with well-represented serve communities today.
Editor’s Note: An Invited Commentary by R.M. environment, and breadth and impact of patient communities. Finally, we Schwartzstein appears on pages 333–335. of research.4–10 Our nation also needs encourage the medical education and physician–citizens who are prepared admissions leadership to adopt 3 practices to use their voices and the trust our to appropriately use MCAT scores in the Medical education is a public good, communities place in them to advocate context of holistic admissions. responsible for preparing the physician not only for better health care but also workforce that our nation needs and for equity in opportunity and in the for producing physician–citizens who social systems on which patients rely. How MCAT Scores Are Currently will participate in and advance our Used 1,2 Fulfilling these obligations to society democracy. To accomplish this mission, requires that all physicians actively Extensive validation studies done on each medical schools must build a physician recognize and address the pernicious iteration of the MCAT exam demonstrate workforce that is diverse and inclusive role that structural racism has played that it is a psychometrically valid with respect to race, ethnicity, and all in creating and sustaining inequities achievement test that predicts applicants’ the rich identities that exist within our in health care, education, and societal likelihood of success in several aspects communities.3 Racially and ethnically 11 of medical education, including medical diverse medical students and physicians opportunity in our country. A critical school coursework, the United States improve access to care, adherence to first step is to examine the ways in which treatment, quality of the educational medical schools use the Medical College Medical Licensing Examination (USMLE) Admission Test (MCAT) to evaluate Step exams, and graduation in 4 or 5 Copyright © 2019 by the Association of American Medical Colleges 12–14 Please see the end of this article for information aspiring physicians and the impact these years. The range of MCAT scores that about the authors. practices have on the profession’s ability are compatible with success in medical 15,16 Correspondence should be addressed to Catherine to build a racially and ethnically diverse school, however, is wide. Recognizing Reinis Lucey, University of California, San Francisco, physician workforce. this, many medical schools use a holistic School of Medicine, 533 Parnassus Ave., Suite U-80, review approach to evaluate applicants, San Francisco, CA 94143; telephone: (415) 815- 1633; email: [email protected]. In this Perspective, we look at how MCAT balancing quantitative assessments of their scores are used in admissions and how academic achievements using MCAT scores Written work prepared by employees of the Federal Government as part of their official duties is, under structural racism and differing access to and undergraduate grade point averages the U.S. Copyright Act, a “work of the United States opportunity create and perpetuate group with qualitative data on their premedical Government” for which copyright protection under differences in MCAT scores. We explore school experiences and personal attributes. Title 17 of the United States Code is not available. As such, copyright does not extend to the contributions how the MCAT exam neither over- nor Still, lower scores on the MCAT exam, of employees of the Federal Government. underpredicts performance among racial even those within the range predictive of and ethnic groups while sharing how success, are associated with lower rates of Acad Med. 2020;95:351–356. 16 First published online August 13, 2019 assigning too much weight to the highest acceptance to medical school. doi: 10.1097/ACM.0000000000002939 MCAT scores in admissions decision Supplemental digital content for this article is making makes it difficult to build medical A persistent affinity for applicants with available at http://links.lww.com/ACADMED/A734. school classes that are representative the highest scores on standardized
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 351 36 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges XXX Perspective
exams is not unique to medical schools. exam are similar to those on the old In this Perspective, we provide a closer The Supreme Court has heard highly exam20,21; on other measures of academic look at these group differences, which publicized legal cases challenging the achievement; and on high-stakes exams, reveal that while the MCAT exam appropriateness of admissions decisions such as the SAT, ACT, GRE, GMAT, predicts medical school performance that are made using characteristics and LSAT.22,23 Overlooking applicants in the same way for students from beyond test scores.17–19 Many of these with anything but the highest scores different backgrounds, the educational cases question why students with contributes to persistent challenges in opportunities afforded to students from lower scores on standardized exams diversifying medical school classes. This kindergarten to the time they take the are admitted to institutions of higher practice is particularly problematic MCAT exam are not equitable. Unequal education while students with higher when the weight accorded to MCAT opportunities in housing, education, scores are rejected. Implicit in this scores in admissions decision making is and other areas of society for different argument is the assumption that higher greater than the weight given to other populations have led to differing levels test scores are indicative of a better of academic achievement, reflected in predictors of students’ success, such as student. The reality is much more mean score differences on high-stakes demonstrated community service, clinical nuanced. examinations and in other measures of and research experiences, and personal academic success between URM and non- competencies.24,25 In evaluating applicants whose MCAT URM students with similar aptitudes. scores fall within the range of scores These unequal opportunities and the Seeking a quick solution, some have that predict success in medical school, resulting differences in achievement have admissions committees must consider called for eliminating the MCAT exam their roots in structural racism. 26–28 other important characteristics that or reporting scores as pass or fail. suggest that a given applicant will Eliminating the exam would prevent contribute to the workforce that is medical schools from using the scores to Structural Racism and Unequal needed by communities across the ensure that applicants have demonstrated Opportunity country. Focusing on workforce needs, sufficient achievement to be ready for Centuries of structural and interpersonal admissions committees may justifiably medical school. A pass–fail scoring racism and bias have contributed to racial select applicants with MCAT scores at system would hinder the work of schools and ethnic disparities in wealth, health, the lower end of the range predicting that employ holistic review admissions and educational opportunity. While overt success in medical school because, for processes to admit applicants with a wide discrimination against people of color example, their rural backgrounds increase range of MCAT scores, including scores was outlawed in the 1960s, hundreds the likelihood that they will practice that would fall below a national pass–fail of years of legalized discrimination in an underserved area, because they cutoff.16 Instead, leaders in medical before that time created the conditions speak the language and understand the education must work to understand the in which minority populations remain culture of a major demographic group root causes of group differences in MCAT significantly disadvantaged, even today.29 in the United States, or because their scores and propose, pilot, and disseminate When first introduced, government LGBTQ+ identity adds an important appropriate countermeasures. programs, such as Social Security,30,31 the perspective to the educational and health Federal Housing Administration loan care environments. Additionally, while program,32 and the GI Bill, implicitly or the MCAT exam is designed to measure Why Group Differences Exist explicitly prevented minority populations 33,34 applicants’ academic preparation for The recognition that mean MCAT scores from receiving benefits, causing medical school, it is not designed to differ between white, black, and Latinx them to endure substantial, sustained 11,35,36 measure or predict their performance populations has led many to view the economic disadvantage. These related to other, essential competencies, exam as intrinsically biased.27 However, and other programs also promoted such as interpersonal skills and psychometric studies show that this residential segregation and prevented home ownership among people of color, communication, professionalism, and is not the case. Psychometric validity ethical behavior, or to take the place of concentrating minority populations exists when a high-stakes exam neither other attributes that nonexam aspects of in low-income neighborhoods with over- nor underpredicts the subsequent the admissions process evaluate. inadequate access to quality housing, performance of different examinee economic/occupational opportunity, populations. Scores from the old MCAT health care, fresh food, quality schools, Group Differences in MCAT Scores exam did not show bias against black and and public safety.11,35 In addition, the Individuals of every race and ethnicity Latinx medical students in predicting criminal justice system continues obtain scores from the low, middle, their success in medical school and on to produce disparate outcomes for and high ranges of the MCAT score licensing exams (i.e., the success rates of people of color, contributing to family scale, but mean scores are lower for students from these races/ethnicities were fragmentation, poverty, and diminished applicants from racial and ethnic not higher than their scores predicted).20 employment opportunities for previously groups underrepresented in medicine Early research on the new exam shows incarcerated individuals.37 (URM) compared with mean scores that new scores predict success in the first for their peers who are from groups year of medical school comparably for These and other discriminatory practices well represented in medicine. These examinees from URM and non-URM have negatively affected economic success group differences on the new MCAT backgrounds.14 in minority populations. In 2016, the
352 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 37 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
median level of wealth for white families shadowing, and other experiences. unequal opportunity on the diversity of was $171,000, while the median wealth Minority students from lower-resourced the nation’s physician workforce using for black families was $17,600. Data colleges and universities may have less 3 critical levers: admissions processes, from 2015 show that 3 times as many access to the necessary prerequisites for pipeline programs, and curriculum. children of color as white children live in medical school, academically beneficial households below the poverty level.38,39 experiences such as research projects, Admissions processes: Use MCAT scores or experienced and accessible health wisely The negative impact of these social professions career advisors.49,50 Medical school leaders must instruct and systems and practices on educational support their admissions committees opportunity is striking. Because the While economics are important, a high to understand and use MCAT scores major source of local government socioeconomic status does not protect appropriately, eschewing the use of such funding for public schools comes from students of color from the negative scores for anything other than identifying property taxes, housing inequality leads effects of structural and interpersonal the achievement level that students to educational inequality.40 Supplemental racism.51 Studies have documented that need to succeed in their institutions. Digital Appendix 1 available at http:// even at low-poverty schools, discipline The MCAT exam enables every medical links.lww.com/ACADMED/A734 includes in K–12 education is more frequent school to identify applicants whose 2 illustrations of the persistent impact and severe for children of color than for current level of achievement may be too of 20th-century discriminatory housing white children,52,53 leading to interrupted low to succeed in their school. Beyond policies on 21st-century educational or terminated school experiences. that cutoff, selecting students based opportunity.41,42 Panel A shows a 1935 Opportunities to participate in gifted on small differences in scores is not map of redlined communities in San and talented programs are more often supported by the data on the reliability Francisco, California, that were deemed denied to minority students in school of the exam.16 Despite this psychometric hazardous for mortgage lending because systems that do not employ sound evidence, admissions officers describe of their high concentration of black selection procedures, a finding that pressure from institutional stakeholders residents.41 Panel B shows a map of low- persists across socioeconomic levels.54 to select students with the highest scores quality schools today concentrated in the Even minority students from middle or because the ranking of the medical school same geographic areas as the redlined high socioeconomic levels can experience depends in part on the mean MCAT score communities in 1935.42 the negative effects of low expectations, of the matriculating class.57 denying them the encouragement Across the nation, black and Latinx and support to pursue educational Medical schools that have assembled children are more likely than white opportunities beyond high school.55,56 classes of capable, diverse students use children to live in poverty, experience several strategies. First, they identify the food insecurity, reside in single-parent These and many other examples of full range of MCAT scores associated households, and grow up in families structural and interpersonal racism with success at their school. Then, where no parent has full-time, year- underpin the observed group differences they consider each applicant’s score in round employment.20 Minority children between URM and non-URM students in context, recognizing that a history of also are more likely to attend low-quality academic achievement and in scores on multiple adverse educational experiences day care and show elevated blood lead high-stakes exams. As we have explained, related to race or ethnicity may lead to levels.20 Black and Latinx students are social, economic, employment, health, scores that are lower than those of other more likely to attend schools with high and criminal justice challenges all applicants but still predictive of success.58 teacher turnover, inexperienced teachers, negatively affect students’ achievement. Furthermore, these schools build a and teachers who are not certified in Not all URM students experience all learning environment in which the the subjects they teach.20,43 In addition, of these trials, but most experience at obstacles to achievement that may have unstable parental employment may least some of them. Despite multiple existed for their students before entry require children to change schools barriers to success, many aspiring medical into medical school are highly attenuated more frequently.20 Black and Latinx students from URM groups demonstrate or eliminated. students are more likely than Asian or substantial achievement, earning MCAT white students to attend a high-poverty scores that are within the range that The University of California, San school44 and to report the presence predicts success in medical school.16 Francisco, School of Medicine, for of gangs in school.45 High schools in Equitable interpretation of MCAT scores example, published data documenting low-income areas are much less likely to requires consideration of the context in that the gap in standardized exam scores offer advanced placement coursework or which each applicant earned those scores, between URM and non-URM students skilled college advisors.46,47 rather than assuming that all applicants narrows at each stage of medical school, had equal opportunities. suggesting that a supportive learning These unequal educational opportunities environment may help URM students continue into college. Lack of family achieve success at a faster rate than their wealth may lead minority students to What Can Be Done? non-URM peers.59 Morehouse School begin their college education in the Addressing this opportunity gap is of Medicine also reported achievement community college system48 and to work daunting for medical educators, but it in fostering the success of students who during school, leaving them with less is not impossible. Leaders in medical entered the school with a wide range of time for studying, unpaid internships, education can address the impact of MCAT scores.60 The success of holistic
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 353 38 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
review in diversifying medical school in preparing undergraduate students to a diverse workforce through holistic classes has led residency programs to be strong, academically qualified medical admissions and help them to develop adopt a similar approach to selecting school applicants; however, such advisors the expertise they need. All medical interns, recognizing that excess weighting are not equitably distributed across the schools must train the next generation of scores on the standardized, high- nation’s higher education institutions. of physicians to understand the existence stakes USMLE Step exams interferes Underresourced institutions are less likely and extensive ramifications of structural with the goal of diversifying residency to provide institutional or financial support racism and the resultant health and programs.61,62 for premedical advising.66 Recognizing the health care disparities. Additionally, value of high-quality health professions recognizing how structural racism leads Achieving greater diversity through advisors, the National Association of to interpersonal bias may help physicians admissions requires making changes Advisors for the Health Professions offers address their own personal biases that to the whole admissions process. advising services free of charge to students contribute to health care inequities.73 Medical schools are exploring multiple from colleges that do not have dedicated Diversifying medical school classes is a strategies for achieving this aim, for health professions advisors.67 critical first step in educating physicians example, employing anonymous voting to work effectively with individuals systems; blinding interviewers to Many medical schools have developed from all populations. Schools also must academic metrics; and using multiple partnerships with communities and establish core competencies related mini-interviews and scoring rubrics schools across the educational continuum to understanding structural racism to give equal weight to experiences, to support URM and socioeconomically and its influence on health and health attributes, and academic metrics. In disadvantaged students interested in care disparities. In addition, structural addition, ensuring diversity in the health professions careers. The success competence and antiracism curricula composition of admissions committees of these endeavors depends on a broad- should be introduced in undergraduate and encouraging admissions committee based institutional commitment to and graduate medical education.74–76 members to complete training in diversity that is sustained with financial mitigating unconscious bias can help support and based in respectful Conclusions them make judgments about academic community engagement. For example, and professional promise given each the University of Illinois at Chicago Structural racism is the result of applicant’s unique context.25,63 Urban Health Program includes centuries of discrimination against initiatives that span elementary school people of color in the United States. Pipeline programs: Enhance through undergraduate education and Its roots are deep and its consequences opportunities for applicants to prepare provides access to and preparation for far-reaching. Medical education and the for medical school all health professions; it is aimed at the profession of medicine are as affected Medical schools and national medical needs of URM students, many of whom by this stain as other social systems are. education organizations must redouble attend underresourced schools.68 Medical The medical profession can successfully their efforts to address the proximate schools also sponsor programs that educate the diverse physician workforce barriers to success for URM students enable prehealth advisors from nearby that our communities need and prepare aspiring to become physicians. Success on undergraduate institutions to learn more all physicians to be the citizens our the MCAT exam requires exam-specific about the medical school admissions democracy needs if we collectively preparation in addition to high-quality process and MCAT exam preparation.69–71 commit to understanding and counteracting the impact of structural higher education. The Association of Other schools embrace their role as racism on medical student selection and American Medical Colleges (AAMC) an anchor institution, leveraging their education and on the provision of health has worked to decrease barriers to test system’s procurement, investment, and care. Embracing new ideas about what preparation by providing free study employment opportunities to improve MCAT scores are desirable may be more guides and tools and reduced-price the educational and economic milieu for acceptable if the purpose behind this registration for applicants with financial those in the surrounding community, necessary mindset change is to mitigate hardship.64 Additionally, the AAMC has including those aspiring to health 72 the effects of society’s structural racism. collaborated with the Khan Academy and professions careers. the Robert Wood Johnson Foundation to Acknowledgments: The authors acknowledge the provide a collection of free, online, video- Curriculum: Educate physician–citizens dedication and contributions of the Medical based tutorials on the topics covered on As a profession and as individuals, College Admission Test Validity Committee the MCAT exam.65 Unfortunately, use of physicians are trusted for their ability to and the following Association of American Medical Colleges staff: Cynthia Searcy, Karen these test preparation resources is lower think critically and advise individuals Mitchell, Lesley Ward, and Jordan Yee Prendez among premedical students from lower and communities about threats to health. (psychometric intern). They thank University socioeconomic backgrounds and those No greater threat to health exists today of California, San Francisco, School of Medicine who attend lower-resourced schools.21 than the disparities in our social systems, students Jazzmin Williams and Laeesha Corneo which shorten lives, obstruct access to for publicizing the maps of San Francisco that These students also may not have ready evidence-based health care, impoverish illustrate the persistent impact of structural access to knowledgeable individuals families, incarcerate generations, and racism on educational quality. In addition, they acknowledge the tremendous efforts who can advise them on best practices attenuate educational achievement. of all medical school leaders, faculty, and to prepare for the MCAT exam. Health The next generation of physicians can administrators who are working to diversify the professions advisors play an important role take on this work if we are able to build physician workforce.
354 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 39 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
Funding/Support: None reported. proficiency Latinos with type 2 diabetes. A legal perspective. Ann Intern Med. JAMA Intern Med. 2017;177:380–387. 2018;169:403–404. Other disclosures: The authors cochair the 11 Bailey ZD, Krieger N, Agénor M, Graves J, 28 Sturman N, Parker M. The elusive grail of Association of American Medical Colleges Medical Linos N, Bassett MT. Structural racism and social inclusion in medical selection. Med College Admission Test Validity Committee. They health inequities in the USA: Evidence and Educ. 2013;47:542–544. receive no compensation for this work. interventions. Lancet. 2017;389:1453–1463. 29 Pager D, Shepherd H. The sociology of 12 Julian ER. Validity of the Medical College discrimination: Racial discrimination in Ethical approval: Reported as not applicable. Admission Test for predicting medical school employment, housing, credit, and consumer performance. Acad Med. 2005;80:910–917. markets. Annu Rev Sociol. 2008;34:181–209. Disclaimers: The views expressed in this article 13 Dunleavy DM, Kroopnick MH, Dowd KW, 30 United States Code: Social Security Act, 42 do not necessarily reflect the views of the Searcy CA, Zhao X. The predictive validity USC. §§ 301–1305 (suppl. 4 1934). Uniformed Services University, the US Army, or of the MCAT exam in relation to academic 31 Bureau of the Census. Fifteenth Census of the the Department of Defense. performance through medical school: United States: 1930. Population: Volume IV. A national cohort study of 2001–2004 Occupations by states. https://www.census. C.R. Lucey is executive vice dean, vice dean for matriculants. Acad Med. 2013;88:666–671. gov/library/publications/1933/dec/1930a- education, and professor of medicine, University of 14 Busche K, Elks ML, Hanson JT, et al. The vol-04-occupations.html. Published 1933. California, San Francisco, School of Medicine, San validity of scores from the new MCAT Accessed July 18, 2019. Francisco, California. exam in predicting student performance: 32 Logan JR. The persistence of segregation Results from a multisite study. Acad Med. in the 21st century metropolis [published A. Saguil was associate dean for recruitment 2020;95:387–395. online ahead of print June 17, 2013]. City and admissions, Uniformed Services University 15 Association of American Medical Colleges. Community. doi:10.1111/cico.12021 of the Health Sciences, F. Edward Hébert School Using Scores From the Old MCAT Exam in 33 Turner SE, Bound J. Closing the gap or of Medicine, Bethesda, Maryland, at the time Medical Student Selection. Washington, DC: widening the divide: The effects of the G.I. this work was completed. He is associate dean, Association of American Medical Colleges; Bill and World War II on the educational regional education, Uniformed Services University 2016. outcomes of black Americans. J Econ Hist. of the Health Sciences, F. Edward Hébert School 16 Association of American Medical Colleges. 2013;63:145–177. of Medicine, San Antonio, Texas, now. He is also vice chair, Medical College Admission Test Validity Using MCAT Data in 2020 Medical Student 34 Herbold H. Never a level playing field: Blacks Committee, Association of American Medical Selection. Washington, DC: Association of and the G.I. Bill. J Blacks High Educ. Winter Colleges, Washington, DC. American Medical Colleges; 2019. https:// 1994–1995:104–108. www.aamc.org/download/498250/data/ 35 Hahn RA, Truman BI, Williams DR. Civil rights usingmcatdatain2020medstudentselection. as determinants of public health and racial and pdf. Accessed July 24, 2019. ethnic health equity: Health care, education, References 17 Regents of the University of California v employment, and housing in the United States. 1 Labaree DF. Public goods, private goods: The Bakke, 438 US 265 (1978). SSM Popul Health. 2018;4:17–24. American struggle over educational goals. 18 Fisher v University of Texas, 579 US __ (2016). 36 Aliprantis D, Carroll D. What is behind Am Educ Res J. 1997;34:39–81. 19 Students for Fair Admissions, Inc v President the persistence of the racial wealth gap? 2 Bhutta ZA, Chen L, Cohen J, et al. Education and Fellows of Harvard Coll. (Harvard Federal Reserve Bank of Cleveland website. of health professionals for the 21st century: Corp.) 261 F supp 3d 99 (D Mass 2017). https://www.clevelandfed.org/newsroom- A global independent Commission. Lancet. 20 Davis D, Dorsey JK, Franks RD, Sackett PR, and-events/publications/economic- 2010;375:1137–1138. Searcy CA, Zhao X. Do racial and ethnic commentary/2019-economic-commentaries/ 3 Boelen C, Woollard B. Social accountability group differences in performance on the ec-201903-what-is-behind-the-persistence- and accreditation: A new frontier for MCAT exam reflect test bias? Acad Med. of-the-racial-wealth-gap.aspx. Published educational institutions. Med Educ. 2013;88:593–602. February 28, 2019. Accessed July 18, 2019. 2009;43:887–894. 21 Girotti JA, Chanatry JA, Clinchot DM, et al. 37 The Sentencing Project. Report to the 4 Hung R, McClendon J, Henderson A, Evans Investigating group differences in examinees’ United Nations on racial disparities in the Y, Colquitt R, Saha S. Student perspectives preparation for and performance on the new U.S. criminal justice system. https://www. on diversity and the cultural climate at a U.S. MCAT exam. Acad Med. 2020;95:365–374. sentencingproject.org/publications/un- medical school. Acad Med. 2007;82:184–192. 22 Camara WJ, Schmidt AE. Group Differences in report-on-racial-disparities. Published April 5 McGee R Jr, Saran S, Krulwich TA. Diversity Standardized Testing and Social Stratification. 19, 2018. Accessed July 18, 2019. in the biomedical research workforce: New York, NY: College Entrance Examination 38 Dettling LJ, Hsu JW, Jacobs L, Moore KB, Developing talent. Mt Sinai J Med. Board; 1999. Report no. 99-5. Thompson JP. Recent trends in wealth- 2012;79:397–411. 23 Sackett PR, Shen W. Subgroup differences holding by race and ethnicity: Evidence 6 Whitla DK, Orfield G, Silen W, Teperow C, on cognitive tests in contexts other than from the survey of consumer finances. Howard C, Reede J. Educational benefits personnel selection. In: Outtz JL, ed. Adverse FEDS Notes. https://www.federalreserve. of diversity in medical school: A survey of Impact: Implications for Organizational gov/econres/notes/feds-notes/recent-trends- students. Acad Med. 2003;78:460–466. Staffing and High Stakes Selection. New York, in-wealth-holding-by-race-and-ethnicity- 7 Morrison E, Grbic D. Dimensions of diversity NY: Routledge/Taylor & Francis Group; 2010. evidence-from-the-survey-of-consumer- and perception of having learned from 24 Ballejos MP, Rhyne RL, Parkes J. Increasing finances-20170927.htm. Published 2017. individuals from different backgrounds: The the relative weight of noncognitive admission Accessed July 18, 2019. particular importance of racial diversity. criteria improves underrepresented minority 39 Patten E, Krogstad JM. Black child poverty Acad Med. 2015;90:937–945. admission rates to medical school. Teach rate holds steady, even as other groups see 8 Saha S, Guiton G, Wimmers PF, Wilkerson L. Learn Med. 2015;27:155–162. declines. Pew Research Center website. http:// Student body racial and ethnic composition 25 Terregino CA, McConnell M, Reiter HI. The www.pewresearch.org/fact-tank/2015/07/14/ and diversity-related outcomes in US medical effect of differential weighting of academics, Black-child-poverty-rate-holds-steady-even- schools. JAMA. 2008;300:1135–1145. experiences, and competencies measured by as-other-groups-see-declines. Published 9 Dunlap JL, Jaramillo JD, Koppolu R, Wright multiple mini interview (MMI) on race and 2015. Accessed July 18, 2019. R, Mendoza F, Bruzoni M. The effects ethnicity of cohorts accepted to one medical 40 Reschovsky A. The future of U.S. public of language concordant care on patient school. Acad Med. 2015;90:1651–1657. school revenue from property tax. Lincoln satisfaction and clinical understanding for 26 Muller D, Kase N. Challenging traditional Institute of Land Policy website. https://www. Hispanic pediatric surgery patients. J Pediatr premedical requirements as predictors of lincolninst.edu/publications/policy-briefs/ Surg. 2015;50:1586–1589. success in medical school: The Mount Sinai future-us-public-school-revenue-property- 10 Parker MM, Fernández A, Moffet HH, Grant School of Medicine Humanities and Medicine tax. Published 2017. Accessed July 18, 2019. RW, Torreblanca A, Karter AJ. Association Program. Acad Med. 2010;85:1378–1383. 41 Nelson RK, Winling L, Marciano R, et al. of patient-physician language concordance 27 Genao I, Gelman J. The MCAT’s restrictive Mapping inequality. Redlining in New Deal and glycemic control for limited-English effect on the minority physician pipeline: America. In: Nelson RK, Ayers EL, eds.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 355 40 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Perspective
American Panorama. https://dsl.richmond. Professions in Honor of Herbert W. Nickens, medical school admissions. J Health Care edu/panorama/redlining/#loc=13/37.7575/- M.D. Washington, DC: National Academies Poor Underserved. 2018;29:9–18. 122.4364&opacity=0.8&city=san-francisco- Press; 2001. 64 Association of American Medical Colleges. ca. Accessed July 18, 2019. 51 Kohli R, Pizarro M, Nevárez A. The AAMC Fee Assistance Program. www.aamc. 42 San Francisco, CA. GreatSchools.org. https:// “new racism” of K–12 schools: Centering org/fap. Accessed July 25, 2019. www.greatschools.org/california/san- critical research on racism. Rev Res Educ. 65 Khan Academy. MCAT test prep. https:// francisco/schools. Accessed February 2017;41:182–202. www.khanacademy.org/test-prep/mcat. 26, 2019. 52 U.S. Department of Education Office for Accessed July 25, 2019. 43 U.S. Department of Education Office for Civil Rights. Civil rights data collection: 66 Oyewole SH. Sustaining minorities in Civil Rights. Civil rights data collection: Data Data snapshot: School discipline. https:// prehealth advising programs: Challenges snapshot: Teacher equity. https://www2. ocrdata.ed.gov/Downloads/CRDC-School- and strategies for success. In: Smedley BD, ed.gov/about/offices/list/ocr/docs/crdc- Discipline-Snapshot.pdf. Published 2014. Stith AY, Colburn L, Evans CH, eds. The teacher-equity-snapshot.pdf. Published 2014. Accessed July 18, 2019. Right Thing to Do, The Smart Thing to Accessed July 18, 2019. 53 U.S. Government Accountability Office. Do: Enhancing Diversity in the Health 44 Orfield G, Chungmei L. Why segregation K–12 education: Discipline disparities for Professions: Summary of the Symposium on matters: Poverty and educational black students, boys, and students with Diversity in Health Professions in Honor of inequality. The Civil Rights Project. https:// disabilities. https://www.gao.gov/products/ Herbert W. Nickens, M.D. Washington, DC: civilrightsproject.ucla.edu/research/k- GAO-18–258. Published 2018. Accessed July National Academies Press; 2001. 12-education/integration-and-diversity/ 18, 2019. 67 National Association of Advisors for the why-segregation-matters-poverty-and- 54 Grissom JA, Redding C. Discretion Health Professions. Find an advisor. https:// educational-inequality. Published 2005. and disproportionality: Explaining the www.naahp.org/student-resources/find-an- Accessed July 18, 2019. underrepresentation of high-achieving advisor. Accessed July 25, 2019. 45 Musu-Gillette L, De Brey C, McFarland J, students of color in gifted programs. AERA 68 Toney M. The long, winding road: One Hussar W, Sonnenberg W, Wilkinson-Flicker Open. 2016;2:1–25. university’s quest for minority health care S. Status and trends in the education of racial 55 Allen Q. Racial microaggressions: The professionals and services. Acad Med. and ethnic groups 2017. National Center for schooling experiences of black middle-class 2012;87:1556–1561. Education Statistics website. https://nces. males in Arizona’s secondary schools. J Afr 69 Morehouse School of Medicine. Morehouse ed.gov/pubs2017/2017051.pdf. Published Am Males Educ. 2010;1:125–143. School of Medicine Pre-Medical Faculty 2017. Accessed July 18, 2019. 56 Gershenson S, Papageorge N. The power of Summit 2018: Developing strategies for a 46 Handwerk P, Tognatta N, Coley RJ, Gitomer teacher expectations: How racial bias hinders diverse health professions workforce. https:// DH, eds. Access to Success: Patterns of student attainment. Education Next. Winter www.msm.edu/events/2018/pre-medical- Advanced Placement Participation in U.S. 2018;18. https://www.educationnext.org/ faculty-summit. Published 2018. Accessed High Schools. Princeton, NJ: Educational power-of-teacher-expectations-racial-bias- July 18, 2019. Testing Service; 2008. hinders-student-attainment. Accessed July 70 Illinois Medical School Admissions 47 The Executive Office of the President. 18, 2019. Consortium. Medical school admissions Increasing college opportunity for low- 57 Association of American Medical Colleges. workshop, University of Illinois at Chicago; income students: Promising models and 2017 admissions officer survey. April 2017. October 31, 2014; Chicago, IL. a call to action. https://obamawhitehouse. Unpublished data. 71 University of South Alabama. Alabama archives.gov/sites/default/files/docs/ 58 Terregino CA, Saguil A, Price-Johnson T, Health Professions Advisor Conference increasing_college_opportunity_for_low- Anachebe NF, Goodell K. The diversity and website. https://www.southalabama.edu/ income_students_report.pdf. Published 2014. success of medical school applicants with departments/academicsuccess/pre-health/ Accessed July 18, 2019. scores in the middle third of the MCAT score al_health_conference.html. Published 2018. 48 Gandara P. Lost opportunities: The scale. Acad Med. 2020;95:344–350. Accessed July 18, 2019. difficult journey to higher education for 59 Teherani A, Hauer KE, Fernandez A, King 72 Vick AD, Baugh A, Lambert J, et al. Levers underrepresented minority students. In: TE Jr, Lucey C. How small differences in of change: A review of contemporary Smedley BD, Stith AY, Colburn L, Evans CH, assessed clinical performance amplify to large interventions to enhance diversity in medical eds. The Right Thing to Do, The Smart Thing differences in grades and awards: A cascade schools in the USA. Adv Med Educ Pract. to Do: Enhancing Diversity in the Health with serious consequences for students 2018;9:53–61. Professions: Summary of the Symposium on underrepresented in medicine. Acad Med. 73 Matthew DB. Just Medicine: A Cure for Diversity in Health Professions in Honor of 2018;93:1286–1292. Racial Inequality in American Health Care. Herbert W. Nickens, M.D. Washington, DC: 60 Elks ML, Herbert-Carter J, Smith M, Klement New York, NY: NYU Press; 2015. National Academies Press; 2001. B, Knight BB, Anachebe NF. Shifting the 74 Metzl JM, Hansen H. Structural competency: 49 Carnevale AP, Strohl J. Separate & Unequal: curve: Fostering academic success in a diverse Theorizing a new medical engagement How Higher Education Reinforces the student body. Acad Med. 2018;93:66–70. with stigma and inequality. Soc Sci Med. Intergenerational Reproduction of 61 Spector AR, Railey KM. Reducing reliance on 2014;103:126–133. White Racial Privilege. Washington, DC: test scores reduces racial bias in neurology 75 Hardeman RR, Burgess D, Murphy K, et al. Georgetown Public Policy Institute; 2013. residency recruitment [published online Developing a medical school curriculum 50 Smedley BD, Stith AY, Colburn L, Evans ahead of print March 29, 2019]. J Natl Med on racism: Multidisciplinary, multiracial CH. The right thing to do, the smart thing Assoc. doi:10.1016/j.jnma.2019.03.004 conversations informed by Public Health to do: Enhancing diversity in the health 62 Aibana O, Swails JL, Flores RJ, Love L. Critical Race Praxis (PHCRP). Ethn Dis. professions. In: Smedley BD, Stith AY, Bridging the gap: Holistic review to increase 2018;28(suppl 1):271–278. Colburn L, Evans CH, eds. The Right Thing diversity in graduate medical education. Acad 76 Gard LA, Peterson J, Miller C, et al. Social to Do, The Smart Thing to Do: Enhancing Med. 2019;94:1137–1141. determinants of health training in U.S. Diversity in the Health Professions: Summary 63 Capers Q, McDougle L, Clinchot DM. primary care residency programs: A scoping of the Symposium on Diversity in Health Strategies for achieving diversity through review. Acad Med. 2019;94:135–143.
356 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 41 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Invited Commentary
Leveraging the Medical School Admissions Process to Foster a Smart, Humanistic, and Diverse Physician Workforce Richard M. Schwartzstein, MD
Abstract An excellent physician has a range of who have demonstrated proficiency importance of intellectual rigor and, talents, including the knowledge and in these areas. With these changes, while pursuing goals for a more diverse critical thinking abilities to work with the scores continue to be predictive of physician workforce, maintain standards rapidly changing biomedical and social student performance in the preclerkship that ensure medicine’s commitment science content of the profession as well curriculum. to patients. The author concludes as the interpersonal and communication with suggestions for how educators skills to build meaningful relationships In this Invited Commentary, the author can work with under-resourced with patients and families. The Medical argues that, as educators struggle colleges and premedical programs to College Admission Test (MCAT) was to define the characteristics of the help disadvantaged students get the revised in 2015 to focus more on “right” candidates for medical school preparation they need to succeed in analytical reasoning skills and behavioral and design processes to identify and medical school and throughout their and social sciences knowledge to admit those applicants, it is important careers. Taking these steps will allow ensure that future physicians have the to consider the message being sent educators to support students, prepare capabilities needed to care for patients in by calls for the MCAT exam to play a them for practice, and fulfill their the 21st century and to allow admissions reduced role in admissions decisions. obligation to the public to produce committees to identify applicants Educators must avoid diminishing the excellent physicians.
Editor’s Note: This is an Invited Commentary use of the new MCAT exam is intertwined As educators struggle to define the on Lucey CR, Saguil A. The consequences of with questions about who should and will characteristics of the “right” candidates structural racism on MCAT scores and medical be the doctors of tomorrow, particularly for medical school and design processes school admissions: The past is prologue. Acad Med. as arguments grow about the importance to identify and admit those applicants, 2020;95:351–356; Terregino CA, Saguil A, Price- of a diverse physician workforce for the we should consider the message we may Johnson T, Anachebe NF, Goodell K. The diversity health of our population. To answer be sending inadvertently by calling for and success of medical school applicants with scores these questions, the medical education the MCAT exam to play a reduced role in in the middle third of the MCAT score scale. Acad community has focused largely on the admissions decisions. Med. 2020;95:344–350; Girotti JA, Chanatry JA, medical school admissions process to Clinchot DM, et al. Investigating group differences identify the “right” students. The Role of the MCAT Exam in in examinees’ preparation for and performance on Admissions the new MCAT exam. Acad Med. 2020;95:365–374; In this issue, Busche and colleagues and Busche K, Elks ML, Hanson JT, et al. The offer evidence that the new MCAT exam Several years ago, I had the good fortune validity of scores from the new MCAT exam in remains a valid predictor of performance to serve on the 5th Comprehensive predicting student performance: Results from a in the preclerkship curriculum and that Review of the MCAT Review Committee multisite study. Acad Med. 2020;95:387–395. it is not systematically biased against (MR5 Committee), convened by the any racial or socioeconomic groups.1 Association of American Medical Students who do well on the new MCAT Colleges (AAMC) to help guide the Performance on the Medical College exam are as well or better prepared for a development of the new MCAT exam. Admission Test (MCAT) is a significant career as a physician as those who took The committee was a large group factor in the decision to admit an applicant the previous version of the exam, and that included faculty and premedical to medical school; thus, the validity and they are likely to reason better and can advisors from colleges and medical apply knowledge of important concepts schools across the United States. Please see the end of this article for information in the social and behavioral sciences. Both public and private institutions about the author. However, whether due to less access to were represented. Over the course of 2 Correspondence should be addressed to Richard M. preparation materials or inequalities in the 3 years that the committee met, Schwartzstein, Beth Israel Deaconess Medical Center, their premedical education,3 students our discussions were wide ranging, 330 Brookline Ave., Boston, MA 02215; telephone: from lower socioeconomic groups do earnest, and reflective of the competing (617) 667-5494; email: [email protected]. not perform as well on the new MCAT priorities of the members.5 We wanted Acad Med. 2020;95:333–335. exam, which has led some to advocate future doctors with intellectual rigor, First published online November 19, 2019 that performance on the MCAT exam is strong critical thinking skills, good doi: 10.1097/ACM.0000000000003091 Copyright © 2019 by the Association of American given too much weight in the admissions communication skills, and who were Medical Colleges process.3,4 humanists.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 333 42 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges XXX Invited Commentary
The committee recommended that the potentially disadvantaging students from 400 patients whom I admitted to the MCAT exam be changed to emphasize lesser known, under-resourced colleges. hospital. In contrast, a typical intern critical thinking skills (e.g., the ability to With the assistance of researchers at the today personally admits fewer than 100 analyze and interpret data) rather than AAMC, the MR5 Committee modeled the patients. The failure of educators to factual recall. In addition, a section on impact of changing the MCAT exam to consider the importance of experience for the behavioral and social sciences was pass/fail. This analysis demonstrated that their learners has led to concerns about added to signal that being an excellent many students from lower socioeconomic the true competence and autonomy of doctor requires more than knowledge groups who were admitted to medical residency graduates.10,11 of the biological and chemical sciences. school in the present system of reporting In retrospect, I think we spent less time actual MCAT scores would have been Entrustable professional activities (EPAs) than we should have on other important excluded in a pass/fail system. Consistent theoretically would set the bar higher traits like curiosity and resilience (or with this observation, Terregino and than “competence,” but a recent review “grit”), as curiosity about the patient as colleagues found that schools that of the literature revealed that the data do a person and about the pathophysiology accepted more applicants with midrange not yet support the use of this approach of her problem is one way to bridge MCAT scores had more diverse as an assessment tool.12 The inter-rater differences between patients and doctors matriculating classes.4 These findings reliability for determining entrustability and to minimize cognitive bias and suggest that the holistic admissions is fair at best.13 Frame of reference error.6 Becoming and being an excellent process works. training,14 which requires a major doctor is challenging intellectually, investment in time and resources for a emotionally, and physically; if passion Graduation rates for these students medical school with many faculty, may and perseverance are the essence of grit,7 remain high. But when only 2% of be necessary to truly attain validity in we surely need those traits in our doctors. students fail to complete medical school EPA decisions. Furthermore, few faculty Of course, there are limits to what one for academic reasons,8 we must also members are willing to commit to stating can learn from a standardized exam score, be sure that we have not lowered our that a student cannot proceed to the next yet holistic admissions processes are expectations and standards during phase of the curriculum because of EPA designed in large part to help us discern medical school while arguing for less ratings. the presence of these characteristics in emphasis on MCAT performance when applicants. making admissions decisions. From my experience and what I have heard from colleagues around the Nevertheless, the MCAT exam is the country, it is difficult to slow down, if starting point. Scores are predictive of Is Competence Enough? not stop, a student’s progression through students’ performance in the first phase Over the last 2 decades, the idea of medical school because of academic of medical school and of their ability admitting applicants who are merely performance issues. A former chair of to learn, digest, and apply the scientific “smart enough” for medicine has the Harvard Medical School promotions principles needed for the practice of emerged. Competency-based medical and review board observed that faculty medicine. However, the premedical education (CBME) is designed to ensure view their students like their patients, curricula available at colleges likely that students do not advance through that is, they never abandon them, affects how applicants perform on medical school unless they are competent; and multiple students over the years the MCAT exam, so individuals from however, “competent” generally has been have described their medical school lower socioeconomic groups and those defined as the bare minimum required experience not as “pass/fail” but as “pass/ attending under-resourced colleges may for acceptability. While the original pass.” Failure is not truly a grading well be disadvantaged when they take notion of CBME was that students would option for most faculty, and students the MCAT exam. Because some medical remain in place until they achieved seem to know this. There are invariably schools place a high value on MCAT competence (or were counseled to pursue mitigating reasons for the poor scores, the exam has been perceived to be another profession if they failed to performance, claims that educators will a barrier to achieving greater diversity in improve after remediation), today much address the deficits in the next phase of the physician workforce.3,4 How then do greater attention is devoted to shortening the curriculum, or concerns about how we determine what role the MCAT exam training by moving students along the student will perceive the setback. Too should play in the admissions process? as soon as they achieve competence.9 often I have heard that deficiencies will Less attention is given to the fact that be addressed by the student’s residency Some members of the MR5 Committee competence is context specific and program. advocated that the new MCAT exam be that experience (i.e., seeing many pass/fail. Others pointed out that further patients under varying conditions) is an While much has been written about distinctions among students are needed important element in the development of “productive failure,” we as educators since there are more students applying meaningful competence. have become intensely protective of our than there are slots available at medical students.15 We do not allow them to fail. schools. If admissions committees cannot The implications of having fewer medical Additionally, if we move a student along assess applicants’ academic performance school experiences, if students move for several years before concluding that using their MCAT scores, they will through the curriculum more quickly, a career in medicine is not right for that default to using other measures, such as are compounded by the changing person, we are reluctant to remove her grade point average and the rigor of the environment at the residency level. As from school because she may be saddled applicant’s undergraduate institution, a medical intern, I evaluated more than with significant debt and no degree.
334 Academic Medicine,MCAT® Vol. is a program95, No. of3 /the March 2020 43 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges Invited Commentary
As educators, we are in a unique position. that the social determinants of health R.M. Schwartzstein is professor of medicine, We have a responsibility to our students lead to untold misery, yet we invest in Harvard Medical School, Boston, Massachusetts. to support and prepare them for practice, treatments for the resulting diseases but we also have a responsibility to the rather than address the root causes of public to produce excellent doctors. the problem. The MCAT exam is an References We know that MCAT scores predict accurate assessment of an applicant’s 1 Busche K, Elks ML, Hanson JT, et al. The performance in the early years of medical preparation in the foundational validity of scores from the new MCAT 1 exam in predicting student performance: school. We should not discount their role principles of the biological, chemical, Results from a multisite study. Acad Med. in identifying who those excellent doctors and social sciences needed for the 2020:95:387–395. may be. study of medicine. If as a community 2 Girotti JA, Chanatry JA, Clinchot DM, et al. our efforts to foster a more diverse Investigating group differences in examinees’ physician workforce are being thwarted preparation for and performance on the new An Alternative Solution MCAT exam. Acad Med. 2020:95:365–374. by socioeconomic factors that impair the 3 Lucey CR, Saguil A. The consequences of About 15 years ago, as discussions ability of talented students to compete structural racism on MCAT scores and about the role of MCAT scores in the effectively, we should address the true medical school admissions: The past is admissions process became more acute, cause of the problem (e.g., inadequate prologue. Acad Med. 2020:95:351–356. 4 Terregino CA, Saguil A, Price-Johnson T, I remember hearing what I thought preparation in high school and college Anachebe NF, Goodell K. The diversity and was a perplexing argument at national to handle the intellectual challenges success of medical school applicants with meetings. Prominent educational leaders of medicine), rather than create a scores in the middle third of the MCAT score seemed to create a false dichotomy: workaround once those students reach scale. Acad Med. 2020:95:344–350. We can have “smart” doctors or medical school. 5 Schwartzstein RM, Rosenfeld GC, Hilborn R, Oyewole SH, Mitchell K. Redesigning “compassionate, humanistic” doctors. I the MCAT exam: Balancing multiple think there is a third option: We can and Potential solutions include reaching perspectives. Acad Med. 2013;88:560–567. should have doctors who are both. out to under-resourced colleges and 6 Simpkin AL, Schwartzstein RM. Tolerating universities with predominantly uncertainty—The next medical revolution? N Engl J Med. 2016;375:1713–1715. With that goal in mind, there is probably underserved student populations to 7 Duckworth A. Grit: The Power of Passion no magic formula for picking the “right” work with those students who entered and Perseverance. New York, NY: Simon and medical students. In fact, there are many college unprepared for the rigorous Schuster, Inc.; 2016. types of “right” students, given the range premedical curricula or with those who 8 Garrison G, Mikesell C, Matthew D. Medical of clinicians, researchers, and educators do not have access to such curricula. school graduation and attrition rates. AAMC Analysis in Brief. April 2007;7:1–2. we need. Yet even in meeting these needs, We also could use new educational 9 Raymond JR Sr, Kerschner JE, Hueston WJ, we cannot make major compromises. technology, such as the HMX online Maurana CA. The merits and challenges of For example, the doctor with a great courses,16 to supplement the premedical three-year medical school curricula: Time academic record and research potential curricula at these under-resourced for an evidence-based discussion. Acad Med. colleges. We could develop more 2015;90:1318–1323. may care for patients one day, so she 10 Pellegrini VD Jr. Perspective: Ten thousand must also have the requisite interpersonal postbaccalaureate programs designed hours to patient safety, sooner or later. Acad and communication skills to graduate. specifically for disadvantaged students Med. 2012;87:164–167. Similarly, the doctor who is a great to help them enter medical school with 11 Hashimoto DA, Bynum WE 4th, Lillemoe communicator also needs the cognitive the necessary academic foundation KD, Sachdeva AK. See more, do more, teach more: Surgical resident autonomy and the acumen to think critically and avoid to succeed. Finally, we can encourage transition to independent practice. Acad the pitfalls of pattern recognition and our own universities to invest faculty Med. 2016;91:757–760. diagnostic error. During our multifaceted time and resources in the development 12 Meyer EG, Chen HC, Uijtdehaage S, admissions process, I believe that it is and implementation of these kinds of Durning SJ, Maggio LA. Scoping review acceptable to take a risk on a promising programs so that all students have the of entrustable professional activities in undergraduate medical education. Acad applicant who may not rank as highly opportunity to master the premedical Med. 2019;94:1040–1049. as someone else in one or another of curricula and demonstrate their talents 13 Kelleher M, Kinnear B, Sall D, et al. the attributes needed to be an excellent on the MCAT exam and beyond in A reliability analysis of entrustment- doctor, but we must also commit to medical school. Together, we can create derived workplace-based assessments a physician workforce that is made up of [published online ahead of print truly maintaining high standards for September 17, 2019]. Acad Med. our students and to working hard to smart, humanistic, and diverse doctors, doi:10.1097/ACM.0000000000002997. improve students’ areas of weakness without making compromises during 14 Newman LR, Brodsky D, Jones RN, during medical school. If remediation is the medical school admissions process. Schwartzstein RM, Atkins KM, Roberts not successful, however, we must redirect DH. Frame-of-reference training: Acknowledgments: The author thanks David Establishing reliable assessment of teaching students to other career paths. Roberts, MD, and Steven Weinberger, MD, for effectiveness. J Contin Educ Health Prof. their review of an earlier draft of this article. 2016;36:206–210. Ideally, we should not need to take risks 15 Lukianoff G, Haidt J. The Coddling of the in the admissions process. As a society, Funding/Support: None reported. American Mind. New York, NY: Penguin we are more likely to fix problems Press; 2018. Other disclosures: None reported. 16 Harvard Medical School. HMX courses. after they arise than to invest in their https://onlinelearning.hms.harvard.edu/ prevention. For example, we know Ethical approval: Reported as not applicable. hmx/courses. Accessed November 5, 2019.
Academic Medicine, Vol. 95, No. 3 / March 2020 MCAT® is a program of the 335 44 | Research to Evaluate the Fairness, Use, and Predictive Validity of the MCAT® Exam Introduced in 2015 Association of American Medical Colleges 20-002C (03/20)