Medical Education) Students, We Computed an Average Grade for Those Courses Completed at UNECOM

edical education students who began UNECOM’s pro- M gram in fall 1996 and fall 1997 (Level 1; N 187; 44% female) and 1994 and 1995 (Level 2; N 86; 40% female). All students signed consent forms for Correlation of scores for the release of the relevant data. For students who took the Level 1 or Level 2 exami- Comprehensive Osteopathic Medical nation more than once, we used scores Licensing Examination with osteopathic from their first sitting. To measure academic performance in the first 2 years medical school grades of UNECOM’s curriculum, each student’s average percentage grade was computed in three ways. First, we calculated STEVE E. HARTMAN, PhD the average grade for courses related BRUCE P. BATES, DO directly to the basic medical sciences SARAH A. SPRAFKA, PhD (gross anatomy, neuroanatomy, embry- ology, histology, virology, parasitology, The authors evaluated construct validity of scores for the Comprehensive Osteo- immunology, and pathology). Second, pathic Medical Licensing Examination (COMLEX–USA), the examination used we calculated the average grade for cours- to evaluate osteopathic physicians for licensure. They computed correlations es with a strong clinical emphasis—our between students’ grades in the first 2 years of osteopathic medical school and their second-year “systems” courses (nervous, scores on the COMLEX–USA Level 1 (N 187) and Level 2 (N 86), as well musculoskeletal, respiratory, cardiovas- as correlations between third- and fourth-year clerkship grades and the COM- cular, hematology, gastroenterology, der- LEX–USA Level 2. Correlations of Level 1 scores with grades for years one, two, matology, renal, and reproductive sys- and the first 2 years combined were .74, .80, and .81, respectively; for Level 2, cor- tems). Finally, we calculated the average relations were .59, .70, and .71. Correlation between clerkship grades and scores grade for all of these courses combined. for the COMLEX–USA Level 2 was .26. The strong correlation between COM- All courses were weighted equally. LEX–USA results and grades for the didactic curriculum in the first 2 years of med- Some students had not been required ical school provides evidence for the construct validity of scores for the COM- to take all courses (for example, some LEX–USA Levels 1 and 2. had completed one or more equivalent (Key words: Comprehensive Osteopathic Medical Licensing Examination, courses before matriculating). For these medical education) students, we computed an average grade for those courses completed at UNECOM. Most students we evaluated he examination used by the Nation- We agree with Wright and Bezruczko1 had completed all of their relevant course- Tal Board of Osteopathic Medical that statistical affinities of a medical work at UNECOM. Even those students Examiners (NBOME) to evaluate newly licensing examination with measures of who did not complete all coursework at trained osteopathic physicians for licen- academic aptitude taken before enroll- UNECOM completed most such work at sure is the Comprehensive Osteopathic ment in medical school (for example, the college. Medical Licensing Examination (COM- undergraduate GPAs and MCAT scores) To monitor clinical proficiency LEX–USA), Levels 1 through 3. In this are irrelevant to the validity of any med- through the third and fourth years, fol- preliminary report, we summarize data ical licensing examination—though they lowing each clerkship a supervisor pertaining to the construct validity of are useful for assessing predictive valid- assessed each student’s competence using scores for Levels 1 and 2 of the COM- ity of preadmission measures. Therefore, a five-point scale to rate 12 criteria. For LEX–USA. we focused instead on correlations each student who had taken the COM- between COMLEX–USA scores (for Lev- LEX–USA Level 2, we measured clini- Dr Hartman is a professor in the Department els 1 and 2) and performance measures cal performance using their average grade of Anatomy, College of Osteopathic Medicine, for the 4 years of osteopathic medical over all 12 criteria and all 15 clerkships. University of New England, Biddeford, Maine, curriculum at the University of New Eng- If our students’ scores on the COM- where Dr Bates is chairman in the Depart- ment of Family Medicine and Dr Sprafka is land College of Osteopathic Medicine LEX–USA Level 1 are a valid measure of Director of Predoctoral Education. (UNECOM). their mastery of “the medical concepts Correspondence to Steve E. Hartman, and principles [that is, medical basic sci- Department of Anatomy, College of Osteo- ences] necessary for understanding the pathic Medicine, University of New England, Methods Biddeford, ME 04005. We compiled scores for the COM- mechanisms of medical problems,”2 the E-mail: [email protected] LEX–USA examinations administered to product-moment correlation between

Hartman et al • Medical education JAOA • Vol 101 • No 6 • June 2001 • 347 those scores and academic performance scores and their grades in the first 2 years The correlation between clerkship during the first 2 years in medical school of medical school. grades and COMLEX–USA Level 2 may should be high. Likewise, if scores on Although no similarly derived corre- have been low in part because several the COMLEX–USA Level 2 are a valid lations with regard to COMLEX–USA clerkship evaluation items related more measure of students’ mastery of “medi- Level 1 scores have been published, our to student behavior than to cognitive cal concepts and principles necessary for values (.74 to .81) compare favorably skills (for example, “ability to work with making appropriate medical diagnoses,”2 with those associated with the United other healthcare professionals”). How- the correlation between those scores and States Medical Licensing Examination ever, consideration of only those items grades for years 1 and 2 should be high. Step 1 (.62 and .73, N 1624; .72 to clearly related to cognitive mastery did lit- Because COMLEX–USA poses ques- .83 with a median of .76, N values not tle to improve the correlation. This, in tions in the context of medical diagnosis provided5) and are consistent with con- turn, may have occurred because of the and treatment (“clinical presentations struct validity for scores of the COM- restricted range of clerkship grades— and physician tasks”2), we predicted that LEX–USA Level 1 examination. At .59 to supervisors tended to use only the upper scores for both Levels 1 and 2 would be .71, correlations of scores on the COM- registers of the five-point scale (unsatis- more highly correlated with grades for LEX–USA Level 2 with grades were mod- factory, marginal, average, above average, our second-year, more clinically oriented erate and, likewise, suggestive of con- outstanding). Also, these grades may courses than with first-year grades. Also, struct validity. have been influenced by factors that have to the degree that (1) third- and fourth- Many students scoring at the low ends uncertain relationships to actual clinical year clerkship grades measure the cog- of the grade scales scored higher on the competence. For example, clerkship nitive component of clinical competence, COMLEX–USA Level 1 than would have grades were correlated (r .24; N and (2) scores on the COMLEX–USA been predicted—assuming that a linear 101) with students’ position on the Level 2 are valid, the former should cor- relationship exists between these mea- extraversion-introversion scale of the relate with the latter. sures. This may reflect lack of agreement Myers-Briggs Type Indicator6; on aver- between these measures of academic apti- age, extraverts were awarded higher Results tude in what is arguably the most impor- grades than were introverts. This corre- Correlations between scores on the tant region of the COMLEX–USA Level lation is not high, but it is as high as the COMLEX–USA Level 1 and grades for 1 scale—where pass/fail decisions are ren- correlation that exists between clerkship first-year basic-science courses, second- dered. If this observation is corroborated grades and the COMLEX–USA Level 2 year clinical systems, and all courses com- following examination of other and larg- (r .26). This contributes to the impres- bined were .74, .80, and .81, respective- er samples, an explanation should be sion that clerkship grades are measuring ly. Correlations between Level 2 and sought. a broader conceptual universe than is these grade measures were .59, .70, and As predicted, however, scores on the the COMLEX–USA Level 2, which is .71, respectively. As predicted, scores for clinically focused COMLEX–USA Lev- primarily a measure of cognitive mas- both Levels 1 and 2 appear to be corre- els 1 and 2 both were more highly corre- tery. lated more strongly with grades for the lated with second-year grades (more clinically focused second-year curricu- reflective of clinical understanding) than Conclusion lum than with those for the first-year first-year grades. This may be partly due Because medical schools set different curriculum. The correlation between to the second-year curriculum being near- internal standards,5 it is important that scores on the COMLEX–USA Level 2 er in time to the administration of both psychometrically sound licensing exam- and clerkship grades was .26 (N 86). COMLEX–USA examinations. Also, the inations be in place to assure professional COMLEX–USA scores and medical integrity and public well-being. One Comments school grades of UNECOM students are expectation with regard to scores on such In descriptions of the COMLEX–USA, not entirely independent measures because examinations is that they will correlate the NBOME specifies that the purpose of several members of our faculty have writ- strongly with students’ performance in the Level 1 examination is to “assess a ten both medical school examinations their medical school curriculum. The fact candidate’s knowledge of the basic sci- and items for the COMLEX–USA. How- that correlations are high between our ences.”3 Administrators and faculty at ever, COMLEX–USA item writers are students’ scores on the COMLEX–USA UNECOM, as part of our college’s mis- asked to submit only items never used at Levels 1 and 2 and their grades for the sion, have worked to develop a curricu- their home college, and more than 99% first 2 years of medical school is evidence lum offering the basic science background of COMLEX–USA items reported on for construct validity of the COM- necessary for students’ success as prac- here were written by individuals unaffil- LEX–USA. ticing physicians. Therefore, one indica- iated with our medical school, as noted in tion of construct validity for scores of a conversation with Boyd Buser, DO, the COMLEX–USA Level 1 would be chairman of the NBOME Product Com- correspondence between our students’ mittee (October 6, 1999).

348 • JAOA • Vol 101 • No 6 • June 2001 Hartman et al • Medical education Acknowledgment The authors thank UNECOM’s Depart- Review ment of Anatomy for financial support, and Dr Boyd Buser for his feedback on the manuscript.

References 1. Wright BD, Bezruczko N. Evaluation of COMLEXÐUSA. Chicago, Ill: MESA Psycho- metric Laboratory; 1999.

2. Bulletin of Information. Chicago, Ill: Nation- al Board of Osteopathic Medical Examiners, This Won’t Hurt a Bit! And Inc; 1999. Other Fractured Truths in 3. Level 1: Examination guidelines and sam- Healthcare ple exam, 1999. Chicago, Ill: National Board of Osteopathic Medical Examiners, Inc; 1999. By Karyn Buxman, RN. 96 pp. Lamoine Press, 4. Elam CL, Johnson MS. NBME Part I versus 63 Pioneer, Suite 200, Hannibal, MO 63401. USMLE Step 1: predicting scores based on 2000. $11.95. preadmission and medical school perfor- mances. Acad Med 1994;69:155. Karyn Buxman, RN, has put together a unique and entertaining series of stories 5. Swanson DB, Ripkey DR, Case SM. Rela- tionship between achievement in basic sci- related to healthcare. In the foreword of ence coursework and performance on 1994 her book, Buxman states her belief that USMLE Step 1. 1994-95 Validity Study Group laughter is the sweetness that helps the for USMLE Step 1/2 Pass/Fail Standards. medicine go down. Presenting stories and Acad Med 1996;71:S28-S30. jokes she has collected over many years, 6. Hammer AL. MBTI applications: A decade the author suggests their use as a way of of research on the Myers-Briggs Type Indica- reducing stress. I believe that she accom- tor. Palo Alto, Calif: Consulting Psychologists plishes that goal. Press; 1996. Vice-president and contributing editor for the Journal of Nursing Jocularity until the publication ceased, Buxman uses con- structive—rather than destructive—humor to relieve stress and perhaps improve body physiology. There are numerous positive physiologic changes associated with laughter, including improved respiratory and cardiovascular system function, relaxed muscles, elevated mood, and possibly a boost to the immune system. In the group setting, the author points out that humor can solidify ties, improve productivity, and serve as a powerfully persuasive form of communication. Reading this book made me laugh and improved my mood after a tough day. In this context, the humor should not be offen- sive and is helpful in relieving stress. I rec- ommend this book without hesitation. The author can be contacted at www.humorx.com.

Gilbert E. D’Alonzo, DO Editor in Chief

Hartman et al • Medical education/Review/Coming in. . . JAOA • Vol 101 • No 6 • June 2001 • 349