AMEE 2017 Abstract Book Monday 28th August

#3L Short Communications: Research in #3L2 (786) Beyond a good story: from Hawthorne Effect to reactivity in health professions education research Location: Room 102

#3L1 (660) Gary Sutkin, University of Missouri‐Kansas City, Kansas City, Medical Education Research in 2006‐2015: A USA Scoping Review Elise Paradis, University of Toronto, Toronto, Canada

Background: The Hawthorne Effect (HE) is used most Lynn Monrouxe, Chang Gung Medical Education Research often to describe a research participant’s response to Centre, Chang Gung Memorial Hospital, Linkou, Taiwan being observed and allegedly threatens the validity of Garrett Ren‐Jie Liu observational research. The objective of this study was to Mojca Babovič gain a more nuanced understanding of the role of the HE

in health professions education (HPE) research. Background: As a rapidly emerging field in Taiwan, Summary of work: We reviewed critiques of the initial Medical Education Research (MER), has already produced Hawthorne Works studies, analyses of the HE in a vast body of evidence. However, there has been no educational and psychological settings, systematic systematic study exploring this field so far. Recognizing reviews, and finally examples in the HPE literature. We the void, our study aims to map the state of the MER then scrutinized examples from our own HPE research, literature originating from Taiwan (2006‐2015). looking for both confirmatory and disproving evidence of Summary of work: Between May–December 2016, a a HE. scoping review of healthcare education manuscripts was Summary of results: Definitions of HE vary immensely, but conducted. Inclusion criteria: English/Chinese language; generally encompass: special attention accorded Taiwanese authors; January 2006 and December 2015. participants, awareness of being in an experiment, and Research Questions included: numbers of published change in routine. Evidence in HPE research is highly papers per year; journal types; most cited articles; most contingent on task and research context. Our research frequently researched topics; research design; and key suggests that the pace of healthcare delivery and the researchers. embeddedness of researchers minimize HE. Summary of results: 6,266 references were found across Discussion: The HE has gained mythical power in 3 databases (MEDLINE+Journals@Ovid, Scopus, Web of observational, particularly HPE, research. Yet the Science), additionally 191 retrieved from 22 key journals. heterogeneity in description of the HE limits its Stage 1: three researchers screened titles/abstracts for interpretability. Research participants’ behavioral duplicates and exclusion/inclusion criteria. Stage 2: from adaptation during observational research likely comes remaining, 890 manuscripts, 19 conference papers and from their understanding of the study and the associated three non‐peer‐reviewed book chapters were excluded expected behavior, rather than mere awareness of being (n=868 texts remained). observed. Discussion: 2008‐2009 publications almost doubled (n=56; Conclusion: The HE has a variable, complex presence in n=94); then steadily increased (104 average yearly HPE observational research. Researchers should remain publications). The majority are published in alert to possible adaptations in participants’ behavior and journals (e.g. Journal of Nursing, Nurse Education Today). report and question these behaviors, while carefully Medical Education also features highly (n=9 pre‐2010; n=21 considering the context. Future research should examine 2011‐onwards). A follow‐up 2016‐replication is underway the longitudinal effects of any HE. for a clearer picture. Content themes will be discussed. Take‐home message: We suggest that reviewers avoid Conclusion: In our scoping review we illuminate key asking about a HE, and instead articulate their specific concepts and main sources of evidence to fully grasp the methodologic concerns. We also suggest that researchers potential of further research in MER for Taiwan. A rigorous avoid a HE by performing sustained observation, cross‐ database of current literature helps us identify areas of checking their own (etic) understanding with the emic highest priority and provides us with a rationale for future view of participants, thus providing more robust accounts grants applications and decision‐making. of participant life. Take‐home message: Amidst this flourishing of publications, the complex climate of MER in Taiwan is a fruitful avenue for exploration. This is required for us to continue to undertake rigorous research in the field, thereby engaging in meaningful dialogues internationally. Our scoping review plays an important part in this endeavour.

AMEE 2017 Abstract Book Monday 28th August

#3L3 (386) #3L4 (294) What Activity Theory has to offer in Medical Education Learning Curves in Health Professions Education Research – an example Research: A Systematic Review

Agnes Elmberger, Department of Learning, Informatics, Neva Howard, University of North Carolina, Chapel Hill, Management and Ethics, Karolinska Institutet, Stockholm, USA Martin Pusic, New York University, New York City, USA Juha Nieminen Rose Hatala, University of British Columbia, Vancouver, Klara Bolander Laksov Canada David Cook, Mayo Clinic, Rochester, USA Background: Lately, there has been an increasing interest in activity theory (AT) within medical education research Background: Learning Curves(LCs) are used in medical (Reid 2015). Often, the arguments for using AT are that it education to represent paths to competence and offers a comprehensive framework bridging the expertise and can be useful tools for competency separation between theory and practice, and that it paradigms. However, research reports of LCs often omit enables the researcher to include both micro and macro important information. We conducted a systematic review structures. of the reporting quality of LC research to identify specific Summary of work: With the aim to study how areas for improvement. pedagogical change takes place in clinical environments, Summary of work: We searched MEDLINE, EMBASE, activity theory was used to illuminate the involved CINAHL, ERIC, Web of Science and Scopus in 2011 for elements in such processes. Qualitative data was collected studies of simulation‐based education. From this in five different clinical contexts where initiatives for convenience sample the authors identified all studies educational change had been taken. reporting a LC. Reviewers worked in duplicate to extract Summary of results: The notion of the activity system information on graphical, statistical, and conceptual with its included elements (Engeström & Sannino, 2010) elements of LCs. was especially helpful in identifying the involved factors Summary of results: 145 of 985 studies reported a LC. and what role they play in educational change in clinical Items for improved reporting included: description of environments. The analysis continued by identifying learning intervention (reported in 51%), graphical interconnections and contradictions between neighboring competency lines (22%), and 13% displayed no LC graph at activity systems, which further deepened the all. 63% statistically analyzed the LC, but only 7% of the understanding. functions chosen used all data offered by the LC. Discussion: AT enabled us to broaden the understanding Discussion: Given that LCs are inherently a visual display for how educational change takes place in clinical of learning, it is striking that the graphs portrayed in up to environments, so that future change initiatives can be 90% of studies underutilized effective graphing techniques. supported. In our experience, one of the challenges with Statistically, methods were used that did not take AT was how to proceed from merely describing results to advantage of hierarchical linear models, the ideal generating new knowledge and add to existing theories. approach to statistical modeling of a LC. Conclusion: Activity theory does not come ready to use. It Conclusion: The reporting of learning curves is often is a theory with many founding principles that needs to be incomplete. Furthermore, the rich conceptual and understood in order to reach its full potential. But when statistical framework offered by learning curves is this is attended to, the theory allows researchers to underutilized. Consistent reporting of key elements and systematically study complex activities within medical consolidation around best practices will improve the use education, such as educational change. of learning curves for both local applications and for Take‐home message: Activity theory holds the activity as generalizable research. unit of analysis and includes the different elements Take‐home message: LCs in Health Professions Education involved in the activity. The theory also includes tools and are poorly reported graphically, statistically and notions to illuminate interconnections between several conceptually. Best practices include: use of graphical tools activities and thus, it offers ways to investigate the activity that depict multiple dimensions of the learning task, from a more systemic perspective. complete reporting of interventions and feedback, communication of important thresholds such as competency and remediation, and use of a mathematical modeling function.

AMEE 2017 Abstract Book Monday 28th August

#3L5 (852) #3L6 (2622) Exploring in‐depth analysis of top‐cited articles in medical Lessons from a Community of Practice education Ellayne Fowler, University of Bristol, Bristol, UK Samy Azer, King Saud University, Riyadh, Saudi Arabia Background: This reflection is on a three‐year University Background: The aim of this study was to analyze the top‐ Fellowship awarded to develop the quality of the medical cited articles in medical education identified by Azer 2015 education research at a multi‐site medical school. Using a and explore the characteristics of these articles that can conceptual framework of Communities of Practice (Lave & provide insight into their assessment. Wenger, 1991) a range of measures were taken to develop Summary of work: The following were analyzed: (1) any work already being done in teaching Academies. correlation between number of grants, number of Summary of work: This work is a case study, reporting on authors, institutes, or countries involved and number of a range of initiatives and analysis of both results and citations, (2) number of female representation in processes. 3 years of successful abstracts at a UK authorship, the representation of none‐medical staff in the conference were analysed using a quality framework authorship, and (3) exploring any association between top developed from literature. Educational interventions and authors and awards in medical education. support were analysed using questionnaires and focus Summary of results: No correlation was found between groups. the number of citations and number of grants, authors, Summary of results: Analysis of successful abstracts institutes, or countries involved. 50% of authors were demonstrate development of quality in terms of methods females. No correlations were found between the number of evaluation (Kirkpatrick, 1967) and purpose of research of females or non‐medical authors and citations. Authors (Cook.et.al, 2008) . Changes in the size of the COP have with two or more articles received the Karolinska led to further revision of intervention methods for Institutet Prize for Research. sustainability through greater work between Academies Discussion: The analysis shows that the higher number of and a medical education course. citations received are not necessarily related to funding, Discussion: Using a theoretical framework has aided working collaboratively with a big team from several reflection on developing educational research in an institutes or several countries. The results indicate that extended medical school faculty. Small interventions in top‐cited articles have presented innovative ideas or support have led to external success at conferences and provided answers to questions/challenges that have faced the introduction of innovations into the medical school medical/health educators. curriculum. Communication and sharing of innovations are Conclusion: The finding that authors of more than one key to further developments. top‐cited article received the Karolinska Institutet Prize for Conclusion: Theoretical framing of educational research Research in Medical Education indicates that this could be issues has led to interventions that support the a parameter in selecting the candidates for the award. development of a COP. While you can’t create a COP you Take‐home message: Careful assessment of top‐cited can support the development of a group with common articles may provide more insight into exploring their goals into a COP which is the basis for ongoing changes in characteristics and present an excellent model for that COP. researchers and medical/health educators that they use in Take‐home message: • COPs take time to grow and need their teaching and research work. to be sustained in view of their changing makeup; • Develop some consensus on quality before attempting larger research projects; • Build on success with a forward looking strategy for research in your programme