EVALUATION OF Teacher Preparation Programs Purposes, Methods, and Policy Options Michael J. Feuer, The George Washington University Robert E. Floden, Michigan State University Naomi Chudowsky, Caldera Research, and Judie Ahn, National Academy of Education National Academy of Education Washington, DC NATIONAL ACADEMY OF EDUCATION 500 Fifth Street, NW Washington, DC 20001 NOTICE: This report was approved by the National Academy of Education Board of Directors. This study was supported by a grant from the National Science Foundation, Award No. 1153848. Any opinions, findings, conclu- sions, or recommendations expressed in this publication are those of the National Academy of Education and the authors and do not necessarily reflect the views of the National Science Foundation. International Standard Book Number 978-0-615-90081-0 Additional copies of this report are available from the National Acad- emy of Education, 500 Fifth Street, NW, Washington, DC 20001; Internet, http://www.naeducation.org. Copyright 2013 by the National Academy of Education. All rights reserved. Suggested citation: Feuer, M. J., Floden, R. E., Chudowsky, N., and Ahn, J. (2013). Evaluation of teacher preparation programs: Purposes, methods, and policy options. Washington, DC: National Academy of Education. The National Academy of Education advances high quality education research and its use in policy formation and practice. Founded in 1965, the Academy consists of U.S. members and foreign associates who are elected on the basis of outstanding scholarship related to education. Since its establishment, the Academy has undertaken research studies that address pressing issues in education, which are typically conducted by members and other scholars with relevant expertise. In addition, the Academy sponsors professional development fellowship programs that contribute to the preparation of the next generation of scholars. EValuation of TeacHER Education PROGRams: TowaRD A FRamewoRK foR InnoVation1 Steering Committee Michael J. Feuer (Chair), Graduate School of Education and Human Development, The George Washington University Deborah Loewenberg Ball, School of Education, University of Michigan Jeanne M. Burns, Louisiana Board of Regents Robert E. Floden, College of Education, Michigan State University Susan H. Fuhrman (Ex Officio), Teachers College, Columbia University Lionel C. Howard, Graduate School of Education and Human Development, The George Washington University Brian Rowan, Institute for Social Research and School of Education, University of Michigan Staff Judie Ahn, Senior Program Officer Naomi Chudowsky, Consultant Nancy Kober, Editor Gregory White, Executive Director 1 It is noted that the title of this project: Evaluation of Teacher Education Programs: Toward a Framework for Innovation, as awarded by the National Science Foundation, differs from the title of this report. v Acknowledgments ublic concern for the effectiveness of teacher preparation programs has sparked renewed interest in the attributes of evaluation systems Pused to gauge their quality. There are many such systems currently in place, with different purposes and consequences, and a growing need to clarify their advantages and drawbacks as the basis for developing new and innovative approaches. That need is the principal impetus for this report, which addresses a number of questions: What are the rela- tive strengths, limitations, and consequences of existing approaches to evaluating teacher preparation programs? How well do evaluation meth- ods align with multiple intended uses of their results? What principles should guide the design, implementation, and interpretation of evalua- tion systems? The work leading to this report was made possible by a generous grant from the National Science Foundation. We are especially grateful to Dr. Janice Earle, Senior Program Director in the Division of Research on Learning, for her interest and wise and careful guidance. The National Academy of Education steering committee formed to undertake this project included Deborah Ball, Jeanne Burns, Robert Floden, Susan Fuhrman (ex-officio), Lionel Howard, and Brian Rowan. I thank them all for their willingness to take time from very busy schedules to help shape the study, contribute their knowledge and experience to our deliberations, and fine-tune key sections of the report. Gregory White, executive director of the National Academy of Education, and Maxine Freund, Associate Dean for Research and External Relations at the Gradu- ate School of Education and Human Development at the George Washing- ton University, provided much needed intellectual and logistical support. vii viii ACKNOWLEDGMENTS I am indebted to the three principal co-authors for their superb contributions. Bob Floden drafted key sections and read every word of the whole report (several times), added subtlety and nuance to com- plex explanations, and never let us stray from our main mission. Naomi Chudowsky produced elegant and eloquent text and made sure our dis- cussions were evidence-based and methodologically defensible. Judie Ahn took responsibility for managing and then integrating the work of our outside authors, organized our workshops and meetings, and stead- fastly and efficiently kept the project on course. In addition, I want to say a special thanks to Nancy Kober, whose fine editorial hand is evident throughout. The Committee commissioned four papers to enrich our deliberations with information regarding the special issues surrounding evaluation of future STEM educators; lessons from evaluation systems in other coun- tries; and lessons from pre-service evaluation in the nursing profession. We are grateful to Suzanne Wilson; John Furlong; Maria Teresa Tatto, Joseph Krajcik, and James Pippin; and Jean Johnson and Christine Pintz, for their excellent papers and their interest and willingness to participate in the project. The Committee convened two workshops in which policy makers, researchers, and evaluation practitioners contributed their expertise. We are grateful to Chad Aldeman, James Cibulka, Susan Fuhrman, Heather Harding, George Noell, Charles Peck, Robert Rickenbrode, Kate Walsh, Nancy Cartwright, Carl Cohn, James Kelly, Richard Shavelson, Lee Shulman, William Trent, Brenda Turnbull, Suzanne Wilson, Joe Conaty, and Patricia Tate. I thank Emerson Elliott for help with fact-checking, and Lorraine McDonnell and Lorrie Shepard for wise counsel early on. Peer review is a linchpin of the National Academy of Education’s process, designed to ensure that its reports are accurate, objective, and responsive to the study charge. I thank the reviewers for their care- ful reading of the draft and their thoughtful insights and suggestions: Christopher Edley, Rick Ginsberg, Daniel Goldhaber, Shirley Malcom, and Brenda Turnbull were selected for their diverse perspectives and technical expertise and served as anonymous, independent reviewers. They were not asked to endorse the contents of the report or review the final draft of the report, responsibility for which rests with the National Academy of Education. The review process was seamlessly overseen by Academy member Judith Warren Little, who was appointed to make sure that reviewers represented a range of relevant perspectives and expertise and that their comments were carefully considered. Michael J. Feuer Chair, Steering Committee Contents Summary 1 1 Introduction: Purposes, Context, and Principles 9 2 The Landscape of Teacher Preparation Program Evaluation 25 3 Program Evaluation: Mapping Approaches to Purposes 61 4 Designing, Using, and Interpreting Teacher Preparation Program Evaluations: Toward A Decision Framework 79 References 99 Appendixes A Workshop Agendas and Participants 111 B Biographical Sketches of Steering Committee Members 117 ix Summary eacher preparation programs (TPPs) are where prospective teach- ers gain a foundation of knowledge about pedagogy and subject Tmatter, as well as early exposure to practical classroom experience. Although competence in teaching, as in all professions, is shaped signifi- cantly by on-the-job experiences and continuous learning, the programs that prepare teachers to work in K-12 classrooms can be early and impor- tant contributors to the quality of instruction. Evaluating the quality and effectiveness of TPPs is a necessary ingredient to improved teaching and learning. Many aspects of the relationship between teacher preparation and instructional quality are not fully understood, and existing approaches to TPP evaluation are complex, varied, and fragmented. Designers and consumers of TPP evaluations could benefit from clear information about the purposes, effects, strengths, and limitations of current evaluation approaches and from guidance for designing and using future evalu- ations. This report, the product of an analysis by a committee of the National Academy of Education, aims to fill that need. THE CURRent Landscape Systems for evaluating TPPs use various types of evidence—each with its particular strengths and limitations—to make inferences 1 2 EVALUATION OF TEACHER PREPARATION PROGRAMS about the quality of the preparation experience and its role in pro- ducing employable, high-quality teachers. Evaluations use a variety of evidence to approximate the aspects of teacher preparation that are not all easily observable or quantifiable. “Inputs,” such as selectivity in admissions, faculty qualifications, the quality and substance of teacher preparation course instruction, and the quality of student teaching experiences, are