Abstract This report identifies mathematics and science curricula as well as professional development models at the middle and high school levels that are effective based on their success in increasing student achievement. The goal of the study was to provide some choice to districts and schools that wanted guidance in selecting a curriculum and that wished to use effectiveness as a selection criterion. Unexpectedly, most middle and high school mathematics and science curricula did not have studies of student achievement with comparison groups, and it proved especially difficult to find effects in either math or science for subgroups by sex, minority status, and urban status. Findings strongly suggest that science curricula is more effective when it is inquiry-based, although math curricula can be effective when standards- or traditional-based. REVIEW OF EVALUATION STUDIES OF MATHEMATICS AND SCIENCE CURRICULA AND PROFESSIONAL DEVELOPMENT MODELS By Beatriz C. Clewell Clemencia Cosentino de Cohen The Urban Institute and Patricia B. Campbell Lesley Perlman Campbell-Kibler Associates, Inc. with Nicole Deterding Sarah Manes Lisa Tsui The Urban Institute and Shay N.S. Rao Becky Branting Lesli Hoey Rosa Carson Campbell-Kibler Associates, Inc. Submitted to the GE Foundation December 2004 Acknowledgments A number of individuals contributed to this effort in various ways. We were fortunate to have the assistance of Gerhard Salinger of the National Science Foundation; Jo Ellen Roseman of Project 2061 at the American Association for the Advancement of Science (AAAS); Joan Abdallah at AAAS; and several staff members at the Center for Science Education at the Education Development Center—Barbara Berns, Jeanne Century, Joe Flynn, Elisabeth Hiles, Jackie Miller, Marian Pasquale, and Judith Sandler —in helping us identify science curricula that might have evaluation studies. We are also deeply indebted to those who reviewed our report and offered useful suggestions for revising it: Jane Butler Kahle at Miami University of Ohio and Linda Rosen of Education & Management Innovations, Inc. Thanks are due to William Bradbury and Cara West, Urban Institute staff who helped in the production of the report. Most of all, we wish to express our appreciation for the responsiveness of curriculum developers and researchers whose curricula are reviewed in this report and who shared evaluation studies with us. Last, but not least, we thank our program officer, Roger Nozaki and his colleague, Kelli Wells, both of the GE Foundation, for their insightful comments and suggestions on the report draft that helped to make this document more user-friendly. We thank the GE Foundation for funding this review and for taking an evidence-based approach to school reform. We think it’s the only way to go. REVIEW OF EVALUATION STUDIES OF MATHEMATICS AND SCIENCE CURRICULA AND PROFESSIONAL DEVELOPMENT MODELS Introduction This report presents the findings of a review of about four hundred studies evaluating mathematics and science curricula and professional development models for middle school and high school. As requested by the GE Foundation, the main goal of this review was to identify, in response to the GE Foundation’s request, mathematics and science curricula as well as professional development models that had been deemed effective based on their success in increasing student achievement. The Foundation’s interest in these findings stems from its desire to initiate a program of funding to foster sustainable improvement in academic achievement of underrepresented and disadvantaged populations. Historically, curriculum choice at the local level has often been made by a committee that decides which curriculum to adopt based on considerations only peripherally related to student achievement—such as state-imposed standards, recommendations of others, cost, and presentations by publishers’ representatives. Choice of professional development models follows a similar pattern. Indeed, there has been very little else available to guide school districts in their curriculum selection process, since for most curricula and textbooks the only data at hand are publishers’ figures on the number of adoptions. That has been changing. There is a growing movement to assess the effectiveness of math and science curricula through various methodologies, including content analyses, comparative studies, case studies, and synthesis studies.1 And while there have been several studies of the effectiveness of professional development practices, very few have measured the effects of these practices on student achievement. In this document, we describe the methodology used to conduct this review, present our findings, and end with a summary of conclusions. To provide an international perspective on these topics, the report includes a brief look at the research on mathematics and science education in three countries that are similar in key dimensions to the U.S. Methodology Criteria for Selecting Evaluation Studies We developed a set of criteria to guide the selection of evaluation studies to be included in our review. Studies were expected to have (1) rigorous methodological design; (2) measures of impact on student outcomes (which include, but are not limited 1 The majority of these efforts have been undertaken by the American Association for the Advancement of Science (AAAS), the National Research Council (NRC), and the U.S. Department of Education. The AAAS study used content analysis, the NRC study did not rate specific curricular math programs, and the U.S. Department of Education study reviewed middle school math programs only. 1 to, test scores); (3) comparative data, cross-sectional or longitudinal, with experimental and quasi-experimental designs preferred over others; and (4) high quality and valid data. We offer several caveats regarding the quality of the evaluation studies that we report in this document, especially those on mathematics and science curricula. Because of the dearth of studies that met our criteria, we were forced to compromise and include a number of evaluations that did not report effect sizes; a few that did not give the statistical significance levels for differences; and several that lacked non-treatment comparison groups. In some cases we were unable to verify the quality of the data on which findings were based. It was also a source of great disappointment that so few of the studies we identified disaggregated findings by sex, race/ethnicity, or urban school location. We believe, however, that taken as a whole the studies that are included here offer useful insight into the condition of mathematics and science curricula in middle and high school. Identification of Curricula/Professional Development Models Using research databases such as the Education Resource Information Center (ERIC), Education Abstracts, and web sites such as Northwest Regional Education Laboratory’s Catalog of School Reform Models, we conducted a literature search of articles and reports pertaining to (1) major mathematics and science curricula used at the middle and high school level; and (2) empirical studies that examine how teacher professional development in science and mathematics affects student outcomes. The review team also gathered and reviewed relevant documents that were not accessible through traditional research outlets. It was much more the case for science than for mathematics that most of the evaluation studies of recent curricula were unpublished reports of evaluations conducted by the program developers. On the other hand, a large number of published mathematics curriculum studies were available for inclusion in this review. Appendix D contains lists of all mathematics and science curricula for which studies were sought. Primary sources of mathematics reform models were the Northwest Regional Educational Laboratory’s database on whole-school reform models2 and Comprehensive School Reform and Student Achievement: A Meta-Analysis.3 Primary sources of the mathematics curriculum were National Science Foundation-funded projects; the U.S. Department of Education’s “What Works Clearinghouse” and the Mathematics Expert Panel; the Mathematical Sciences Education Board’s Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials; and the American Association for the Advancement of Science’s Project 2061. 2 http://www.nwrel.org/scpd/catalog/index.shtml 3 Borman, G. D., G. M. Hewes, L. T. Overman, and S. Brown. 2002. Comprehensive School Reform and Student Achievement: A Meta-Analysis. CRESPAR Report No. 59. Baltimore, Md.: CRESPAR/Johns Hopkins University. http://www.csos.jhu.edu/CRESPAR/techReports/report59.pdf, 2 In order to ensure broad coverage of the science curriculum studies, we contacted experts on science curricula at organizations such as the Education Development Center, the Technology Education Research Center (TERC), the National Science Foundation, Project 2061, and others. In view of how few published evaluations of science curricula we were able to identify, we attempted to locate more recently developed curricula that might not yet have produced published evaluation studies. Once these curricula were identified through conversations with experts in the field, we obtained contact information on the developers of these programs and approached each of them to ascertain whether or not they had evaluation data or reports on the effectiveness of their curricula that met our established criteria. Several