NCEE 2009-4052 U.S. DEPARTMENT OF EDUCATION Achievement Effects of Four Early Elementary School Math Curricula Findings from First Graders in 39 Schools Achievement Effects of Four Early Elementary School Math Curricula Findings from First Graders in 39 Schools February 2009 Roberto Agodini Barbara Harris Sally Atkins-Burnett Sheila Heaviside Timothy Novak Mathematica Policy Research, Inc. Robert Murphy SRI International Audrey Pendleton Project Officer Institute of Education Sciences NCEE 2009-4052 U.S. DEPARTMENT OF EDUCATION U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences Sue Betka Acting Director National Center for Education Evaluation and Regional Assistance Phoebe Cottingham Commissioner February 2009 This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO- 0112/0003. The project officer was Audrey Pendleton in the National Center for Education Evaluation and Regional Assistance. IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the reports. This publication is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should be: Roberto Agodini, Barbara Harris, Sally Atkins-Burnett, Sheila Heaviside, Timothy Novak, and Robert Murphy (2009). Achievement Effects of Four Early Elementary School Math Curricula: Findings from First Graders in 39 Schools (NCEE 2009-4052). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. To order copies of this report, • Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 1398, Jessup, MD 20794-1398. • Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, call 800-872-5327 (800-USA-LEARN). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 800-437-0833. • Fax your request to 301-470-1244. • Order online at www.edpubs.org. This report also is available on the IES website at http://ies.ed.gov/ncee. Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113. ACKNOWLEDGMENTS This study was made possible by the collaboration and hard work of many individuals beyond the study authors. We appreciate the willingness of the participating districts, schools, and teachers to use the study’s curricula, and to respond to the data requests that are the basis for this report. We also appreciate the willingness of the curriculum publishers to take part in this evaluation. We benefited from useful comments of the study’s technical working group: Richard Askey, Douglas Clements, Thomas Cook, Lynn Fuchs, Tom Loveless, Kevin Miller, Donald Rock, and Hung-Hsi Wu. We thank Melissa Thomas for helping to direct all phases of the student testing and other data collection efforts, and for her contributions to the development of the survey instruments. Alejandra Lopez helped design the teacher surveys. Tom Barton, Tim Bruursema, and Kristina Rall helped design the data collection training programs and manage the data collection. Season Bedell-Boyle and Loring Funaki coordinated the large team of student testers. Mark Brinkley, Andrew Frost, and Joel Zief provided systems support, Erin Slyne programmed the computer- assisted test instruments, and Donsig Jang provided sampling support. We thank Carol Razafindrakato for his programming expertise, and Emily Sama Martin for processing the teacher-reported adherence data. Brian Gill provided useful comments on an earlier version of the report. Marjorie Mitchell and William Garrett produced the report. Last, but far from least, we thank the team of site recruiters who diligently worked to secure the study’s first cohort of participating districts and schools. The recruiters included several report authors, Alex Bogin, Larissa Campuzano, John Deke, Patricia Del Grosso, Benita Kim, Jeffrey Max, and Melissa Thomas. The Authors iii DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST The research team for this evaluation consists of a prime contractor, Mathematica Policy Research, and a main subcontractor, SRI International. Neither organization nor its key staff have financial interests that could be affected by the findings from the evaluation. None of the study’s Technical Working Group members, which were convened by the research team to provide advice on key features of the study, have financial interests that could be affected by the evaluation’s findings. v CONTENTS Chapter Page EXECUTIVE SUMMARY .......................................................................................... xvii I INTRODUCTION AND STUDY FEATURES ................................................................1 A. THE NEED FOR A LARGE-SCALE STUDY OF MATH CURRICULA ............1 B. DESIGNING THE EVALUATION AND SELECTING THE CURRICULA .......3 1. Selecting the Curricula .....................................................................................4 2. Evaluation Design ............................................................................................5 C. RECRUITING PARTICIPANTS ............................................................................6 1. Suitable Districts ..............................................................................................6 2. Recruiting Districts and Schools ......................................................................7 3. Characteristics of Participants ..........................................................................9 D. RANDOM ASSIGNMENT AND STATISTICAL POWER ..................................9 E. DATA COLLECTION ..........................................................................................14 1. Outcome Measure ..........................................................................................14 2. Other Data Collection ....................................................................................16 F. FUTURE PUBLICATION PLANS .......................................................................18 II CURRICULUM IMPLEMENTATION ..........................................................................19 A. CONTEXT FOR CURRICULUM IMPLEMENTATION ....................................20 B. TEACHER CURRICULUM TRAINING .............................................................24 1. All Teachers Attended Initial Training on Their Assigned Curriculum ........24 2. Ninety-Six Percent of Teachers Attended Follow-Up Training on Their Assigned Curriculum .....................................................................................27 3. Other Sources of Professional Development .................................................28 vii CONTENTS (continued) II (continued) C. SCHOOL-BASED INSTRUCTIONAL SUPPORT ..............................................28 1. Seventy-Three Percent of Teachers Had Access to a Math Coach ................29 2. Teachers Reported Having a Supportive Instructional Environment ............31 D. SOME BASICS ABOUT TEACHER USE OF THE ASSIGNED CURRICULUM .....................................................................................................31 1. Nearly All Teachers (99 percent in the fall and 98 percent in the spring) Reported Using Their Assigned Curriculum .....................................31 2. Eighty-Eight Percent of Teachers Completed at Least 80 Percent of Their Curriculum ...........................................................................................33 3. One-Third of Teachers Supplemented with Other Materials .........................33 4. Saxon Teachers Spent One More Hour on Math Instruction per Week ........37 E. TEACHER ADHERENCE TO THE ESSENTIAL FEATURES OF THE CURRICULA .........................................................................................................37 1. Descriptions of the Curricula and Teacher Adherence ..................................38 a. Investigations .............................................................................................39 b. Math Expressions .......................................................................................42 c. Saxon ..........................................................................................................45 d. SFAW ........................................................................................................48 2. Content Coverage ...........................................................................................50 III CURRICULUM EFFECTS ON FIRST GRADE ACHIEVEMENT .............................53 A. METHODS USED TO CALCULATE CURRICULUM EFFECTS ....................53 B. RELATIVE EFFECTS OF THE CURRICULA ...................................................58 1. Student Math Achievement was Significantly Higher in Math Expressions and Saxon Schools than in Investigations and SFAW Schools ...........................................................................................................60 2. Some Curriculum Differentials Also Exist in Several Subgroups .................61 C. NEXT STEPS FOR THE STUDY .........................................................................68