NCEE 2010-4018 U.S. DEPARTMENT OF EDUCATION Evaluation of the DC Opportunity Scholarship Program Final Report Evaluation of the DC Opportunity Scholarship Program Final Report June 2010 Patrick Wolf, Principal Investigator, University of Arkansas Babette Gutmann, Project Director, Westat Michael Puma, Chesapeake Research Associates Brian Kisida, University of Arkansas Lou Rizzo, Westat Nada Eissa, Georgetown University Matthew Carr, Westat Marsha Silverberg, Project Officer, Institute of Education Sciences NCEE 2010-4018 U.S. Department of Education U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance John Q. Easton Acting Commissioner June 2010 This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO-0126. The project officer was Marsha Silverberg in the National Center for Education Evaluation and Regional Assistance. IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the reports. This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should be: Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa, and Matthew Carr. Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. To order copies of this report, • Write to ED Pubs, U.S. Department of Education, P.O. Box 22207, Alexandria, VA 22304. • Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, call 800-872-5327 (800-USA-LEARN). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 1-877-576-7734. • Fax your request to 703-605-6794. • Order online at www.edpubs.gov. This report also is available on the IES website at http://ies.ed.gov/ncee. Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113. Contents Page Acknowledgments ...................................................................................................................................... xiii Disclosure of Potential Conflicts of Interests ............................................................................................ xiv Executive Summary .................................................................................................................................... xv 1. Introduction ...................................................................................................................................... 1 1.1 DC Opportunity Scholarship Program ................................................................................ 1 1.2 Mandated Evaluation of the OSP ........................................................................................ 4 1.3 Contents of This Report .................................................................................................... 12 2. School and Student Participation in the OSP ................................................................................. 15 2.1 School Participation .......................................................................................................... 15 2.2 Student Participation ......................................................................................................... 21 3. Impacts on Key Outcomes At Least Four Years After Application to the Program ...................... 29 3.1 Analytic and Presentation Approaches ............................................................................. 29 3.2 Impacts Reported Previously (Through Year 3) ............................................................... 33 3.3 Impacts on Student Educational Outcomes After At Least Four Years............................ 34 3.4 Impacts on Reported Safety and an Orderly School Climate ........................................... 43 3.5 Impacts on School Satisfaction ......................................................................................... 47 3.6 Chapter Summary ............................................................................................................. 51 4. Exploratory Analysis of OSP Intermediate Outcomes At Least Four Years After Random Assignment .................................................................................................................................... 53 4.1 Selection and Construction of Intermediate Outcomes ..................................................... 53 4.2 Intermediate Outcomes Across Evaluation Years ............................................................ 54 4.3 Impact of the OSP on Intermediate Outcomes At Least Four Years After Random Assignment ....................................................................................................................... 57 4.4 Chapter Summary ............................................................................................................. 61 5. Exposure, Awareness, and Response of DCPS and Private Schools to the OSP ........................... 63 References ................................................................................................................................................... 73 Appendix A. Research Methodology....................................................................................................... A-1 Appendix B. Benjamini-Hochberg Adjustments for Multiple Comparisons ......................................... B-1 Appendix C. Sensitivity Testing .............................................................................................................. C-1 iii Contents (continued) Page Appendix D. Relationship Between Attending a Private School and Key Outcomes ............................. D-1 Appendix E. Detailed ITT Tables ........................................................................................................... E-1 Appendix F. Exploration of Whether Parents Get What They Seek From School Choice ..................... F-1 Appendix G. To What Extent Are Treatment Effects of the OSP Observed Across the Outcome Test-Score Distribution? Quantile Regression Analysis of the OSP ................................. G-1 Appendix H. Intermediate Outcome Measures ........................................................................................ H-1 iv List of Tables Page Table ES-1 Features of Participating OSP Private Schools Attended by the Treatment Group in 2008-09 ..................................................................................................................... xxvii Table 1-1 OSP Applicants by Program Status, Cohorts 1 Through 6, Years 2004-09 ....................... 3 Table 2-1 Features of Private Schools Participating in the OSP by Participation Status, 2004-05 through 2008-09 ................................................................................................. 17 Table 2-2 Features of Participating OSP Private Schools Attended by the Treatment Group in 2008-09 ......................................................................................................................... 18 Table 2-3 Characteristics of School Attended by the Impact Sample, Year of Application and 2008-09 ...................................................................................................................... 20 Table 2-4 Percentage of the Impact Sample Still in K-12 by Type of School Attended: At Baseline and 2008-09 ........................................................................................................ 27 Table 2-5 Percentage of the Impact Sample Attending Schools Identified Between 2003 and 2005 as in Need of Improvement (SINI): Baseline and 2008-09...................................... 28 Table 3-1 Overview of the Analytic Approaches .............................................................................. 30 Table 3-2 Impact Estimates of the Offer and Use of a Scholarship on the Full Sample: Academic Achievement, 2008-09 ..................................................................................... 35 Table 3-3 Impact Estimates of the Offer and Use of a Scholarship on Subgroups At Least Four Years After Application: Academic Achievement ................................................... 39 Table 3-4 Estimated Impacts in Months of Schooling From the Offer and Use of a Scholarship for Statistically Significant Reading Impacts After At Least Four Years ................................................................................................................................. 40 Table 3-5 Impact Estimates of the Offer and Use of a Scholarship on Students Forecasted in Grade 12 or Above by 2008-09: Percent with High School Diploma, 2008-09 ............... 42 Table 3-6 Impact Estimates of the Offer and Use of a Scholarship on the Full Sample and Subgroups: Parent Perceptions of Safety and an Orderly School Climate, 2008-09 ........ 45 Table 3-7 Impact Estimates