EARLY ACHIEVERS STANDARDS VALIDATION STUDY Final Report: May 2016 childcare quality & early learning CENTER FOR RESEARCH AND PROFESSIONAL DEVELOPMENT Prepared by: Janet Soderberg, Ph.D., Director of Research & Evaluation, CQEL Gail E. Joseph, Ph.D., Principal Investigator & Director, CQEL Sara Stull, Ph.D., Research Associate, CQEL Nail Hassairi, Ph.D. Student, Research Assistant, CQEL with contributions from: Songtian (Tim) Zeng, Alice Porter, DaHye Keum and April Boyce Scientific Reviewers: Milagros Nores, Associate Director, National Institute for Early Education Research (NIEER) Min Sun, Assistant Professor of Educational Policy, University of Washington Deborah McCutchen, Professor of Educational Psychology and Associate Dean for Research, University of Washington This work was supported by funding from the Department of Early Learning as part of the Race-to-the Top Early Learning Challenge Grant of 2011. This is a publication from the Childcare Quality and Early Learning Center for Research & Professional Development (CQEL). CQEL is a research center of the College of Education at the University of Washington. CONTENTS Figures ii Tables ii Acknowledgements iii Executive Summary 1 Introduction 6 1. Early Achievers at a Glance 6 2. QRIS Validation 9 3. Early Achievers Validation Study 10 Data collection team 11 Study recruitment 11 Sampling 12 Selection criteria 12 4. Participating Providers and Programs 13 5. Participating Children and Families 15 6. Assessment Measures and Procedures 17 Program quality measures 17 Inter-rater reliability (IRR) 19 Data collection procedures 20 Individual child assessments 22 Data collection procedures 24 7. Child, Family, and Program Characteristics 25 8. Analysis 26 9. Study Results: Child Outcomes 27 10. Study Results: Research Evaluation Questions 35 Early Achievers Quality Rating Level and Child Outcomes 36 Early Achievers Quality Standards and Child Outcomes 38 Early Achievers Observational Measures of Quality and Child Outcomes 39 Extended Analyses 42 11. Summary of Early Achievers and Child Outcomes 45 12. Survey of Provider and Family Experiences in Early Achievers 47 13. Study Limitations 59 14. Recommendations from the Statewide Internal Standards Validation Study of Early Achievers 60 References 61 Appendix A: Early Achievers Rating Structure 66 Appendix B: QRIS Validation Studies 69 Appendix C: Imputation 72 Appendix D: Sensitivity Analysis 80 Appendix E: Score Models and Results Tables 84 Appendix F: Early Achievers Standard Indicator Factor Analysis 134 Appendix G: Spline Analysis 144 Appendix H: Extended Analyses 149 Appendix I: Director/FCC Owner Survey 159 Appendix J: Teacher/provider Survey 231 Appendix K: Parent Survey 292 i FIGURES 1.1 A Logic Model for QRISs 3.1 Data Collection Timeline 4.1 Number of Participating Programs 4.2 Map of Participating Programs by Child Care Aware Regions 4.3 Program Rating Levels 5.1 Participating Children by Program Type 5.2 Participating Children by Program Rating Level 9.1 Percentage of Entering Kindergarten Children at or Above the Mean/Established Age-Expected Skill Level 9.2 Percentage of Low-income Entering Kindergarten Children at or Above the Mean/Established Age-Expected Skill Level 10.1 ECERS-R Item 25 (Nature/Science) TABLES 4.1 Classroom Characteristics 5.1 Demographics of Children 6.1 Classroom Observation Measures 6.2 Data Collector Inter-rater Reliability Rates by Instrument 6.3 Descriptive Statistics for Classroom Quality Measures 6.4 Validation Study Constructs and Individual Child Assessment Measures 9.1 Descriptive Statistics—Individual Child Assessment Scores 9.2 Descriptive Statistics—EHS/HS/ECEAP 9.3 Descriptive Statistics—Individual Child Assessment: Spanish Battery 9.4 Preschool Full Sample Fall and Spring Scores by Assessment and Income Status 9.5 Infant/Toddler Full Sample Fall and Spring Scores by Assessment and Income Status 9.6 Preschool Full Sample Differences Across Fall and Spring Measures 9.7 Preschool Change Scores for Preschoolers from Low-income Families 9.8 Gains for Full Infant/Toddler Sample 9.9 Gains for Infant/Toddler Children from Low-income Families 10.1 Early Achievers Classroom/FCC Home Environment and Interactions Component of the Facility Curriculum and Learning Environment and Interactions Standard Area 10.2 Alternate Early Achievers Rating Structure 11.1 Summary Table: All Children in All Sites 12.1 Directors/Owners/Teachers: Highest Education Level Completed 12.2 Directors/Owners/Teachers: Current Annual Salary 12.3 CCC Directors’/FCC Owners’/Teachers’ Top Three Reasons Children Were Unenrolled in Past 12 Months 12.4 CCC Directors/FCC Owners/Teachers: Primary Curriculum in Classroom 12.5 Teachers’/Directors/Owners’ Satisfaction with the Early Achievers Rating Process 12.6 Directors/Owners: Positive Changes in Program Practices Since Early Achievers Enrollment 12.7 Teachers: Positive Changes in Program Practices Since Early Achievers Enrollment 12.8 Parents/Guardians: Highest Level of Education Completed 12.9 Parents/Guardians: Highest Level of Education Completed (Low-income Group Only) 12.10 Parents/Guardians: Most Important Reasons for Selecting Child Care Program 12.11 Parent Ratings of Importance of Teacher/Provider Practices 12.12 Parents’ Observations of Teacher/Provider Practices 12.13 Frequency of Parent Participation in Activities with their Children ii Acknowledgements We would like to thank the Department of Early Learning for sponsoring this evaluation. We wish to acknowledge the Early Achievers Validation Study Research Team who assisted with this study, including Sara Stull, Da Hye Keum, Nail Hassairi, Songtian (Tim) Zeng, Britta Bergan, CJ Thomas, James Tanliao and Kevin Cummings. We would also like to thank the members of our data collection team who conducted assessments with children across the study year: Lea Bachman, Peggy Johnson, Kristen Nicholson, Amy Gutierrez, Natalie Ceballos, Natalie Nichols, Maryam Diriye, Andrea Turner, Colleen Conroy, Jenny Sun , Tammy Park, and Beth Endresen. Additionally, we are grateful for the contributions of all CQEL staff who were instrumental in conducting classroom quality observations throughout the course of the study. The authors would like to offer appreciation to all those who supported the work in various capacities. This includes: Derek Wang, DeEtta Simmons, Johnna Holt, Jamie Phillips, Virginia Tse, Angela Haberman, Catherine Duffy, Donna Chang, Robin Scarlet, and Mark McCarthy. We would like to thank Prak Nisarat, Sara Stull, April Boyce and Da Hye Keum who provided assistance with final report preparation. Finally, we are grateful for all of the participants- CCC directors, FCC owners, providers, and teachers- who allowed us to visit their sites during this important study. We are indebted to the parents and families who chose to participate, and to the children themselves. iii Early Achievers Standards Validation Study childcare quality & early learning CENTER FOR RESEARCH AND PROFESSIONAL DEVELOPMENT EXECUTIVE SUMMARY The Statewide Internal Standards Validation Study of Early Achievers addressed whether elements of Washington State’s Quality Rating and Improvement System (QRIS) for early care and education are associated with measurable gains in children’s outcomes across developmental domains. With this validation study, Washington joins a handful of other states that have attempted to examine whether sites that receive higher ratings are actually producing better outcomes in terms of child development (Karoly, 2014). The University of Washington Childcare Quality & Early Learning Center for Research & Professional Development (CQEL) conducted this study from June 2014 through December 2015 in partnership with the Washington State Department of Early Learning (DEL). Nearly every state in the nation has created a QRIS to raise the quality of early care and education. Washington State’s QRIS is called Early Achievers. At the time of this study Early Achievers was a voluntary program, but since the passage of the Early Start Act in 2015, participation has been mandated for all child care sites serving children with subsidies as well as Early Childhood Education and Assistance Program (ECEAP) sites. With support from a federal Race to the Top Early Learning Challenge grant, Early Achievers went statewide beginning July 1, 2012. Its goal—consistent with the overarching purpose of QRIS—is to help early learning professionals offer high-quality child care that supports children’s learning and development. Early Achievers is comprised of elements including a standard area rating structure, a quality rating system (quality levels 1–5), coaching and professional development, incentives to attain higher levels of quality, and information sharing among families. In the validation process, independent observers assess quality across standard areas that include Child Outcomes, Family Engagement, Professional Development, and Curriculum and Staff Supports. These ratings are intended to distinguish meaningful levels of quality and subsequent improved outcomes for children. 1 The validation study Our validation study employed research methods including records review, surveys with early learning professionals and parents and families, observational assessments of center care classrooms and family child care homes, and individual direct child assessments to explore three key research questions (shown below). Focus areas of the study were provider characteristics and program quality, child outcomes (residual gains in children’s learning