Final Grantee Report: Texas 21st Century Learning Centers Program Cycle 8, Year 4 (AY2016-17) YES Prep Public Schools, Inc. Durand Research and Marketing Associates, LLC July 31, 2017 Acknowledgements: The firm of Durand Research and Marketing Associates, LLC, wishes to express its gratitude to Christa Queen-Sutherland, YES Prep’s ACE Program Director, to Maria Maradiaga, Yes Prep’s ACE Family Engagement Specialist, and to the eight ACE site coordinators for their invaluable assistance with this evaluation. The substantial, continuous, and always-timely support of these individuals, their patience in answering many questions, their facilitation of visits to program sites, and their ready willingness to go “above and beyond” are particularly appreciated. Texas 21st Century Community Learning Centers Cycle 8, Year 4 (AY2016-17) YES Prep Public Schools Table of Contents I. Executive Summary…………………………………………………………………………………………………………1 II. Introduction and Purpose of Program……………………………………………………………………………..6 III. Evaluation Plan/Strategy……………………………………………………………………………………………… 15 IV. Program Implementation and Support Strategy…………………………………………………………….20 V. Program Participation ………………………………………………………………………………………………… 30 VI. Program Intermediate Outcomes………………………………………………………………………………… 41 VII. Program Impacts……………………………………………………………………………………………………………52 VIII. Stakeholder Perceptions……………………………………………………………………………………………… 71 IX. Principal Evaluation Findings, Evaluator Commentary, Recommendations…………………….92 X. Assessment of Evaluator Recommendations and Site Coordinator Commentary…………..101 XI. Evaluator Information…………………………………………………………………………………………………..101 XII. References Utilized in the Evaluation…………………………………………………………………………….104 XIII. Appendices Appendix A: Grantee and Centers’ Logic Models (1st Recommended Deliverable)………….109 Appendix B: Implementation and Alignment (2nd Recommended Deliverable) ………………211 Appendix C: Outputs: Activities and Participation (3rd Recommended Deliverable) ……….229 YES Prep Evaluation, Cycle 8, Year 4 I. Executive Summary In this report, results are presented from both a process and an outcomes evaluation of the 21st Century Community Learning Centers program of the YES Prep Public Schools, Inc., Cycle 8, Year 4 (AY2016-17). Beginning in 2013 the YES Prep Public Schools, Inc. (hereafter simply “YES Prep”), received grant funding from the Texas Education Agency (TEA) for an after-school program under provisions of the Federal “No Child Left Behind Act” as reauthorized in 2002. This Federal program was subsequently replaced in 2015 by the “Every Student Succeeds Act,” which shifted more responsibility away from the Federal government to the states. Under provisions of these acts, YES Prep received a funding Cycle 8 grant to provide 21st Century Community Learning Centers (21st CCLC) programming at eight (8) campuses or “program center sites.” These eight program center sites included East End, Fifth Ward, Southeast, Gulfton, North Central, North Forest, Northside, and Southwest. All of these sites are located in Harris County, Texas, in the greater Houston metropolitan area. In developing its after-school program, YES Prep designed and implemented a program in furtherance of the following purposes of the Federal legislation: “…to provide all children significant opportunity to receive a fair, equitable, and high-quality education, and to close educational achievement gaps.” (Section 1001, Every Student Succeeds Act). In doing so, YES Prep carried out activities in four component areas required for what TEA “re-branded” from 21st Century Community Learning Centers to “Afterschool Centers on Education” (hereafter referred to as ACE in this report). These four required component areas were Academic Support, Enrichment, Family Services, and College and Career Readiness. Among the most notable process evaluation findings about the YES Prep ACE Cycle 8, Year 4 (AY 2016-17) program, findings reached by the program’s external evaluator, Durand Research and Marketing Associates, LLC, were the following -- 1. The grantee (YES Prep ACE) supported the program’s theory of action as set forth by TEA and its consultants as well as the implementation of that theory at each of the eight program sites. The grantee did so with through resources, leadership, guidance, and staffing – among other ways. 2. The logic models for the grantee and for each center represented quite well the program in resources, implementation practices, outputs, outcomes and intended, ultimate impact. Moreover, each of the logic models was modified appropriately in light of program changes, especially changes following the fall term (See Appendix A to this report.) 3. The YES Prep ACE program was well-implemented and as intended. 4. The evaluation team found considerable site to site variation in total operating budgets and in staffing. Such variation was fully expected in that students’ needs differed across the centers (as identified by a needs assessment), variations in program activities and in teaching resources were found from site to site, and differences were found in “student mix,” especially in economic disadvantages and the proportion at-risk. Finally, the team also expected and identified economies of scale with regard to budgets and numbers of participants that appeared to influence site to site variability in budgets. 3 | Page YES Prep Evaluation, Cycle 8, Year 4 5. The YES Prep Cycle 8, Year 4, ACE program served a total of 2862 students, an increase of 12 students over the Year 3 Program and of 508 students over the Year 1 program. Of the Year 4 students attending the program, 1234 or about 43% were “regular participants.” This latter number represented an increase of 49 regular participants from program Year 3 and 516 from program Year 1. Finally, during the current academic year, the YES Prep ACE program also had 811 total adults in attendance. 6. Consistent with Federal, state and program intentions, the ACE program at YES Prep enrolled students diverse in ethnicity, but also in gender, in being at-risk, in being economically disadvantaged, in having limited English proficiency, and in having English as their second language. 7. All student “regular participation,” adult participation, type of activity (activity component), and program operations requirements were fully met or more generally exceeded at all of the eight centers. 8. Considerable variation was found across the centers in the number as well as in the percentage of time for each activity component type (i.e., Academic Support, Enrichment, Family Services, and College and Career Readiness). Further, this variation was found both in the spring and in the fall terms. Such variation was expected by the evaluation team given that activities at a center and in an academic term were a function of differences in students’ needs (as measured by a needs assessment), student-parent “voice and choice” surveys, alternative scheduling choices, and differences in students’ grade level “mix” among a number of considerations. 9. Surveys capturing the perceptions of program stakeholders revealed a strong reservoir of support and approval on the part of parents and students; ratings of either excellent or good on the part of site coordinators in the ACE program having met its intended objectives; and generally positive program perceptions among ACE instructors. 10. The ACE program did an excellent job of keeping stakeholders and their perceptions informed. The program utilized quite a variety of means and media to do so. The evaluation team found particularly commendable the development and implementation of an extremely informative and attractive YES Prep ACE Web site (http://www.yesprep.org/ACE). Below, the most notable YES Prep ACE Cycle 8, Year 4 outcomes evaluation findings by the Durand Research and Marketing Associates, LLC, evaluation team are summarized: 1. While varying by ACE site and by subject area, the evaluation team found, through an examination of data on a panel of students (that is, the exact same students) from the fall of 2015 through the spring of 2017, grade improvements for each center in each of four subjects: reading, math, science, and social studies. Moreover, the team also found evidence across the ACE centers of grade “maintenance” or “reinforcement,” a result that was even more enhanced if one also considers those students for whom there was “no change needed” in grades (i.e., “A” grades at the start and end of the study period). 2. On average, the percentage of students passing courses was higher in the spring of 2017 than in the fall of 2015 among the same panel of students. 3. In the course of analyzing data available in TEASE, the evaluation team compared “regular” (30 days or more of program participation) and “non-regular” YES Prep ACE 4 | Page YES Prep Evaluation, Cycle 8, Year 4 participants from the two-year panel for the outcomes of changes in school day absences, reading, math, science, and social studies grades as well as changes in in non-criminal referrals. The team found that regular participants showed improvements compared to non-regular ones in all of the outcomes. However, this was not found to be true at all of the centers. In other words, we did find some variations across the centers in the associations between outcomes and regular vs. non-regular participation. 4. But are these differences in outcomes between “regular” and. “non-regular” ACE participants really “program outcomes” or do they derive from something else, especially background or