February 15, 2018

Evaluation of 2018 Extended

Learning Opportunities-Summer

Adventures in Learning (ELO

SAIL) in Title I Schools

Nyambura Susan Maina, Ph.D. Natalie Wolanin, M.Ed.

Program Evaluation Unit

850 Hungerford Drive Rockville, Maryland 20850 301-740-3000

Dr. Jack R. Smith Superintendent of Schools

Dr. Janet S. Wilson Associate Superintendent, Office of Shared Accountability

Published for the Office of Shared Accountability Copyright © 2018 Montgomery County Public Schools, Rockville, Maryland February 2019 Office of Shared Accountability Program Evaluation Unit

Highlights: Evaluation of 2018 Extended Learning Opportunities‒Summer Adventures in Learning (ELO SAIL) in Title I Schools Nyambura Susan Maina, Ph.D. and Natalie Wolanin, M.Ed.

Background positive, statistically and practically significant impacts All rising K–2 students in Montgomery County Public were found for all students overall, students receiving Schools’ (MCPS) Title I schools were eligible to attend the FARMS and ESOL services, as well as for most of the Extended Learning Opportunities Summer Adventures in MCPS focus groups (Figure 1). Learning (ELO SAIL) program regardless of their level of reading and mathematics skills. The 2018 ELO SAIL was Figure 1. Effects of 2018 ELO SAIL program on fall reading offered for 5 weeks (25 days) in 21 schools; 17 schools performance (K) or avoiding summer loss (Grade 1 and 2) offered an extended day program. relative to non-ELO SAIL peers Group Measure K Grade 1 Grade 2 *** *** *** Purpose and Scope of Study DIBELS d = .43 d = .30 d = .15 The study examined: a) differences in reading and All students * *** ** AP-PR mathematics performance for Kindergarten students with and d = na d = .27 d = .16 without ELO SAIL experiences and b) whether the 2018 ELO *** *** *** Students receiving DIBELS d = .46 d = .34 d = .18 SAIL program lessened summer academic loss in reading and FARMS services * *** * mathematics for Grade 1–2 students who attended. The study AP-PR addressed four questions: d = na d = .30 d = .16 *** *** * Students receiving DIBELS 1. What were the demographic characteristics of students d = .42 d = .34 d = .14 ESOL services who participated in summer 2018 ELO SAIL? * *** ** AP-PR 2. What were the attendance patterns for students who d = na d = .23 d = .20 Students receiving ** * participated in summer 2018 ELO SAIL? DIBELS Special Education d = .20 d = .42 d = .33 3. How did students who attended ELO SAIL perform on services AP-PR d = na d = .10 d = .11 the fall reading assessments, DIBELS and MCPS AP- * PR, compared with peers who did not attend ELO Non-FARMS All DIBELS d = .34 d = .14 d = - .53 Other Student SAIL? * Groups AP-PR 4. How did students who attended ELO SAIL perform on d = na d = .48 d = .50 the fall mathematic assessment MAP-P, compared with * * DIBELS peers who did not attend ELO SAIL? Non-FARMS Black d = .43 d = .36 d = .12 or African American d = - AP-PR d = na d = .17 .20 Methodology *** DIBELS A quasi-experimental design was used to compare the Non-FARMS d = .51 d = .15 d = .16 Hispanic/Latino * performance of ELO SAIL participants and non-participants; AP-PR only students who attended 17 days or more of ELO SAIL d = na d = .07 d = .09 *** * * were included in the participant group. Analyses of student DIBELS FARMS All Other d = .79 d = .81 d = .74 outcomes statistically controlled for demographic differences Student Groups * AP-PR and performance before ELO SAIL (Grade 1 and 2). d = na d = .27 d = .54 *** *** *** DIBELS Key findings FARMS Black or d = .50 d = .39 d = .36 African American * * AP-PR  More than one-half of eligible students (51%, N=3,684) d = na d = .33 d = -.14 *** *** ** voluntarily participated in the 2018 ELO SAIL program. DIBELS FARMS d = .43 d = .32 d = .16  Most (76%) were receiving FARMS services, 59% were Hispanic/Latino * *** ** AP-PR receiving ESOL services, and 11% were receiving d = na d = .29 d = .23 special education services. Green shading indicates positive, statistically (*p< .05, **p < .01, ***p <  Over 75% attended at least 20 of the 25 days. .001) and practically significant effect (d=>0.15, cut-off for practically meaningful effects).  Indicates groups with <50 students. N/A indicates  Participation varied by school (48–64%). that effect sizes are not available.  Overall, K–2 students who participated in ELO SAIL demonstrated stronger reading skills as measured by fall DIBELS and AP-PR assessments and avoided summer reading loss at higher rates than those who did not. Across the grade levels and measures for reading,

1

February 2019 Office of Shared Accountability Program Evaluation Unit

Highlights: Evaluation of 2018 Extended Learning Opportunities‒Summer Adventures in Learning (ELO SAIL) in Title I Schools Nyambura Susan Maina, Ph.D. and Natalie Wolanin, M.Ed.

 Overall, K–2 students who participated in ELO SAIL sizes, enrichment extended day program activities, demonstrated stronger mathematics skills as measured English language development. by fall MAP-P RIT scores and avoided summer reading loss at higher rates than those who did not. Across grade 3. Continue to use a variety of approaches to increase levels, statistically and practically significant impacts of and encourage participation in ELO SAIL— 2018 ELO SAIL on performance in K, and avoiding particularly explaining benefits of ELO SAIL to all summer loss for Grade 1 and 2, were found for all stakeholders and emphasizing that the primary goal students overall, students receiving FARMS and ESOL of ELO SAIL is academic improvement. services, as well as most MCPS focus groups (Figure 2). 4. Replicate the outcome study over time using Figure 2. Effects of 2018 ELO SAIL on fall MAP-P different student population to see if the observed performance (K) or avoiding summer loss (Grade 1 and 2) findings are stable. relative to non-ELO SAIL peers Group Measure K Grade 1 Grade 2 5. Include in future studies a formative evaluation *** *** *** component that incorporates teachers’ and parents’ All students MAP-P d = .24 d = .34 d = .27 feedback regarding their experience with the *** *** *** program. Students receiving FARMS services MAP-P d = .29 d = .30 d = .27 6. Share and store ELO SAIL participation and Students receiving *** *** *** attendance data in the Performance Matters ESOL services MAP-P d = .28 d = .33 d = .22 platform. Students receiving * Special Education services MAP-P d = .09 d = .09 d = .30 Non-FARMS All Other Student ** Groups MAP-P d = .01 d = .51 d = .24 Non-FARMS Black or African American MAP-P d = .30 d = .32 d = .14

Non-FARMS * *** ** Hispanic/Latino MAP-P d = .28 d = .53 d = .41 ** * FARMS All Other Student Groups MAP-P d = .38 d = .50 d = .08 *** *** * FARMS Black or African American MAP-P d = .43 d = .42 d = .24 *** *** *** FARMS Hispanic/Latino MAP-P d = .25 d = .24 d = .29 Green shading indicates positive, statistically (*p< .05, **p < .01, ***p < .001) and practically significant effect (d=>0.15, cut-off for practically meaningful effects).

Recommendations 1. Continue to provide ELO SAIL to rising K–2 students.

2. Sustain and work on increasing the positive impacts of ELO SAIL by continuing to utilize many of the strengths of the program cited as best practices in literature for voluntary summer programs, such as: early planning, using MCPS teachers, small class

2

Table of Contents

List of Tables ...... iii List of Figures ...... iv Executive Summary ...... v Purpose and Scope of Study ...... v Summary of Methodology ...... v Summary of Findings ...... vi Conclusion ...... x Recommendations ...... xi Introduction ...... 1 Background ...... 1 Overview of the 2018 ELO SAIL Program ...... 1 Planning and Coordination of the 2018 ELO SAIL Program ...... 2 Staffing and Cost of ELO SAIL 2018 ...... 4 Literature Highlights ...... 5 Scope and Evaluation Questions ...... 7 Purpose of Study ...... 7 Evaluation Questions ...... 8 Methodology ...... 8 Study Design ...... 8 Study Sample ...... 9 Data Sources ...... 9 Fall 2018–2019 Outcome Measures: Literacy ...... 10 Fall 2018–2019 Outcome Measures: Mathematics ...... 11 Procedures for Analyses ...... 11 Strengths and Limitations ...... 12 Results ...... 13 Evaluation Question 1. What are the demographic characteristics of students who participated in summer 2018 ELO SAIL? ...... 13 Evaluation Question 2. What are the attendance patterns for students who participated in summer 2018 ELO SAIL? ...... 15

Program Evaluation i Evaluation of the ELO SAIL Program Evaluation Question 3. How did students who attended ELO SAIL perform on fall DIBELS and MCPS AP-PR assessments compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL on the reading performance vary by student subgroups? ...... 16 Performance on Fall DIBELS Assessment ...... 16 Performance on Fall MCPS AP-PR ...... 19 Summary of Findings for Question 3...... 24 Evaluation Question 4. How did students who attended ELO SAIL perform on the fall MAP-P mathematics assessments compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL on mathematics performance vary by student subgroups? ...... 26 Summary of Findings for Question 4...... 28 Discussion and Conclusion ...... 29 Recommendations ...... 31 Acknowledgements ...... 32 References ...... 33 Appendices ...... 36 Appendix A: 2018 ELO SAIL Schools and Staff Allocation for 2018 ...... 36 Appendix B: K–2 Beginning of Year Benchmark Goals for DIBELS ...... 37 Appendix C: AP-PR Fall Target Reading Levels ...... 38 Appendix D: 2018 ELO SAIL Participation by School...... 39 Appendix E: Summary Statistics for Fall DIBELS Composite Score by Grade ...... 41 Appendix F: Summary Statistics for Fall MAP-RIT Scores by Grade ...... 44

Program Evaluation ii Evaluation of the ELO SAIL Program List of Tables

Table 1. Number and Percentage of Students Who Attended 2018 ELO SAIL by Grade ...... 14 Table 2. Background Characteristics of Students ELO SAIL Participants and Non-Participants15 Table 3. Adjusted Mean Fall DIBELS Composite Scores For Kindergarten Students by ELO SAIL Participation ...... 17 Table 4. Adjusted Mean Fall DIBELS Composite Scores For Grade 1 Students by Student Subgroups and ELO SAIL Participation...... 18 Table 5. Adjusted Mean Fall DIBELS Composite Scores For Grade 2 by Student Subgroups and ELO SAIL Participation ...... 19 Table 6. Descriptive Statistics and Results of Chi-square Test for Kindergarten Subgroups who were Reading or Pre-reading on Fall AP-PR, by ELO SAIL Participation ...... 21 Table 7. Estimated ELO SAIL Program Effects Associated with Avoiding Summer Loss in Reading (AP-PR): Grade 1 Students Relative to Non-ELO SAIL Peers...... 22 Table 8. Estimated ELO SAIL Program Effects Associated with Avoiding Summer Loss in Reading (AP-PR): Grade 2 Students Relative to Non-ELO SAIL Peers...... 23 Table 9. Summary of 2018 ELO SAIL Program Effects on Fall Reading Performance Relative to Non-ELO SAIL Students by Grade and Outcome Measure ...... 25 Table 10. Mean Fall MAP-P RIT Mathematics Score for Kindergarten Students by ELO SAIL Participation ...... 26 Table 11. Mean Gain in MAP-P RIT Mathematics Score from Spring- to-Fall 2018 for Grade 1 by ELO SAIL Participation ...... 27 Table 12. Mean Gain in MAP-P RIT Mathematics Score from Spring-to-Fall 2018 for Grade 2 by ELO SAIL Participation ...... 28 Table 13. Summary of 2018 ELO SAIL Program Effects on Fall Mathematics Performance Relative to Non-ELO SAIL Students by Grade ...... 29 Table D1. 2018 ELO SAIL Participation by School ...... 39 Table D2. Summary Statistics on Full Days Attended ...... 40 Table E1. Summary Statistics DIBELS Composite Scores by Subgroups: Kindergarten ...... 41 Table E2. Summary Statistics DIBELS Composite Scores by Subgroups: Grade 1 ...... 42 Table E3. Summary Statistics DIBELS Composite Scores by Subgroups: Grade 2 ...... 43 Table F1. Summary Statistics Fall MAP-RIT Scores: Kindergarten ...... 44 Table F2. Summary Statistics for Fall MAP-RIT Scores and Change from Spring to Fall 2018: Grade 1 ...... 45 Table F3: Summary Statistics for Fall MAP-RIT Scores and Change from Spring to Fall 2018: Grade 2 ...... 46

Program Evaluation iii Evaluation of the ELO SAIL Program List of Figures Figure 1. ELO SAIL and non-ELO SAIL participants by Focus Group...... vii Figure 2. Effect size plots of 2018 ELO SAIL program on performance of the fall DIBELS assessment relative to non-ELO SAIL peers...... viii Figure 3. Effect size plots of 2018 ELO SAIL program on avoiding summer loss in reading (maintaining or increasing spring 2018 AP-PR book level) relative to non-ELO SAIL for Grades 1 and 2 students...... ix Figure 4. Effect size plots of 2018 ELO SAIL program on fall MAP-P mathematics performance (Kindergarten) or net spring-to-fall 2018 gain in MAP-P RIT scores (Grade 1 and 2) relative to non-ELO SAIL peers...... x Figure 5. MCPS initiatives for preventing summer learning loss 2018...... 2 Figure 6. Suggested daily instructional models for 2018 ELO SAIL half-day option...... 3 Figure 7. Suggested daily instructional models for 2018 ELO SAIL extended day option...... 4 Figure 8 Attendance pattern: distribution of ELO SAIL participants by full days attended...... 16 Figure 9. Performance of Kindergarten who were reading (AP-PR book >3) on fall AP-PR assessments by ELO SAIL participation...... 20

Program Evaluation iv Evaluation of the ELO SAIL Program Executive Summary

Extended Learning Opportunities‒Summer Adventures in Learning (ELO SAIL) is a voluntary summer program offered in Montgomery County Public Schools (MCPS) Federally Funded Title I schools. The primary goal of ELO SAIL is to alleviate summer academic loss and promote continued learning. All K–2 students in Title I schools were eligible to attend the ELO SAIL program regardless of their level of reading and mathematics skills. In 2018, the ELO SAIL program was offered for 5 weeks (25 days) in 21 schools. Unlike prior years, the 2018 ELO SAIL program offered an extended day option for 6.5 hours each weekday in 17 of the schools. Funding for the ELO SAIL program are provided through Title I of the Every Student Succeeds Act of 2015.

With a goal to avert the achievement loss that students may experience in summer, the specific purposes of 2018 ELO SAIL were to:  Preview and work on literacy and mathematics skills for the next grade  Maintain and improve skills in literacy and mathematics  Expose students to science, arts, and physical literacy through the full-day ELO program  Familiarize young students with school and school routines (kindergarten students)  Build students’ confidence and joy for learning  Strengthen basic skills that are preconditions of later learning  Provide continuing English language instruction for speakers of other languages

Purpose and Scope of Study

This evaluation, conducted by the Office of Shared Accountability, was requested by the Office of the Chief Academic Officer. The purpose of the study was to a) examine differences in reading and mathematics performance for kindergarten students with and without ELO SAIL experiences and b) examine whether the 2018 ELO SAIL program lessened summer academic loss in reading and mathematics for Grades 1–2 students who attended.

This study was guided by the following evaluation questions: 1. What were the demographic characteristics of students who participated in summer 2018 ELO SAIL? 2. What were the attendance patterns for students who participated in summer 2018 ELO SAIL? 3. How did students who attended ELO SAIL perform on the fall reading assessments, Dynamic Indicators of Basic Early Literacy Skills® (DIBELS) and MCPS Assessment Program‒Primary Reading (AP-PR), compared with peers who did not attend ELO SAIL? 4. How did students who attended ELO SAIL perform on the fall mathematic assessment Measures of Academic Progress‒Primary Grades (MAP-P), compared with peers who did not attend ELO SAIL?

Summary of Methodology

A quasi-experimental design was used to compare the performance of students who participated in ELO SAIL (N=3,684) and non-ELO SAIL peers (N=3,171) on districtwide measures used to assess student learning for K–2 students. The analytical sample varied by grade and measurement; only students who attended 17 days or more of ELO SAIL were included. Typical of a quasi-

Program Evaluation v Evaluation of the ELO SAIL Program experimental design, multivariate statistical models were used to control for demographic characteristics and baseline equivalence on prior achievement (Grade 1 and 2) between the two groups so that any differences in performance could be more reliably associated with the ELO SAIL program. Further, two-way contingency table analyses (Pearson’s chi-square), logistic regression, and/or Analysis of Covariance (ANCOVA) models were used as appropriate. Lastly, effect sizes for group differences were computed and used to quantify the magnitude of ELO SAIL impacts to determine whether the detected effects were practically meaningful in an educational setting (d>=0.15) and to facilitate comparisons across the outcome measures and grade-levels. Descriptive summary statistics were computed for characteristics of students who attended ELO SAIL and for their attendance patterns. At the subgroup level, analyses were disaggregated by students receiving Free and Reduced-price Meals System (FARMS), English for Speakers of Other Languages (ESOL), or Special Education services, and the MCPS monitoring and focus groups: Non-FARMS All Other Student Groups (monitoring group), Non-FARMS Black or African American, Non-FARMS Hispanic/Latino, FARMS All Other Student Groups, FARMS Black or African American, and FARMS Hispanic/Latino.

Summary of Findings

Evaluation Question 1. What were the demographic characteristics of students who participated in summer 2018 ELO SAIL?

A total of 3,684 students participated in the 2018 ELO SAIL program, representing more than one half of the 7,126 K–2 students attending Title I schools as of September 30, 2018. The ELO participants were equitably distributed across grade levels. Looking within grade, the highest participation was in kindergarten (57%) followed by Grade 1 (53%) and Grade 2 (51%). The participation rates by school varied from 48% to 64%, indicating a 16-point difference. Of the 3,684 students who participated in ELO SAIL, 53% were male, 66% were Hispanic/Latino, 23% were Black or African American, 6% were Asian, 4% White, and about 2% were multiple races and ethnicities. Most (76%) were receiving FARMS services, 59% receiving ESOL services, and 11% receiving special education services. The percentage of ELO SAIL participants and nonparticipating peers across MCPS focus groups were similar with the exception of the monitoring group where the percentage of non-ELO SAIL students was twice that of the ELO SAIL group (Figure 1).

Program Evaluation vi Evaluation of the ELO SAIL Program 100 90 80 70 60 54.3 48.2 50 40 30 13.9 15.2 13.9 20 11.1 10.8 Percentage of students of Percentage 7.1 7.2 10 6.3 5.9 5.9 0 Non-FARMS Non-FARMS Non-FARMS FARMS All FARMS Black FARMS All Other Black or Hispanic/Latino Other Student or African Hispanic/Latino Student Groups African Groups American American

ELO SAIL (N=3,684) Non-ELO SAIL (N=3,171)

Figure 1. ELO SAIL and non-ELO SAIL participants by Focus Group.

Evaluation Question 2. What were the attendance patterns for students who participated in summer 2018 ELO SAIL?

Analyzing the student attendance data was intended to clarify the proportion of attendees who got a sufficient dose of ELO SAIL. Nearly all students (99%) who registered for ELO SAIL also attended. On average, students attended for 20–21 days, representing 80–85% of the available days, across all the subgroups, with a median of 23–24 days attended across the student subgroups. Over three fourths (75%) attended for 20 or more days. This finding suggests that Title I program efforts were successful in monitoring and encouraging high attendance among participants.

Evaluation Question 3. How did students who attended ELO SAIL perform on the fall reading assessments DIBELS and MCPS AP-PR, compared with peers who did not attend ELO SAIL?

Kindergarten, Grade 1, and Grade 2 students who participated in ELO SAIL demonstrated stronger reading skills as measured by fall DIBELS and AP-PR assessments than those who did not. Across the grade-level and measures for reading, differences were most pronounced in students receiving FARMS services, K–2 students overall, and students in the FARMS Black or African American and FARMS Hispanic/Latino focus groups.

Performance on Fall DIBELS. ELO SAIL participation had a positive statistically significant and practically meaningful effect on the performance of the fall DIBELS assessment for: Kindergarten (d=.43), Grade 1 (d=0.30), and Grade 2 (d=0.15). Regardless of grade and subgroup, ELO SAIL participants had significantly higher DIBELS composite scores relative to their non-ELO SAIL peers. Positive and practically meaningful effects (d>0.15) of the ELO SAIL program were found among the students receiving FARMS and ESOL services as well as students in the monitoring group (kindergarten), Non-FARMS Black or African American (K, 1), Non-FARMS

Program Evaluation vii Evaluation of the ELO SAIL Program Hispanic/Latino (Kindergarten), FARMS All Other Student Groups (Kindergarten), FARMS Black or African American (K, 1, 2), and FARMS Hispanic/Latino MCPS focus groups (K, 1, 2) (Figure 2).

Kindergarten Grade 1 Grade 2

0.85 0.79 0.8 0.75 0.7 0.65 0.6 0.55 0.51 0.5 0.5 0.45 0.43 0.43 0.43 0.39 0.4 0.36 0.36 0.34 0.32 0.35 0.3 0.3

Effect Effect Sizes 0.25

0.2 0.15 0.16 0.15 0.1

0.05 n/a n/a n/a n/a n/a n/a n/a 0 All Non-FARMS Non-FARMS Non-FARMS FARMS All FARMS Black FARMS All Other Black or Hispanic/Latino Other Student or African Hispanic/Latino Student Groups African Groups American American

Figure 2. Effect size plots of 2018 ELO SAIL program on performance of the fall DIBELS assessment relative to non-ELO SAIL peers. Note. Only statistically significant (p<0.5) results are shown. The dashed horizontal line at 0.15 represents the cut-off line for a meaningful/“small” effect. N/A indicates that results for these groups are not shown because they are not statistically significant.

Performance on AP-PR. Participating in the 2018 ELO SAIL program was associated with significantly higher percentages of kindergarten students who were reading (AP-PR book level >3) by fall 2018 (11% vs. 8%). Further, attending ELO SAIL was associated with statistically higher percentages among Non-FARMS Hispanic/Latino, FARMS All Other Student Groups, FARMS Black or African American, and FARMS Hispanic/Latino focus groups who were either reading or pre-reading by fall AP-PR.

The analyses for Grades 1 and 2 examined students who maintained or improved on their maximum AP-PR book level from spring 2018, avoiding summer loss in reading. After controlling for differences in demographic characteristics, the analyses showed that, significantly higher proportions of Grade 1 (d=0.27) and Grade 2 (d=0.16) students were more likely to have maintained or increased their spring maximum book level, thus avoiding summer loss (p<0.05). Figure 3 shows how the magnitude of the estimated impact of the ELO SAIL program on reducing summer loss in reading varied across student focus groups. The effects of participating in the ELO

Program Evaluation viii Evaluation of the ELO SAIL Program SAIL program on avoiding summer loss in reading among two focus groups group in Grade 1 and one focus group in Grade 2 were statistically and practically significant. The benefits were also statistically and practically significant for students receiving ESOL and FARMS services in Grade 1 and 2.

Grade 1 Grade 2 0.35 0.33

0.3 0.29 0.27

0.25 0.23

0.2 Effect Effect sizes 0.16 0.15

0.1

0.05 n/a n/a n/a n/a n/a n/a n/a n/a n/a 0 All Non-FARMS All Non-FARMS Non-FARMS FARMS All FARMS Black or FARMS Other Student Black or African Hispanic/Latino Other Student African American Hispanic/Latino Groups American Groups

Figure 3. Effect size plots of 2018 ELO SAIL program on avoiding summer loss in reading (maintaining or increasing spring 2018 AP-PR book level) relative to non-ELO SAIL for Grades 1 and 2 students. Note. Only statistically significant (p<0.5) results are shown. The dashed horizontal line at 0.15 represents the cut-off line for a meaningful/“small” effect. N/A indicates that results for these groups are not shown because they are not statistically significant.

Evaluation Question 4. How did students who attended ELO SAIL perform on the fall Measures of Academic Progress—Primary Grades Mathematics (MAP-P) assessment compared with peers who did not attend ELO SAIL?

The findings show strong positive and practically significant impacts of the 2018 ELO SAIL program on the mathematics performance for students in kindergarten, Grade 1, and Grade 2 students.

Kindergarten. After controlling for differences in demographic characteristics, the mean fall MAP-P RIT score in mathematics was significantly higher and practically significant for ELO SAIL participants relative to nonparticipants (p<.000, d=0.24). With the exception of kindergarten students in the monitoring group, the fall mathematics RIT scores were statistically higher and educationally meaningful among the kindergarten students who attended the 2018 ELO SAIL program across 4 of the 5 focus groups (Figure 4) and among students receiving FARMS and ESOL services.

Program Evaluation ix Evaluation of the ELO SAIL Program Grades 1 and 2. The analyses for Grades 1 and 2 examined changes in the RIT scores from spring to fall 2018. Analyses for all Grades 1 and 2 found significantly higher and educationally meaningful effects of participating in the ELO SAIL program relative to non-ELO SAIL peers (p<.000, d=.34 and d=.27). Though the effect sizes of ELO SAIL on reducing summer loss in mathematics varied by grade and student focus groups (Figure 4), Grade 1 (d=0.24–0.53) and Grade 2 (d=0.24–0.41); they were practically significant for most groups, indicating that ELO was very influential in maintaining or improving mathematics skills of students who attended.

Kindergarten Grade 1 Grade 2 0.6

0.55 0.53 0.51 0.5 0.5

0.45 0.43 0.41 0.42 0.4 0.38 0.34 0.35 0.29 0.3 0.27 0.28 0.24 0.24 0.250.24 0.25

0.2 Effect Effect sizes 0.15

0.1

0.05 n/a n/a n/a n/a n/a n/a 0 All students Non-FARMS All Non-FARMS Non-FARMS FARMS All FARMS Black or FARMS Other Student Black or African Hispanic/Latino Other Student African Hispanic/Latino Groups American Groups American

Figure 4. Effect size plots of 2018 ELO SAIL program on fall MAP-P mathematics performance (Kindergarten) or net spring-to-fall 2018 gain in MAP-P RIT scores (Grade 1 and 2) relative to non-ELO SAIL peers. Note. Only statistically significant (p<0.5) results are shown. The dashed horizontal line at 0.15 represents the cut-off line for a meaningful/“small” effect. N/A indicates that results for these groups are not shown because they are not statistically significant.

Conclusion

The findings demonstrate that participation in the 2018 ELO SAIL program provided advantages for maintaining or improving reading and mathematics skills compared with not participating. The findings related to benefits of ELO SAIL mirror findings by other researchers showing that when measured a few months after the program, the impact of voluntary summer programs similar to the ELO SAIL program are more evident in mathematics than reading; and that even after participating, some students still experience summer loss. As such, minimizing learning losses represents a positive outcome, even if those students still have a net summer setback. Indeed, these positive impacts on reading and mathematics skills for K–2 students confirm that the ELO SAIL

Program Evaluation x Evaluation of the ELO SAIL Program program represents an essential program because early patterns in academic achievement have profound implications for later learning.

Recommendations

The recommendations are based on findings in this study and research to best practices for optimizing the impact of voluntary summer programs.

1. Due to positive findings for maintaining or improving reading and mathematics skills compared with not participating, continue to provide ELO SAIL to K–2 students.

2. Sustain and work on increasing the positive impacts of ELO SAIL by continuing to utilize many of the strengths of the program cited as best practices in literature for voluntary summer programs such as: early planning for ELO SAIL, using MCPS teachers who are conversant with Common Core State Standards and curriculum offered in MCPS, small class sizes, enrichment extended day program activities, English language development, free transportation and meals, having well documented records of participation and attendance, and promoting practices that emphasizes the strengths in what children impacted by poverty have rather than what they do not (cultural competency).

3. Continue to use a variety of approaches to increase and encourage participation in ELO SAIL a. Explain the benefits and observed impacts of the ELO SAIL program to staff and participating families. In particular, draw attention to the positive impact of the ELO SAIL program. Knowing whether ELO SAIL is able to improve mathematics and reading skills and which areas the improvements were largest would be valuable information to parents, teachers, and staff who refer or who provided the program.

b. Examine communication about the goals of ELO SAIL and possible misconceptions; continue to emphasize that the primary goal of ELO SAIL is academic improvement. Literature shows that limited understanding and misconceptions about voluntary programs such as ELO SAIL program may be at the heart of non-participation.

c. Tap into the potential of kindergarten participation. Kindergarten is the gateway to school; if the trends in 2018 continued, more than half of the incoming kindergarten students would access ELO SAIL and attain the benefits early on.

d. Make a concerted effort to work with other county-wide summer programs, particularly linking ELO SAIL students to libraries and recreation throughout the summer, which may encourage higher interest and participation and sustain the benefits of the ELO SAIL programs.

e. Explore factors preventing families from accessing ELO SAIL.

4. Share and store ELO SAIL participation and attendance data in the Performance Matters Unify platform. This will allow attendance data to be readily available and usable to schools and other staff in MCPS.

Program Evaluation xi Evaluation of the ELO SAIL Program 5. Replicate the outcome study over time using different student population to see if the observed findings are stable.

6. Include in future studies a formative evaluation component that incorporates teachers and parents’ feedback regarding their experience with the program.

Program Evaluation xii Evaluation of the ELO SAIL Program Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Evaluation of 2018 Extended Learning Opportunities‒Summer Adventures in Learning (ELO SAIL) in Title I Schools Nyambura Susan Maina, Ph.D. and Natalie Wolanin, M.Ed.

Introduction

At the request of the chief academic officer, the Office of Shared Accountability (OSA) conducted an evaluation the 2018 Extended Learning Opportunities‒Summer Adventures in Learning (ELO SAIL) program. ELO SAIL is voluntary summer learning program for students entering K–2 in Federal Title I schools in Montgomery County Public Schools (MCPS). The ELO SAIL program is one of four programs that are an integral part of MCPS’ strategic objective of “Expanding Options and Access to Programs” with the primary objective of reducing summer learning loss and the academic achievement gaps (MCPS, 2018a). These programs support the strategic priority, “to increase the access to programs and resources so that all children are academically successful.”

In 2018, the ELO SAIL program was offered for 5 weeks (25 days) in 21 schools. For the first time since its inception in 2002, ELO SAIL offered an extended day option of 6.5 hours each weekday in 17 schools. The remaining four Title I schools featured a four-hour instructional day. Funding for the ELO SAIL program are provided through Title I of the Every Student Succeeds Act of 2015(ESSA).

Background

Overview of the 2018 ELO SAIL Program

Goal of ELO SAIL. The goal of ELO SAIL is to prevent the achievement loss that students may experience in summer, also known as summer slide/summer learning loss and prepare students for the next grade level by focusing on reading, writing, and mathematics using MCPS Curriculum 2.0 with a focus on common core state standards (MCPS 2018a).

With a goal to reduce summer loss that students may experience in summer, the specific purposes of 2018 ELO SAIL were to:  Preview and work on literacy and mathematics skills for the next grade  Maintain and improve skills in literacy and mathematics skills  Expose students to science, arts, and physical literacy through the full-day program  Familiarize young students with school and school routines (kindergarten students)  Build students confidence and joy for learning  Strengthen basic skills that are preconditions of later learning  Provide continuing English language instruction for speakers of other languages

Program Evaluation 1 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Figure 5. MCPS initiatives for preventing summer learning loss 2018. Source: MCPS (2018c): Superintendent’s Administrative and Supervisory Meeting, August 22, 2018

Expected student outcomes. The first expectation is that the majority of the K–2 students in eligible Title I schools will participate in ELO SAIL with high attendance. Second, the expectation is that students who attend the ELO SAIL program will maintain or advance their literacy and mathematics and be ready to succeed at the next grade level. In the end, the expectation is that ELO SAIL attendees will continue to meet or exceed classroom, local, and external grade-level targets specified in the Evidence of Learning (EOL) framework (MCPS, 2018b).

Planning and Coordination of the 2018 ELO SAIL Program

Target population and recruitment. The target population for the 2018 ELO SAIL program comprised students who entered kindergarten, Grade 1 or Grade 2 in the fall of the 2018–2019 school year. Participation in ELO SAIL was voluntary and all K–2 students in 21 of the 231 Title I schools were eligible to attend regardless of their levels of performance in reading and mathematics (MCPS, 2018b). Registration for 2018 ELO SAIL. Registration took place at the student’s school during spring 2018. Upon registering their students, parents or guardians were asked to sign a confirmation form which serves as both as a reminder of program dates and as a commitment for the child to attend. Schools were expected to closely monitor: 1 ) attendance and contact students who did not attend the first three days of ELO SAIL and 2) student participation throughout the 25 days of the program.

Summer 2018 ELO SAIL Full-Day and Half-Day Options

1 Two MCPS Title I schools, Oak View and Cresthaven are Grades 3–5 schools/intermediate schools.

Program Evaluation 2 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

ELO SAIL half-day program. Four schools implemented a four-hour instructional day of reading and mathematics (Figure 6). As illustrated in Figure 2, schools had the option of utilizing a variety of daily instructional models. The suggested daily instructional schedule included reading for 1 hour 30 minutes to 2 hours and mathematics for 1 hour to 1 hour 30 minutes depending on school and grade level.

Figure 6. Suggested daily instructional models for 2018 ELO SAIL half-day option. Source: MCPS, ELO SAIL training modules, 2018.

ELO extended day program. Out of the 23 MCPS schools designated as Federal Title I schools during the 2018–2019 school year, 17 were selected for the extended ELO SAIL program based on the proportion of students receiving Free and Reduced-price Meals System (FARMS) services (>68%). The ELO SAIL extended day program was possible in 2018 because of greater flexibility within ESSA to provide a well-rounded educational program for all students. In addition, the program was responding to research about the importance of extended learning opportunities for all students, especially students impacted by poverty.

After the morning academic program (Figure 6) available to all ELO SAIL students, students at 17 sites participated in an extra two hours to expand curricula offerings to include arts, physical literacy, and sciences. The recommended instructional offerings and schedule varied by grade (Figure 7). At every site, the schedule for kindergarten students was expected to include 40–45 minutes each of Hands-On Science, Physical Literacy, and Music. The Grade 1 instructional schedule included one hour of Hands-On Science and one hour of Physical Literacy, and the Grade 2 schedule included one hour of Physical Literacy and one hour of Coding.

Program Evaluation 3 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Figure 7. Suggested daily instructional models for 2018 ELO SAIL extended day option. Source: MCPS, ELO SAIL training modules, 2018.

Staffing and Cost of ELO SAIL 2018

Staffing. Principals are responsible for recruiting staff and was based on student enrollment and student needs (MCPS, 2018a). In summer 2018, the classroom staffing ratios for ELO SAIL classrooms were set as follows:  Kindergarten = 1:18  Grades 1–2 = 1:18  Paraeducators = 1:75

Appendix A details the number of classroom teachers, teachers of English for Speakers of Other Languages (ESOL) and special education teachers at each ELO SAIL site. In addition, the 17 extended day ELO SAIL sites had an Extended Day Coordinator who was responsible for overseeing extended day activities. A variety of supporting staff were on hand at each ELO SAIL site and included a school secretary, paraeducators, food service satellite manager and cafeteria worker, and clerical support for registration. Each site was managed by an ELO administrator, typically the assistant principal.

Volunteers. All together, the number of volunteers per school ranged from 12 to 25, for a total of 358 volunteers (Appendix A). The volunteer recruitment process for ELO SAIL was coordinated

Program Evaluation 4 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability by the Division of Pupil Personnel Services in the Office of Student and Family Support and Engagement (OSFSE) from March 28 through May 18, 2018.

Per student cost of 2018 ELO SAIL program. The program was offered at no cost to families, as federal and state funds were used to support the program. All students received breakfast and lunch at no cost, and transportation to and from the program was provided. The Per Pupil Allocation (PPA) for the ELO extended day program was $963.91.

Professional learning and preparation. In preparation for 2018 ELO SAIL, training was offered to all ELO SAIL literacy and mathematics teachers as well as extended day teachers. The purpose of training all teachers was to ensure that teachers were knowledgeable about the most recent ELO SAIL curricula, instructional expectations, and program structures. Title I funds were allocated for stipends for teachers participating in ELO SAIL training. Training was also provided for all food service satellite managers,’ clerical support for registration, staff members, and summer attendance secretaries. Registration for the ELO SAIL related professional development was available on the Professional Development Online (PDO) platform. The following hours were provided for ELO staff planning and preparation. • ELO Classroom Teachers (literacy and mathematics) = 4 hours • ELO Extended Day Teachers (arts and sciences) = 2 hours • ELO Summer Attendance Secretaries = 2 hours

Approximately 2 hours per week were spent on collaborative planning for each grade level.

Historical overview and ELO SAIL program changes over time. The implementation of the ELO SAIL initiative commenced in July 2002 in 17 Title I schools. The target population and reach has fluctuated over time, primarily in response to changes in funding. In the period between 2002 and 2018, the target population varied as follows:  2002–2011: K–4  2012: K–2; reduction based on the loss of stimulus funds from the US Department of Education  2013: K– 5; target grades increased because of an increase in funding  2014–2018: K–2; grade-level coverage reduced to respond to the economic instability that came with sequestered federal funds

ELO SAIL continued to operate in all Title I schools for a duration of four weeks until 2016 (Cooper-Martin et al., 2016). In 2017 and 2018, ELO SAIL was offered for 25 days or 5 weeks.

Literature Highlights

This section covers literature highlights on the phenomenon of summer slide and studies on voluntary summer learning programs offered by school districts as a means of stemming summer learning loss.

Program Evaluation 5 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Why summer slide is a concern. The summer slide, a loss of academic progress during the summer break between school years, has been confirmed through numerous studies since 1906 (McLaughlin, B, & Smink, J., 2009; White, 1906). Educators and researchers observed that students arrived back at school in the fall worse off in reading or mathematics than they finished in the spring, causing teachers to give up weeks of class time, or more, to make up for that loss. A variety of compelling research makes it clear that learning and achievement are perishable (Augustine et al. 2016; RAND, 2011; Alexander, 1997; Borman & Dowling, 2006; Reardon, Robinson-Cimpian, & Weathers, 2015). These studies support the conclusions that:  Learning loss has a large role in creating the achievement gap.  There is a strong link between socioeconomic status and the loss of reading and mathematics skills experienced over the summer.  The impact of the summer slide contributes to a more pronounced achievement gap between low and higher-income children because low-income students suffer disproportionate learning loss over the summer.  The average student loses a month of academic-calendar learning each summer.  Older students lose more over the summer than younger ones.  Students see greater academic dips in mathematics than in reading.  The summer slide results in wasted class time reviewing old material rather than introducing new material.

Cumulative impact of summer slide on students impacted by poverty. To some, the observable widening achievement gap cannot necessarily be attributed to inadequate schooling, but rather to a lack of exposure to academic and stimulating experiences during the summer months for many children; children have fewer opportunities for cultural, athletic, and outdoor activities during the summer than their more affluent peers (Augustine, et al., 2016; Alexander, 1997; Coombs and McIntyre, 2016). Also widely reported is that the effects of summer accumulate; “with the achievement gap between children impacted by poverty and less impacted peers at 9th grade being traced primarily to differential summer learning over the elementary years” (Alexander 1997; McCombs et al., 2012; Allington & McGill-Franzen, 2013). The faucet theory further explains this phenomenon: when the faucet is turned on, all students enjoy similar learning rates, but when school is not in session for the summer and the “faucet” is turned off, children have less access to resources, resulting in lower learning rates than their middle-class counterparts (Quinn, McIntyre, J., & Gomez, 2016; Alexander, 1997). In particular, students impacted by poverty are reported to lose vast amounts of learning over the summer when they don’t have access to the same enriching activities as their higher-income peers, such as vacations, visits to museums and libraries, or even just time spent with family discussing academic concepts or everyday events. As such, summer programs that include enrichment activities address another related challenge—inequity.

Impact of prior years of ELO SAIL programs on academic outcomes. Several studies of ELO SAIL have been completed by MCPS since the inception of ELO (Cooper-Martin & Zhao, 2016a; Kola et al., 2004; Wang, 2007). These findings showed there was a positive impact on reading performance in fall 2015 for kindergarteners when the two categories, reading and pre-reading, were combined and compared to non-readers. There was little evidence of a positive impact on reading in fall 2015 for Grade 1 and Grade 2 students; this was similar to findings of ELO SAIL sessions in 2012, 2013, and 2014. However, a higher percentage of ELO SAIL 2015 participants

Program Evaluation 6 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability receiving FARMS avoided summer loss compared to nonparticipants in both Grade 1 and Grade 2 (Cooper-Martin & Zhao, 2016). For analyses of effect of ELO SAIL on mathematics skills, the same study compared fall 2015 Measures of Academic Progress-Primary Grades (MAP-P) RIT Scores to prior spring RIT scores (before ELO SAIL). Findings showed there was evidence of a positive impact from ELO SAIL 2015 on mathematics performance in the fall for both Grade 1 and Grade 2 students, similar to analyses of previous ELO SAIL sessions. Further, there was evidence for a positive impact of ELO SAIL on fall performance for FARMS recipients. These evaluations of ELO SAIL in previous years frequently found positive results recipients of FARMS and ESOL services.

The impact of voluntary, district-run summer programs, that include academics and enrichment activities, on low-income elementary students. One notable randomized experimental study examined the short and long-term effects of voluntary summer programs and the characteristics of the programs that were effective across five districts’ programs for two summers (RAND, 2016). This study found strong evidence that voluntary summer learning programs can produce short- term gains in mathematics. Students who attended the programs entered school in the fall with stronger mathematics skills than those who did not. Indeed, high attenders (those who attended at least 20 days) in the first summer gained a significant and substantial benefit in mathematics and outperformed the control students at a rate representing about 15 percent of an average annual gain in mathematics. The study also found promising evidence that after two consecutive summers, students with high attendance (20 or more days per summer) outperformed their peers in mathematics and English Language Arts (ELA) and displayed stronger social-emotional competencies (RAND, 2017; McCombs, Sloan, Augustine, Schwartz, Martorell, & Zakaras, 2014).

What programmatic factors influence student outcomes? Three studies of summer programs offered by districts concluded that the best summer programs have small classes, encourage regular attendance, provide individualized instruction, and offer a combination of academic and enrichment activities (Augustine Sloan McCombs, Schwartz, & Zakaras, 2013; Augustine, & Thompson, 2017). These researchers cautioned that putting these elements in place required careful planning. Without early and careful planning, and including district leaders and program planners, summer program often suffered from logistical problems and poor instruction. The researchers point out that many problems identified by the researchers—from weak teacher training to ineffective transportation—could be traced to a rushed planning process. The researchers recommended: 1) creating schedules that protect instructional time and 2) investing in instructional quality.

Scope and Evaluation Questions

Purpose of Study

The purpose of the study was to a) examine differences in reading and mathematics performance for kindergarten students with and without ELO SAIL experiences and b) examine whether the 2018 ELO SAIL program lessened summer academic loss in reading and mathematics for Grades

Program Evaluation 7 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

1–2 students who attended. The fall MCPS Assessment Program in Primary Reading (AP-PR), Dynamic Indicators of Basic Early Literacy Skills® (DIBELS), and MAP-P Mathematics, were used to examine academic outcomes associated with attending the 2018 ELO SAIL program.

Evaluation Questions

This study was guided by the following evaluation questions: 1. What are the demographic characteristics of students who participated in summer 2018 ELO SAIL? 2. What are the attendance patterns for students who participated in summer 2018 ELO SAIL? 3. How did students who attended ELO SAIL perform on the fall reading assessments, DIBELS and MCPS AP-PR, compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL reading performance vary by student subgroups? a. How did Kindergarten students who attended ELO SAIL perform on the fall reading assessments compared with peers who did not attend ELO SAIL? b. Was ELO SAIL effective in maintaining/improving the spring 2018 AP-PR book levels of Grade 1 and 2 program participants relative to that of nonparticipating peers? 4. How did students who attended ELO SAIL perform on the fall mathematic assessment MAP-P, compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL math performance vary by student subgroups? a. How did Kindergarten student who attended ELO SAIL perform on the fall MAP-P assessment compared with peers who did not attend ELO SAIL? b. Was ELO SAIL effective in maintaining/improving the spring 2018 MAP-P RIT scores for Grade 1 and 2 ELO SAIL program participants relative to that of nonparticipating peers?

Methodology

Study Design

The analyses for evaluation questions 3 and 4 were designed to: a) describe the performance of ELO SAIL participants relative to the comparison group and b) measure the net short-term effects of the ELO SAIL program on fall student achievement. Because students were not randomly assigned to the ELO SAIL program, the study used a quasi-experimental design to compare the performance of ELO SAIL and non-ELO SAIL on fall MCPS AP-PR, DIBELS, and MAP-P mathematics assessments. Notably, the non-participants were fewer than participants, therefore drawing a matching sample could not be achieved. The intact group of non-participants was used as a comparison group. Multivariate statistical models were used to statistically control for demographic characteristics and baseline nonequivalence on prior achievement where applicable (Grade 1 and 2) between the two groups (Bordens & Abbott, 2008; Shadish, Cook, & Campbell, 2002). To achieve this, a propensity score was computed to statistically control for any nonequivalence between the two groups and to isolate the program effects of ELO SAIL

Program Evaluation 8 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability participation so that any differences in academic performances could be more accurately associated with the ELO SAIL program. Subsequently, the propensity scores were categorized into quintiles and included as a covariate in the ANCOVA and logistic regression models described in the analyses procedures section. The propensity score is described as a conditional probability that a student will be “treated” (voluntarily enroll in ELO SAIL) based on an observed group of covariates (Lueleen, Shadish & Clark, 2006). A separate propensity score was estimated for each grade-level.

Study Sample

All K–2 students enrolled in 21 Title I schools (N = 7,126) in the fall of 2018–2019 make up the study sample.

ELO SAIL treatment sample. The ELO SAIL participants (treatment group) consisted of 3,6842 students who attended the 2018 ELO SAIL program for at least 1 day and were still enrolled in Title I schools during fall 2018. The analytical samples varied by grade and measures:  Fall 2018 DIBELS measurement included: K, Grades 1 and 2 students who attended ELO SAIL for 17 days or more  Fall 2018 AP-PR measurement included: kindergarten students who attended 17 days or more; Grade 1 and 2 students who attended 17 or more and had spring and fall AP-PR (summer loss in reading)  Fall 2018 MAP-P measurement included: kindergarten students who attended 17 days or more; Grade 1 and 2 students who attended 17 days or more and had spring 2018 and fall 2018 MAP-P scores (summer loss in math) Non-ELO SAIL comparison sample. The comparison group, non-ELO SAIL, comprised the 3,171 K–2 students who did not attend the 2018 ELO SAIL program. The earlier plan to create a matched sample was not realized because the nonparticipants were fewer than participants. The analytical samples varied by grade and measures and were the same as the ELO SAIL treatment sample. Data Sources

The data analyzed for this study were compiled from Title I program records and MCPS databases. These measures included: a) demographic characteristics, b) records of ELO SAIL participation and attendance, and c) spring 2018 and fall 2018 performance in reading and mathematics performance.

Student demographic characteristics. Student information representing student gender, race/ethnicity, ESOL level, and receipt of special education and FARMS services during the 2018– 2019 school year were used to describe student groups and disaggregate data for analyses. These data were also used to compute the monitoring group and MCPS focus group categories as follows:

2 119 ELO SAIL participants who were attending non-Title I schools during fall 2018 were excluded.

Program Evaluation 9 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

 Non-FARMS All Other Student Groups (monitoring group)  Non-FARMS Black or African American  Non-FARMS Hispanic/Latino  FARMS All Other Student Groups  FARMS Black or African American  FARMS Hispanic/Latino

ELO SAIL participation. ELO SAIL participation records were used to identify students who attended ELO SAIL. A dichotomous variable was created to flag students who participated for 17 days or more. The group of students who had attended 17 days or more was used in the analytical sample for examining effects of ELO SAIL on fall academic outcomes. Students who did not attend ELO SAIL (non-ELO SAIL) were the comparison group.

Fall 2018–2019 Outcome Measures: Literacy

DIBELS. DIBELS is a set of procedures and measures that assess the understanding and development of early literacy skills (Dynamic Measurement Group, Inc., 2016). DIBELS are designed to be short (one minute) fluency measures used to regularly monitor the development of early literacy and early reading skills. Starting in the 2018–2019 school year, DIBELS are administered to all K–5 students three times a year (MCPS, 2018c). The fall mean composite measure was used to compare performance of ELO SAIL and non-ELO SAIL peers (Appendix B). The DIBELS Composite Score is a combination of multiple DIBELS scores and provides the best overall estimate of the student’s early literacy skills and/or reading proficiency (DIBELS, 2010).

MCPS AP-PR. AP-PR is a set of measures, developed and field-tested by the district, that examines K–2 students’ knowledge of basic reading concepts, their understanding of grammar, their ability to represent sounds in writing, their recognition of common words, and their ability to read and comprehend text. Students are expected to reach grade specific target reading levels in fall (Appendix C). To examine summer slide in reading for Grade 1 and 2 students, a measurement of the difference between the fall 2018 book level (after ELO SAIL) and maximum book level for spring 2018 (prior to ELO SAIL) was computed; then a dichotomous variable created to distinguish students who maintained or increased the maximum book level (avoided summer loss) from peers who showed evidence of summer loss.

For kindergarten students, only the fall 2018 AP-PR data were available and used. The analysis for kindergarten could not examine their summer learning loss per se because students did not have measures for spring 2018 (prior to ELO SAIL). In addition, being early in the school year, the vast majority (90%) did not have book level information (Appendix C). Therefore, AP-PR data were used to categorize kindergarten students into the following three levels of performance: 1. Reading: students who have attempted book level (> Level 3) with 90% accuracy and their fall oral retell is greater than 2, thus have met the fall reading benchmark. 2. Pre-reading: students who passed print concepts or reading behaviors (print concepts and reading behaviors combined) 3. Non-reading: students who were not ready for print concepts or attempted but did not pass

Program Evaluation 10 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Lastly, a dichotomous indicator variable was created to distinguish: a) students who were reading or pre-reading and b) non-reading students.

Fall 2018–2019 Outcome Measures: Mathematics

Measures of Academic Progress—Primary Grades (MAP-P) is a computer-adaptive mathematics assessment developed by Northwest Education Association (NWEA). Scores on MAP tests are reported on a Rasch Unit (RIT) scale. RIT scores are vertically equated, range from 100 to 300 and are used to measure student achievement and growth. To examine summer slide in mathematics skills, a measure of the difference (change) between fall 2018 and spring 2018 (prior to ELO SAIL) MAP-P RIT scores was computed for students in Grade 1 and 2. Since kindergarten students did not have MAP-P scores prior to the ELO SAIL program, fall MAP-P RIT scores were used in the analyses for kindergarten.

Procedures for Analyses

The analytical sample and procedures varied by evaluation question and outcome measure.

Analyses procedures for Evaluation Questions 1 and 2. To address Questions 1 and 2, descriptive statistics were used to summarize information on the characteristics of the ELO SAIL participants and non-ELO SAIL peers and attendance patterns for attendees.

Analyses procedures for Evaluation Question 3 and 4. The following procedures were used to answer evaluation Questions 3 and 4.

 Computation of propensity scores. A propensity score was computed to statistically control for any nonequivalence between ELO SAIL participants and non-ELO SAIL participants. Propensity scores were computed for each grade using the demographic indicators, including gender, race/ethnicity, and receipt of ESOL, FARMS, and special education services. Subsequently, the propensity scores were categorized into quintiles and included as a covariate in the ANCOVA models.

 Analyses for DIBELS composite scores for K, Grades 1 and 2. To estimate the effect of ELO SAIL on reading performance on the fall DIBELS assessment, ANCOVA procedures were utilized. Student data were analyzed separately for K, Grades 1 and 2. The dependent (outcome) variables were the fall DIBELS composite scores, controlling for differences in demographic characteristics with the propensity score. The independent variable was attending ELO SAIL for 17 days or more or nonparticipation in ELO SAIL.

 Analysis of summer loss in reading using AP-PR. The analytic sample for Grade 1 and 2 analyses was limited to students who had spring 2018 and fall 2018 AP-PR data. The proportions of Grades 1 and 2 ELO SAIL and non-ELO SAIL students who maintained or improved on prior spring 2018 maximum book level were compared, using binary logistic regression, while controlling for student demographic characteristics. Odds ratios from

Program Evaluation 11 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

logistic regression were used to compute effect sizes to measure the magnitude of differences in rates of students who avoided summer loss in reading. When the odds ratio is equal to one, it means that ELO SAIL and non-ELO SAIL peers were performing at comparable levels. Odds ratio greater than 1 indicate that the ELO SAIL participants were avoiding summer loss at higher rates than their non-ELO SAIL peers were. Where applicable, the analyses examined how the proportions of students avoided summer loss in reading level varied by student characteristics (FARMS, ESOL, and special education) and Focus Group categories. To examine the effect of ELO SAIL on reading skills for kindergarten students, the proportions of ELO SAIL and non-ELO SAIL kindergarten students who were reading or pre-reading were compared using two-way contingency table analyses (Pearson’s chi-square).

 Analysis of summer loss in math using MAP-P RIT scores. To examine summer slide in mathematics for Grade 1 and 2 students, ANCOVA procedures were used to analyze change (net gain) in MAP-P mathematics RIT scores from spring to fall 2018 among ELO SAIL and non-ELO SAIL participants, while controlling for differences in demographic characteristics with the propensity score. The analytic sample was limited to students who had MAP-P scores for spring 2018 and fall 2018. The independent variable was ELO SAIL participation for 17 days or more or nonparticipation. For kindergarten students, fall MAP-P mathematics RIT scores were used to compare performance of ELO SAIL participants and non-ELO SAIL peers.

 Effect sizes. Effect sizes were computed3,4 to establish how meaningful the group differences were to educators (American Psychological Association, 2010; Cohen, 1988). In evaluating educational programs, effect sizes >= 0.15 are considered important by educational researchers (Lipsey, 2012; Cohen, 1988).

Strengths and Limitations

Strengths. This study has much strength, particularly the large sample size that includes all K–2 students in Title I schools. It is also commendable that the student records from the Title I programs were complete with unique student identifiers and records of full days and partial days attended which made it easy to: a) identify students who did not participate in 2018 ELO SAIL and b) merge the data on ELO SAIL attendees with other student information and fall assessment data from MCPS. Because students were not randomly assigned to the ELO SAIL program, threats to internal validity of the results were addressed through a quasi-experimental design. In the design, multivariate statistical analyses were used to control for group differences. Further, for analyses related to Grade 1 and 2, AP-PR and MAP-P data for spring 2018 (before ELO SAIL) and fall 2018 (after ELO SAIL) were readily available, which facilitated the analysis of summer loss. These analyses also included effect sizes, making it easy to depict the impact of ELO SAIL by grade, assessment, and across student subgroups. Lastly, the study used Evidence of Learning (EOL) measures that are used to assess student learning for all students across MCPS.

3 The formula for Cohen d = (Mt – Mc)/SD. 4 Effect size (logit d) = ln (OR)/pi/sqrt 3.

Program Evaluation 12 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Limitations. The ELO SAIL participants were self-selected through opting to attend ELO SAIL. Because students were not randomly assigned to ELO SAIL; the chief threat to the validity of the results is a selection bias, an inherent threat to validity of program effects in any impact assessment using non-equivalent (not randomized) group designs. In this study, the nonequivalence was controlled for using a propensity score. However, the computation of the propensity was limited to data that are available for all students in MCPS databases leaving out many confounding factors that could influence the outcome. For example, it is unknown how the non-ELO SAIL students spent their summer. MCPS District Mathematics Assessments scores, based on what students are learning during the fall quarter, were not available at the time of these analyses. Unlike the case for literacy, whereby AP-PR and DIBELS assessments were available, this study relied on the MAP-P, a generalized assessment of mathematics as a measure of fall mathematics performance. Also, no prior DIBELS data were available therefore, students performance on DIBELS before the ELO SAIL program was not compared to their performance after ELO SAIL. This study does not measure summer learning loss for Kindergarten students as such. Instead, the performance of kindergarten students in the fall is compared to non-ELO SAIL participants, while controlling for demographic characteristics. No prior achievement data were available for kindergarten students, therefore, their reading and mathematics skills before the ELO SAIL program cannot be directly measured and compared to their performance after ELO SAIL. Lastly, there are many benefits of ELO SAIL that may have affected students in ways that are not captured in this study. For example, English language development, or other gains from the extended day program option and field trips. The 2018 ELO SAIL program provided students with a variety of Hands-On Science, Music, and other enriching and exciting opportunities they may not have had otherwise. Results

The results are presented in order of evaluation question. First, the descriptions of ELO SAIL participants and their attendance patterns are presented (questions 1 and 2). The results on academic performance (questions 3 and 4) are presented with regard to performance on fall 2018– 2019 DIBELS, AP-PR, and MAP-P mathematics assessment by grade-level and focus group categories.

Evaluation Question 1. What are the demographic characteristics of students who participated in summer 2018 ELO SAIL?

Table 1 shows that a total of 3,684, or 52%, of the 7,1265 kindergarten, Grade 1, and Grade 2 students enrolled in Title I schools, participated in the 2018 ELO SAIL program. This attendance count is different from the original count of 3,803 students from Title I program records because it excludes 119 students who were not attending Title I schools in the fall. The participations rates were very similar across the grade-levels with the highest participation by kindergarten students (57%).

5 September 30, 2018 Official Enrollment file.

Program Evaluation 13 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 1. Number and Percentage of Students Who Attended 2018 ELO SAIL by Grade Eligible Participated Grade N n % Kindergarten 2,242 1,287 57.4 Grade 1 2,323 1,240 53.4 Grade 2 2,265 1,159 51.2 All 7,126 3,684 51.7

Appendix D (Table D1) presents data on participation rates for 2018 ELO SAIL by school. The participation rates ranged from 48–64%; in the majority of the schools (N=12), the ELO SAIL participation rates were greater than 50 percent.

Characteristics of students who attended 2018 ELO SAIL. Table 2 shows the characteristics of students who attended ELO SAIL and their non-ELO SAIL peers. Of the 3,684 students who participated in ELO SAIL, 53% were male, 66% were Hispanic/Latino, 23% were Black or African American, 6% were Asian, 4% White, and about 2% were multiple races and ethnicities. Most (76%) received FARMS services, 59% received ESOL services, and 11% received special education services (Table 2), which are higher proportions than students in the non-ELO SAIL group. Conversely, there were lower proportions of white and multiple race students in ELO SAIL compared to non-ELO SAIL peers. Additionally, the distribution of students by the focus groups was generally similar between the ELO SAIL and non-ELO SAIL groups with the exception of the monitoring group, non-FARMS All Other Student Group. The percentage of students in the monitoring groups among the non-ELO SAIL participants was twice that of the ELO SAIL group. Also, there was a higher percentage of students in the FARMS Hispanic/Latino group among the ELO SAIL participants compared to the non-SAIL group. Finally, there were slightly more ELO SAIL students in Grade 2 and slightly less in kindergarten compared to students who did not attend ELO SAIL.

Program Evaluation 14 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 2. Background Characteristics of ELO SAIL Participants and Non-Participants Attended ELO Non-ELO SAIL SAIL (N =3,684) (N =3,442) Student characteristics n % n % Grade 1 1,240 33.7 1,184 34.4 Grade 2 1,159 31.5 1,210 35.2 Grade Kindergarten 1,285 34.9 1,048 30.4 Male 1,964 53.3 1,753 50.9 Gender Female 1,720 46.7 1,689 49.1 Asian 231 6.3 168 4.9 Black or African American 829 22.5 746 21.7 Race/ethnicity White 132 3.6 235 6.8 Hispanic/Latino 2,429 65.9 2,186 63.5 Two or more 61 1.7 91 2.6 No 902 24.5 1,055 30.7 FARMS receipt Yes 2,782 75.5 2,387 69.3 No 1,505 40.9 1,683 48.9 ESOL receipt Yes 2,179 59.1 1,759 51.1 Special Education No 3,291 89.3 3,156 91.7 receipt Yes 393 10.7 286 8.3 Monitoring Group Non-FARMS All Other Student Groups 233 6.3 480 13.9 Non-FARMS Black or African American 260 7.1 204 5.9 Non-FARMS Hispanic/Latino 409 11.1 371 10.8 Focus Groups FARMS All Other Student Groups 219 5.9 248 7.2 FARMS Black or African American 561 15.2 479 13.9 FARMS Hispanic/Latino 2,002 54.3 1,660 48.2 Note. American Indian and Pacific Islander students were included in the total but not reported separately.

Evaluation Question 2. What are the attendance patterns for students who participated in summer 2018 ELO SAIL?

Figure 8 shows the general attendance patterns for the 25 days of 2018 ELO SAIL program. Among the 3,803 students who signed up for ELO SAIL, 99% attended for one or more days. The data show that the majority of the students attended for 20 or more days, indicating that 75% were high attenders. An additional 14% attended at least 16 of the 25 days or 80% of the time. Less than 1 percent did not attend any ELO SAIL session.

Program Evaluation 15 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

80 74.8

70

60

50

40

30

20 14.3 10 6.3 2.7 1.4 0.6 0

20–25 days 16–19 days 11–15 days 6–10 Days 1– 5 Days No Show Perccent students who attended ELOSAIL

Number full days attended (N=3,803)

Figure 8 Attendance pattern: distribution of ELO SAIL participants by full days attended.

Table D2 (Appendix D) displays the summary statistics on full days attended by student subgroups. On average, students attended for 20–21 days, representing 80–85% of the available days, across all the subgroups, with a median of 23–24 days attended across the student subgroups.

Evaluation Question 3. How did students who attended ELO SAIL perform on fall DIBELS and MCPS AP-PR assessments compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL on the reading performance vary by student subgroups?

Performance on Fall DIBELS Assessment

Starting with the 2018–2019 school year, DIBELS will be administered in the fall, winter, and spring to students in kindergarten, Grade 1, and Grade 2. To answer research question 3, one-way ANCOVA was conducted to compare fall DIBELS composite scores for ELO SAIL participants and non-ELO SAIL peers, while controlling for demographic characteristics.

Kindergarten. Table 3 presents the adjusted mean fall DIBELS composite scores and effects sizes for ELO SAIL participation. The unadjusted means and standard deviations are presented in Appendix E, Table E1. On average there was a statistically and practically significant difference in fall DIBELS composite scores between ELO SAIL and non-ELO SAIL (F (1, 2160) = 110.26, p=0.00, d=.43), indicating that ELO SAIL participants in Kindergarten outperformed their peers

Program Evaluation 16 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability on the fall DIBELS scores. Further analyses showed that the effect of ELO SAIL was positive and practically significant for students receiving services, students in each of the five focus groups, and the monitoring group (ranging from d=0.2 to d=0.79). The difference for students receiving special education services met the established cut-off for educational significance (d > .15) but was not statistically significant.

Table 3. Adjusted Mean Fall DIBELS Composite Scores For Kindergarten Students by ELO SAIL Participation ELO SAIL Non-ELO SAIL ELO SAIL Program Effect Adjusted Effect Adjusted Std Adjusted Std Mean Sig Size Characteristics N Mean Error N Mean Error Difference F (p) (d) All Kindergarten Students 1066 40.4 0.77 1097 29.1 0.76 11.3 110.26 .000 0.43 FARMS recipients 806 37.8 0.86 795 26.4 0.87 11.4 87.60 .000 0.46 ESOL recipients 649 32.5 0.90 580 22.5 0.95 10.0 58.60 .000 0.42 Special Education recipients 98 33.0 2.42 86 28.2 2.58 4.8 1.83 .178 0.20 Non-FARMS All Other Student Groups 70 54.3 3.06 109 45.3 2.44 9.0 5.17 .024 0.34 Non-FARMS Black or African American 68 54.4 3.17 67 42.8 3.19 11.6 6.69 .011 0.43 Non-FARMS Hispanic/Latino 122 39.6 2.15 126 26.5 2.12 13.1 18.78 .000 0.51 FARMS All Other Student Groups 62 50.8 3.27 53 28.9 3.55 21.9 20.24 .000 0.79 FARMS Black or African American 151 50.4 2.05 175 37.7 1.90 12.7 19.95 .000 0.50 FARMS Hispanic/Latino 593 33.0 0.91 567 22.9 0.93 10.1 59.32 .000 0.43 Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.

Grade 1. Table 4 presents the adjusted mean fall DIBELS composite scores by ELO SAIL participation and the effects sizes for the impact of ELO SAIL. . The adjusted mean difference between the two groups was statistically and practically significant F (1, 2085 = 47.47), p=.000, d=0.30) in favor of Grade 1 ELO SAIL participants. The unadjusted means and standard deviations are presented in Appendix E, Table E2. The analyses at the subgroup level, indicated that the differences in fall DIBELS composite scores were statistically significant for every subgroup of students receiving services and in four of the five focus groups. Additionally, the magnitude of ELO SAIL effects on the fall DIBELS assessment varied (ranging from d=0.15 to d=0.81) and were practically significant for all students receiving services, and all of the focus groups. The monitoring group, Non-FARMS All Other Student Groups, was just below the practically significant effect size cutoff (d=.14).

Program Evaluation 17 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 4. Adjusted Mean Fall DIBELS Composite Scores For Grade 1 Students by Student Subgroups and ELO SAIL Participation ELO SAIL Program Effect ELO SAIL Non-ELO SAIL Adjusted Effect Adjusted Std Adjusted Std Mean Sig Size Characteristics N Mean Error N Mean Error Difference F (p) (d) All Grade 1 Students 1031 95.3 1.35 1057 82.2 1.33 13.1 47.44 .000 0.30 FARMS recipients 795 90.8 1.46 770 76.7 1.49 14.1 45.68 .000 0.34 ESOL recipients 612 81.7 1.57 557 68.4 1.65 13.3 33.84 .000 0.34 Special Education recipients 115 82.8 3.96 92 64.5 4.43 18.3 9.38 .002 0.42 Non-FARMS All Other Student Groups 19 118.1 10.53 61 111.7 5.87 6.5 0.29 .593 0.14 Non-FARMS Black or African American 75 115.1 5.30 73 98.3 5.37 16.8 4.87 .029 0.36 Non-FARMS Hispanic/Latino 97 94.4 4.26 114 87.5 3.93 6.9 1.42 .235 0.15 FARMS All Other Student Groups 11 105.4 13.35 12 62.6 12.78 42.7 5.34 .032 0.81 FARMS Black or African American 168 110.5 3.39 151 93.1 3.58 17.5 12.44 .000 0.39 FARMS Hispanic/Latino 575 84.3 1.53 579 71.7 1.53 12.7 34.20 .000 0.32 Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.  indicates groups with <50 students. In spite of statistically significant results, the small sample sizes make the results difficult to interpret because uncertainty in estimates of effect size is much greater for small groups of students than it is for large ones.

Grade 2. Table 5 presents adjusted mean fall DIBELS composite scores for Grade 2 students by ELO SAIL participation. The corresponding unadjusted means and standard deviations are presented in Appendix E, Table E3. On average, there was a statistically significant difference in the fall DIBELS scores between Grade 2 ELO SAIL participants and non-ELO SAIL peers (F (1, 2058) =47.44, p = .000), in favor the ELO SAIL group. The resultant effect size suggested a small practical significance (d=0.15).

Looking across student subgroups revealed that the ELO SAIL and non-ELO SAIL differences were statistically significant for each subgroup of students receiving FARMS, ESOL, and special education services (p<0.05). The differences were also statistically significant among FARMS All Other Student Groups, FARMS Black or African American and FARMS Hispanic/Latino focus groups. The corresponding small to moderate effect sizes (d>0.15) confirm that the better performance of Grade 2 ELO SAIL participants was practically meaningful among students

Program Evaluation 18 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability receiving FARMS and special education services and students in the FARMS All Other Student Groups, FARMS Black or African American, and FARMS Hispanic/Latino focus groups. In the monitoring group, the difference favored the non-SAIL group and was practically significant; although there was no statistically significant difference in the performance of students in the monitoring group.

Table 5. Adjusted Mean Fall DIBELS Composite Scores For Grade 2 by Student Subgroups and ELO SAIL Participation

ELO SAIL Non-ELO SAIL ELO SAIL Program Effect Adjusted Effect Adjusted Std Adjusted Std Mean Sig Size Characteristics N mean Error N Mean Error Difference F (p) (d) All Grade 2 Students 990 157.7 2.78 1071 143.7 2.67 14.0 13.12 .000 0.15 FARMS recipients 773 147.5 3.07 787 131.6 3.04 15.8 13.32 .000 0.18 ESOL recipients 598 122.8 3.42 544 111.3 3.59 11.6 5.39 .020 0.14 Special Education recipients 123 113.0 7.61 80 84.8 9.45 28.2 5.40 .021 0.33 Non-FARMS All Other Student Groups 19 166.5 24.35 37 224.5 17.32 -58.0 3.69 .060 -0.53 Non-FARMS Black or African American 64 196.7 11.91 62 185.0 12.11 11.7 0.46 .498 0.12 Non-FARMS Hispanic/Latino 95 173.2 8.78 139 158.1 7.25 15.1 1.76 .186 0.16 FARMS All Other Student Groups 16 155.3 21.82 16 87.5 21.82 67.8 4.59 .041 0.79 FARMS Black or African American 160 183.3 6.27 164 153.5 6.19 29.8 11.37 .001 0.36 FARMS Hispanic/Latino 555 134.7 3.49 551 120.6 3.50 14.0 8.00 .005 0.16 Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.  indicates groups with <50 students. In spite of statistically significant results, the small sample sizes make the results difficult to interpret because uncertainty in estimates of effect size is much greater for small groups of students than it is for large ones.

Performance on Fall MCPS AP-PR

Percentage of Kindergarten students reading, pre-reading, or not reading. Figure 9 describes the percentage of kindergarten students who were reading, pre-reading, or not reading as measured by fall AP-PR, by ELO SAIL participation. The results displayed in Figure 9 indicate that higher percentages of ELO SAIL participants were either reading (11% vs. 8%) or pre-reading (33% vs. 24%) compared with non-ELO SAIL peers. Conversely, there was a significantly lower proportion of ELO SAIL participants among the non-readers (56% vs. 68%); these results were statistically 2 significant (χ (2, N=2036), 34.91, p<0.05).

Program Evaluation 19 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

100 11.4 7.5 90

80 24.4

32.9 70

60

50

40 68.0 30 55.7

20

Percent of Kindergarten Students 10

0 ELOSAIL (N=1251) Non-ELOSAIL (N=974)

Not reading Pre-reading Reading

Figure 9. Performance of Kindergarten who were reading (AP-PR book >3) on fall AP-PR assessments by ELO SAIL participation.

Proportion of Kindergarten students reading or pre-reading by student subgroups. Table 6 compares the percentage of kindergarten students who were reading or pre-reading (combined) in the fall, as measured by AP-PR, by participation in ELO SAIL. Significant differences were found between all Kindergarten ELO SAIL participants and nonparticipants and for students receiving FARMS services, ESOL services, and students in the Non-FARMS Hispanic/Latino, FARMS All Other Student Groups, FARMS Black or African American, and FARMS Hispanic/Latino focus groups (Table 6).

Program Evaluation 20 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 6. Descriptive Statistics and Results of Chi-square Test for Kindergarten Subgroups who were Reading or Pre-reading on Fall AP-PR, by ELO SAIL Participation

ELO SAIL Non-ELO SAIL Pearson- Chi-Square Characteristics N n % N n % χ2 All 1251 554 44.3 954 305 32.0 34.512* FARMS recipients 946 392 41.4 681 187 27.5 33.750* ESOL recipients 754 245 32.5 490 99 20.2 22.420* Special Education recipients 115 38 33.0 75 18 24.0 1.786 Non-FARMS All Other Student Groups 82 51 62.2 102 58 56.9 0.535 Non-FARMS Black or African American 82 52 63.4 63 30 47.6 3.618 Non-FARMS Hispanic/Latino 141 59 41.8 108 30 27.8 5.268* FARMS All Other Student Groups 70 46 65.7 47 16 34.0 11.323* FARMS Black or African American 180 110 61.1 155 59 38.1 17.696* FARMS Hispanic/Latino 696 236 33.9 479 112 23.4 15.081* *p < .05

Grade 1. As specified in the Methods section, the analysis for Grade 1 and 2 examined whether ELO SAIL participants were more likely to avoid summer loss ( i.e. maintain or improve spring 2018 AP-PR book levels) relative to non-ELO SAIL peers. To compare effect sizes for avoiding summer slide, odds ratio (OR) from logistic regression were converted to Cohen's d statistics. Table 7 displays percentages of Grade 1 students who avoided summer loss in reading, and estimated effects of the ELO SAIL program on reducing summer loss. Overall, Grade 1 ELO SAIL participants (71% vs. 62%) were more likely to avoid summer loss (p < 0.05, d=0.27). These results were statistically and practically significant.

Program Evaluation 21 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 7. Estimated ELO SAIL Program Effects Associated with Avoiding Summer Loss in Reading (AP-PR): Grade 1 Students6 Relative to Non-ELO SAIL Peers ELO SAIL Program Effect Non- ELO SAIL ELO SAIL (N=972 (N=714)

No Odds No summer Loss Summer B ratio Loss (Std (No N N Error) Wald summer Sig. Effect n % n % Characteristics loss) (p) Size .487 20.826 All Grade 1 Students 972 693 71.3 714 442 61.9 (.107) 1.628 .000 0.27 730 492 .536 511 70 285 57.9 FARMS recipients (.123) 19.103 1.710 .000 0.30 520 330 .418 341 65.6 184 55.8 ESOL recipients (.145) 8.293 1.519 .004 0.23 Special Education 98 48 .174 64 65.3 31 64.6 recipients (.380) .209 1.190 .647 0.10 Non-FARMS All 66 81 Other Student 53 80.3 59 72.8 .872 Groups (.453) 3.705 2.391 .054 0.48 Non-FARMS Black 73 55 .314 57 78.1 40 72.7 or African American (.419) .563 1.369 .453 0.17 Non-FARMS 103 86 .134 72 69.9 58 67.4 Hispanic/Latino (.316) .180 1.144 .671 0.07 FARMS All Other 55 26 .493 39 70.9 16 61.5 Student Groups (.523) .888 1.638 .346 0.27 FARMS Black or 167 100 .597 133 79.6 68 68 African American (.289) 4.282 1.817 .039 0.33 FARMS 508 366 .529 339 66.7 201 54.9 Hispanic/Latino (.142) 13.840 1.698 .000 0.29 Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.  indicates groups with <50 students. In spite of statistically significant results, the small sample sizes make the results difficult to interpret. Uncertainty in estimates of effect size is much greater for small groups of students than it is for large ones.

The sizable odds ratio (OR>1) and effect sizes (d>0.15) for students receiving FARMS and ESOL as well as the FARMS Black or African American and FARMS Hispanic/Latino focus groups were significantly (statistically and practically) more likely to have avoided summer loss relative to their non-ELO SAIL counterparts.

Grade 2. Table 8 displays the proportions of Grade 2 students who avoided summer loss in reading by ELO SAIL participation. Overall, Grade 2 ELO SAIL participants were more likely to avoid summer loss than non-ELO SAIL peers (p<0.05, d=0.16). At the focus group and service receipt subgroup level, statistically significant and practically meaningful impacts of ELO SAIL were found only

6 Students with both spring 2018 and fall 2018 AP-PR information.

Program Evaluation 22 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability among students receiving FARMS or ESOL services and among students in the FARMS Hispanic/Latino focus group.

Medium effect sizes in favor of ELO SAIL participants maintaining or improving their maximum book level from the spring were found among students in the Non-FARMS All Other Student (d =.50) and FARMS All Other Student (d = .54) focus groups although the differences were not statistically significant. To the contrary, ELO SAIL participants in the Non-FARMS Black or African American focus group were less likely to avoid summer loss in reading than their non- ELO SAIL counterparts although those differences were not statistically significant.

Table 8. Estimated ELO SAIL Program Effects Associated with Avoiding Summer Loss in Reading (AP-PR): Grade 2 Students Relative to Non-ELO SAIL Peers ELO SAIL Non-ELO SAIL ELO SAIL Program Effect

No Summer No Summer Loss Loss Odds B ratio (std (No N n % N n % Error) Wald summer Sig. Effect Characteristics loss) (p) Size .298 All Grade 2 Students 953 735 78.6 822 614 74.7 (.115) 6.667 1.347 .010 0.16 723 559 77.3 601 437 72.7 .296 FARMS recipients (.129) 5.232 1.344 .022 0.16 534 400 74.9 413 281 68 .369 ESOL recipients (.147) 6.293 1.447 .012 0.20 Special Education 119 81 68.1 56 36 64.3 .195 recipients (.344) .321 1.215 .571 0.11 Non-FARMS All 59 51 86.4 67 67 77.6 Other Student .904 Groups (.514) 3.096 2.47 .079 0.50 Non-FARMS Black 62 54 87.1 47 48 89.6 -.374 or African American (.653) .319 0.691 .572 -0.20 Non-FARMS 85 67 78.8 107 104 76.9 .166 Hispanic/Latino (.352) .223 1.181 .637 0.09 FARMS All Other 52 48 92.3 57 57 82.5 .974 Student Groups (.641) 2.308 2.649 .129 0.54 FARMS Black or 166 130 78.3 127 126 84.1 -.256 African American (.310) .680 0.774 .408 -0.14 FARMS 511 385 75.3 417 420 68.1 .413 Hispanic/Latino (.149) 7.647 1.511 .006 0.23 Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.  indicates groups with <50 students. In spite of statistically significant results, the small sample sizes make the results difficult to interpret. Uncertainty in estimates of effect size is much greater for small groups of students than it is for large ones.

Program Evaluation 23 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Summary of Findings for Question 3

Table 9 presents a summary of statistically and practically meaningful effects of the 2018 ELO SAIL program in reading and how they varied by outcome measure and grade. The effects of the 2018 ELO SAIL program on the fall performance of K–2 students on the DIBELS assessments were positive and both practically and statistically significant for all K–2 students as well as all subgroups in kindergarten (although special education was not statistically significant) and most subgroups in Grade 1.

Also, a significantly (statistically and practically) higher percentage of Grade 1 and 2 ELO SAIL participants maintained or increased their spring maximum book level on AP-PR, thus avoiding summer loss, compared to non-ELO SAIL participants. The positive and practically educational meaningful impacts of the 2018 ELO SAIL program on reducing summer loss in reading as measured by AP-PR were most pronounced for Grade 1 and 2 students receiving ESOL and FARMS services and students in two focus groups--FARMS Black or African American for Grade 1 and FARMS Hispanic/Latino for Grades 1–2.

Program Evaluation 24 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table 9. Summary of 2018 ELO SAIL Program Effects on Fall Reading Performance Relative to Non-ELO SAIL Students by Grade and Outcome Measure Group Measure Kindergarten Grade 1 Grade 2 *** *** *** DIBELS d = .43 d = .30 d = .15 All students * *** ** AP-PR d = na d = .27 d = .16 *** *** *** DIBELS Students receiving FARMS services d = .46 d = .34 d = .18 * *** * AP-PR d = na d = .30 d = .16 *** *** * DIBELS Students receiving ESOL services d = .42 d = .34 d = .14 * *** ** AP-PR d = na d = .23 d = .20 ** * Students receiving Special Education DIBELS d = .20 d = .42 d = .33 services

AP-PR d = na d = .10 d = .11 * DIBELS d = .34 d = .14 d = - .53 Non-FARMS All Other Student Groups * AP-PR d = na d = .48 d = .50 * * DIBELS d = .43 d = .36 d = .12 Non-FARMS Black or African American

AP-PR d = na d = .17 d = -.20 *** DIBELS d = .51 d = .15 d = .16 Non-FARMS Hispanic/Latino * AP-PR d = na d = .07 d = .09 *** * * DIBELS d = .79 d = .81 d = .74 FARMS All Other Student Groups * AP-PR d = na d = .27 d = .54 *** *** *** DIBELS d = .50 d = .39 d = .36 FARMS Black or African American * * AP-PR d = na d = .33 d = -.14 *** *** ** DIBELS d = .43 d = .32 d = .16 FARMS Hispanic/Latino * *** ** AP-PR d = na d = .29 d = .23 * p< .05, ** p < .01, *** p < .001; Notes. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold. Green shading indicates positive, statistically and practically significant effect.  indicates groups with <50 students. In spite of statistically significant results, the small sample sizes make the results difficult to interpret. Uncertainty in estimates of effect size is much greater for small groups of students than it is for large ones.

Program Evaluation 25 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Evaluation Question 4. How did students who attended ELO SAIL perform on the fall MAP-P mathematics assessments compared with peers who did not attend ELO SAIL? Did the effects of the ELO SAIL on mathematics performance vary by student subgroups?

For kindergarten students, fall MAP-P mathematics RIT scores were used to compare the performance of ELO SAIL and non-ELO SAIL peers. For Grade 1 and 2, one-way ANCOVA was conducted for the spring (prior to ELO SAIL) to fall (after ELO SAIL) change in MAP-P RIT scores.

Kindergarten. Table 10 shows the adjusted mean fall RIT scores (after controlling for demographics) and effect sizes associated with ELO SAIL participation among all kindergarten students and subgroups of students. The corresponding unadjusted means and standard deviations are presented in Appendix F, Table F1. The difference in fall RIT scores between the two groups of all kindergarten students was statistically significant (F (1, 2056) =33.3, p=.000), in favor of ELO SAIL as well as practically significant with a small effect size of .24. At the subgroup level, the ELO SAIL participants achieved significantly higher fall RIT scores than non-ELO SAIL peers in most subgroups. No significant differences were detected for students receiving special education services and students in the Non-FARMS All Other Student Groups, and Non-FARMS Black or African American focus groups. The effect sizes (0.24–0.43) show that the effect of participating in ELO SAIL was practically significant for every student subgroup with the exception of students receiving special education services and the monitoring group (d < .15).

Table 10. Mean Fall MAP-P RIT Mathematics Score for Kindergarten Students by ELO SAIL Participation ELO SAIL Non-ELO SAIL ELO SAIL Program Effect Adjusted Effect Adjusted Std Adjusted Std Mean Sig Size Characteristics N Mean Error N Mean Error Difference F (p) (d) All Kindergarten Students 1016 138.65 0.38 1041 135.6 0.38 3.1 33.30 .000 0.24 FARMS recipients 763 137.4 0.42 754 134.0 0.42 3.4 32.58 .000 0.29 ESOL recipients 606 134.8 0.46 538 131.6 0.48 3.2 23.53 .000 0.28 Special Education recipients 92 134.3 1.35 80 133.1 1.45 1.1 0.33 .566 0.09 Non-FARMS All Other Student Groups 67 146.6 1.54 104 146.5 1.23 0.2 0.01 .935 0.01 Non-FARMS Black or African American 67 142.9 1.61 65 138.8 1.64 4.1 3.10 .080 0.30 Non-FARMS Hispanic/Latino 119 138.7 1.10 118 135.1 1.10 3.6 5.35 .022 0.28 FARMS All Other Student Groups 61 143.8 1.86 54 138.2 1.98 5.5 4.05 .047 0.38 FARMS Black or African American 145 142.2 0.97 169 137.1 0.90 5.1 14.37 .000 0.43 FARMS Hispanic/Latino 557 135.4 0.45 531 132.6 0.46 2.8 18.92 .000 0.25 Note. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.

Program Evaluation 26 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Grade 1. Table 11 presents the adjusted mean spring-to-fall changes in MAP-P RIT scores, the adjusted mean differences, and magnitude of differences (effect sizes) for ELO SAIL program participation. The corresponding unadjusted means and standard deviations are presented in Appendix F, Table F2. Comparing the estimated mean gains showed that the average gain in RIT scores made by ELO SAIL participants (mean=5.00) was significantly higher than the non-ELO SAIL group (mean=2.1) (F, (1, 1936 =54.99); p=.000), indicating less summer loss in mathematics for ELO SAIL participants relative to non-ELO SAIL peers. The resultant effect size of 0.34 indicates that the higher performance of ELO SAIL Grade 1 students was also practically significant. At the subgroup level, the ELO SAIL participants made higher average net gains in RIT scores ranging from .8 points for students receiving special education services to 5.3 points for FARMS All Other Student Groups. The effect sizes (0.24–0.53) show that the effect of participating in ELO SAIL was practically significant for every student subgroup with the exception of students receiving special education services (d < .15).

Table 11. Mean Gain in MAP-P RIT Mathematics Score from Spring- to-Fall 2018 for Grade 1 by ELO SAIL Participation ELO SAIL Non-ELO SAIL ELO SAIL Program Effect Adjusted Adjusted Adjusted Effect Mean Std Mean Std Mean Sig Size Characteristics N Gain Error N Gain Error Difference F (p) (d) All Grade 1 Students 1002 5.0 0.27 934 2.1 0.28 2.9 54.99 .000 0.34 FARMS recipients 773 4.7 0.31 679 2.1 0.33 2.6 31.95 .000 0.30 ESOL recipients 593 4.9 0.36 504 1.9 0.39 3.0 31.09 .000 0.33 Special Education recipients 114 2.7 0.87 88 1.9 0.99 0.8 0.36 .552 0.09 Non-FARMS All Other Student Groups 64 5.5 1.10 86 0.9 0.94 4.5 9.13 .003 0.51 Non-FARMS Black or African American 72 6.0 0.89 63 3.6 0.95 2.5 3.61 .060 0.32 Non-FARMS Hispanic/Latino 93 6.8 0.88 106 2.2 0.82 4.6 14.69 .000 0.53 FARMS All Other Student Groups 51 6.1 1.44 31 0.8 1.87 5.3 4.79 .032 0.50 FARMS Black or African American 164 5.9 0.68 121 2.2 0.79 3.7 12.78 .000 0.42 FARMS Hispanic/Latino 558 4.2 0.36 527 2.1 0.37 2.1 15.79 .000 0.24 Note. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.

Grade 2. Table 12 presents the adjusted mean spring-to-fall changes in MAP-P RIT scores, the adjusted mean differences, and magnitude of differences (effect sizes) for Grade 2 ELO SAIL and non-ELO SAIL peers. The corresponding unadjusted means and standard deviations are presented in Appendix F, Table F3. There was a significant difference in net gains in RIT scores between the two groups (F, (1, 1952 =36.67); p=.000), in favor of ELO SAIL. By gaining an average of 3.5 RIT score points, students who attended ELO SAIL maintained or improved their spring RIT

Program Evaluation 27 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability score an average of 2.1 points higher than non-ELO SAIL peers. The corresponding effect size of 0.27 confirms that the higher performance of Grade 2 ELO SAIL participants was practically significant. This pattern of higher performance was both statistically and practically significant for students receiving FARMS, ESOL, and Special Education services, and students in the Non- FARMS Hispanic/Latino, FARMS Black or African American, and FARMS Hispanic/Latino focus groups. The effect size for students in the Non-FARMS Black or African American focus groups was just below the designated cutoff (d = .14).

Table 12. Mean Gain in MAP-P RIT Mathematics Score from Spring-to-Fall 2018 for Grade 2 by ELO SAIL Participation ELO SAIL Non ELO SAIL ELO SAIL Program Effect Adjusted Adjusted Adjusted Effect Mean Std Mean Std Mean Sig Size Characteristics N Gain Error N Gain Error Difference F (p) (d) All Grade 2 Students 972 3.5 0.25 987 1.4 0.24 2.1 36.67 .000 0.27 FARMS recipients 760 3.5 0.28 726 1.4 0.29 2.1 26.50 .000 0.27 ESOL recipients 584 3.7 0.33 511 2.0 0.35 1.7 12.68 .000 0.22 Special Education recipients 124 2.3 0.82 79 -0.4 1.03 2.7 4.35 .038 0.30 Non-FARMS All Other Student Groups 59 2.8 1.02 75 0.9 0.91 1.9 1.84 .178 0.24 Non-FARMS Black or African American 62 2.9 1.01 55 1.8 1.07 1.1 0.56 .456 0.14 Non-FARMS Hispanic/Latino 91 4.5 0.74 131 1.5 0.62 3.0 9.36 .002 0.41 FARMS All Other Student Groups 56 2.7 1.09 65 2.0 1.01 0.7 0.20 .659 0.08 FARMS Black or African American 159 3.3 0.58 152 1.6 0.59 1.7 4.41 .037 0.24 FARMS Hispanic/Latino 545 3.6 0.33 509 1.3 0.34 2.3 22.95 .000 0.29 Note. Effect sizes (d > 0.15) indicate that impact of ELO SAIL is practically significant in an educational setting; practically significant effects are in bold.

Summary of Findings for Question 4

Table 13 presents a summary of statistically significant and practically meaningful impacts of the 2018 ELO SAIL program on maintaining or improving in mathematics skills by grade. The effects of the 2018 ELO SAIL program on the fall performance of K–2 students on MAP-P were positive and both statistically and practically significant for all K–2 students as well as most subgroups. Among all kindergarteners, the mean RIT score in mathematics from the fall MAP-P test was significantly higher and practically significant for ELO SAIL participants relative to nonparticipants (p<.000, d=0.24). Similarly, the fall mathematics RIT scores were significantly higher and practically meaningful (d>0.15) for ELO SAIL participants among the kindergarten students receiving FARMS or ESOL services, and students in the Non-FARMS Hispanic/Latino,

Program Evaluation 28 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

FARMS All Other Student Groups, FARMS Black or African American, and FARMS Hispanic/Latino focus groups.

For all Grade 1 and Grade 2 students, ELO SAIL participants averaged statistically higher and practically meaningful net gains in spring-to-fall MAP-P RIT scores than non-ELO SAIL peers. The positive and practically educationally meaningful impacts of ELO SAIL on mathematics were most pronounced for students receiving ESOL and FARMS services; students in four focus groups in Grade 1; and students receiving special services and three focus groups in Grade 2.

Table 13. Summary of 2018 ELO SAIL Program Effects on Fall Mathematics Performance Relative to Non-ELO SAIL Students by Grade . Group Measure Kindergarten Grade 1 Grade 2 *** *** *** All students MAP-P d = .24 d = .34 d = .27 *** *** *** Students receiving FARMS services MAP-P d = .29 d = .30 d = .27 *** *** *** Students receiving ESOL services MAP-P d = .28 d = .33 d = .22 Students receiving Special Education * services MAP-P d = .09 d = .09 d = .30 ** Non-FARMS All Other Student Groups MAP-P d = .01 d = .51 d = .24

Non-FARMS Black or African American MAP-P d = .30 d = .32 d = .14 * *** ** Non-FARMS Hispanic/Latino MAP-P d = .28 d = .53 d = .41 * * FARMS All Other Student Groups MAP-P d = .38 d = .50 d = .08 *** *** * FARMS Black or African American MAP-P d = .43 d = .42 d = .24 *** *** *** FARMS Hispanic/Latino MAP-P d = .25 d = .24 d = .29 * p< .05, ** p < .01, *** p < .001. Notes. Effect size d >= .15 was considered a finding of practical significance and are bolded. Green shading indicates positive, statistically and practically significant effect.

Discussion and Conclusion

This study provides analyses of the impact of the 2018 ELO SAIL program on academic outcomes for K–2 students in Title I schools, by end of the fall quarter. The first key finding is that more than one half (52%) of K–2 students in Title I schools attended ELO SAIL. This result is laudable and is the first step for students to be able to access a voluntary opportunity provided by the school district. Second, more than three fourths of students who attended were high attenders—they

Program Evaluation 29 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability attended more than 20 of the 25 days of the 2018 ELO SAIL program. Third, the findings from this study show that overall, 2018 ELO SAIL participants maintained or improved on their reading or mathematics skills, thus avoiding summer loss when their performance was compared with: 1) non-participating peers and 2) how they were performing at the end of the school year before the ELO SAIL program.

The findings are consistent with recent research from MCPS and others who document stronger impacts in mathematics and reading during the semester following a summer program among low- income students and also that the impacts tend to vary by student subgroups (Augustine, et al., 2017; Cooper et al., 2016).

These findings validated the relevance of summer programming for students impacted by poverty. Put in a larger MCPS context, it is noticeable from the end of year performance that students in the FARMS subgroups, under which the majority of students in Title I schools fall, are over represented among students “not learning enough” (MCPS 2019). In some schools, close to one half of the students had not met their grade-level targets at the end of the school year; making them highly susceptible to summer loss. Consequently, ELO SAIL can only maintain or improve on the reading and mathematics skills that students had at the end of the 2017–2018 school year. While the lack of statistically significant findings in some instances, possibly related to small sample sizes, is often disappointing, minimizing some summer losses represents a step forward and a positive change, even when some ELO SAIL participants still have a net summer setback. Therefore, while analyzing these results vis a vis the impact of ELO SAIL, it is also important to look at the larger context of learning gaps that existed at end of the school year; and recognize the value of reducing potential learning loss, especially for students who were not making sufficient progress during the school year. Lastly, the data at hand do not explain what is driving the varying impacts of ELO SAIL across grade-levels and subgroups.

Conclusions. The findings demonstrate that participation in the 2018 ELO SAIL program provided advantages for maintaining or improving reading and mathematics skills compared with not participating. The findings related to benefits of ELO SAIL mirror findings by other researchers showing that when measured a few months after the program, the impact of voluntary summer programs are more evident in mathematics than reading; and that even after participating, some students still experience summer loss. As such, minimizing learning losses represents a positive outcome, even if those students still have a net summer setback. Indeed, these positive impacts on reading and mathematics skills for K–2 students confirm that the ELO SAIL program represents an essential program because early patterns in academic achievement have profound implications for later learning.

Program Evaluation 30 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Recommendations

The recommendations are based on findings in this study and research to best practices for optimizing the impact of voluntary summer programs.

1. Due to positive findings for maintaining or improving reading and mathematics skills compared with not participating, continue to provide ELO SAIL to K–2 students.

2. Sustain and work on increasing the positive impacts of ELO SAIL by continuing to utilize many of the strengths of the program cited as best practices in literature for voluntary summer programs such as: early planning for ELO SAIL, using MCPS teachers who are conversant with Common Core State Standards and curriculum offered in MCPS, small class sizes, enrichment extended day program activities, English language development, free transportation and meals, having well documented records of participation and attendance, and promoting practices that emphasizes the strengths in what children impacted by poverty have rather than what they do not (cultural competency).

3. Continue to use a variety of approaches to increase and encourage participation in ELO SAIL a. Explain the benefits and observed impacts of the ELO SAIL program to staff and participating families. In particular, draw attention to the positive impact of the ELO SAIL program. Knowing whether ELO SAIL is able to improve mathematics and reading skills and which areas the improvements were largest would be valuable information to parents, teachers, and staff who refer or who provided the program.

b. Examine communication about the goals of ELO SAIL and possible misconceptions; continue to emphasize that the primary goal of ELO SAIL is academic improvement. Literature shows that limited understanding and misconceptions about voluntary programs such as ELO SAIL program may be at the heart of non-participation.

c. Tap into the potential of kindergarten participation. Kindergarten is the gateway to school; if the trends in 2018 continued, more than half of the incoming kindergarten students would access ELO SAIL and attain the benefits early on.

d. Make a concerted effort to work with other county-wide summer programs, particularly linking ELO SAIL students to libraries and recreation throughout the summer, which may encourage higher interest and participation and sustain the benefits of the ELO SAIL programs.

e. Explore factors preventing families from accessing ELO SAIL.

4. Share and store ELO SAIL participation and attendance data in the Performance Matters Unify platform. This will allow attendance data to be readily available and usable to schools and other staff in MCPS.

Program Evaluation 31 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

5. Replicate the outcome study over time using different student population to see if the observed findings are stable.

6. Include in future studies a formative evaluation component that incorporates teachers and parents’ feedback regarding their experience with the program.

Acknowledgements

The authors would like to thank Dr. Shahpar Modarresi, supervisor, Program Evaluation Unit (PEU), for her guidance through the planning of the evaluation and review of the report; Ms. Nichelle Owens, Supervisor, Title I programs and her staff for collaboration in planning, availing pertinent information about the 2018 ELO SAIL program, and providing feedback to the draft report; for Ms. Maria Allendes, office assistant, PEU, for technical support; Mr. Tung T. Do, data systems operator II, Title I Programs, for availing a variety of data on the ELO SAIL program; Ms. Missy Gumula, senior reporting specialist; Testing Unit for availing data on student enrollment; Ms. Marilyn Powell, technical analyst, Applied Research Unit, for processing and availing the fall DIBELS data; Mr. Juan Carlos Davila, evaluation support specialist PEU Unit, for compiling AP-PR and MAP-P data for this study and peer review; and Ms. Julie Wade, evaluation specialist, PEU, for peer review of the report.

Program Evaluation 32 Evaluation of the 2018 ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

References

Alexander, K., Entwisle, and Olson, S., (2007): Lasting Consequences of the Summer Learning Gap. Retrieved November 10, 2018 from https://www.ewa.org/sites/main/files/file- attachments/summer_learning_gap-2.pdf.

Allington, Richard L.; McGill-Franzen, Anne; Camilli, Gregory; Williams, Lunetta; Graff, Jennifer; Zeig, Jacqueline; Zmach, Courtney; and Nowak, Rhonda, “Addressing Summer Reading Setback among Economically Disadvantaged Elementary Students.” Reading Psychology, Vol. 31, No. 5, October 2010, pp. 411–427.

Augustine, & Thompson (2017). , Making Summer Last: Integrating Summer Programming into Core District Priorities and Operations. Santa Monica, CA: RAND Corporation https://www.rand.org/pubs/research_reports/RR2038.html.

Augustine, Sloan, McCombs, Schwartz, & Zakaras (2013). Getting to Work on Summer Learning: Recommended Practices for Success, Santa Monica, Calif.: RAND Corporation, RR-366-WF, 2013. As of January 19, 2018: https://www.rand.org/pubs/research_reports/RR366.html.

Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan Schweig, Andrew McEachin, and Kyle Siler-Evans (2016). Learning from summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth. Retrieved October 21, 2018 from: https://www.rand.org/pubs/research_reports/RR1557.html. Augustine, Catherine H.; McCombs, Jennifer S.; Pane, John F.; Schwartz, Heather L.; Schweig, Jonathan; McEachin, Andrew; and Siler-Evans, Kyle (2016). Kids Who Attend More Benefit More: Voluntary Summer Learning Programs. Retrieved October 21, 2018: https://www.rand.org/pubs/research_briefs/RB9924.html. Bordens, K.S. & Abbott, B.B. (2008). Research design and methods: A process approach. (7th Ed.). San Francisco: McGraw Hill. Borman, D., & Maritza Dowling, N., (2006). Longitudinal Achievement Effects of Multiyear Summer School: Evidence from the Teach Baltimore Randomized Field Trial,” Educational Evaluation and Policy Analysis, Vol. 28, No. 1, 2006, pp. 25–48. Cooper-Martin, E. and Zhao H. (2016a). Impact of the Extended Learning Opportunities Summer Adventures in Learning (ELO SAIL) Program on Student Academic Performance: Part 2, summer 2015. Rockville, Maryland. Montgomery County Schools.

Cooper-Martin, E., Wolanin N., Jang S., Modarresi, S. and Zhao H. (2016b). Impact of the Extended Learning Opportunities Summer Adventures in Learning (ELO SAIL) Program on Student Academic Performance: Part 1, Results from fall 2012 to fall 2015. Rockville, Maryland. Montgomery County Schools.

Program Evaluation 33 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Kim, James S., and David M. Quinn (2013). The Effects of Summer Reading on Low-Income Children’s Literacy Achievement from Kindergarten to Grade 8: A Meta-Analysis of Classroom and Home Interventions,” Review of Educational Research, Vol. 83, No. 3, 2013, pp. 386–431. Lipsey, Mark W.; Puzio, Kelly; Yun, Cathy; Hebert, Michael A.; Steinka-Fry, Kasia; Cole, Mikel W.; Roberts, Megan;. Anthony, Karen S. and Busick, Matthew D. Translating the Statistical Representation of the Effects of Education Interventions into More Readily Interpretable Forms, Washington, D.C.: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education, NCSER 2013-3000, 2012. Luellen, J.K., Shadish, W.R., & Clark, M.H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29(6), 530–558. McLaughlin, B. and Smink, J. (2009). Summer Learning: Moving from the Periphery to the Core. Denver, Colorado: Education Commission of the States. Progress of Education Reform, Vol. 10, No. 3. McCombs, Jennifer Sloan John F. Pane, Catherine H. Augustine, Heather L. Schwartz, Paco Martorell, and Laura Zakaras, Ready for Fall? Near-Term Effects of Voluntary Summer Learning Programs on Low-Income Students’ Learning Opportunities and Outcomes, Santa Monica, Calif.: RAND Corporation, RR-815-WF, 2014. Retrieved October 20, 2018 from http://www.rand.org/pubs/research_reports/RR815.html Mcfachin, A., & Atterbery, A. (2016). The Impact of Summer Learning Loss on Measures of School Performance. Retrieved October 20, 2018 from https://www.rand.org/pubs/external_publications/EP66495.html. MCPS (2018b) Allocations for Summer Programs. March, Weekly Principals Memoranda: March, 18, 2018 Montgomery County Public Schools (2019). Learning, accountability and results data dashboards. https://www.montgomeryschoolsmd.org/data/learning-accountability-and-results.html

Montgomery County Public Schools (2018a). MCPS Strategic Plan: FY 2017-2018. https://www.montgomeryschoolsmd.org/campaigns/Strategic-Planning- 2017/index.html. Montgomery County Public Schools (2018c). A Commitment to Learning, Accountability and Results. Retrieved from https://www.montgomeryschoolsmd.org/campaigns/Strategic- Planning-2017/index.html.

Montgomery County Public Schools (2018c). A & A Meeting, August 22, 2018. https://www.montgomeryschoolsmd.org/data/learning-accountability-and- results.html.

Program Evaluation 34 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Quinn, D.M., Cooc, N., McIntyre, J., & Gomez, C.J. (2016). Seasonal dynamics of academic achievement inequality by socioeconomic status and race/ethnicity: Updating and extending past research with new national data. Educational Researcher, 45(8), 443-453. http://journals.sagepub.com/doi/abs/10.3102/0013189X16677965?journalCode=edra McCombs RAND (2011). Summer Counts Making Summer Programs Part of the Learning Equation. https://www.rand.org/pubs/research_briefs/RB9599/index1.html

Reardon, Sean F.; Robinson-Cimpian, Joseph P., and Ericka S. Weathers, “Patterns and Trends in Racial/Ethnic and Socioeconomic Achievement Gaps,” in Helen F. Ladd and Margaret E. Goertz, eds., Handbook of Research in Education Finance and Policy, 2nd edition, New York: Routledge, 2015, pp. 491–509. Rosenbaum, P., & Rubin, D. (1983). Biometrika, Vol. 70 (1). (Apr., 1983), 41–55. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental & quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin. Shafer (2016). Why kids lose math knowledge, and how families can work to counteract it. Retrieved October 21, 2018 from: https://www.gse.harvard.edu/news/uk/16/06/summer-math-loss and https://digitalpromise.org/2017/05/18/preventing-math-summer-slide/

Wang, H. (2007). Evaluation of 2007 Extended Learning Opportunities Summer Adventures in Learning (ELO SAIL) Program: Implementation and Outcomes. Rockville, MD: Montgomery County Public Schools.

White, W. (1906). Reviews before and after vacation. American Education, 185–188.

Program Evaluation 35 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendices

Appendix A: 2018 ELO SAIL Schools and Staff Allocation for 2018

Special Extended Food Classroom ESOL SISa School Name Education Volunteers Para Day Secretary Service Teachers Staff Teacher Teachers Coordinator Manager Arcola 11 1 1 0 25 3 1 1 1 Bel Pre 15 1 1 0 13 4 Half day 1 1 Capt. James E. Daly 9 1 1 0 15 2 Half day 1 1 Gaithersburg 12 1 1 1 15 3 1 1 1 Georgian Forest 9 1 1 0 20 2 1 1 1 Harmony Hills 13 1 1 0 15 3 1 1 1 Highland 9 1 1 0 13 4 1 1 1 Jackson Road 12 1 1 1 21 4 1 1 1 JoAnn Leleck at Broad Acres 12 1 1 0 15 3 1 1 1 Kemp Mill 9 1 1 0 12 2 1 1 1 New Hampshire Estates 12 1 1 0 12 3 1 1 1 Rolling Terrace 12 1 1 1 15 3 Half day 1 1 Roscoe R. Nix 12 1 1 0 15 3 1 1 1 Sargent Shriver 11 1 1 1 10 3 1 1 1 South Lake 14 1 1 0 25 4 1 1 1 Summit Hall 11 1 1 0 20 2 1 1 1 Twinbrook 9 1 1 0 15 2 Half day 1 1 Washington Grove 8 1 1 0 17 2 1 1 1 Watkins Mill 11 1 1 0 20 3 1 1 1 Weller Road 10 1 1 1 20 2 1 1 1 Wheaton Woods 8 1 1 0 25 2 1 1 1 Total 229 21 21 5 358 59 17 21 21 aSupplemental Intervention Support Teacher

Program Evaluation 36 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendix B: K–2 Beginning of Year Benchmark Goals for DIBELS

Source. DIBELS® Next Benchmark Goals and Composite Score © Dynamic Measurement Group, Inc. August 31, 2010 http://www.boe.faye.k12.wv.us/docs/DIBELSNextBenchmarkGoals.pdf

Program Evaluation 37 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendix C: AP-PR Fall Target Reading Levels

Program Evaluation 38 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendix D: 2018 ELO SAIL Participation by School

Table D1. 2018 ELO SAIL Participation by School Number Number % Attended School of Students Attended ELO SAIL ELO SAIL Arcola Elementary School 338 167 49.4 Bel Pre Elementary School 500 243 48.6 Capt. James E. Daly Elementary School 268 129 48.1 Gaithersburg Elementary School 385 202 52.5 Georgian Forest Elementary School 290 159 54.8 Harmony Hills Elementary School 333 213 64.0 Highland Elementary School 265 144 54.3 Jackson Road Elementary School 337 203 60.2 JoAnn Leleck Elementary School at 354 Broad Acres 208 58.8 Kemp Mill Elementary School 213 114 53.5 New Hampshire Estates Elementary School 342 192 56.1 Rolling Terrace Elementary School 356 172 48.3 Roscoe R. Nix Elementary School 464 225 48.5 Sargent Shriver Elementary School 369 225 61.0 South Lake Elementary School 395 196 49.6 Summit Hall Elementary School 310 175 56.5 Twinbrook Elementary School 294 140 47.6 Washington Grove Elementary School 183 112 61.2 Watkins Mill Elementary School 353 185 52.4 Weller Road Elementary School 287 140 48.8 Wheaton Woods Elementary School 219 140 63.9 Not in Title I school --* 112 Total 6855 3796

Program Evaluation 39 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table D2. Summary Statistics on Full Days Attended Group Full Days Attended N Mean Standard Max. Median Mode Deviation Grade Grade 1 1,236 21 5 25 22 24 Grade 2 1,157 21 4 25 22 24 Kindergarten 1,281 21 5 25 22 24 Race Asian 231 22 4 25 23 24 Black or African American 823 21 4 25 23 25 White 133 21 5 25 22 24 Hispanic 2,424 21 5 25 22 24 Two or More 61 21 4 25 22 25 FARMS No 901 21 4 25 22 24 Yes 2,773 21 5 25 22 24 ESOL No 1,502 21 5 25 22 24 Yes 2,172 21 4 25 22 24 Special No 3,282 21 5 25 22 24 Education Yes 392 21 4 25 22 23 Focus Non-FARMS All Other 228 21 4 25 23 24 Group Student Group Non-FARMS Black or 261 21 4 25 23 24 African Am. Non-FARMS 412 20 5 25 21 23 Hispanic/Latino FARMS all Other Student 199 21 4 25 23 24 Group FARMS Black or African 562 21 4 25 23 25 American FARMS Hispanic/Latino 2,012 21 5 25 22 24

Program Evaluation 40 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendix E: Summary Statistics for Fall DIBELS Composite Score by Grade

Table E1. Summary Statistics DIBELS Composite Scores by Subgroups: Kindergarten ELO SAIL Non-ELO SAIL Standard Standard Characteristics N Mean Deviation Maximum Range Median N Mean Deviation Maximum Median Gender Male 567 40.0 26.5 114 114 39 570 29.6 25.3 107 24 Female 502 39.6 25.6 113 113 38 543 29.6 24.8 117 25 Asian 70 55.5 25.1 114 111 54 45 36.0 28.5 107 30 Black or African American 222 51.7 25.2 113 113 53 250 39.0 25.5 117 37 Race White 45 49.2 30.0 107 106 50 78 44.2 23.5 91 46 Hispanic/Latino 715 34.0 24.1 113 113 33 706 23.7 22.6 104 18 Two or More 16 47.0 23.9 100 86 45 32 42.7 27.4 101 38 FARMS No 263 47.1 27.6 114 114 46 305 37.2 26.3 105 37 Yes 806 37.5 25.2 113 113 37 808 26.7 23.9 117 22 ESOL No 420 50.5 25.4 114 114 52 525 38.2 24.9 105 37 Yes 649 33.0 24.1 104 104 31 588 21.9 22.6 117 15 Special No 970 40.6 26.1 114 114 39 1022 29.7 25.2 117 24 Education Yes 99 32.7 24.3 113 113 33 91 28.5 23.4 86 25 Non-FARMS All Other 70 52.6 27.8 114 113 53 109 46.4 25.1 105 47 Student Groups Non-FARMS Black or 71 54.2 27.2 112 112 54 69 43.0 25.7 102 43 African American Non-FARMS 122 39.9 26.0 113 113 38 127 26.2 23.8 87 21 Focus Group Hispanic/Latino FARMS All Other Student 62 51.1 26.5 102 102 52 55 28.6 24.2 107 23 Groups FARMS Black or African 151 50.6 24.2 113 113 53 177 37.5 25.3 117 36 American FARMS Hispanic/Latino 593 32.7 23.5 94 94 31 576 23.2 22.4 104 17

Program Evaluation 41 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table E2. Summary Statistics DIBELS Composite Scores by Subgroups: Grade 1 ELO SAIL Non-ELO SAIL Standard Standard Characteristics N Mean Deviation Maximum Range Median N Mean Deviation Maximum Median Gender Male 563 95.9 46.2 285 280 95 626 83.0 45.6 270 78 Female 476 92.2 40.2 263 262 90 611 83.0 40.6 260 81 Asian 69 114.6 36.4 221 183 114 45 98.7 41.8 256 96 Black or African 244 111.1 45.9 277 268 110 248 96.1 46.0 270 93 American Race White 31 111.5 47.3 227 209 116 79 100.8 49.4 260 99 Hispanic/Latino 675 85.0 40.5 285 284 83 841 75.7 39.5 240 73 Two or More 20 105.0 38.9 199 161 98 28 111.6 48.4 217 105 FARMS No 238 107.0 47.1 266 257 106 329 98.3 45.6 260 96 Yes 801 90.4 41.7 285 284 89 908 77.5 41.0 270 75 ESOL No 423 111.7 42.5 285 276 108 573 99.9 43.3 270 96 Yes 616 82.2 40.1 263 262 80 664 68.4 37.5 209 67 Special No 922 95.8 42.9 285 284 94 1116 84.8 43.4 270 82 Education Yes 117 81.7 47.0 237 226 74 121 65.7 37.8 230 61 Non-FARMS All Other 65 121.5 40.3 227 189 119 107 108.2 47.2 260 104 Student Groups Non-FARMS Black or 76 113.2 48.5 255 246 110 80 101.4 46.1 212 102 African American Non-FARMS 97 92.7 46.7 266 257 86 142 89.0 42.6 209 88 Focus Group Hispanic/Latino FARMS All Other 55 101.1 36.4 198 180 102 48 85.1 44.2 217 79 Student Groups FARMS Black or African 168 110.1 44.8 277 264 109 167 93.6 45.8 270 90 American FARMS Hispanic/Latino 578 83.7 39.2 285 284 82 693 73.0 38.4 240 70

Program Evaluation 42 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table E3. Summary Statistics DIBELS Composite Scores by Subgroups: Grade 2 ELO SAIL Non-ELO SAIL Standard Standard Characteristics N Mean Deviation Maximum Range Median N Mean Deviation Maximum Median Gender Male 536 152.4 94.2 435 435 160 616 144.0 95.0 444 151 Female 461 153.1 89.2 449 449 159 595 150.0 94.2 451 156 Asian 65 195.6 83.0 352 346 201 77 201.2 74.8 328 205 Black or African 226 183.8 82.5 380 380 193 264 164.9 90.1 443 171 American Race White 35 153.6 87.4 355 353 150 59 185.1 121.1 449 184 Hispanic/Latino 654 136.8 91.2 449 449 139 791 129.2 91.4 451 136 Two or More 17 194.1 115.3 442 437 188 31 210.9 84.9 451 202 FARMS No 219 183.7 94.3 442 442 192 314 185.7 101.6 451 189 Yes 778 144.1 89.4 449 449 151 897 133.3 88.2 451 140 ESOL No 396 198.9 85.6 449 446 206 597 185.5 89.4 451 185 Yes 601 122.3 82.8 404 404 128 614 109.3 83.9 431 115 Special No 872 158.5 91.2 449 449 164 1117 151.7 93.8 451 157 Education Yes 125 111.9 86.8 321 321 117 94 86.7 84.3 449 66 Non-FARMS All Other 59 195.1 95.1 442 436 197 90 225.5 93.2 451 234 Student Groups Non-FARMS Black or 65 191.7 89.0 355 355 215 71 192.7 99.6 443 199 African American Non-FARMS 95 171.4 96.8 440 439 172 153 159.5 99.8 444 160 Hispanic/Latino Focus Group FARMS All Other 58 170.3 84.8 355 353 181 79 166.7 85.6 314 179 Student Groups FARMS Black or 161 180.7 79.9 380 377 189 187 155.4 84.3 351 158 African American FARMS 559 130.9 89.0 449 449 133 631 122.5 87.5 451 129 Hispanic/Latino

Program Evaluation 43 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Appendix F: Summary Statistics for Fall MAP-RIT Scores by Grade

Table F1. Summary Statistics Fall MAP-RIT Scores: Kindergarten Fall RIT MAP- P

Standard Characteristics N Mean deviation Median Race Asian 115 144.4 15.2 144 Black or African American 472 139.8 12.4 139 White 123 144.7 12.8 143 Hispanic/Latino 1,421 134.6 11.5 134 Two or More 48 145.4 13.2 143 FARMS No 568 140.9 13.7 140 Yes 1,614 135.7 11.9 135 ESOL No 945 141.8 12.4 141 Yes 1,237 133.3 11.4 133 Special No 1,992 137.4 12.5 137 Education Yes 190 133.7 13.1 132 Focus Non-FARMS All Other Student 179 146.5 13.0 146 Group Groups Non-FARMS Black or African 140 140.9 13.6 140 American Non-FARMS Hispanic/Latino 249 136.9 12.7 136 FARMS All Other Student Group 117 141.2 14.5 139 FARMS Black or African 328 139.5 11.8 140 American FARMS Hispanic/Latino 1,169 134.1 11.2 134

Program Evaluation 44 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table F2. Summary Statistics for Fall MAP-RIT Scores and Change from Spring to Fall 2018: Grade 1 Change RIT Fall RIT MAP- P

Standard Standard. Characteristics N Mean Deviation. Median Range Mean Deviation Median Race Asian 114 3.7 9.9 5.00 68.00 114 165.2 15.1 166 Black or African American 492 4.3 8.9 4.00 81.00 492 161.6 14.5 163 White 110 3.1 8.5 4.00 52.00 110 170.6 18.8 172 Hispanic/Latino 1,516 3.3 8.6 3.00 83.00 1,516 155.7 14.6 156 Two or More 48 3.1 10.1 3.00 67.00 48 166.7 13.6 167 FARMS No 567 3.9 8.6 4.00 86.00 567 164.1 16.2 166 Yes 1,709 3.4 8.8 4.00 84.00 1,709 156.5 14.6 157 ESOL No 996 3.7 8.5 4.00 86.00 996 164.9 14.3 166 Yes 1,280 3.4 8.9 4.00 84.00 1,280 153.3 14.2 154 Special No 2038 3.7 8.7 4.00 91.00 2038 159.0 14.9 160 Education Yes 238 2.4 9.2 3.00 65.00 238 152.4 17.6 153 Focus Group Non-FARMS All Other 172 2.7 8.8 4.00 67.00 172 170.7 16.5 172 Student Groups Non-FARMS Black or 156 4.8 8.0 5.00 63.00 156 164.1 14.9 166 African American Non-FARMS 239 4.2 8.8 4.00 78.00 239 159.5 15.2 161 Hispanic/Latino FARMS all Other Student 103 4.3 10.2 5.00 66.00 103 161.8 15.9 164 Group FARMS Black or African 335 4.0 9.3 4.00 74.00 335 160.5 14.1 161 Am. FARMS Hispanic/Latino 1,271 3.2 8.5 3.00 81.00 1,271 155.0 14.3 155

Program Evaluation 45 Evaluation of the ELO SAIL Program

Montgomery County Public Schools 4/9/2019 Office of Shared Accountability

Table F3: Summary Statistics for Fall MAP-RIT Scores and Change from Spring to Fall 2018: Grade 2 Change RIT

Standard Standard Characteristics N Mean Deviation Median Range N Mean Deviation Median Race Asian 142 2.9 7.4 3.00 44.00 142 187.5 15.6 189 Black or African American 490 2.4 7.5 2.00 74.00 490 180.9 14.8 180 White 94 1.1 8.9 0.00 44.00 94 187.4 15.8 188 Hispanic 1,445 2.6 7.7 3.00 66.00 1,445 176.6 15.3 178 Two or More 48 1.5 7.7 1.00 33.00 48 188.7 14.9 189 FARMS No 533 2.3 7.5 2.00 47.00 533 184.0 17.2 185 Yes 1,675 2.5 7.8 2.00 74.00 1,675 177.5 14.8 178 ESOL No 993 1.9 7.2 2.00 66.00 993 185.6 14.4 186 Yes 1,215 2.9 8.0 3.00 73.00 1,215 173.7 14.6 175 Special No 1,989 2.6 7.5 3.00 74.00 1,989 180.0 15.3 180 Education Yes 219 1.3 9.2 2.00 66.00 219 170.5 16.7 171 Focus Non-FARMS All Other Student 149 1.7 7.7 2.00 36.00 149 191.9 15.8 193 Group Groups Non-FARMS Black or African 136 2.4 7.7 2.00 46.00 136 183.7 16.4 185 American Non-FARMS Hispanic/Latino 248 2.6 7.2 2.00 46.00 248 179.5 16.8 180 FARMS All Other Student 137 2.3 8.3 2.00 44.00 137 182.9 14.1 183 Groups FARMS Black or African 348 2.4 7.4 2.00 74.00 348 180.0 14.0 179 American FARMS Hispanic/Latino 119 2.6 7.8 3.00 66.00 1,190 176.1 14.9 177

Program Evaluation 46 Evaluation of the ELO SAIL Program