Overview of the DPS-UCD Collaborative ELA Studies

Overview of the DPS-UCD Collaborative ELA Studies The DPS-UC Denver Collaborative research has involved three key components: (1) individual case studies of successful schools, (2) a longitudinal study examining ELLs’ trajectory toward academic and linguistic proficiency, and (3) extension studies that arose as unanswered questions from the case and longitudinal studies. The primary research questions guiding these studies were: What are the characteristics of schools that are successfully serving a high number of English Language Learners? What are the long-term outcomes of variables that influence ELL’s achievement? What is the trajectory of ELL students in DPS? The primary purpose of this report is to describe the findings from the longitudinal study. Phase I: DPS UCD ELA Exemplary Schools Case Study Denver Public Schools joined into a collaborative with the University of Colorado at Denver (UCD) in August 2010. Through this collaboration, a number of UCD faculty members acted as co- investigators in a study of practices within schools in DPS in which ELLs were outperforming their peers in other schools in the district. The purpose of the study was to provide guidance to Denver Public Schools (DPS) and other districts in improving the educational performance of English language learners by describing practices currently used in six DPS schools in which English language learners were experiencing the most academic success.

What emerged as central to the success of these schools was the attention and thoughtful integration of practices that support English language learners throughout the entire school. Each of these schools appeared to have a language lens through which they view students and practice. They have developed a coherent approach to educating students, and display high expectations for students and staff, supported by a culture of engagement and instruction geared toward making the content accessible to students. The success of these schools appears to be linked to how well schools integrate ELA services and the “language lens” into the overall school culture and classroom instruction, rather than a particular program or series of programs. The following practices emerged as key components contributing to the success experienced by six diverse schools in working with English language learners:

 General Successful School Practices: Schools experiencing success in educating learners of English as an additional language share key characteristics of successful schools in general, including, coherence in vision, values, and actions; high expectations for students and teachers; a climate of safety, respect, and supported collaboration for teachers; and a safe community for children.

 Successful Practices for English Language Learners: Schools in this study experiencing success in educating learners of English as an additional language shared, in varying degrees, the following characteristics specific to the needs and strengths of learners of English as an additional language: a “language lens;” a focus on academic language; a focus on literacy and/or bi-literacy; teachers who share an understanding of second language acquisition (SLA); active use of data to guide instruction in language, literacy, and content; a culture of continuous learning/improvement; purposeful professional development and coaching; strong distributed and/or instructional leadership and committed and well- trained teachers; and active engagement of parents.

 Systematic, School-Wide Integration of Attention to Language Acquisition into Whole School Culture: The school-wide second language learning/instruction practices that contribute to the success of learners of English as an additional language are supported and enhanced by practices that characterize successful schools and integrated into the schools’ cultures. Please see the full reports for more information: http://testing.dpsk12.org/

Phase II: Longitudinal Study The case study represented the first stage of a larger, broader study to look more deeply at ELLs in DPS. The second stage was the longitudinal study, the purpose of which was to examine the impact of DPS ELA services and student characteristics on English language acquisition and academic achievement over time. The longitudinal study constituted a quantitative study whereby archival data were analyzed to determine students’ trajectories in transitioning toward English instruction and exiting program services.

Research Question: What are the characteristics and trajectories of ELL students in DPS in respect to English acquisition and academic progress?

Study Description Five cohorts of students were examined using five years of archival data. Due to limitations in the dataset, data prior to 2006-07 was not examined. A total of 29,151 Kindergarten to 12th grade students were tracked through 2010-11 for the study. By including multiple cohorts researchers could examine students’ trajectories through elementary, middle and high school as well as at the transitions from elementary to middle and middle to high school. The five cohorts included: 1. Kindergarten – 4th grade (students who were Kindergarteners in 2006-07) 2. 1st – 5th grades (students who were in 1st grade in 2006-07) 3. 3rd – 7th grades (students who were in 3rd grade in 2006-07) 4. 6th – 10th grades (students who were 6th graders in 2006-07) 5. 9th – 12th grades (students who were 9th graders in 2006-07) Cohort # Students 2006-07 Grade2007-08 2008-09 Grade2009-10 2010-11 Level Grade Level Level Grade Level Grade Level

Kinder-4th 3,102 Kinder 1st 2nd 3rd 4th

1st-5th 3,066 1st 2nd 3rd 4th 5th

3rd-7th 2,716 3rd 4th 5th 6th 7th

6th-10th 1,999 6th 7th 8th 9th 10th

9th-12th 2,105 9th 10th 11th 12th Analyses

The longitudinal study included an analysis of the language and academic achievement trajectories of English learners in DPS and an examination of the profiles of these students. The study focused on analyzing the relationship between a student’s background experience and resources (e.g., socio-economic status, primary language, academic background) and the student’s language and achievement trajectories. Student performance, including CELA and CSAP status and growth, were used to determine the students’ language and academic trajectories. ELA programs and transition between and out of services were examined to see what impact, if any, ELA program had on a student’s trajectory. Student demographics were analyzed to identify how student trajectories differed based on student characteristics (e.g., free or reduced price lunch).

Outcome variables:

Time to increase a proficiency band on CELA and CSAP Average scale score change on CELA and CSAP Program exit High school outcome (graduation, drop out)

Independent variables:

Instructional approach in the classroom (ELA Program) Initial CELA score (English proficiency) Demographics (language, FRL)

Summary of Findings

Key findings from the study include:

 Students tend to “plateau” at higher levels of English proficiency, and show less growth on CELA at transition points between education levels.

 ELLs who start at lower levels on CELA tend to make little progress on CSAP.

 There is some evidence to support the effectiveness of ELA program services, with serviced ELLs showing greater growth on CELA and CSAP than non-serviced ELLs.

 Students who exit from ELA program services, on average, start at higher CELA levels and score similar to never ELLs on CSAP.

 Many EL students leave DPS (e.g., drop out) in secondary grades, even exited students who tend to show higher academic performance.

The findings from the longitudinal study are broken down into the following overall sections:

 English Language Proficiency (CELA)

 Academic Achievement (CSAP)

 Student Progression through ELA Programs  ELA Program Effectiveness

 Exited Student

 Graduation & Dropout

 Additional Analyses

One major outcome of the longitudinal study was the development of the ELA Trajectory (based on CELA), which provides a target trajectory for ELLs based on their starting proficiency level. Another “spin-off” from the study was an analysis comparing CSAP Lectura and CSAP Reading performance by CELA level. This analysis informed district recommendations regarding language of assessment. These outcomes are discussed in the findings section below under Additional Analyses. Finally, as a result of the findings from this study, DPS worked with DPS-UCD Collaborative members to develop a literature review responding to and informing the ELA Trajectory. A brief review of this research follows the Findings section. Definition of Terms

English Language Learner (ELL): Unless otherwise specified, “ELL” in the report refers to all English language learners in DPS, including students receiving services, students not receiving services (e.g., students whose parents opted them out of services), and students who are exited from program services. Only non-exited students take the CELA, therefore, in the analysis of CELA growth trajectories the population described is non-exited ELLs (including students not receiving services).

Exited ELLs: The population described in this report as “exited ELLs” includes students who are in the first year of monitoring after program exit (“Monitor year 1”), students who are in the second year of monitoring after exit (“Monitor year 2”), and students who have been re-designated (“Exited 3+ years”).

Non-ELLs or Never ELLs: For comparison, corresponding data for native English speakers are included throughout this report where possible. These students are referred to as “non-ELLs” or “never ELLs” and are native English speaking students who have never received ELA program services (not including parent opt outs or students eligible to receive services but not receiving them). Similar cohorts (Kindergarten-4th grade; 1st-5th grade; 3rd-7th grade; 6th-10th grade; 9th-12th grade) of never ELL students were tracked from 2007 through 2011.

Limitations There were a number of study limitations, including limitations known in advance, and limitations that emerged during data collection and analysis. ELA program placement determinations cannot be made before 2002-03 due to issues in the historical data tracked, and there are many challenges in these data prior to 2006-07. For example, prior to 2006-07 there was not a category for non- exited parent opt outs (also referred to as PPF3s); that is, PPF3s were reported as non-exited ELLs. As a result, the study was limited to 2006-07 through 2010-11. Program data are based on state designations. State designations most closely reflect what services students are eligible to receive; however, these determinations are not always a direct reflection of actual services rendered. To create a closer representation of services actually rendered, for the purpose of this study, ELL program status was based on state designation, parent placement option, whether the student was placed in a classroom with an ELA-S or ELA-E qualified teacher, and services offered at the school students attended. The ELA Department was consulted if there were inconsistencies in these data. However, despite an improved indicator of student ELA placement, the indicator still does not always reflect actual services rendered (e.g., level of fidelity to the DPS ELA model) nor the quality of services rendered.

Additional limitations included:

A large amount of data were missing (e.g., due to high student mobility). Retention (in grade) data are extremely complex, particularly at the high school level and were therefore not analyzed in this study. Special education status was not included as a covariate due to low numbers of students with disabilities in the study population. Home literacy (a variable that could have acted as a covariate) data are not collected for students in DPS. Parent education (a potential covariate) data are not collected in DPS.

Findings Study Participants

The table below provides an overview of the students in each of the cohorts. Cohort # Students # % FRL** % Female % Spanish % Speaking a % Non- Enrolled Speaking Refugee* Spanish/Non- all years Language Refugee of the Language study speaking Kinder-4th 3,102 1,899 77% 48% 91% 5% 4%

1st-5th 3,066 1,933 85% 50% 91% 6% 4%

3rd-7th 2,716 1,694 87% 50% 90% 7% 4%

6th-10th 1,999 1,166 91% 49% 91% 6% 3%

9th-12th 2,105 1,093 84% 49% 89% 6% 5%

* Refugee status is currently not collected in DPS. This variable was generated by the research team by coding languages based on historical refugee status. ** By comparison, 49%-55% of never ELLs in the same grade configurations as the cohorts qualified for free or reduced price lunches. English Language Proficiency

English proficiency in DPS is measured by the Colorado English Language Assessment (CELA). The CELA assesses the four domains of language (reading, writing, listening, speaking; the overall score is a composite of the four domains). There are five levels of proficiency on CELA, ranging from Level 1 (formerly referred to as Beginning) to Level 5 (formerly referred to as Above Proficient). Reaching CELA Level 5 is the primary criterion for exiting from ELA program services (in addition to other criteria, such as CSAP performance). All non-exited EL students (including parent opt-outs) in DPS are tested on CELA. Even students whose parents opt them out of program services must technically “exit” from the ELA program by meeting the exit criteria. CELA Overall was used in this study as the measure of language development and proficiency.

One major finding from the study was that longitudinal gains on CELA decreased as students moved up in performance bands and grade levels. This was especially true at transition points from 5th grade to 6th grade and from 8th to 9th grade. The first figure below, which combines data across all of the cohorts, shows a decrease in the average change in scale score points as students moved to higher grades, with major dips in scale score change at the transition points from elementary to middle and middle to high school. In fact, there was negative change in scale scores between the 8th and 9th grade year.

When examining how students in the study cohorts progressed on CELA over the course of the study, we see that students in the lower grades were more likely to increase proficiency bands – and to start at a lower level on CELA. For example, 92% of students in the Kindergarten to 4th grade cohort scored at CELA Levels 1 or 2 in Kindergarten. Over 80% of the students in the Kindergarten cohort increased two or more proficiency levels over the course of the five year study period. However, nearly one in five (18%) students in that cohort stayed at the same CELA level or increased only one level in five years.

The graphs below show that the number of students still taking CELA (not exited from the program) declines with each increasing grade level. However, the data show that the majority of the students who continue to be CELA tested (serviced and not serviced students) in secondary grades show little progress toward increasing CELA proficiency bands. These students were more likely to start at higher CELA levels. For example, nearly 70% of students in the 6th grade cohort scored at CELA levels 3 or 4 in 2007, and about half (53%) of the students in secondary grades stayed at CELA Levels 3 or 4 over the course of the study. The phenomenon of students making little progress from CELA Levels 3 and 4 is referred to within the district as the “plateau” phenomenon.

Students who started in 2007 at different proficiency bands on CELA differed by FRL status. For example, students who scored at CELA Levels 4 or 5 in 1st grade were less likely to qualify for free or reduced price lunches than those entering at Levels 1, 2 or 3. Analysis includes students whose CELA scores could be tracked over the course of the five years; *four years for the 9th-12th grade cohort.

Analysis includes students with five consecutive years of CELA scores; *four years for the 9th-12th grade cohort.

Academic Achievement (CSAP)

While CELA was the measure of English proficiency, the Colorado State Assessment Program (CSAP) was the primary measure of academic achievement. The graphs below display how students in the 3rd, 6th and 9th grade cohorts progressed on CSAP in comparison to never ELLs; the dotted lines represent the minimum scale score associated with being Proficient on CSAP. The first three graphs in each cohort series represent the cohort as a whole, while the fourth graph shows the progress of students who scored at CELA Levels 1, 2 or 3 in 2007.

Students in the 3rd – 7th grade cohort appear to be closing the gap to CSAP proficiency, as well as with never ELLs, in Reading and Writing, while the gap to proficiency on CSAP Math slightly widened in 7th grade. Never ELLs also experience a dip in performance in Math in 7th grade. Third graders scoring 1, 2 or 3 on CELA in 06-07 appear to be closing the gap to proficiency on CSAP Reading. However, these students made virtually no progress toward CSAP proficiency in Writing or Math.

Similar patterns exist for the 6th and 9th grade cohorts, with more dramatic gaps between ELLs (and never ELL) and CSAP proficiency in the upper grades, especially for Writing and Math. Students scoring 1, 2 or 3 on CELA in 06-07 made virtually no progress toward proficiency on CSAP between 9th and 10th grades. While never ELLs consistently scored higher than ELLs, they experienced similar patterns of slowing growth and a widening proficiency gap in CSAP Math and Writing as ELLs.

Analysis includes students with five consecutive years of CSAP scores.

Analysis includes students with five consecutive years of CSAP scores; ELL students began at CELA Level 1, 2, or 3 in 3rd grade. Analysis includes students with five consecutive years of CSAP scores.

Analysis includes students with five consecutive years of CSAP scores; ELL students began at CELA Level 1, 2, or 3 in 6th grade.

Analysis includes students with two consecutive years of CSAP scores.

Analysis includes students with two consecutive years of CSAP scores; ELL students began at CELA Level 1, 2, or 3 in 9th grade. Student Progression through ELA Programs DPS follows a Transitional Native Language Instructional (TNLI) model and an English as a second language (ESL) model to serve students in the district who are English language learners. The TNLI model refers to the provision of instruction in the students’ native language (for Spanish-speakers only) in core subject areas so that the student is continuing to build literacy and academic skills while learning English. These services are referred to as ELA-S. ESL in DPS is a model of instruction that provides supported English instruction and English language development to ELLs who speak a language other than Spanish. ESL services are also provided to Spanish-speaking ELLs who have transitioned out of TNLI, or whose parents selected to have them enrolled in these services. This is referred to as ELA-E. Additionally, there are dual language schools in which all students, regardless of their home language, participate in services, receiving instruction in both Spanish and English. Based on findings from the longitudinal study, the majority of ELL students receiving services are following the DPS ELA model, transitioning from ELA-S to ELA-E in elementary school and exiting services by secondary school. For example, 64% of the students in the Kindergarten cohort who started in ELA-S had either transitioned to ELA-E or exited program services by 4th grade; 73% of the 1st grade students who started in ELA-S transitioned to ELA-E or exited the program by 5th grade. Fifty-seven percent of 6th grade students had previously exited ELA program services, and 73% of the 6th grade cohort students had exited by the end of the study period (10th grade). For the high school cohort, 74% had exited as of 9th grade and four out of five students in the cohort had exited by 12th grade. The series of graphs below show how students progressed through ELA programs throughout the study period.

While the majority of students are transitioning through or exiting from the program, some students are staying in the same ELA program for many years. For example, 41% of students in the Kindergarten cohort who began in either ELA-S or ELA-E in 2007 remained in the same ELA program for all five years of the study. The students who remained in the same program (ELA-S or ELA-E) for multiple years were more likely to start at a lower CELA level than students who transitioned or exited the program.

Kindergarten – 4th Grade Cohort

1st – 5th Grade Cohort

3rd – 7th Grade Cohort

6th – 10th Grade Cohort 9th – 12th Grade Cohort

Kindergarten, 1st and 3rd Grade Cohorts

* Kinder-4th grade cohort: 50% stayed in ELA-S; 50% in ELA-E

** 1st-5th grade cohort: 69% stayed in ELA-E; 18% stayed in ELA-S

*** 3rd-7th grade cohort: 96% stayed in ELA-E; 4% stayed in ELA-S ELA Program Effectiveness

Impact of ELA Program on Language Acquisition Due to limitations in program data prior to 2007, as well as differences in the study populations served by ELA programs (e.g., differences by FRL status, primary language), conducting an evaluation of DPS ELA programs is a challenge. The following series of graphs display progress on CELA by ELA program in comparison to the students’ target trajectory (ELA target trajectories are explained in the Additional Analyses section under Target Trajectories for CELA). The target trajectories are indicated by dotted green lines, which represent the minimum scale score associated with the CELA proficiency level targets for each grade level. The sample sizes by program type were too small to conduct these analyses for the 9th grade cohort.

On whole, these graphs show that students whose parents opt them out of services and students at schools that do not offer services start at higher levels on CELA than students who stayed in the same program throughout the study. The graphs also show promising findings whereby, on average, students are meeting their targets. The few exceptions include students who stayed in the same program all five years of the study, especially those students in ELA-S all five years in elementary school, and the students in ELA-E throughout the study period in the upper grades.

Also displayed below are the CELA Overall median growth percentiles for students in each cohort by program type. Overall, serviced students (ELA-S/ELA-E) showed greater growth than students who remained opted out of program services or in a school without services throughout the five years of the study.

Due to small sample sizes, caution should be taken in interpreting these findings.

Kinder-4th Grade Cohort

Analysis includes students with five consecutive years of CELA scores.

Analysis includes students with four or five consecutive years of CELA scores.

Analysis includes students with three or four consecutive years of CELA growth percentiles.

1st-5th Grade Cohort

3rd- 7th Grade Cohort

6th – 10th Grade Cohort Analysis includes students with five consecutive years of CELA scores.

Analysis includes students with two to five consecutive years of CELA scores.

Analysis includes students with four consecutive years of CELA growth percentiles.

While a simple analysis of average CELA scale scores and growth percentiles over time by program provides some information about how students are acquiring English proficiency, it does not provide the full picture. On whole, students who opt out of services are less likely to qualify for free or reduced price lunches. Additionally, students who exited services within the timeframe of the study and students who did not receive services (parent opt outs and students in schools that do not offer services), on average, started at higher CELA levels than serviced students and students who remained in program services longer. As a result of these variations within the student population, repeated measures ANOVA analyses were run, holding starting CELA levels and FRL status constant. These analyses were only run for students whose primary language was Spanish. Two levels of analyses were conducted: the first compared students who received program services with those who did not despite being eligible for services. The second compared serviced students by program type.

Results from these analyses revealed that Spanish speaking students receiving ELA services showed greater growth in English language proficiency compared to students not receiving services. However, the impact of ELA services on growth declined over time. For example, for the Kindergarten cohort, the impact of ELA services was greatest in first grade, and declined gradually in subsequent grades. The size of the effect on overall CELA scores of receiving ELA services was .27 in Grade 1 (a small to moderate effect), .25 in Grade 2, .16 in grade 3 (a small effect), and .08 in grade 4 (a negligible but statistically reliable effect). Similar results were revealed for the 1st and 3rd grade cohorts.

When examining serviced students by program type, the analysis of the Kindergarten, 1st grade, and 3rd grade cohorts showed greatest growth for students in ELA-S and ELA-E (transitioned) or Dual Language programs, followed by students in ELA-S, then by students in ELA-E services. While statistically significant, the effect sizes were small. Because sample sizes for students in the upper grades become smaller and smaller as students have already exited from the program, analyses were not conducted for the 6th or 9th grade cohorts. *Serviced students = ELA-S, ELA-E, Dual Language; Non-serviced students = parent opt out, services not offered at school. Differences were significant (at least p<.05).

Impact of ELA Program on Student Achievement A similar analysis was conducted by serviced versus non-serviced students and by program type using CSAP scale scores as the outcome variable. Again, FRL status and initial CSAP performance were held constant. Results showed that serviced students in all cohorts outperformed non- serviced students in CSAP growth in Reading. The impact of ELA services was mixed in respect to Math and Writing CSAP growth. For example, serviced students in the 1st-5th grade cohort showed greater gains on CSAP Reading and Writing than non-serviced students, yet results were not statistically significant for CSAP Math. The effect sizes were small to moderate, decreasing with each increase in grade level. Similar to CELA gains, the Kindergarten, 1st grade, and 3rd grade cohorts appeared to show evidence of greater growth among students in ELA-S and ELA-E (transitioned) or Dual Language programs than for students in ELA- S or ELA-E alone (stronger relationship for CSAP Reading than in Writing and Math). Sample sizes for ELA-S and Dual Language programs were too small in upper grades to run these analyses for the 6th and 9th grade cohorts.

*Serviced students = ELA-S, ELA-E, Dual Language; Non-serviced students = parent opt out, services not offered at school. Differences were significant (at least p<.05). Exited Students

Another key question of the longitudinal study was how students perform after exiting from ELA program services. Students who exited ELA services were more likely to score at CELA Levels 3, 4, or 5 in 2006-07 (91-100% of students in all cohorts except Kindergarten; 66% of the Kindergarten cohort students who exited started at Levels 3, 4 or 5). Furthermore, exited students were less likely to qualify for free or reduced price lunches and students were less likely to speak Spanish or a refugee language. Exited students were more likely to have received ELA services (vs. parent opt- out or no services offered at their school).

Due to these differences in student characteristics, an analysis to compare exited students with never ELLs was run. A sample of exited ELLs and never ELs was pulled, matching students on 2006- 07 CSAP scores and FRL status. Results revealed that when FRL status and initial CSAP scores were held constant, exited students perform similarly or slightly higher than never ELLs on CSAP.

Analysis includes students with two consecutive years of CSAP scores. Graduation & Dropout

The final set of analyses conducted for the longitudinal study was an examination of leave codes (e.g., graduated, transferred) for the 6th and 9th grade cohorts. Results from these analyses revealed that many students left DPS (e.g., dropped out) by the end of the study period, even exited ELLs who tend to show higher performance.

Among 6th-10th graders, 35% of the cohort was no longer in DPS by Grade 10 (i.e., transferred out or dropped out). A higher percentage of non-exited students left by grade 10 (48%) than exited students (28%). Of the 35% of the cohort who left, only 2% were coded as having dropped out. However, 18% of the cohort were reported as enrolled in DPS at the end of the year but did not appear in DPS the following year.

For the 9th-12th grade cohort, 48% was no longer in DPS by grade 12 in 2009-10. Among non-exited students, 52% were no longer in the cohort by 12th grade. Forty-four percent of exited students were no longer in DPS by grade 12. Slightly over one-third (35%) of the 9th grade cohort graduated, 36% dropped out or left without an exit code (7% were coded as having dropped out in 2010), 13% transferred outside the district or entered a GED program, and 17% were still enrolled in a fifth year of high school.

By comparison, the graduation rate for all students in DPS in 2010 was 52%, with 53% of non-ELLs and 43% of ELLs graduating. This was in contrast to a statewide graduation rate of 72%; 49% of the students with limited English proficiency graduated statewide, while 75% of non-ELLs in the state graduated in 2010. The 2010 dropout rate for ELLs in DPS was 5%; the dropout rate for all students in DPS was 6.4%, higher than that of the ELL subgroup. The statewide dropout rate for all students was 3%, and the rate for ELLs was 6%.

When ELLs who graduated from high school were compared to students who dropped out in respect to CELA, no major differences were observed. EL students who graduated and EL students who dropped out averaged between Level 3 and Level 4 on CELA Overall in grades 9 and 10. This may indicate that drop out may be more related to factors other than language acquisition.

Graduates and dropouts were also compared using CSAP as the outcome variable. ELLs who graduated averaged near the cutoff for Proficient in CSAP Reading in grades 9 and 10. Students who dropped out averaged near Partially Proficient in grades 9 and 10. On CSAP Writing, students who graduated and students who dropped out averaged between Partially Proficient and Proficient, yet ELLs who dropped out showed no improvement in Writing from grade 9 to grade 10. Finally, EL students who graduated averaged just below Partially Proficient in Math in grades 9 and 10, while students who dropped out averaged Unsatisfactory in Math in grades 9 and 10. Both graduates and dropouts made little growth from grade 9 to 10. Never ELLs (not graduates vs. dropouts) are shown for comparison. Additional Analyses

Target Trajectories for CELA One of the primary outcomes to emerge from the longitudinal study was the ELA Trajectory. The ELA Trajectory represents a new way of thinking about student progress on CELA. The trajectory was developed based on district, state, and national research, to assess whether or not English Language Learners are on pace in acquiring English proficiency (as defined by achieving CELA Overall Level 5). Students meeting their target trajectory are referred to as “On-Track.”

The ELA Trajectory analyses occurred as a result of the proposition by the Colorado Department of Education (CDE) for a six year trajectory to CELA Level 5 (see below). However, the question remained as to whether this was a reasonable number of years. Therefore, longitudinal study researchers examined multiple cohorts of students in Kindergarten through grade 12 to determine how long they took to advance on CELA to Level 5. The conclusion of these analyses was that, on average, students at every grade level and every starting CELA level adhered to CDE’s proposed six year trajectory.

The trajectory can be used to determine the number of years a student has to advance to the next performance level based on their first CELA Overall score. For example, if a student started at Level 2 in 2007, they should advance to Level 3 or higher in 2008, Level 4 or higher by 2010, and Level 5 by 2012. The ELA Trajectory is only for CELA Overall, not individual CELA domains.

In order to provide information to principals and school staff about how their students are performing in relation to the ELA Trajectory, user-friendly reports have been developed and rolled out in the Principal Portal. The reports summarize student information, as well as provide student level reports that can exported into Excel so that principals can further examine the data.

In addition to these reports, a number of analyses were conducted based on students’ on-track status. Analyses with CELA 2012 data revealed that of the non-exited EL students in DPS with more than one CELA score (i.e., not new to CELA), 53% were on-track to reach CELA Level 5. Of all the students who were off-track, 62% scored at CELA Level 3 or 4, while only 12% were at CELA Level 1. The percentage of students who were on-track decreased with each subsequent grade in elementary school, and there was a large drop in the percentage of on-track students in both 6th and 9th grades. Comparing students by FRL status, Paid Lunch students had the highest percentage on-track. The largest gap was between the Paid and Free Lunch students at nine percentage points.

Finally, one cohort of Kindergarten through 5th grade students was compared by program type. All placement options except those students who stayed in ELA-S throughout elementary school were shown to be following the ELA Trajectory. Researchers were not surprised to see that the ELA-S students who had not transitioned out of services within five years were the only placement option not following the trajectory (it is important to note the small sample size of this subgroup). This is consistent with the program findings discussed above. All of these ELA-S students qualified for free or reduced price lunches, and 18% were considered students with disabilities. Students who transitioned from ELA-S to ELA-E had the highest percentage on-track.

* 25% of students with a NS were in Kindergarten in 2012.

The New to CELA category is not reported here and not included in the denominator.

Note: On-Track status categories consist of On-Track, Off-Track, and New to CELA. The New to CELA category is not reported here and not included in the denominator. Kindergarten students reported here have Kindergarten scores in multiple years.

Note: On-Track status categories consist of On-Track, Off-Track, and New to CELA. The New to CELA category is not reported here and not included in the denominator.

The target trajectory lines indicate the lower and upper range of scores needed at each grade level to be considered On-Track. State ELL Placement designations were used to determine ELA Placement. Analysis includes students with CELA scores all six years. “ELA-S” includes students who had an ELA Placement of ELA-S for five or more years. “ELA-E” includes students who had an ELA Placement of ELA-E for all six years. “ELA-S to ELA-E” includes students who had an ELA Placement of ELA-S in Kindergarten and ELA-E in 5th grade and spent at least two years in ELA-S placement; “Parent Opt Out” includes students who had an ELA Placement of Parent Opt Out for four or more years.

State ELL Placement designations were used to determine ELA Placement. Analysis includes students with CELA scores all six years. “ELA-S” includes students who had an ELA Placement of ELA-S for five or more years. “ELA-E” includes students who had an ELA Placement of ELA-E for all six years. “ELA-S to ELA-E” includes students who had an ELA Placement of ELA-S in Kindergarten and ELA-E in 5th grade and spent at least two years in ELA-S placement; “Parent Opt Out” includes students who had an ELA Placement of Parent Opt Out for four or more years.

Language of Assessment Another analysis made possible by the longitudinal study as a result of cleaning the ELA data and tracking cohorts of ELLs over time was an analysis by CSAP testing language. ELLs who were in 2nd grade in either 2006-07 or 2007-08 and took CELA were combined into one cohort and tracked forward to 3rd, 4th and 5th grades. The analyses showed that selecting the appropriate language of assessment (TCAP Reading vs. Lectura) based on a body of evidence can result in higher 3rd, 4th, and 5th grade TCAP scores. Generally, students who scored at CELA Levels 1 – 4 in 2nd grade performed better in their native language (Spanish) on the 3rd grade TCAP. Students who scored at CELA Levels 1 – 4 and took the 3rd grade TCAP Lectura scored higher on TCAP Reading in 4th and 5th grades. The assumption is that for these students instruction continued in Spanish. These students continued to have a greater percentage scoring Proficient or Advanced on the 4th and 5th grade TCAP Reading. DPS leadership is using these analyses to inform educators as to how to determine the language of assessment for students so that students are assessed in the most appropriate language.

Literature Review: CELA Plateau Phenomenon DPS-UCD Collaborative members from UC Denver (Shannon Svaldi and Honorine Nocon) conducted a literature review regarding why DPS students experience a “plateau” at CELA Levels 3 and 4 as well as at transition points between grade levels, which was a primary finding of the longitudinal study. A brief review of the research in the field as well as recommendations to address these issues has been presented to leadership, a Research Brief was developed, and an extensive paper is currently in development. Workshops have been conducted with district leadership and school staff delving into the findings and brainstorming how to use the research in practice. Below is a brief review of the research; the language is taken directly from Ms. Svaldi and Ms. Nocon’s summary. The review of the literature suggests a complex answer to the question regarding why ELLs are experiencing “plateaus” at higher levels of English proficiency and at transition points between education levels. The answer involves developmental, socio-cultural, institutional, and assessment factors. Some plateauing at CELA Levels 3 and 4 reflects normal second language development, i.e., slower growth at higher levels of proficiency. Long term plateaus at CELA Levels 3 and 4 as well as the plateau/drop from grade 8 to 9 are associated with more complex content and language as students age and become more proficient in English, and for the older students, the stronger influence of sociocultural and institutional factors accompanies developmental changes. This suggests the need for adapting classroom, school and district approaches to meet the specific needs of these learners. These recommendations are summarized below. For more details from the literature review: http://testing.dpsk12.org/reseach_eval/research/research_findings/research_findings.html

What can we do in schools ? • Develop coherent and consistent institutional practices and beliefs: Coherence of approach to learners of language and language instruction, i.e., a shared language lens, is more important than the language of instruction.

• Cultivate the shared belief that all children can learn and that knowing more than one language is an asset.

• Build a positive, safe school environment.

• Provide a meaningful and academically challenging curriculum and hold high expectations for all students.

• Involve parents: Home literacy practices in L1 or L2, such as frequency of book reading with children and taking children to the library, also contribute to later L2 oral language and literacy development.

• Increase school‐parent outreach/involvement/interaction/community building in early grades. This outreach should include encouragement to read (and talk) in the home language especially and English (if the family members are relatively fluent).

What can we do in schools and districts ? • Provide professional development opportunities for content and ELA teachers in the structure, use, and function of English (relevant to their areas) so that they can intentionally and explicitly guide learners.

• Provide professional development opportunities for content area and ELA teachers about how languages work and how languages influence each other.

• Educate teachers and school leaders in second language (L2) acquisition and L2 instructional models. • Educate teachers and school leaders in strategies for transitioning learners to higher level courses and content that ensure (a) transition is not premature or (b) if transition is premature, other supports are in place.

What can we do in classrooms ? • Increase academic oral language use in upper grades and all grades for more advanced speakers in the content areas to support literacy and academic language.

• Use direct instruction of language and opportunities to interact in the language AND with the content in mainstream classrooms.

• Provide frequent opportunities for beginning, intermediate, and advanced learners of English to communicate in English. The purposes that actually motivate learners must be the primary drivers of language instruction ‐ not generic language structures. Acquisition of structures is supported by the need and desire to communicate.

• Use ongoing formative assessment of oral and written language as well as content knowledge to guide instruction and learning.

Implications & Next Steps Dissemination of Study Findings

Longitudinal study findings have been distributed widely throughout DPS. Presentations were given to senior DPS leadership, ELA Department staff, Teacher Effectiveness Coaches (TECs), Data Assessment Partners (DAPs) and School Improvement Partners (SIPs), and to principals at TNLI designated schools, to name a few.

Additionally, ARE produced a number of research briefs highlighting key findings that emerged as a result of the study. The first was about the ELA Trajectory and the second highlighted the literature review from UCD regarding the CELA plateau. Workshops with district and school leadership were conducted regarding the “plateau” literature review, delving into the findings and planning for how to integrate the recommendations for action into district and school practice. Additionally, leadership is working to roll out recommendations regarding the language of assessment protocol.

ELA Trajectory reports have been integrated into the Principal Portal, and provide school, region, and district level information. The school-level reports summarize how ELLs in the school are performing in relation to the target trajectory: whether they are on-track, off-track (and how long they’ve been at their current CELA level), or whether they are new to CELA. The school summary reports are followed by student-level information detailing ELLs’ on-track status, ELA program placement and other demographics, current CELA levels by domain, and most recent TCAP Reading and Writing performance levels. These reports can be used in conjunction with other data to determine what supports or interventions are needed school-wide, as well as what supports the ELL students need to be on-track to achieve English proficiency. Longitudinal study researchers worked with DAPs and SIPs to develop a protocol for interpreting and using ELA Trajectory reports, in addition to other CELA reports. Trainings have been conducted throughout the district on the CELA interpretation protocol. Furthermore, the ELA Department is working with the Department of Technology Services and ARE to develop both an ELA Dashboard and ELA “report card” that highlights key ELA data indicators and guides educators in interpreting and acting upon the data.

Recommendations for Further Data Collection: Language of Instruction

As a result of the longitudinal study, researchers have been working to help improve the data ELA systems in DPS. One such example of this effort is the potential development of a tool to begin collecting data on language of instruction and actual services delivered in DPS ELA-S and ELA-E classrooms. Currently, language of instruction is determined based student ELA program placement designations. These designations are our best guess based on a number of variables, including PPF, school designation, HLQ, teacher designation, course designation, test scores and more. These classifications assume that students designated as “State Bilingual” receive some level of native language (Spanish) instruction and those designated as “State ESL” receive instruction primarily in English with additional supports in their native language if available.

However, (1) these are our best guess based on the data available, and do not necessarily reflect the services the student is actually receiving, and (2) even if they do reflect the services rendered, the amount or dosage of instruction provided in each language is still not tracked. Therefore, researchers are challenged to truly evaluate the DPS ELA model since we do not have a more precise measure of what is implemented in classrooms. In order to be able to answer research questions such as “How effective are schools in transitioning students to English?” information on actual language of instruction and dosage of language is needed at different times throughout the year.

To address this need, an online data entry application such as OASIS could be used to collect this information. Teachers would be asked to periodically complete three to six simple questions each month (e.g., Thinking about your literacy instruction this month, did you teach primarily in Spanish or in English?). To date, the ARE department has developed a series of questions in collaboration with the ELA department; questions can be pushed out to teachers based on teacher and course designation (ELA-E, ELA-S, ELA-S/E, dual language) and grade level. Next steps include working with the ELA department to work out a project plan, determine the exact business rules regarding how to push out the tool to teachers, develop and pilot the tool, and promote and monitor the application among school staff. UCD-DPS Collaborative

A number of extension studies, grant applications, and manuscripts are underway in DPS based on findings from the case studies and longitudinal study. For example, some extension studies currently being conducted by DPS-UCD Collaborative researchers include studies of math and literacy, Empowering Teachers-as-Researchers, formative assessment practices, and leadership practices. Extension studies are conducted based on researcher interest, alignment with DPS priorities, and available resources. For more information about these ongoing activities, please contact the ARE department. Next Steps

A number of next steps have been identified related to the longitudinal study. These next steps are described below.

ELA Teacher Effectiveness Project The Human Resources and Planning Departments are involved in developing and executing an analysis of the effectiveness of ELA teachers. The project is still in development. The project lead has worked extensively with longitudinal study researchers to align research questions to the ELA research occurring in DPS. These analyses will describe the current teacher population, including teachers’ preparation programs and retention in DPS, and will develop a model to predict student achievement.

Achievement Gaps Research Project Longitudinal study researchers have worked in collaboration with other ARE researchers to develop a project to study the achievement gap in DPS more extensively. The project, which is currently in development, will examine students who qualify for free or reduced price lunches and ELLs who have closed the achievement gap with non-FRL/never ELL students by 5th grade. The primary research questions of this study are:

 What are the school related factors that contributed to closing the gap?

 What are the parent/family related factors that contributed to closing the gap?

 What are the time use differences between students who closed the gap and students who did not close the gap?

This mixed methods study emerged from analysis of the gaps in DPS between at-risk student populations, including ELLs. Results showed that the academic performance gap between ELL and never ELL students existed when students entered DPS and on whole persisted throughout students’ academic career, indicating that students come in with certain disadvantages. While this was the overall trend, there were instances in which students closed the gap. This study will examine students who closed the gap, and contrast those students with the population who did not. The study will take place in the 2012-13 school year.

Transition from CELA to Access With the adoption of the WIDA standards in Colorado, the state has also adopted the assessment associated with those standards. Researchers in ARE are working to develop a transition plan from the CELA to the Access assessment, which will inform future analyses regarding ELLs in DPS.

Researcher in Residence As in the 2010-11 school year, a researcher in residence from UC Denver will be hosted in the ARE department. This UCD professor, who has extensive experience in mining and analyzing longitudinal datasets and hierarchical linear modeling, will be involved in key research projects and initiatives throughout the district.

New ELA Longitudinal Study Through negotiations regarding the Modified Consent Decree (MCD), a new longitudinal study has been proposed that will continue through 2015 to examine how ELLs are performing in DPS after the multiple changes (e.g., parent placement form options, teacher training) related to ELA have been implemented.

Conclusion The longitudinal study revealed a number of trends regarding DPS ELLs. The study findings provided evidence to support ELA programs, as well as areas of need among EL students, such as students experiencing plateaus at higher levels of English proficiency. Students who have not transitioned or exited from program services by secondary school appear to show less growth toward English proficiency.

Based on the study findings, a target trajectory for ELLs was developed and reports have been disseminated to school staff regarding the “on-track” status of their students. Additionally, a literature review was developed by UCD, which informs the plateau phenomenon and target trajectory, and provides extensive recommendations to district and school staff regarding best practices. These findings continue to be distributed throughout the district, including extensive conversations and planning sessions about how to improve practice within DPS.

Literature Review References Abedi, J. (2009). Validity of assessments for English language learning students in a national/international context. Estudios sobre Educacion, 16, 167-183.

Dixon, L., Zhao, J., Shin, J., Wu, S. et al. (2012). What we know about second language acquisition: A synthesis from four perspectives. Review of Educational Research, 82(1), 5-60.

Fox, F.E., Levitt, P., & Nelson, C.A. (2010). How the timing and quality of early experiences influences the development of brain architecture. Child Development, 81(1), 28-40.

Genesee, F., Lindholm-Leary, K., Saunders, W. & Christian, D. (2006). Educating English Language Learners. New York, NY: Cambridge University Press.

Hakuta, K. (2011). Educating language minority students and affirming their equal rights: Research and practical perspectives. Educational Researcher, 40(4), 163-174.

Lantolf, J.P., & Thorne, S.L. (2006). Sociocultural theory and the genesis of second language development. New York: Oxford University Press.

Lenroot, R., & Giedd, J. (2011). Annual Research Review: Developmental considerations of gene by environment interactions. Journal of Child Psychology and Psychiatry, 52, 429–441. Lillard, A., & Erisir, A. (2011). Old dogs learning new tricks: Neuroplasticity beyond the juvenile period, Developmental Review. 31, 207-239.

Solano-Flores, G. (2008). Who is given tests in what language by whom, when, and where? The need for probabilistic views of language in the testing of English language learners. Educational Researcher, 37(4), 189-199.