Horizonte School Improvement Grant Evaluation

Horizonte School Improvement Grant Evaluation Final Report March 2014 by Angelina Castagno and Charles Hausman

Sources of Data As part of the ongoing evaluation of the School Improvement Grant at Horizonte, one evaluator spent three days at Horizonte interviewing a range of students, parents, teachers, coaches, counselors, social workers, and administrators at the end of January 2014. The purpose of this visit was to collect qualitative data for the final year of SIG. Interviewees included the following:  5 parents who represented at least four different racial/ethnic backgrounds and children who had been at the school from anywhere between two months and two years;  2 coaches;  3 counselors or social workers;  5 administrators representing both the school and the central office;  21 teachers representing every subject area, including teachers from the main school building (including night school) and three outside sites; and  14 students from the main school building who represented at least three different racial/ethnic backgrounds and who had attended Horizonte from anywhere between two months and two-and-a-half years.

Most of these interviews occurred in focus groups, but five of them were individual interviews. Groups were primarily organized by similar content areas (for teachers) and/or roles within the school (e.g., counselors and social workers). All interviews took place at the Horizonte main building site. During the interviews, the evaluator took detailed notes and captured participant responses verbatim as often as possible. All participants were informed that their anonymity would be maintained, direct quotes would be used in reports and publications but would not be attributed to individuals, and they would have access to the final evaluation report. All statements below that are in quotation marks are direct quotes from respondents during interviews or focus groups.

In addition to the above qualitative data, quantitative data from five surveys are utilized to support the findings in this report. These surveys include:  Spring 2014 Mid-Year Teacher Survey (February 2014, N=38),  AdvancED Staff Survey (January 2014, N=43),  AdvancED Parent Survey (January 2014, N=87),  AdvanceED Student Survey (January 2014, N=195), and  Spring 2013 SIG Teacher Survey (March 2013, N=34). The specific survey utilized to generate tables and graphs presented appears under each graph and table. AdvancED survey data were utilized in this report to prevent unnecessary duplication of surveys, thus reducing stakeholder’s completion time and avoiding disruption during end of year accountability testing.

1 In an effort to tell the full story that the qualitative and quantitative data encompassed, we have organized this report around three overarching questions: Who/what is Horizonte?, What impact did SIG have on Horizonte?, and Who/what will Horizonte be post-SIG?

Who/What is Horizonte?

Teachers, administrators, and other staff at Horizonte share an awareness and an ownership over their institutional identity and responsibility. This was aptly described by one leader who noted that “awareness has been the biggest benefit of SIG…awareness by the school themselves about the learners they have, and it helped people see that they could impact student learning…there has been greater awareness by leaders and staff at all levels.” There is a common language and vision across the various constituent groups that make up the school (students, parents, teachers, administrators, other staff). This sense of who/what Horizonte is and the significant evolution of the school’s identity over the past few years was articulated by many teachers, staff, and administrators. The following three quotes, all from different individuals, speak to this:  “When we give schools some funding and let schools decide what’s best, we can do amazing things for our students. SIG allowed us to ask who our students are and then cater all of our programs to our kids…when you give that local control and give them funding, give people in the building power to find ways to help the students, they really can do amazing things.”  “There is a real camaraderie among teachers…the students and parents feel good about what we’re doing; we’ve tried to communicate a lot with parents about what we’ve been doing, and it’s fun to see some of the successes.”  “There’s been a paradigm shift at this school, and it’s evident as we talk with parents and students and teachers…there is improvement and integrity of academic rigor. We used to be known as a holistic focused school where everyone was welcomed; we want to keep that of course, but at the same time you see that the instructional capacity and curriculum have had a tremendous shift.”

Survey data further reinforced that a shared vision, one focused on student success, exists across stakeholder groups at Horizonte. Specifically, 97.6% of staff and 89.5% of parents strongly agreed or agreed “Our school’s purpose statement is clearly focused on student success.” Similarly, 85.1% of staff strongly agreed or agreed “Our school’s purpose statement is based on shared values and beliefs that guide decision-making.”

In this section, we identify and describe the prominent themes that emerged from data collected during the third and final year of the SIG; these themes are illustrative of Horizonte’s institutional identity.

Academic Rigor

One of the most prominent themes that emerged from our interviews across every constituent group was “academic rigor.” Although students and parents talked about rigor

2 using different terms than teachers, staff, and administrators, they all pointed to the importance of academic rigor and the shared vision and value around rigor. As one staff member summed up, “We’re really focused on academic rigor. It was there with some teachers, and really lacking with others; it had no consistency from place to place and site to site…we’re doing a whole lot better preparing students academically.” Most teachers, staff, and administrators felt that the focus on rigor was a new focus that had been established and solidified over the past few years at Horizonte. As one teacher aptly described, “In terms of what students can do in the classroom, it’s like a 180 degree change. I really remember last year asking students to write or analyze documents and students seemed really resistant to it. And now I can ask them to write and they know what to do. And they feel positive about doing things that are challenging, and they like reading…now it’s just like this is what we do.” Since previous years of SIG were focused on the establishment of common curricula and assessments, efforts in year 3 focused on rigor in many content areas. For example, Language Arts had previously established common curricula and assessments that were aligned, so during year 3 of SIG they were working to “increase the rigor and complexity of that curriculum.”

As indicated in the following graphs, the majority of students, parents and teachers agree that Horizonte is a school in which rigorous teaching and learning occur. For example, 63.2% of students agreed or strongly agreed “My school provides me with challenging curriculum and learning experiences,” while only 6.2% disagreed or strongly disagreed. It is important to emphasize that over 30% responded neutral on this item, so there is still room for significant improvement from the perspective of students. 74.9% of students agreed or strongly agreed “In my school, a high quality education is offered,” while only 4.7% disagreed or strongly disagreed.

3 2014 AdvancED Student Survey

A higher percentage of parents than students rated Horizonte as rigorous. Specifically, 83.4% of parents agreed or strongly agreed “Our school has high expectations for students in all classes,” while only 4.8% disagreed. In addition, 80.7% of parent respondents agreed or strongly agreed “All of my child’s teachers give work that challenges my child,” while only 1.2% disagreed.

2014 AdvancED Parent Survey

4 Finally, 95.6% of teachers expressed some level of agreement (16.2% slightly agreeing) with “Teachers have high expectations for student learning.” Unfortunately, one teacher strongly disagreed, and another slightly disagreed.

2014 SIG Mid-Year Teacher Survey

When asked in the interviews and focus groups to elaborate on what rigor meant, teachers explained that there used to be a lot of “worksheets,” “poor interaction,” “credit for filling out little stuff,” and “too much focus on the social work stuff at the expense of academics.” Currently, however, most believed that students were engaged in “higher level thinking” and “more difficult reading,” and that teachers held “higher expectations” and maintained “greater consistency in grading.” In our conversations with students about the academic rigor at Horizonte, they initially communicated that the work “was easy.” It is important to understand, however, what students meant by “easy.” Through probing questioning and conversation, it became clear that “easy” did not mean less academically rigorous; instead “easy” meant accessible, relevant, and “challenging but not stressful.” As one student explained, “It’s tough but doable” and another added “When it’s hard, teachers come and give you help.” Parents also talked about the resources available to support the emphasis on academic rigor; they mentioned resources such as peer tutoring, college prep classes, and extra support from teachers outside of class time. An administrator similarly articulated that “Easy is not about rigor; it is about access and meeting the kids where they are at.” In summary, students described teaching and learning at Horizonte as rigorous and easy, with easy referring to their opportunity to be successful as a result of the support provided by school personnel. Support takes the form of differentiated and personalized instruction, targeted interventions, and school based services.

Quantitative data were consistent with these descriptions. For example, 72% of students agreed or strongly agreed “All of my teachers use a variety of teaching methods and learning activities to help me develop the skills I need to succeed.” Similarly, but at a lower frequency, 60.1% of students agreed or strongly agreed “All of my teachers change their teaching to meet my learning needs,” leaving room for improvement.

5 2014 AdvancED Student Survey

Horizonte parents provided further evidence of teacher’s providing personalizing instruction to enable students to be successful. 82.2% of parents agreed or strongly agreed “All of my child’s teachers meet his/her learning needs by individualizing instruction.”

6 In addition to personalized learning, students and parents identified support programs and services as critical to student success. As indicated in the graphs below, 86% of students and 82.7% of parents agreed or strongly regarding the availability of support services and programs based on identified needs and their positive contribution to student success.

2014 AdvancED Student Survey 2014 AdvancED Parent Survey

As highlighted in the tables below, teachers provided the final evidence validating that differentiated, personalized teaching and other support services and programs were enhancing student success. The numbers in each column represent the percentage of teachers providing each level of agreement. On average, approximately 87% of teachers agreed or strongly agreed with items regarding targeted interventions for students, individualized assistance for students, high quality student support services, and personalized instruction—all of which meet the individual learning needs of students.

SD=Strongly Disagree D=Disagree SlD=Slightly Disagree SlA=Slightly Agree A=Agree SD=Strongly Agree

Indicator SD D SlD SlA A SA Students who need them are being provided targeted instructional 0 2.6 5.3 13.2 52.6 26.3 interventions. Teachers are available to give students the assistance they need with 0 2.6 5.3 5.3 52.6 34.2 assignments. 2014 Mid-Year SIG Teacher Survey

SD=Strongly Disagree D=Disagree N=Neutral A=Agree SD=Strongly Agree

Indicator SD D N A SA Our school provides high quality student support services (e.g., 0 2.4 4.9 36.6 56.1 counseling, referrals, educational and career planning.) All teachers in our school personalize instructional strategies and 0 2.3 9.3 48.8 39.5 interventions to address individual learning needs of students. 2014 AdvancED Staff Survey

7 Horizonte leaders made pointed efforts to ensure that the majority of teachers were teaching in their content area as a strategy to enhance rigor. This is more challenging at the smaller sites, but school leaders felt that significant progress had been made in this area. Teachers also noted that they appreciated the distinct teaching assignments so that most (at the main site, at least) have fewer class preps. In the past, for example, a single teacher may have taught four different math classes, but now math teachers are assigned to either 9th and 10th or 11th and 12th. A concerted effort also was made during the final year of SIG to “slow down some of the transfers” into the school so students now typically only join classes during particular times each term. This was described by teachers and other staff as having had a tremendous impact on the ability to maintain structure and consistency within classes.

Student Success

Another major theme from conversations with every constituent group was the emphasis on student success at Horizonte, which was embedded in the common vision described earlier in this report. Success was talked about in a number of ways, but all agreed that the school had a common goal of student success, which included that students would be successful academically and socially at Horizonte but also after they graduated from Horizonte. One teacher stated there is a “culture of expecting excellence from both teachers and students,” and another added “The kids expect to work and they want to work.” Success includes clearly articulated high expectations (i.e., academic rigor), as well as focusing on the growth and development of students who have often been labeled in appropriately as “failures” in other school settings. As one teacher explained, “Some kids who never had success before are now getting it…and they know it’s not fluff success.” Parents also articulated the shared focus on student success, and it was one of the things they most appreciated about Horizonte. One parent noted that when she met with teachers, they focused on “what my student can do” to succeed rather than “focusing on the negative stuff.”

The emphasis on student success also includes student accountability. As parents noted, “students have accountability…kids are held accountable for their actions, both good and bad.” Parents are kept informed of their students’ progress, including immediate phone calls home if a student is not at school. As one parent explained, “This makes you feel like your kids are safe.” But parents and students are also kept informed about the growth and positive things students are doing at Horizonte. In other words, student success is intentionally celebrated at the school. Most classroom doors have posted signs with students’ names who have had good attendance and who have passed all their classes. This is an effective means of public recognition, and it is one that students also noted in our conversations with them. Other public recognition of success that was identified in our data included award ceremonies, gift cards, and phone calls home.

Parent survey data provide additional evidence of the extent to which teachers help parents understand their child’s progress (80.1% strongly agree or agree). Survey results also indicate that learning at Horizonte is a shared responsibility (74.7% strongly agree or agree).

Students identified a number of reasons they were successful at Horizonte despite previously experiencing little to no success in other schools. These included:  “more one-on-one time with teachers”,  “more opportunities to get into college”,  “feedback from teachers,” especially related to “the good things we’re doing and the things we can get caught up with”,  the school “gives back to the community” and provides “lots of things for the community that really help out”,  “failing here is not an option” and “failing is not acceptable here”,  “they don’t judge you here”,  “they have conversations with everyone and get to know us”,  “everyone [including students] motivates everyone to do good in this school”, and  “we get to be more open” and “we can express ourselves”

Students and parents spoke about the “smaller classes” and “more individual attention” that occur at Horizonte. These were important factors contributing to the perception that students are more successful and that teachers and leaders at Horizonte “care about student success.” In addition to the personalized instruction and support services described earlier, relationships were a common variable students and parents identified as enabling student success.

Consistent with this emphasis on relationships, on the 2013 Spring SIG Survey, 88.7% of teachers expressed slight to strong agreement with “I am satisfied with the way students are treated by teachers.” 85.9% of teachers expressed agreement with “I am satisfied with the way students are treated by administration, and 92.3% of teachers agreed “I am satisfied with the way students are treated by counselors.” With the belief that respect is reflective, it is important to emphasis that students reported being treated with respect. Specifically, only 14.8% of students disagreed or strongly disagreed that “In my school, all students are treated with respect.” However, 26.5% of students responded “Neutral” to this

9 item, suggesting some disconnect between teachers and students, as well as room for progress in this area.

2014 AdvancED Student Survey

Research has identified the importance of every student having a least one adult advocate at school. As noted in the graphs below, a clear majority of parents (87.7%) and students (72.5%) believe they have an advocate at Horizonte.

Common time before the school year was consistently identified by teachers and other staff as integral to the establishment of a shared vision and common language on student success throughout the school. There was widespread concern about that time not being available once SIG funds were gone. As one teacher noted, “Once that time is gone, I’m afraid the consistency will go away,” and another explained, “that philosophy came with training, with quality professional development, and there was interaction and modeling.”

Students talked at length about how teachers, staff, and administrators at Horizonte focus on success post-graduation from Horizonte as well. As one student noted, “Before I came here, I was 110% sure I wouldn’t go to college, and now I know I will.” Other students agreed, and every student with whom we spoke said that they planned to pursue additional education or training after graduating from Horizonte. This wasn’t a vague notion of

10 “college”; instead, students talked about specific colleges and programs in which they were interested, and many had actually visited those sites. As one student elaborated, “Now I have a better idea of what I want to do.” Over the life of the SIG, there has clearly been a cultural shift at Horizonte in which all stakeholders have moved from the expectation that all students graduate from high school to all students graduate career or college ready. As a result of rigorous and relevant instruction and healthy relationships, parents and their students report feeling prepared for the future.

Acceptance and Validation of Students and Families

There is a clear understanding among Horizonte teachers, leaders, and other staff about what Horizonte does for and with students and families. There is a cultural norm that is inclusive of families and youth. Staff understand that “Every interaction most families have had with schools has been negative,” and they work hard to reverse that trend. Every family meets with school leaders before their child enrolls at Horizonte, and that conversation centers on the academic rigor at the school and the expectation that every student will succeed. This is followed up with frequent “positive phone calls” to families, which Horizonte staff explain is “a new concept for most families.” Parents talked at length about this aspect of Horizonte, and especially how this makes Horizonte unique. One explained, “It’s way way different [from our previous school]; it’s more respectful, the teachers and the kids. When you walk in, you feel like are you very welcome anytime.” Another parent concurred and explained how students are accepted and validated at Horizonte: “They don’t judge the students; they accept them. Some students come with gang affiliations or whatever, and they don’t treat them like they’re stupid. It doesn’t matter why they fell behind; they are just wanting them to succeed.”

Survey data reinforced the acceptance and inclusiveness of families. For example, 81.5% of teachers agreed or strongly agreed “Our school provides opportunities for stakeholders to be involved in the school,” while 94.9% of parents/guardians expressed some level of agreement that “Parents/guardians feel welcome in this school.”

The acceptance and validation of students and families, and their inclusiveness in the schooling process, are obviously related to the theme of student success discussed earlier

11 in this report. Part of why students are able to experience success is that they are validated as human beings capable of learning and capable of growth. One student captured this sentiment well when he explained, “Everybody has potential to have greatness in this school…for some of us, it didn’t work out as well at our other schools…but here we know we have that potential. Here they give us that opportunity to bring out that potential.”

Collaboration

Teachers across every subject area articulated the sense of unity and a common vision within the school; one noted, “As a staff, we are working together better; we have common goals, and we’re headed in the same direction; each year of the grant it has progressively gotten better.” And another teacher concurred saying, “We’re all really focused in the same direction.” Another agreed by saying, “When we’re all on board and we set an expectation, it’s been great.” Parents also spoke about the collaborative work at Horizonte. Many agreed with the one mother who noted that teachers “come together to get help for” students if and when that is needed.

This level of collaboration was validated further by the survey data. For example, 78.2% of teachers agreed or strongly agreed with “In my school, teachers work together to improve student learning,” while only 4.1% disagreed or strongly disagreed. Similarly, 83.5% of parents agreed or strongly agreed “All of my child’s teachers work as a team to help my child learn,” while only 3.5% disagreed.

2014 AdvancED Teacher Survey 2014 AdvanceED Parent Survey

Structured time for collaboration among teachers, staff, and administrators was identified as absolutely critical for sustaining authentic collaboration. As one teacher noted, “In past years, we talked about collaborating, but we never had time for it. Time is so crucial for collaboration, and it’s one thing that teachers don’t have enough of.” This notion of structured time to collaborate is further discussed below in the section on Professional Learning Communities.

Some content areas have been more productive and successful with their collaborative time, and others expressed a desire for that. In year 3 of SIG, there was increased

12 collaboration between social studies and language arts, and teachers in both content areas talked about the benefits they have noticed in their classrooms. Teachers are benefitting from additional resources and collaborative conversations, and students are noticing the curricular alignment and transferring skills between the different content area classes. Teachers explained that “The consistency has been really good for students,” and that “The language they are hearing is the same.” Science, on the other hand, is an area that still needs to work more collaboratively to deliver a more unified curriculum and develop a set of common assessments.

There was some concern among teachers about an absent shared understanding regarding student behavior. As has been true in previous years, some teachers felt that there were inconsistencies among teachers and between teachers and administrators regarding behavior expectations and follow through. As one example, teachers described how the “cell phone policy has been on the agenda for four meetings” but was still not addressed in a faculty meeting. Teachers with whom we spoke were not overly concerned about the particular contents of behavior policies, but they were concerned that policies were clear and that everyone was “enforcing them uniformly.” There were also a small number of teachers who felt that some teachers had “given up as far as enforcement.”

Data-based Decision-making

The use of data to guide student placement was identified by a number of people with whom we spoke. One teacher summed up this practice by saying, “Kids are being placed in classes where they should be…and that has been hugely beneficial.” Another concurred, noting, “I don’t know how we did it without this data; we just guessed at what students needed, but now we can actually use data to make decisions.” The use of data also was described as important for helping students build self-confidence because data “motivates them when they can actually see it.”

On the 2014 Mid-Year SIG Teacher Survey, teachers were asked to report their level of agreement regarding whether data were being utilized for several specific purposes. Collectively, the results indicate widespread use of data to inform instruction, with the greatest emphasis on measures of student growth over time as opposed to achievement at one point in time. Their responses are in the following table.

Indicator SD D SlD SlA A SA Teachers use data to track the achievement of individual students. 0 2.6 2.6 23.7 39.5 31.6 Teachers use data to track the achievement of specific groups of 0 5.3 5.3 36.8 21.1 31.6 students (e.g., Special Ed, English Learners, racial/ethnic groups.) Teachers evaluate student achievement against benchmarks related 0 2.6 0 39.5 18.4 39.5 to the Utah Common Core. Teachers use assessment to measure student progress over time (e.g., 0 0 0 18.4 26.3 55.3 pre and post-test scores, gain scores.) Data on student performance from common assessments are utilized 2.7 2.7 5.4 21.6 29.7 37.8 on a regular basis to inform instruction.

2014 Mid-Year SIG Teacher Survey

13 School Image

Horizonte still seems to have a negative reputation among many in the Salt Lake community who are not personally connected to the school. Some of the parents talked about how they did not want their child to attend Horizonte initially, and some students concurred with this sentiment. However, all agreed that “Once people actually come here and see it, they see why it’s so good.” The “good things” include those mentioned previously in this report, but especially the overall culture of acceptance and validation, the pointed focus on student success, and the resources provided to support students and families.

Students were passionate about the misperception of Horizonte as a “bad school with bad kids,” and they were clear about their own responsibility for helping to change that reputation. As one said, “It’s our job as students who are here to put Horizonte on the map as a good place,” and another added “We need to prove them wrong about what Horizonte is about.” A third student noted “We’re the ones that work harder,” and still another explained that “We have people here that want a second chance and want to succeed in life.”

Importantly, most staff believed that significant improvements have been made in terms of how the other high schools in the district seem to perceive Horizonte; one teacher noted, for example, that “We’ve done a lot to change their view of us.” The biggest change, according to some Horizonte staff, is the pointed focus on academic rigor. To continue to improve the perception held by those external to the school, Horizonte personnel continue their focus on rigor and student success, while intentionally telling its positive stories in as many venues as possible. Rebranding takes time and should continue to be an explicit goal.

What impact did SIG have on Horizonte?

The themes described above illustrate what makes Horizonte, Horizonte. They are cultural in the sense that they give the school its unique identity. When considering what impact the School Improvement Grant had on Horizonte, it is difficult to parse out specific impacts of SIG from elements of the school that happened during SIG but maybe were less directly related to SIG. In some ways, this is the point since SIG implies a holistic and systemic approach to school improvement. In other words, the items above describing who/what Horizonte is may (or may not) be intimately tied to the overall impact of SIG, but they clearly interact with SIG initiatives. In our interviews with the various constituents at Horizonte, we talked about some specific, concrete elements that staff believed were directly attributable to SIG. Similarly, the 2014 Mid-Year SIG Survey asked teachers to express their level of agreement that SIG directly led to the achievement of several specific outcomes. In this section, we share data about each of the most frequently referenced and agreed upon elements.

Coaches

14 For teachers, the addition of content area coaches was consistently described as the most positive element of SIG. For staff, administrators, and teachers, coaches were described as “essential” to the work that happened under SIG, especially given the overlap in timing with transitioning to the Utah Common Core and SAGE. Some also noted how important the math coach was since many math teachers at Horizonte did not have a degree in math (although this shifted by the third year of SIG). In addition to coaching responsibilities, there was an effort during year 3 of SIG to develop “Horizonte curriculum materials” so that teachers would have a bank of materials for teaching the Utah Common Core. Teachers also expressed an interest in this effort, noting, “We want a bank of high quality curricular materials aligned with the standards.” Coaches talked about using their time to “sift through and identify the best curricular tools that are aligned with the core.” They viewed this as an important form of support that they were able to offer teachers, and teachers agreed. Providing students access to the core curriculum and using curricular and instructional materials are fundamental to student success. All but one teacher reported some level of agreement that they teach the Utah Common Core, which is still in its infancy. The coaches provided extensive support to enable the transition to this new and very different curriculum.

Year 3 of SIG seems to have had much less observations of teachers by coaches. Coaches expressed that their time had been focused on other forms of support so that the formal observations and modeling that had happened in previous years was not taking place as much during this final year of SIG. They did, however, express an interest and commitment to returning to that model of coaching where observations and feedback cycles occur frequently.

Survey data on the impact of coaches is extremely positive. We contend that the most important role of coaches is to work with classroom teachers to improve instruction, and thereby student achievement. As displayed in the following graph, all but one teacher reported some level of agreement with “Support provided by the language arts and math coaches has helped me improve my teaching.”

2014Mid-Year SIG Teacher Survey

15 Furthermore, when asked to rate the extent to which SIG expenditures on coaches had a positive impact on the school, all teachers agreed at some level that resources allocated to coaches had a positive impact, with 75.7% strongly agreeing.

Professional Learning Communities (PLC’s)

The PLC’s were consistently described as essential to the work that happened under SIG. Although some teachers and staff felt that PLC’s “might have happened without SIG,” most agreed that “SIG gave us the focus and direction to really make those work.” What seemed key to the success of PLC’s was that they were structured into the school day so that teachers could meet collaboratively, while students were in groups with the counseling and social work staff gaining important knowledge and skills from that group. An added benefit to this scheduling was that it gave the counseling/social work staff additional access and opportunities to connect with students. The importance of these relationships to student success was emphasized earlier in this report.

The PLC’s, as well as the all-school professional development, also were identified as contributing to “greater cohesion” among the staff from the main site and the outside sites. This time together resulted in “the same curriculum and trainings, which has been invaluable.” This greater consistency and integration across all school sites has been an important accomplishment over the past three years, and it is likely attributable to both the shared PLC and PD time and a SIG funded position explicitly for those purposes (see the following section).

The PLC time in previous years had been more professional development-based, but there was an effort during year 3 of SIG to ensure that it was collaborative work time among teachers with the coach serving as a facilitator or “conductor” rather than a “formal leader.” Teachers and other staff now distinguish between PLC’s and professional development and note the importance of both. Weekly collaborative meetings through the PLC structure were identified as ongoing and very necessary to maintain the consistency and rigor in the academic program of the school. Professional development, on the other hand, was identified as important as well but as primarily done individually or at specific times throughout the year (i.e., before the school year begins).

Survey data clearly indicates overwhelming teacher support for PLC’s. On all six items assessing PLC’s, the largest percentage of teachers strongly agreed that PLC’s were having a positive effect on the outcomes assessed—aligned instructional practices, enhanced student learning, improved teaching, understanding of the core curriculum, and increased classroom rigor. Clearly, each of these outcomes contribute to greater student achievement.

Indicator SD D SlD SlA A SA Teachers work in Professional Learning Communities (PLCs) to 0 0 0 0 19.4 80.6 develop and align instructional practices. PLC participation is an effective way to enhance my students’ 0 0 0 5.4 35.1 59.5 learning. PLC participation has positively affected my teaching. 0 0 0 5.4 29.7 64.9

16 PLC participation has helped me understand the Utah Common Core. 0 2.7 0 8.1 16.2 73.0 PLC participation has helped increase rigor in the classroom. 0 0 2.7 8.1 21.6 67.6 PLC participation is a good use of professional time. 0 0 0 8.1 18.9 73.0 2014 Mid-Year SIG Teacher Survey

Additional Role of the Counselor/Administrator for the Outside Sites

Almost every staff member, teacher, and administrator interviewed voiced that there was an “increased energy in terms of social work and counseling” during the past couple of years at Horizonte. This was most strongly felt at the outside sites, and teachers from those sites have consistently identified the additional counselor/administrative person as vital to the smooth operations and success at those sites. As one teacher explained, “There is constantly a need for folks to be out at the sites.” In addition to the responsibilities at the sites, this person also worked with others to develop a Comprehensive Guidance curriculum for all teachers to use in Advisory.

Consistent with the qualitative data and the earlier points regarding the greater consistency and integration across school sites, there was near unanimous strong agreement that the allocation of SIG funds to this position positively impacted the school. The results are presented in the following graph.

Additional On-site Data Specialist

School leaders, other staff, and most teachers described this position as invaluable to the decision-making that has occurred over the past three years at Horizonte. Coaches and administrators adamantly emphasized that it was integral to their work. As one coach explained, “If data is being effectively used to alter instruction and be more responsive, then it is worth a position.” Teachers benefitted from training on using data, as well as help seeing “how and why the data matters.” Counselors and administrators benefitted from seeing which groups of students were not being successful in order to make changes to students’ programs to boost success among those groups. As just one example, students in Directed Studies were identified as not doing well on state exams, so staff switched to an

17 online curriculum. Most staff agreed that these students were being much better served under this new approach. Data also were identified as being critical to efforts at seeking additional funding for the school as a whole and for particular groups of students. There was a general sense that “if we just rely on the district (i.e., on the central office data personnel), we won’t get what we need when we need it,” given the volume of data central office has to process and the number of schools they support with a relatively small staff.

As highlighted in the graph below, all but one teacher expressed agreement that allocating SIG funds for an on-site data coordinator positively impacted the school. 64.9% of teachers strongly agreed.

Continuous Assessment

Throughout the past couple of years, there have been ongoing conversations between teachers, coaches, and others about what common assessments should look like. Some teachers felt that there had been a shift from “test fatigue” during year 2 of SIG to “tests just went away” during year 3 of SIG. Because of very different accounts by different people, it was difficult to determine from the qualitative data the degree to which common assessments were or were not occurring in the various subject areas on an on-going basis, so this is an area that school leaders, coaches, and teachers should revisit in order to ensure fidelity to school improvement goals. As one teacher explained, “This year the data was becoming more helpful [because of the development of common assessments and rubrics], but now we’re back to doing things on our own.” Teachers in some content areas felt that data from assessments were not being returned to teachers quickly enough for it to be useful. There was a clear message that data are necessary for student success, but it must be relevant to the teaching and it must be received almost immediately in order to guide instruction.

The math department made some changes to their assessments during year 3 of SIG. They had conversations about “what do we want to get out of this” and decided to move away from strictly multiple choice assessments. They have collectively built common

18 assessments as well as common rubrics that have been used during year 3. This is a significant change from the model the math department had in previous years when the coach developed multiple choice pre and post assessments that teachers used. Language arts also had common assessments that were being used, and their focus during year 3 was primarily on increasing the rigor of their curriculum and related assessments (as described in the “academic rigor” section above).

Instructors

Everyone interviewed agreed that the impact of instructors “totally depended on the instructor.” Almost everyone also agreed that greater clarity regarding the expectations of instructors was needed, as well as training for the instructors and for the teachers so that they knew how to best utilize the instructors. As one staff member articulated, “I wish we as a SIG group had discussed the use of instructors and done training.” Some teachers felt that the loss of instructors mid-year during year 3 of SIG was a significant loss, and others believed they “were fine without them.” Teachers noted that the most helpful aspect of having instructors was to “help with behaviors” and “to get kids caught up after being absent.” Some teachers also noted that if instructors could not be in every class, they should be focused on the 9th and 10th grade classes since they felt students in these grades needed more individualized attention and guidance.

The quantitative data indicated greater support for instructors than the qualitative data did. The vast majority of teachers expressed some level of agreement that the allocation of SIG resources positively impacted the school (94.7%), but the magnitude of that agreement was less than it was for the other SIG expenditures with less than half of teachers (47.4%) strongly agreeing that instructors positive impacted the school.

Other Resources

A new and sustainable computer lab was a recent addition to the school. This was identified as a “huge change” that has allowed more “cross curricular activities, writing, journaling, math,” as well as “allowing all classes to use computer technology to enhance every class.” A budget for art supplies was also identified as having a tremendous impact on what students can do in art classes.

Student Learning

The extent to which schools improve student learning is the ultimate litmus test by which the impact of most school improvement initiates initiatives is are assessed. With that in mind, teachers were asked on the 2014 survey to express their level of agreement that SIG initiatives improved student learning. As displayed in the following graph, 100% of teachers expressed some level of agreement that SIG initiatives improved student learning. In fact, the largest percentage of teachers strongly agreed (50%) and 89.5% strongly agreed or agreed.

19 2014 Mid-Year SIG Teacher Survey

Collectively, the data collected for this evaluation provide clear evidence that SIG resources were allocated in strategic ways and the initiatives made possible by SIG resources resulted in positive outcomes for Horizonte students, families and teachers. This perspective was communicated consistently by teachers, academic coaches, counselors and social workers, and school and central office administrators. Although families and students were not asked to directly attribute indicators of school effectiveness to SIG initiatives, their input regarding the outcomes targeted by SIG was highly favorable as well.

Who/What will Horizonte be post-SIG?

While this question of who/what Horizonte will be after SIG is not necessarily answerable at this point, it is important to consider what lessons have been learned over the past three years. We talked with the various Horizonte constituents about what visions and/or concerns they had for “life after SIG,” and we share some of their insights below. Overall, most people with whom we spoke agreed that the focus on academic rigor, the commitment to student success, and the acceptance and validation of students and families were qualities of Horizonte that were “here to stay.” Although most people expressed a sincere desire for these elements of Horizonte’s institutional identity to be sustained, many expressed concern that they might be difficult to maintain without some of the resources that SIG brought to the school. As one leader summarized, Horizonte will need to figure out how to sustain “teachers’ beliefs that they can impact students” and provide them with the necessary resources to do so. Importantly, conversations began during the past few years about how to best meet the needs of students, how to use data to guide those decisions, and how to implement effective instruction and rigorous curriculum. All of those conversations “seem more natural now and it isn’t as threatening; Horizonte is at a point where they are comfortable talking about that.”

Much of the concern, however, lies in how to sustain those conversations and the collaborative work if time is not structured and coaches are not available to facilitate them. As one teacher articulated, “This whole thing has been about collaboration and if we don’t have time for collaboration, how are we going to do what we do?” When reflecting on resources, education stakeholders tend to focus on money and materials. However, it is

20 critical to think of time as a scarce resource in schools. That said, it most often requires money—funds like those provided via the SIG—to provide educators with the capital to best serve students. The evidence collected for this evaluation indicate that academic coaches, collaborative time in PLC’s for teachers, the ability to adapt quickly based on site data, and an additional counselor to integrate off campus sites and coordinate support services for students collectively enhance student learning. Sustainability is always a challenge when large scale grant funds run out, especially in a state such as Utah that spends less per pupil than all other states. That said, maintaining the most critical supports that positively impacted the school in general and student learning more specifically should be the goal.

In addition to these considerations for Horizonte, school and district leaders also need to continue to think about who/what Horizonte is in relation to the other high schools and the district as a whole. As one leader explained, “Horizonte will need to articulate what they are and what they are not.” Similarly, school leaders explained that “kids are successful here because of the systems and supports we have…the kids have not changed so hopefully other schools can see that and look internally too.” This is the sort of systemic change throughout the district that School Improvement Grants are intended to initiate.