RTI Made EZ

Evaluation of “RTI Made EZ” at Paulding County High School

Evaluation Report of the Programs Use and Effectiveness at Paulding County High School

Prepared by Jeff Wallace

Prepared for Chris Nichols

October 24th, 2011

______Page 1 Executive Summary

In the fall of 2011 I, Jeff Wallace evaluated “RTI Made EZ,” a program that was in use in the Paulding County School District. The program was relatively new to the county and for the purpose of the evaluation the focus was on its use and implementation at Paulding County High School.

“RTI Made Easy” was designed to be a more efficient way for Math and English

Teachers to implement the federally mandated “Response to Intervention” program. It can be used by those teachers to spot struggling students at an early stage, and hopefully implement alternative measures that allow the student(s) to become more successful.

Since the “RTI” program generally involves underclassmen at the high school level, this evaluation focused on those teachers and administrators involved with freshmen and sophomore specifically in the areas of math and language arts.

The purpose of this evaluation was to determine the extent of positive impact the program had and if the targeted teachers at Paulding County High School had been properly trained. The extent for which the program was implemented was a focus as well.

The evaluation questions used to guide this evaluation are as follows.

1) Was the program positively impacting student success at PCHS?

2) How was the program diffused to teachers and administrators?

3) In what ways was the program used?

______Page 2 4) What support was available for teachers and administrators as it related to the

program?

5) What barriers were being faced with the implementation of this program?

The participants in this evaluation consisted of eleven classroom teachers and two school administrators. Each participant worked at Paulding County High School and each responded to a request to participate. Of the thirteen total participants, all were asked to complete a survey designed to collect qualitative data. Each survey was anonymous.

Additionally, five were chosen to participate in an interview process meant to collect more in-depth qualitative data.

The key findings taken from the surveyed teachers indicated that the results of the program were varied. A small majority of (54%) of respondents claimed that the program was meeting its goals while (46%) indicated otherwise. The surveys indicated that most of the respondents thought that more time needed to be devoted to training.

Proof of this was shown by (77%) of surveyed answering that not enough training was provided. The findings also indicated that the program was not being utilized with all RTI students. (76%) of respondents indicated that they used this program with less than half of their students. The reason presumably was that many students that had an active RTI file were not in need of assistance in the specific class of the teacher responding.

______Page 3 Upon analyzing the interview questions, it became clear that that those using the system had common positive and negative experiences. As a whole, those interviewed thought that in theory the program could be successful. Each stakeholder interviewed conceded that lack of training was a shortcoming, but also indicated that other issues also stood in the way of increased use of the program. Of these other issues, most interviewed claimed a lack of pressure by the school and county lead to less use on their part. When asked to explain her use or lack thereof, one teacher said “if they

(administration) don’t think it’s important then why would we”. That teacher went on to explain the various other programs that administration places a higher priority on.

Another common problem expressed was a lack of cohesion between the classroom teachers, the special education teachers, and the administrators. Those interviewed said that all work they had done with the program had been on an individual basis. The administrator who participated in the interview process suggested that the program was not technically a special education program and thus is generally left to the regular education teachers. Of the other four interviewed, each was a regular education teacher and they made no mention of working with any special education employees. They did however raise the question of administration involvement and what their exact role should be. One teacher stated “the general feeling is that we (math and english) are being asked to carry more of the burden than some of the other teachers and administrators”. This was a common theme among the teachers interviewed as the program only calls for certain teachers to use it.

______Page 4 The positive aspects of the program were described in a couple of different ways.

First, the overall feeling of those interviewed was that the program gave educators a quicker way to identify students in need of additional help. The ease of the software was the highlight to most and led one teacher to state that “anything that speeds my job up is better than the alternative”. This was a common theme among those in the classroom, as in years past they were left to do manual calculations and track down the progress of their RTI students.

Based on the data collection of the survey and interview participants the final report contains the following conclusion and recommendations:

1) Increased Teacher and Administrator training. Since the needs of various

stakeholders involved vary, offer specific training in place of a broad overview.

2) Better defined objectives and protocol for teachers asked to participate in the “RTI

Made EZ” process.

3) Increased interaction between regular education teachers and special education

teachers.

4) A more active role in the implementation process for on-site administrators.

______Page 5 “RTI” Made EZ” at Paulding County High School

Introduction

Paulding County High School is located in Dallas, Georgia which is approximately 30 minutes west of Atlanta. The school served 1350 students ranging from ninth to twelfth grades. It was a part of the Paulding School District which included 33 schools and had a total of 28,500 students. PCHS opened in 1969 and was the only school in the county until 1992. The school had 96 certified teachers and was receiving a state grant for low income and low achieving schools. Low achievement was not new in education, but for

Paulding County High School it had come swiftly, as they were a “Georgia School of

Excellence” in 2003. Changing demographics coupled with changing district attendance lines brought change to the school quickly. Just as all other schools were mandated,

PCHS, and the district as a whole had tried to use Response to Intervention (RTI) as a means of correcting the issues at hand. Interventions, formal and informal, had been common in schools for many years. According to Martinez, (2011)More recently

Response to Intervention (RTI) has been implemented to ensure that all students receive any needed instruction and interventions to achieve academic success. Because of this,

PCHS was working within RTI to find ways to better serve the students and the challenges they faced academically. RTI cannot be characterized by one educational program or curriculum, but rather a transformation in the way that systems, schools, and professionals operate (Sansosti, 2008). For this reason, the Paulding School district decided to go beyond the minimum requirements set forth by state and federal laws and purchased software that was meant to enhance the RTI process.

______Page 6 “RTI Made EZ” is a program that enables administrators and teachers alike to track students’ progress and needs in math and language arts through information collecting spread sheets. The county trained the RTI administrator at each school on the software, and it was the job of that administrator to train the teachers. The software used grades entered by teachers to alert the proper administrator and teachers that the student was falling behind or needed assistance in a certain area. Both administrators and teachers had access to the information and could use it to proceed in the best interest of the student. It was the hopes of the district that this approach would be superior to the “wait till failing” approach that was used in the past.

I, Jeff Wallace, was the evaluator. I was a US History teacher at PCHS where I was entering my 10th year. This evaluation was part of my program evaluation class for my

EDS in Instructional Technology. The client for this evaluation was PCHS assistant principal, Chris Nichols. Mr. Nichols wanted me to explore the overall use, understanding, and effectiveness of the “RTI Made EZ” program. It was his concern that though the software can be useful, if it is not understood or implemented effectively, then its purpose will not be fully served. The overall goal was to determine how this technology was being diffused and to what extent it was having a positive impact on students.

The basic format of this evaluation was in case study form. A case study program evaluation’s main purpose is to provide stakeholders and their audiences with an authoritative, in-depth, well-documented explication of the program (Stufflebeam, 2000).

To this end, there was an examination of what was being done within the district and the school in order to implement “RTI Made EZ” and accomplish stated goals. Given that

______Page 7 “almost all words in the English language that end in “ing” refer to process or means”

(Kaufman, 2006), this evaluation examined words such as using, accomplishing, failing, meeting, and implementing as they related to the program. This however should not and was not be the only focus of the evaluation. Because useful evaluations focus on ends and not just means (Kaufman, 2006), the evaluation also identified what was actually being accomplished by the program. In order to complete a well-rounded evaluation, it was imperative that each level of the program and each stakeholder within the program be included.

Purpose

The Paulding school district purchased the software “RTI Made EZ” in an attempt to better help those students who are deficient in certain areas of Math and English. The county as did Reschly (2009) believed the application of RTI principles significantly enhances the likelihood of resolving common academic and behavioral problems and improving system outcomes. If this is true, then the faster the areas of concern are identified, the faster the problem(s) can start being solved.

The purpose of this evaluation was to determine the overall effectiveness of the program as a whole. From a school standpoint, was information delivered in an easy to understand format and was proper time given to master the software? From a classroom standpoint, was information delivered in a way that teachers could see the benefit and ease of using the software for student achievement? From the students standpoint did the

______Page 8 information used by teachers and administrators actually increase achievement? What barriers were there, and how could those be eliminated? The ultimate goal of the evaluation was to streamline the process so that the purpose of the program can be better achieved.

Objectives

1) Explore the level of understanding that school administrators and teachers had for

“RTI Made EZ” and evaluate the effectiveness on student achievement.

2) Explore the barriers that stood in the way of the program becoming more

effective.

Evaluation Questions

1) Was the program positively impacting student success at PCHS?

2) How was the program diffused to teachers and administrators?

3) In what ways was the program currently in use?

4) What support was available for teachers and administrators as it related to the program?

5) What barriers were being faced with the implementation of the program?

Participants

The participants for this evaluation were PCHS teachers that were responsible for the tracking of students in Math and English classes. Teachers ranged from 25 -55 years of

______Page 9 age and consisted of eight females and five males. Teacher experience ranged from 5-

25+ years and two administrators were part of the participants. 62% of participants had an advanced degree.

Design and Procedures

At the end of the first nine weeks, Math and English teachers and two RTI administrators were asked to participate in a qualitative survey. The survey was delivered through e-mail, or taken online through Google docs. The survey helped determine the demographics as well as the rate of use that each participant had in his or her classroom.

After the survey, four teachers and one administrator were asked to participate in an interview process. The interview consisted of more detailed questions that couldn’t be answered in a multiple choice survey. Those asked to participate in the interview were those that had the most experience with the program.

Instruments

Evaluation District Grade Questions Survey Interview Reports from the GDOE 1) Impact on X x X

success. 2) Program X X

Diffusion 3) Program X

Use 4) Support X X

available 5) Barriers X

______Page 10 Faced

Survey- The survey was designed to collect qualitative data in various ways.

Stakeholders answered a series of questions that gave the demographics of those questioned. They then answered a series of multiple choice and yes/no questions that gave an indication as to their feelings about how the program was taught to them, their experience and the usefulness of the program. Additionally, they were asked about the support they were given, as well as the amount of time they devoted to the program on a weekly basis.

Interview- The interview was designed to receive more in-depth qualitative information. Teachers asked to participate in the interview process were asked questions of more depth. These included but were not limited to things such as, how has this program aided your students, how do you use this for increased student success, how have your students reacted to this program, and what barriers hinder this program from being more successful?

State Reports- Each year the State puts out the EOCT and graduation test scores for the entire county and school by school. These results were used to obtain quantitative data by comparing yearly test scores for students with disabilities.

Data Analysis

In an effort to determine the overall effectiveness of “RTI Made EZ”, the following steps were taken.

1) Analyzed and charted the results of individual surveys as well as interview responses.

2) Analyzed surveys as a whole and created a chart to indicate the averages as it relates to

______Page 11 the answers.

3) Analyzed the interviews as a whole and looked for patterns within the answers.

Documented these patterns within a chart.

4) Created a list based on the interview answers showing the barriers that teachers faced,

and a list of possible solutions to those problems.

5) Determined based on answers from interview and surveys whether there had been

sufficient training and continued professional learning in order to allow teachers to

be successful.

6) Used county test score results to determine if the program was having a positive effect

on the students of PCHS.

Summary of Key Findings

The Evaluation questions were each analyzed and used in the formation of the report.

Inferences were made about the questions through use of a survey as well as individual interviews

Evaluation Question 1

Was the program positively impacting student success at PCHS? This question was used to determine if students were actually gaining from the program and its intended purpose. In order to answer this question quantitatively, results from the 2010 Georgia

High School Graduation Test were compared to those of 2011. Since not all RTI students are in special education, and not all special education students have had RTI’s in the near past, results of both “Students with Disabilities” (SWD) and all students that took the test were examined. In examining the results of the Graduation test, it became clear that

______Page 12 students made marked improvement in both Math and English. Students with disabilities increased their test scores in Math from a 14.6% pass rate in 2010, to a 50% pas rate in

2011. SWD’s saw their English scores rise from a 51.1% pass rate in 2010, to 79.2% in

2011. Given the great strides made by Students with Disabilities, it came as no surprise that the pass rate of “all test takers” also increased. The pass rate for all math test takers in 2010 was 55.7%, while 2011 saw that rate climb to 85.1%. English test scores for all test takers also saw an increase, going from an 85.7% pass rate in 2010 to 93.7% in 2011.

Figure 1 High School Graduation results from 2010 and 2011.

100 93.7 85.1 85.7 85.71 79.2 71.43 55.7 57.14 50 51.1 42.86 28.57 14.6 14.29 0 SWD Math All Students S WD A ll Students Math English English SWD Math All Students SWD All Students Math E nglish English

For qualitative data, the survey results and interviews were dissected to find common themes about student success. 77% of stakeholders surveyed said they felt the program at the very least had “some” positive impact. During the interviews, most of the stakeholders affirmed the survey which suggested a positive impact on students. Though

______Page 13 the degree to which students were impacted can and was debated, most agreed that anything that streamlines detection of student needs can do nothing but help.

How was the program diffused to teachers and administrators? This question was used to identify how the teachers and administrators were taught the program and the various methods used in the diffusion. In total for the 2011 school year the teachers had received one official training session. Many of those surveyed were not satisfied with what seemed like an insufficient amount of time for a program that they were asked to use. Of the stakeholders surveyed, 54% said that they did not receive enough training with only 31% indicating that they felt they had been given enough. 15% of those asked to use the program had no training at all due to schedule conflicts or absences on the day of training. Though unofficial training was offered on an individual basis, most did not pursue. Only 8% of stakeholders followed the initial training by seeking out further training on their own.

When asked about the diffusion of the program in interviews, feelings were mixed.

Two of those interviewed, teacher 2 and administrator 1 felt that the program did not need intense training and that “all you have to do is play with it to figure it out”. This however was far from the majority opinion as the other three stakeholders (each teachers) interviewed felt differently. It was their opinion that more detail needed to be given as well as examples of real-life use.

Figure 2 Teachers feeling toward training time allotted.

______Page 14 15.38%

30.77% None

53.85%

Figure 3 Actual time in training

46.15%

4.33 30.77%

15.38% 7.69%

0 0 5-30 minutes 30-60 minutes 60+ minutes

In what ways was the program currently in use? This question was used to determine the ways and amount of time that teachers were utilizing the program. As the surveys began coming in, it became very evident that the program was being used to a different degree by most of the respondents. 62% of teachers indicated that they actively use the software for 15-30 minutes a week, while 38% indicated that they used the program little

______Page 15 to none. When completing the data it was odd that some would use the software more than others, but during the interview process reasons emerged.

Figure 4 Teachers Use of the program weekly

Those interviewed had differing opinions about the use of the program in their room with each agreeing that it is not a one-size fits all. One teacher summed it up by stating

“the use is dependent on your students”. She went on to say that it is just the luck of the draw as to which students you have and whether or not they have an RTI file. Even if they have a file they may be doing well, while another teacher has eight such students doing poorly in one class. This assessment of the situation was backed up by the survey which indicated that just because students had an open RTI file, it did not necessarily mean they were being aided through the program. Since the program is only designed to

______Page 16 expose students in need of assistance, those that do well will not be approached by their teacher for additional help.

Figure 5 % of RTI students being served by the program

50+% of RTI Students 0% of RTI 25-50% of RTI Students Students 7.69% 23.08% 15.38%

23.08% 30.77% 0-10% of RTI 10-25% of RTI Students Students

What support was available for teachers and administrators as it related to the program? This question was used to identify the help available for teachers that needed assistance with the program. Given the number of official training sessions for this program was one, it was thought that additional help would be required. When questioned through the survey, stakeholders had differing opinions on the amount and quality of help available for them. The majority of those asked 69% said that they felt only one person in the building fully understood the program. It was assumed at the time

______Page 17 and later affirmed in the interviews that the training facilitator was thought to be that person. 23% of respondents felt that there were multiple people that had full understanding, and 8% of respondents felt that no one had full understanding. Survey participants surprisingly had opposing views on the ease of finding help, even though

92% felt at least one or more person(s) in the building had full knowledge. 46% claimed that it was difficult to find help or get questions answered, while another 46% felt just the opposite, claiming it was easy. It can only be assumed that other factors were playing a role with those seeking help. Such factors could have been RTI administrator was out of office or attending to other duties.

During the interview process stakeholders agreed that in general it was easier to try and figure a problem out on their own as opposed to trying to track down answers. They also agreed that special education needed to play a larger role in aiding this service. In many cases as an administrator put it “special education is told to not get involved with

RTI’s and to let the regular education teachers handle it”. This topic is one that needs further evaluation as it came up repeatedly amongst those interviewed.

Figure 6 Ease of finding help for problems with the program

______Page 18 Impossible

Hard 6

Easy 6

Very Easy 1

0 4.33 8.67 13

What barriers were being faced with the implementation of the program? This question was used to determine what was keeping the program from being more successful and fully realizing its designed purpose. When those surveyed were asked what factors kept them from using the program more often, there were two responses that stood out. 46% of stakeholders said that there was a lack of pressure by the school administrators for teachers to make consistent use of the software. This lack of pressure it is assumed, leads to the perception of a lack of importance. 38% of stakeholders said that other classroom responsibilities just did not leave time to devote to the proper use of the program.

During the interview process two of the participating teachers also stated at least a small amount of resentment among the Math and English teachers as they were being

______Page 19 asked to take on more work while others subjects were not. Time and accountability were two other things discussed by most interviewed. One teacher stated that “there is only so much time in a period, so most focus on those things they are held accountable for”. It is assumed that the statement referred to the teacher observation checklist and the fact that

RTI Made EZ” is not on it.

Figure 7 Factors behind decreased use of the program

Other classroom responsabilities 38.46%

Lack of understanding of expectations 15.38%

Do not feel it helps students

No pressure to use it 46.15%

0 4.33 8.67 13

Conclusions and Recommendations

______Page 20 This evaluation report examined the understanding of the “RTI Made EZ” program by teachers and administrators at Paulding County High School and the barriers that kept it from becoming more successful. According to Howland & Wedman (2004), technology integration is a longitudinal process that requires ongoing commitment and staff development opportunities. Because of this, and regardless of any feelings of individual teachers within the evaluation, continued training and modeling will be needed. The teachers and administrators in this evaluation as a whole have displayed an emergent sense of understanding of the program and will only improve in the future.

The diffusion of this program to teachers was found to not be detailed enough and lacking in time. Fonseca (2011) believes it is not enough to talk about the possibilities of technology. Rather, it is essential to make them visible and to model them for all attendees. Most of the participants in this evaluation felt that they were being asked to use the program without being given adequate training. One session of covering the basic components is not enough for teachers to become proficient in any technology, much less one that is brand new to them. The lack of time devoted to training coupled with a lack of true leadership from an administrator standpoint made it difficult for teachers to attain the expectations and goals of the program.

The positive effect that this program had on students was backed up with both qualitative and quantitative data. State results on the English and Math Georgia High

School Graduation Test show great gains in both Math and English from school year

2010 to school year 2011. While it is impossible to determine if the improvements in the

GHSGT were a direct result of “RTI Made EZ”, the fact that the sub-group “Students

With Disabilities” made such great gains can be used as further indication of the

______Page 21 program’s success. Teachers and administrators affirmed the positive impact through surveys and interviews with the large majority indicating that they were seeing at the minimum some positive effects within their classes.

Ultimately the program’s success will hinge on further training and a better understanding by those that are asked to participate in the program. To this point the participating teachers see this as one of many tools that can be used for student success, but one that the school is not pressuring teachers to use. Given all the expectations placed on educators, teachers quickly set-up a mental priority checklist. At this time, “RTI Made

EZ” is down that list for most participants of this evaluation, and will only ascend if the school decides to put more emphasis on it.

Based on the data collection of the survey and interview participants the final report contains the following conclusion and recommendations:

1) Increased Teacher and Administrator training. Since the needs of various stakeholders involved vary, offer specific training in place of a broad overview.

2) Better defined objectives and protocol for teachers asked to participate in the RTI made easy process.

3) Increased interaction between regular education teachers and special education teachers.

4) A more active role in the implementation process for on-site administrators.

______Page 22

References

Martinez, R. (2011). Response to intervention: how is it practiced and perceived? International Journal of Special Education, 26(1), 44-52. Retrieved from http://www.internationaljournalofspecialeducation.com

Sansosti, F. J. (2008). Viewing response-to-intervention through an educational change paradigm: what can we learn? California School Psychologist, 13, 55-66. Retrieved from http://www.casponline.org

Reschly, D. J. (2009). Teacher preparation for response to intervention in middle And high schools. tq research & policy brief. National Comprehensive Center for Teacher Quality, 28. Retrieved from http://www.tqsource.org

Stufflebeam, D. L. (2000). Evaluation models viewpoints on educational and human services evaluation. (2 ed., p. 54). Norwell: Kluwer Academic Publishing.

Kaufman, R. (2006). Practice evaluation for educators. (p. 29-30). Thousand Oaks: Corwin Press. Retrieved from http://books.google.com/books/feeds/volumes?q=0-7619-3197-x

Howland, J., & Wedman, J. (2004). A process model for faculty development: individualizing technology learning. Journal of Technology and Teacher Education , 239-263.

Fonseca, D. (2011). EduCamp Colombia: Social networked learning for teacher training. The International Review Of Research In Open And Distance Learning, 12(3), 60-79. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/884

Appendix 1

______Page 23 Teacher Survey Program Evaluation Survey Thank you for taking the time to answer this short survey! * Required

1) Male or Female? *

Female

2) Age? *

3) Years teaching, including this one? *

4) Highest degree? *

Bachelors

Masters

Specialist

Doctorate

5) Subject you teach? *

English

Neither

6) How well would you say you have been trained on "RTI Made Ez"? *

0 - None

1 - Not enough

______Page 24 2 - Enough

3 - More than enough

7) How much time to you devote to "RTI Made EZ" on a weekly basis *

0- None

1- 0-15 minutes

2 - 15-30 minutes

3 - 30-60 minutes

4 - 60+ minutes

8) What percentage of your students with open RTI's do you utilize this software with? *

0 - none

1 - 0-10%

2 - 10-25%

3 - 25-50%

4 - 50%+

9) How easy is it to get help if you do not understand a part of ther software? *

0 - Impossible

1 - Hard

2 - Easy

3 - Very easy

10) How many people would you say really understand the program at your school? *

11) What keeps you from using this software more than you do? *

Other classroom responsabilities.

Don't understand exactly what they want me to do with it.

I dont feel it helps the students

______Page 25 No pressure to use it

12) In your opinion what impact has the program had on student success? *

0 - none

1 - little

2 - some

3 - a lot

13) How much time have you had to spend at "RTI Made EZ" training this year?

1- none

2- 0-30 mins

3- 30-60 mins

60+ mins

14) Where do you go for help if you have a question about the program?

Try to figure it out myself

counselor in the building

administrator in the building

County level official

16) On a scale of 0-5 with 5 being the best, how would you rate this program for helping students? *

17) On a scale of 0-5 with 5 being the easiest, how easy is this program to use? *

15) In your opinion is the program attaining its goals? *

18) On a scale from 0-5 with 5 being the most, how much change needs to occur with the program overall? *

19) On a scale from 0-5 with 5 being the most, how serious does your school take this program? *

20) On a scale from 0-5 with 5 being the most, what are the chances you would use this program if not made to? *

Submit

______Page 26 Appendix 2

Stakeholder Interview

1) How were you trained in “RTI Made EZ”?

______

2) What do you feel could have made the training experience better?

______

3) How do you use the program in the classroom?

______

4) What would lead you to use it more often in the classroom?

______

______Page 27 5) What barriers get in the way of this program being more successful in your opinion?

______

6) How do you feel the program has impacted your students’ success?

______

7) What is the sense of teachers as to the importance of this program?

______

8) What could be done from the counties standpoint to improve the program?

______

9) What could be done from the schools standpoint to improve the program?

______

______Page 28 ______

______

10) What could be done from the students’ standpoint to improve the program?

______

11) What resources are at your disposal if you have questions or concerns about the program?

______

12) What is the logical next step(s) for this program in order to continue student success?

______

______Page 29