MIAMI UNIVERSITY the Graduate School

MIAMI UNIVERSITY The Graduate School

Certificate for Approving the Dissertation

We hereby approve the Dissertation

Laurie Ann Banks

Candidate for the Degree

DOCTOR OF PHILOSOPHY

______Andrew Saultz, Director

______Thomas Poetter, Reader

______Kathleen Knight Abowitz, Reader

______Molly Moorhead, Reader

______William Boone, Graduate School Representative

ABSTRACT

UNDERSTANDING IMPLEMENTATION, STUDENT OUTCOMES, AND EDUCATIONAL LEADERSHIP RELATED TO OHIO’S THIRD GRADE READING GUARANTEE

Laurie A. Banks

This study explores the outcomes of policy implementation from the perspective of the policy makers and the educators who are charged with implementation, specifically examining the decision-making process for district leaders during implementation. The study identifies the outcomes from the perspective of the policy-maker utilizing accountability measures, while examining the decision-making process by district leaders during implementation particularly focusing on doing what is “right” and what is “good” as defined by Strike (2008). The author provides an overview of Ohio’s Third Grade Reading Guarantee (TGRG). Ohio’s TGRG is one of many literacy policy’s in America that include retention as a consequence for students not able to demonstrate proficiency by the end of third grade. Quantitative and qualitative data provide a robust data set to inform the scholarship around policy and implementation from a dual perspective. The study reveals the statewide trends in reading proficiency in third grade did not change after implementation, only after a new type of assessment was administered during SY 15-16, resulting in a drastic decline in proficiency as assessed through high-stakes assessment. The K3 literacy measure, an accountability measure for schools and districts tied to Ohio’s Third Grade Reading Guarantee, had a significant negative correlation to student demographics across all three years of initial implementation. This study found one cohort of students placed on Reading Improvement and Monitoring Plans (RIMPs) in a local education agency (LEA), after being found not on track in third grade, and then promoted to fourth grade, were assessed as fourth graders and found still to be off track as assessed by the fall diagnostic. Interviews were conducted with six educational leaders from an LEA. Those interviewed were asked to reflect on the implementation of Ohio’s Third Grade Reading Guarantee and the decisions they tackled during implementation to ensure they acted as functionaries, focusing on what is “right” and “good” (Strike, 2008). The data set was evaluated utilizing Strike’s (2008) work on ethical leadership and decision making. The author concludes the study with policy recommendations and considerations for those in K-12 Leadership.

UNDERSTANDING IMPLEMENTATION, STUDENT OUTCOMES, AND EDUCATIONAL LEADERSHIP RELATED TO OHIO’S THIRD GRADE READING GUARANTEE

A DISSERTATION

Presented to the Faculty of

Miami University in partial

fulfillment of the requirements

for the degree of

Doctor of Philosophy

Department of Educational Leadership

Laurie A. Banks

The Graduate School Miami University Oxford, Ohio

2018

Dissertation Director: Andrew Saultz, Ph.D.

Laurie A. Banks

2018

TABLE OF CONTENTS CHAPTER 1: WHY READING PROFICIENCY BY GRADE 3? 1

INTRODUCTION 1 POLICY DEVELOPMENT 2 OHIO’S THIRD GRADE READING GUARANTEE 4 TEACHER CREDENTIAL 5 LITERACY ACCOUNTABILITY: WHAT’S YOUR GRADE? 6 THE DIAGNOSTIC 7 READING IMPROVEMENT AND MONITORING PLAN: THE RIMP 8 PROFICIENT IN READING? 9 THE CONSEQUENCE: RETENTION 9 EXEMPTIONS 10 CHAPTER 2: LITERATURE REVIEW 11

EARLY LITERACY 11 RETENTION 13 RETENTION AND LITERACY POLICY 16 ETHICAL DECISION MAKING BY SCHOOL LEADERS 19 CHAPTER 3: METHODS 24

RESEARCH QUESTIONS AND DATA SOURCES 25 WHAT ARE THE STATEWIDE TRENDS IN READING PROFICIENCY IN THIRD GRADE IN YEARS BEFORE AND AFTER IMPLEMENTATION OF THIRD GRADE READING GUARANTEE? 25 WHAT IS THE RELATIONSHIP BETWEEN STUDENT DEMOGRAPHICS AND A SCHOOL’S PERFORMANCE ON THE K-3 LITERACY MEASURE? 29 WHAT IS THE RELATIONSHIP BETWEEN THE 3RD GRADE AND 4TH GRADE READING PROFICIENCY AS MEASURED BY THE DIAGNOSTIC MEASURE FOR STUDENTS WHO RECEIVED READING INTERVENTION THROUGH A READING IMPROVEMENT AND MONITORING PLAN DURING THEIR 3RD GRADE YEAR OF SCHOOLING? 31 HOW DID ONE SCHOOL LEA LEADERSHIP TEAM IMPLEMENT THE OHIO’S THIRD GRADE READING GUARANTEE? 32 LIMITATION 32 CHAPTER 4: RESULTS 34

WHAT ARE THE STATEWIDE PROFICIENCY AND RETENTION TRENDS? 34 IS THERE A RELATIONSHIP BETWEEN STUDENT DEMOGRAPHICS AND A DISTRICT’S PERFORMANCE ON THE K3 LITERACY MEASURE? 37 WHAT IS THE RELATIONSHIP BETWEEN 3RD GRADE AND 4TH GRADE READING FOR STUDENTS ON A READING IMPROVEMENT AND MONITORING PLAN IN ONE LEA? 41 HOW DID ONE LEA APPROACH THE IMPLEMENTATION OF OHIO’S THIRD GRADE READING GUARANTEE? 43 POLICY AND COMPLIANCE 44 RESOURCES 45 RETENTION 45 LEADERSHIP AND DECISIONS 46 CHAPTER 5: DISCUSSION 49

PERSONAL NARRATIVE 49 IMPLICATIONS FOR POLICY MAKERS AND EDUCATIONAL LEADERS 52 IMPLICATIONS FOR STUDENTS 53 POLITICAL GAMING 54 THE AIM 55 POLICY RECOMMENDATIONS 56 REMOVE RETENTION 56 ALL LEARNERS 57 iii

TARGETED APPLICATION 57 CLOSING THOUGHTS 57 APPENDIX A: READING COMPETENCIES 59 APPENDIX B: 2014-2015 THIRD GRADE READING GUARANTEE PROMOTION PERCENTAGE 67 APPENDIX C: INTERVIEW PROTOCOL 71 APPENDIX D: TGRG AND STUDENTS WITH DISABILITIES 72 REFERENCES 73

LIST OF TABLES Table 1.0 Research-Based Reading Instruction Programs for 2014-2015 Table 2.0 Approved Vendor Assessments Initial Implementation Table 3.0 Exemptions from retention under TGRG Table 4.0 Diagnostics Utilized to Determine On-Track or Not-On-Track Table 5.0 Performance Level Descriptors Grade 3 Reading Achievement Test Table 5.1 Grade 3 English Language Arts AIR Performance Descriptors Table 5.3 Third Grade Reading Guarantee Level Scale Score Ranges Table 6.0 Statewide results of third grade reading guarantee Table 7.0 Statewide correlation between district student demographics and NR Table 8.0 Correlation between student demographics and K3 Literacy Grade Table 9.0 Statewide correlation between student demographics, exemptions, & retention Table 10.0 LEA demographic information for third grade cohort Table 11.0 LEA correlation for diagnostic and high-stake assessment for students on Table 12.0 LEA promotion criteria met for 3rd grade cohort Table 13.1 Frequency of codes from six interviews Table 13.2 Frequency of meetings during initial implementation years Table 13.3 Frequency of codes from district literacy team notes

LIST OF FIGURES

Figure 1: Strike’s Ethical Decision Making Figure 2: Visual Model of Study Figure 3: K3 Literacy Calculation Figure 4: Ohio's third grade proficiency trends Figure 5: Statewide district K3 literacy grades

Chapter 1: Why reading proficiency by grade 3?

Introduction Over the course of the past decade and half, education policy, particularly at the state level, has focused on literacy with the intent of re-emphasizing efforts on students developing proficient literacy skills by third grade. Ohio followed this trend with the implementation of the Third Grade Reading Guarantee. This policy, fully implemented in the 2013-2014 school year, aimed to increase the reading proficiency of kindergarten through third grade students in Ohio. The policy was unfunded and support to local education agencies (LEAs) for implementation was minimal.

Ohio’s Third Grade Reading Guarantee (TGRG), Ohio Senate Bill 21, was signed into law in June 2012. TGRG is charged with identifying students kindergarten through grade 3 who are reading below grade level to target interventions to increase students’ reading skills to a proficient level set by the Ohio Department of Education (ODE) by the end of third grade (ODE, 2015). The law provides a framework for LEAs to focus and track literacy skills beginning in kindergarten through third grade for students. Ohio’s policymakers utilize the policy developed in Florida, and over 30 other states, working to increase literacy skills in students by third grade (Workman, 2014). The rationale for this policy is evidence of a predictive relationship between literacy competency by the end of third grade and future success or failure in school (Hernandez, 2011; Feister & Smith, 2010). In addition, literacy competency by third grade increases the likelihood of a student being successful at achieving college and career readiness as compared to their non-proficient peers (ODE, 2016).

This study investigates the results of the initial implementation years across local education agencies (LEAs) in Ohio. It provides an understanding of the implementation process through the work of one LEA leadership team seeking to identify decisions and challenges experienced during those initial years. The following questions guide the investigation:

❏ What are the statewide trends in reading proficiency in third grade in years before and after implementation of third grade reading guarantee? ❏ What is the relationship between student demographics and a LEA’s performance on the K3 literacy measure? ❏ What is the relationship between the 3rd grade and 4th grade reading proficiency as measured by the diagnostic measure for students who received reading intervention through a Reading Improvement and Monitoring Plan during their 3rd grade year of schooling in one LEA? ❏ How did one LEA leadership team implement the Ohio’s Third Grade Reading Guarantee?

Positionality influences research, or truth seeking, in approach, questions, and perspective in the construction of knowledge from data analysis. As a full-time 1

administrator in K-12 education serving on a Literacy Leadership Team during the SY 14-15 and SY 15-16, my intent of research on TGRG is to illuminate the implications of this policy and the outcomes, specifically student literacy. In addition, the aim is to give a practitioner voice to the implementation of this policy seeking to identify what practices districts had to employ to align with policy requirements and what decisions district administrators were faced with through implementation. My own experience with this policy in the implementation and monitoring clearly fuels my desire to engage in this research. I provide a reflective narrative of my own experience with this policy from an administrative perspective in the discussion section of this paper. Literacy is a civil right necessary for accessing opportunities for students now and as adults in the future through reading and writing. These skills are a primary mode of communication in all areas of life, professional and personal. They allow individuals to access information regarding the world around them, as well as providing a way for individuals to articulate their knowledge and thoughts. Literacy has significant value extending to skill valued by the larger community as evident in research (NIH, 2000) policy (Workman, 2014) and media (Gilchrist, 2015; Fields, 2012; O’Donnell, 2014). The intent of the inquiry is to create knowledge on the outcomes of literacy policy, intended and unintended, with the aim of informing policy, practice, and future research aligning with Lasswell’s designation of “policy science.” Research and intent paired have the potential to create power and influence. The aim is to go beyond justification of policy (Kaplan, 2009) and move toward an informative extension of what is currently understood through the measured variables’ relationships and the practitioner context of capability. This chapter focuses on the background knowledge of TGRG and how it became policy in Ohio. The second chapter reviews the existing literature related to literacy policies, literacy development, retention, and ethical decision making. The third chapter details the methodology of the study. The final two chapters review the findings and provide insight through discussion of implications and recommendations.

Policy Development TGRG, SB 21, was developed by policy makers in Ohio and approved into law by Republican Governor John Kasich and lawmakers in the spring of 2012 (Fields, 2012). Governor Kasich saw this as a critical component to his education agenda - ensuring that all students read at a proficient level and holding LEAs responsible for achieving this standard (2012). The policy is built on the premise that students are learning to read through third grade, but beyond third grade they are reading to learn and therefore must be proficient in reading to move beyond grade 3. The unspoken secondary goal was to end “social promotion” of students who were not able to demonstrate proficient reading skills. This law was part of a larger education package that included other education issues such as vouchers, teacher evaluation, and online education.

This is not Ohio’s first attempt at developing policy around a minimum competency level. In 1997, under the leadership of Governor George Voinovich, the

"fourth grade reading guarantee" became law through Senate Bill 55 of the 122nd General Assembly requiring school LEAs to retain students in fourth grade who failed the state's fourth grade reading proficiency test, beginning with students entering fourth grade after July 1, 2001 (Partin, 2011). The law included two criteria for exemptions: (1) disabled students who are excused from the test by their individualized education programs (IEPs) and (2) students who fail the test but whose reading teachers and principals nonetheless judge them to be academically prepared for fifth grade (Chester, 2001). It also required assessing each student at the end of first, second, and third grade to identify those reading below their grade level. All students identified to be below grade level were to be provided intervention services as needed during first, second, and third grade, including intensive summer reading programs following third grade for students who needed them. Parent groups and teacher unions acted against the policy, which at the time was significant due to strong local control and extensive local funding for Ohio’s schools. This law was ruled an unfunded mandate in 2001 by the Ohio Supreme Court (The Thomas B. Fordham Institute, 2011). The law was amended and “watered-down” to minimum requirements and the removal of retention based on performance on fourth grade state reading assessment. A Memorandum sent from Mitchell D. Chester (2001), Assistant Superintendent of Ohio at that time, identified the new expanded options for students who score below the “proficient” level, district policy for intervention services, additional opportunities to take the Fourth-grade proficiency test in reading, and reporting the highest score received through EMIS, as well as identify the change to new assessment, Achievement Test, in 2003-2004. What changed in Ohio to create an acceptance of such a policy? Research indicating the significance of reading by third grade on outcomes for students in the future led to the focus on this instrumental skill and the timeline of its development. The growing trend of holding schools accountable for student growth and learning can be attributed with trend of the development of third grade reading policies across the United States. Policy makers, educators, and community members agree that literacy is a fundamental skill all schools should be focused on developing in their students. The trend data in reading proficiency, paired with the identification of literacy as an essential skill, creates a sense of urgency requiring response. These two factors, the discourse identifying our education system as failing and in need of a solution, create a demand for policies to address the failure.

Governor Kasich and policy makers in Ohio utilized a report from Professor Donald Hernandez, City University of New York’s Hunter College, which looked at a national database of nearly 4,000 students born between 1979 and 1989. The report found that students who do not read proficiently by third grade are four times more likely to drop out of school by age 19 than students who can read proficiently (Bloom, 2012). However, even Hernandez himself cautioned policymakers focused on aiming to have students pass a test and not on the development and continuous improvement on reading skills (2012). Hernandez also found that students living in poverty are three times more likely to drop out of school if not reading well by third grade (2012). Governors Kasich and Voinovich weren’t the first to push for such a policy. Today a total of 32 states and the District of Columbia have some form of policy focusing on

reading proficiency by third grade. Of these 32 states, 14 states require retention of students based on performance on reading assessment (Rose, 2012).

Another voice speaking out in the conversation is the non-partisan Education Commission of the States (ECS), established by James Conant, asserting retention of students doesn’t equate to improvement in reading skills for students (Bloom 2012). Success experienced by states and schools having implemented similar policies to TGRG have focused on small class sizes and intensive, personalized instruction. The ECS suggests that the urgency created by these policies can lead to identification and intervention without the need for retention (2012).

The voices missing from the conversation and development of the policy are educators. This policy requires LEAs to make significant changes with no funding and no support from the policymakers. Ohio educators are experiencing high rates of change, as are other states, with new generation assessments, adoption of common core, and new graduation requirements to name a few. The accountability for results has escalated, shifting the K-12 education system that once allowed for more local control focused on specific needs of their learning community and celebrating growth measures determined by the learning community, to a one size curriculum measured by one size assessment.

Ohio’s Third Grade Reading Guarantee TGRG became law in 2012 with a focus on reading competency leading to academic success and increased opportunity for students, ensuring that every struggling reader gets the support needed to learn and achieve. The reading competencies (see Appendix A) identified by the legislation as fundamental to instruction and instructional competencies for educators include: phonological awareness, phonics, fluency, vocabulary, comprehension, differentiated instruction, research-based instructional practices, appropriate instructional materials, and assessment.

The policy has several elements that required LEAs to evaluate practices and adapt to maintain compliance with the regulations. These elements include teacher qualifications, diagnostic assessments, Reading Improvement Monitoring Plans (RIMPs), minimum competency to advance to fourth grade (retention), exemptions and the K3 Literacy Component, an accountability measure. TGRG was mandated with no funding allocated for implementation for LEAs, leaving them to reallocate existing funds and resources to maintain compliance. Ohio completed the third year of implementation in the 2015-2016 school year, but its second year of full implementation with retention.

Governor John Kasich stated, “If you can’t read you might as well forget it” (Fields, 2012). He declared that social promotion, or allowing a student to advance to the next grade despite academic challenges maintaining a student’s social connection to peers, only allows students to get “lapped” and that the state of Ohio will not delay in retaining in third grade if students can’t read (2012). The discussion of the potential harm to students’ social emotional well-being when retained was absent from the bold claims made by the governor. The policy requires students not meeting promotion

requirements to be retained in third grade. The following pages will inspect Third Grade Reading Guarantee as a policy detailing the various components, including reporting requirements and accountability measures.

Teacher Credential TGRG requires that students entering third grade who have been retained or on a Reading Improvement Monitoring Plan must have a teacher with a minimum of one year of teaching experience. In addition, the law requires the teacher to have one of the following qualifications: hold a reading endorsement; master’s degree in reading; “above expected” rating for value added in reading two consecutive years; pass a praxis exam on reading; “most effective” rating in reading two consecutive years based on approved assessments of student growth; alternative credential approved by the department or holds an educator license for teaching grade preK-3 on or after July 1, 2017. These credentials serve as indicators that the teacher has enough experience to help improve student literacy.

The credential standards established by TGRG have had and continue to have great implications for LEAs and their teachers. Initially, LEA leadership teams had to evaluate the license and experience of staff in the fall of 2012 and determine who was qualified to provide the intense reading instruction to intervene with struggling readers. LEAs that have the qualified teachers were good to move forward with planning and enacting the other guidelines of the policy. LEAs that had staff that did not meet the guidelines submitted a “staffing plan” to ODE identifying how they planned to address credentialing.

Teachers who had previously been identified as “highly qualified” were now at risk for being moved to a different teaching assignment if they did not take steps to meet the new credential guidelines. The cost of obtaining the reading endorsement identified to meet the credential guidelines, along with the praxis exam, was not funded through the mandate. This cost was either absorbed by the teachers themselves or by the LEAs who financed the opportunity for employees. The requirements placed on educators has not resulted in an increase in compensation.

The Third grade Reading Guarantee mandates the Ohio Department of Education to provide a list of approved Research-Based Reading Instruction Programs for school LEAs to utilize to prepare educators to address reading deficiencies ensuring alignment with adopted reading competencies (see Appendix A). Table 1 displays the Research- based Reading Instruction Programs approved by ODE for credentialing. There is no funding to finance the training of teachers utilizing these programs. ODE allowed extensions for compliance with teacher credentialing through the 2015-2016 school year if the LEA provides an updated staffing plan each spring.

Table 1.0: Research-Based Reading Instruction Programs for 2014-2015 Intervention AFTCRIP (American Federation of Teachers Comprehensive Reading Instruction Program) CLLIP (Collaborative Language and Literacy Instruction Program) IMSE Comprehensive Orton-Gillingham Training (30-Hour) Wilson Langue Training TESOL Endorsement (For ELL Instruction Only) Reading Recovery IMLSEC (International Multisensory Structured Language Education Council) Literacy Collaborative LLI (Leveled Literacy Intervention System) National Board

The credential guidelines have consequences beyond K-12 and into colleges and universities preparing the next generation of teachers. TGRG has led to a new required rigorous exam on reading instruction for all teacher candidates for grade levels PK-3 or 4-9. This policy, and others like it, are prompting universities and colleges to evaluate their teacher preparation programs and determine if their programs are effective and setting future teachers up for success, which will be measured by student outcomes and impact not only teacher evaluation, but LEA report cards, too.

Literacy Accountability: What’s your grade? In this era of accountability, this policy falls in line promising to measure the outcomes of TGRG as a component of the State Report Card for LEAs in Ohio. The Literacy Component measures literacy improvement on school LEAs’ State Report Cards as of 2014 and gives each LEA a letter grade. The measure analyzes the data reported by LEAs on the fall diagnostic and the ELA (English Language Arts) fall assessment, looking for evidence that LEAs are moving students from not-on-track to on-track in reading proficiency. The measure also accounts for student performance from year to year with LEAs receiving credit for students who were previously not-on-track who move to on-track grade to grade. The final component of the measure is accounting for students who were not on a Reading Improvement Monitoring Plan (RIMP) or removed from one and unable to meet the minimum level identified by the state. The proficient level of performance on the third grade ELA assessment was a score of 400, but the level allowed for minimum competency for TGRG requirements was 392 for the 2013-2014 year, and 394 for the 2014-2015 school year. The policy fails to provide a rationale regarding the minimum score competency designated, which are below the proficient cut score. The state average will represent a letter grade of “C” on the A-F scale and will change with yearly averages in the state. LEA leadership teams are working to refine the process outlined by the policy while maintaining compliance and ensuring students are reading at proficient levels resulting in a high mark, which will communicate to the community that the school district is successful teaching their children to read.

The Diagnostic Implementation of the policy begins with administering a reading diagnostic assessment. Ohio school LEAs must select an approved reading screener or diagnostic to administer to all students in kindergarten through third grade by September 30th each year to determine if the student is on-track based on the previous year’s end-of-year standard of the assessment tool. The policy also requires a diagnostic to be completed in math for grades 1 and 2, and writing for grades 1-3, but the law does not have requirements with the use of this data beyond reporting to ODE the results. All diagnostics must be completed for any student who moves into the LEA within a period of 30 days.

The purpose of the reading diagnostic is to identify if the student has the reading skills that align with meeting the academic standards. ODE does not provide funding to LEAs for the reading diagnostic assessments. The ODE provides screener assessments and full measure assessments they designed to be utilized if LEAs choose. ODE also developed a list of vendor assessments that can be utilized for the diagnostic requirement of TGRG. Table 2.0 displays the approved vendor assessments for SY 14.15, which are identified as comparable to the assessment provided by ODE. However, the 12 vendor assessments and the assessment provided by ODE are administered in different formats (on-line, paper-pencil), measuring different reading skill sets (fluency, comprehension, decoding, and vocabulary), and varying in length of time. LEAs are responsible for reporting the number of students assessed to be on-track and not-on-track. The cut scores for identifying these designations are at the LEA’s discretion. The not-on-track designation leads to the next component of implementation, which is developing Reading Improvement Monitoring Plans for students. The vendor list has changed each year, which has been problematic for LEAs working to develop consistent systems of assessment and intervention.

Table 2.0: Approved Vendor Assessments Initial Implementation Vendor Assessment Form Amplify Wireless Generation mClass: DIBELS Next mClass: DIBELS Next Cambium DIBELS Next DIBELS Next DIBELS Next Cost Curriculum & Associates iReady Diagnostic iReady DX Scantron Performance Series Reading Performance Series Reading Pearson Clinical Assessment Woodcock Reading Mastery WRMT-III Form C Tests NWEA MAP for Reading Assessments MAP for Reading Assessments Pearson Clinical Assessment AIMSWEB AIMSWEB

Pearson School Developmental Reading Developmental Reading Assessment Assessment Pearson School Group Reading Assessment and Group Reading Assessment Diagnostic Evaluation (GRADE) Renaissance Learning STAR Reading Enterprise STAR Reading Renaissance Learning STAR Early Literacy Enterprise STAR Early Literacy Scholastic, Inc. Scholastic Reading Inventory Scholastic Reading Inventory (SRI) (SRI) (Retrieved at http://education.ohio.gov/Topics/Early-Learning/Third grade-Reading-Guarantee/Third grade-Reading-Guarantee-LEA- Resources on 11/30/14)

Reading Improvement and Monitoring Plan: The RIMP The law requires parent notification for any student identified as not-on-track by the diagnostic assessment to explain the process and the possible consequences. The teachers and other building personnel develop a Reading Improvement Monitoring Plan, or RIMP, for each student identified as not-on-track. A RIMP is required to be developed for any student who does not meet the cut score identified by the LEA on the diagnostic assessment administered for on track and off track, but can be written for any and all students. The law allows for the flexibility to determine what the cut off is for on- track at the LEA level, which has implications on the K3 Literacy measure on the LEA’s Report card, despite the lack of consistency and standardization.

The RIMPs are written at the building level and must address the following elements: reading deficiency, intervention, progress monitoring, core curriculum, parent involvement, and language acknowledging. If the student does not meet minimum level of competency, the student will be retained, or the new “positive” language being used is level of competency needed to advance to fourth grade. The plans must be completed within 60 days of the completion of the diagnostic assessment. The intervention is the other element, besides the diagnostic results, that is reported to ODE. The intervention should be specific and tailored to the deficiency identified. The policy, as mentioned earlier, requires all students in third grade who have a RIMP to have a teacher who is highly qualified in reading instruction as determined by state standards.

Students must remain on a RIMP until they either test on-track on the fall reading diagnostic or score a 400 (proficient level) on the third grade ELA assessment. The Ohio Achievement Assessment (OAA) was used during the initial two years of full

implementation, school year 2013-2014 and 2014-2015. In 2015-2016 Ohio moved toward the new English Language Arts (ELA) AIR assessment, which has a reading sub- score utilized to determine promotion.

Proficient in Reading? LEAs administer the state assessment in the fall and then again in the spring for all third grade students. This provides two opportunities to demonstrate proficient level (400), or the state dictated cut score for competency. For the 2014-2015 school year, the cut score was 394. However, if students did not pass either of these assessments, they were provided another opportunity to take the state assessment in the summer or pass an alternative assessment.

The alternative assessments are defined by ODE as summative reading assessments that align to the end of year third grade standards. ODE currently recognizes three assessments for this purpose: Iowa Assessments Form F Level 9, Northwest Evaluation Association (NWEA)’s Measurement of Academic Progress (MAP) and Terra Nova 3. The use of these assessments is optional for LEAs and is unfunded. They can be administered twice during the school year and once during the summer. This provides each third grade student up to six opportunities to meet the minimum competency level of reading dictated by ODE.

Summer programming that is intensive, explicit, and systematic is recommended for students who are at risk for retention based on their performance on the first two state assessments. Summer programming is not funded through the mandate, but the financing of the instruction can be pulled from other pots of money, including Title I funding. However, Title I funds can only be used to purchase interventions, not on specific mandates or outside providers. During the summer, students have an opportunity to take the state assessment once, and if the LEA chooses, they will be able to take one of the approved ODE assessments to show minimum competency as required to advance to fourth grade.

The Consequence: Retention Third grade students who do not demonstrate the minimum level of reading competency on the assessments are retained. However, the policy allows for LEAs to provide split instruction. The 3rd/4th split allows student retained to receive core instruction, except reading, at a fourth-grade level and receive intensive reading instruction at a minimum of 90 minutes a day on a third grade level. In addition to the 90 minute of reading instruction daily, school LEAs must provide the option to parents to identify an outside provider and fund those external services. Students can continue to take an alternative assessment to meet the competency level determined sufficient for being promoted to 4th grade. However, the state puts the mid-year promotion policy on the LEAs to develop. Another important consideration is what students are missing from the core instruction during these intervention periods, as the implications for students socially and emotionally isolated from their peers. While no one would argue against the

importance of proficient reading skills, this should not be done without consideration for the methods established to accomplish the goal.

Exemptions There are exemptions to the Third grade Reading Guarantee. The only students exempt from the reading diagnostic are students with intellectual disabilities, and this was a revision to the law for the 2014-2015 school year. The exemptions for the consequences of TGRG (see table 3.0) include students who are Limited English Proficient, special education students with language in their IEPS specifically exempting them from retention, and students who have already been retained and received intensive remediation for two years.

Table 3.0: Exemptions from retention under TGRG ▪ A student who is an English learner enrolled in U.S. schools for fewer than three full school years and with fewer than three years of instruction in an English as a second language program

▪ A student whose IEP specifically exempts the student from retention under the Third Grade Reading Guarantee (more information is available in the Students with Disabilities section of this manual)

▪ A student who demonstrates reading competency on an alternative assessment for reading approved by the Ohio Department of Education (see detailed guidance below)

▪ A student whose IEP shows that the student has received intensive remediation in reading for two years, and the student was previously retained in any of grades kindergarten through grade 3

▪ A student who has received intensive remediation in reading for two years and was previously retained in any of grades kindergarten through grade 3.**

**A student that is promoted under this exemption must continue to receive intensive reading instruction in grade 4. This instruction shall include an altered instructional day that includes specialized, diagnostic information and specific, research-based reading strategies for the student that have been successful in improving reading among low- performing readers (ODE, 2017)

Education policy at the state level has a significant impact on LEAs at the local level. Many times, the impact is an unintended consequence of which policy makers are unaware. In addition, many times the intent of the policy is aligned with the mission of public education, while the implementation is merely a distraction and burden on resources. In the following chapter, a review of existing literature on reading policy, literacy development, retention, and ethical decision making will be explored. This review provides the foundation for the methodology of this research.

Chapter 2: Literature Review

This chapter examines the existing literature around early literacy development, third grade reading policies, retention, and the ethical decision making for school leaders providing the context for this study. The literature reviewed on early literacy development, third grade reading policies, and retention provides foundational evidence to the trend of statewide third grade reading policy development and the outcomes related. The ethical decision making by school leaders reflects the increasing pressure to focus on students first, while still being held to stringent accountability measures and meeting the demands of other stakeholders, including school boards, staff, and community members. Looking at policy and implementation requires the examination of both the intent and the outcomes experienced, intended and unintended, to truly evaluate its effectiveness in meetings its aim.

Early Literacy Early literacy, for the purpose of this literature review, will include literacy development and research related to children birth through age 8. This age range is critical to understanding the importance of early literacy development prior to entrance into K-12 public education, which typically occurs at age 5, specifically the impact of the early learning experiences and how this influences young learners’ initial experiences in public education. The age range extends to age 8 as this is the typical age of a third grader, which is the age students transition from learning to read to reading to learn. Third grade reading proficiency is predictive of future success or failure in school as identified by previous research (Hernandez, 2011, Feister & Smith 2010).

In Early Warning! Why Reading by the End of Third Grade Matters, Feister (2010) provides disturbing evidence of the lack of literacy skills among 4th graders with 67% of them performing below grade level proficiency. Poverty is identified as the leading indicator of literacy skills, which are a leading indicator of future success, especially graduation from high school. The National Research Council asserts that “academic success, as defined by high school graduation, can be predicted with reasonable accuracy by knowing someone’s reading skill at the end of third grade. A person who is not at least a modestly skilled reader by that time is unlikely to graduate from high school” (p. 9). It is estimated to cost $260,000 in lost earnings, taxes and productivity for every student who does not graduate from high school (Feister, 2010). The report critiques state testing systems for determining reading proficiency levels, as these are subject to state’s definition of proficiency and vary from state to state. Research presented in this report also indicates a relationship between reading level and behavioral and social difficulties for students (Feister, 2010, p. 9).

Feister’s (2010) report, supported by the Annie E. Casey Foundation, states the “readiness gap” that impacts the development of early literacy skills begins at birth for children born with low birthweight, prematurely, with congenital health problems, or affected by prenatal exposure to toxic substances (p. 15). The gap extends as children experiencing these risk factors approach entering kindergarten. Additional factors

impacting this gap include limited access to resources, opportunities for developmentally appropriate experiences, and adequate health care, typically leading to a lack of readiness when entering school.

The Annie E. Casey Foundation (2010) offers areas of focus to continue our nation moving forward in developing a literate populace with ample opportunities for participation and leadership in our democracy. The first component is intervening prior to the first school experience. Students need to be ready to go to school and engage in learning opportunities. Issues that cause the readiness gap include low birth weight, prematurity, congenital health problems, and/or exposure to toxic substance. These gaps continue because of children’s lack of access to resources and opportunities for development in all domains. Rose and Schimke (2012) also speak to the critical time from birth to age five and the research that specifically shows the implications early intervention has on brain development, suggesting a need for a systematic and coordinated approach to early care and education.

Upon entering school there are other factors that can impact this gap, specifically chronic absences and the stress of living in poverty. The following recommendations for policy makers and educators were provided by Rose and Schimke (2012):

1. Develop a coherent system of early care and education that aligns, integrate, and coordinates what happens from birth through third grade so children are ready to take on learning tasks associate with fourth grade and beyond. 2. Encourage and enable parents, families, and caregivers to play their indispensable roles as co-producers of good outcomes for their children. 3. Prioritize, support, and invest in results-driven initiatives to transform low- performing schools into high-quality teaching and learning environments in which all children, including those from low-income families and high-poverty neighborhoods, are present, engaged, and educated to high standards. 4. Find, develop, and deploy practical and scalable solutions to two of the most significant contributors to the under-achievement of children from low-income families- chromic absence from school and summer learning loss.

Coulton, Fischer, and Kim (2015) investigated the pathway to proficiency from birth through 3rd grade in the Cleveland Metropolitan School District. Their research specifically investigated the impact of individual, family, and neighborhood characteristics, social service receipt, mobility, and school experiences on early literacy development. The research team utilized a cohort design to estimate the effects on kindergarten readiness, as measured by the standardized Kindergarten Readiness Assessment-Literacy (KRA-L) and 3rd grade reading proficiency as measured by the standardized Ohio Achievement Assessment. Their investigation revealed that males, Hispanics, children with low birth weight, low SES, mothers without a high school diploma, transiency, and ELL students are all more likely to have lower readiness scores on KRA-L (2015). The third grade reading proficiency performance was positively related to access to early childhood care services, while male, low-birth weight, African- American, English Language Learner, mothers without high school diplomas, low SES,

truancy, transiency, and disability all had a negative relationship on reading proficiency performance (2015).

The National Institute for Literacy (NIFL), The National Center for Family Literacy (NCFL), and cooperating agencies from the Partnership for Reading, addressed early literacy development and implications for early intervention for children ages birth to five in the National Early Literacy Panel (NELP) (National Center for Family Literacy, 2008). The NELP sought to identify interventions, parenting activities, and instructional practices that promote the development of children’s early literacy skills. The panel identified size precursor skills to literacy: alphabet knowledge, phonological awareness, rapid automatic naming, rapid automatic naming of objects or colors, writing or writing name, and phonological memory. They identified five categories of intervention effective in developing early literacy: code-focused intervention, shared- reading intervention, parent and home programs, preschool and kindergarten programs, and language-enhancement programs. The NELP noted the lack of quality research in early literacy and identifying a need for high-quality research to extend our knowledge of early literacy.

The literature clearly indicates the significance of early literacy skills. It identifies the need to ensure this begins at birth, prior to entering the public-school system at kindergarten. The research has clearly indicated factors that have substantial implications for the development of these literacy skills for our youngest learners. In addition, the work from NELP has clearly identified the precursor skills necessary. The literature clearly identifies the evidence necessary to inform policy and develop systems and/or services to prepare our young learners prior to entering the public-school system in kindergarten. However, this literature and the continued national discussion around the significance of early intervention has not led to the overwhelming support and overhaul education needs to see the potential of our early learners fully blossom.

Retention The review of the literature surrounding retention is vast, but requires thorough reading and consideration. Some of the research focuses on short-term academic outcomes for students who are retained, while other research focuses on the long-term academic outcomes for students. There is limited research investigating the impact of retention on short-term or long-term social emotional outcomes for students. The literature reviewed provides insight into existing evidence of impact of retention. The literature presented provides an overview of the different angles of investigation, but common themes emerge revealing negative outcomes for students.

As American schools evolved to graded classes during the 19th century, they began to practice retention modeling British schools, which had been utilizing retention to correct academic deficiencies as early as the 16th century (Rose & And Others, 1983). Retention, or the repetition of a grade level by a student as a result of unmet social or academic standards, was common practice until social scientists began to question the implications on students’ social emotional well-being in the 1930s. As a result, the rate of retentions began to slow and the idea of “social promotion” became more prevalent.

Social promotion refers to the practice of a student being promoted to the next grade level with his/her peers, without having met the minimum academic standards in order to maintain the social emotional well-being of the student. The debate regarding retention and social promotion has endured for decades and continues.

Reasons identified as cause to retention include academic deficiencies, social- emotional delays, low parental involvement, policy requirements, and skill deficiencies (Hoff, Peterson,, & Strawhun, 2014). Retention has been linked disproportionately to certain subgroups already identified to be at-risk, including English Language Learners (ELL), socio-economic status (SES), transient population, truancy, and discipline (Jimerson & Renshaw, 2012, Owings & Magliaro, 1998). Anderson, Jimerson, and Whipple (2005) conducted a study of stressful life events at home and school for first, third, and sixth grade students. The students were asked to rate 20 life events on level of stressfulness. Across all grade levels, events rated most stressful were loss of a parent, academic retention, blindness, being caught stealing, having an accident in class, poor performance on a report card, having an operation, parental conflict, and being sent to the principal (Anderson, Jimerson, & Whipple, 2005). Sixth graders in this study identified retention has the most stressful life event (2005).

Larsen and Akmal (2007) conducted a qualitative study on grade-level advancement/retention policies and procedures to reveal how retention policy and procedure are disconnected from retention research. Through open-ended interviews and review of documents related to policy/procedure, they explored two themes present in previous and current literature: (1) policy on retention is not linked to research on retention, and (2) the ethical dilemmas faced by those implementing the policies on retention. The second theme will be explored in greater detail ahead. The overall findings of the study revealed that many middle-level teachers and administrators (grades 6-8) believe retention policy lead to unintended negative consequences for students not meeting the minimum requirement for promotion to the next grade level, failing to take into account social, familial, and community forces, which staff have little control over. Larsen and Akmal (2007) suggest that LEAs evaluate policies of retention and promotion, taking into account research recommendations in order to determine what best meets the identified needs of students not meeting minimum competencies.

Research has focused on the effects of grade retention on students. Roderick (1994) used a history analysis to study how grade retention influenced graduation outcomes in one urban school system. The study found that repeating a grade in elementary school was associated with increases in the odds of dropping out, even after controlling for differences in background and post retention grades and attendance. The author concluded that the impact of being overage for a grade during adolescence may explain a large proportion of the higher dropout rates among retained youth (Roderick, 1994).

The non-partisan Education Commission of the States (ECS), established by James Conant, has argued against retention policies, stating that retention doesn’t equate to improvement in reading skills for students (Bloom 2012). The repetition of a grade

with similar instructional practices does not allow students to develop the necessary reading skills and flourish into achieving students. The literacy policies of other states and cities that are leading to student success have focused on small class sizes and intensive, personalized instruction. Some of them forgoing the inclusion of retention into their policies, such as Massachusetts. The ECS suggests that the urgency created by these policies, particularly those that have the punitive clause of retention, can lead to identification and intervention without the need for retention (2012).

Jimerson, Anderson, and Whipple (2002) completed a comprehensive review of research on dropout rates in relation to retention. Their research involved an evaluation of 17 studies, criteria for selection included professional publication and the results must address association between dropout status and retention, which suggest retention is “one of the most powerful” predictors of dropout status in high school (p. 441). They found that as previous research indicated, a short-term benefit of retention is likely to dissipate and lead to student dropout. All 17 studies identified early grade retention as one of the most powerful predictors of later school withdrawal (2002). The authors quote Peg Dawson who wrote in a book review “…it could be said, that we’ve won the battle but lost the war” (p. 452). This quote and the research completed by Jimerson, Anderson, and Whipple (2002) align with the short-run and medium-run research on Florida’s policy by Greene and Winters (2012), both suggesting the initial outcomes, the battle, for academic performance are positive, but the long-term impact, the war, is suspect at best. It is important to note that the researchers did not perceive retention as a single indicator leading to dropout. They found some students have certain risk factors (e.g., low SES, low parental level of education) which place them at a higher risk of retention, as well as high school dropout (p. 454).

John Hattie (2009) published his work on student achievement and effect size. He synthesizes over 800 meta-analyses concerning influences and achievement of school- aged students. Hattie (2009) summarizes his analyses around retention:

Retention has been found to have a negative effect of academic achievement in language arts, reading, mathematics, work-study skills, social studies, and grade point average. Promoted students score better than retained students on social and emotional adjustment, and behavior, self-concept, and attitude towards school (p. 97).

He identifies the effect size as -0.16. The desired effect size is greater than 0.40, indicating more than a year’s growth for a student. The negative effect size indicates the loss of achievement and growth. Hattie (2009) poses the question, “The only question of interest relating to retention is why it persists in the face of this damning evidence” (p.98).

The literature overwhelmingly indicates the negative impact of retention on students in both their social emotional development and their academic development. While one may assume that an additional year in the same grade level, receiving instruction on the same information, may lead to increase academic performance, the evidence provided by research does not agree. Our youngest learners are being retained with no promise of something different that will lead to their future success. The

evidence and the significant detrimental implications for our youngest learners cannot be overlooked.

Retention and Literacy Policy A significant component of many literacy policies is the consequence of retention for students. Seventeen states and multiple cities including Washington, D.C., Chicago, and New York, currently have or have passed literacy policies that require (or will require) retention when the students are unable to demonstrate proficiency in reading by the end of third grade, while an additional 10 states do not require but do allow for retention (National Conference of State Legislatures, 2016). The support for this approach to literacy in the United States is evident in the Campaign for Grade Level Reading, the National Governors Association’s A Governor’s Guide to Early Literacy, and the development of more and more state policies directly addressing early literacy.

Florida began the trend for addressing literacy proficiency through retention at the state level in 2002, but Chicago’s public-school system began this work in 1996. Chicago’s policy went beyond third grade to include third, sixth, and eight grade proficiency “gates” for promotion to the next grade. Chicago utilized the students’ score on the Iowa Tests of Basic Skills (ITBS) to determine promotion leading to nearly one in five third graders and one in ten sixth graders and eighth graders being retained (2005). Roderick and Nagaoka (2005) investigated Chicago’s policy specifically looking at the performance of retained third and sixth grade students from 1998 to 2000 two years after the promotional gate. They sought to answer three central questions using same-grade comparisons:

1. Did the extra year of instruction allow retained students to raise their test scores to meet the promotional standards their second time in the same grade? 2. How did the retention under high-stakes testing and Chicago’s use of multiple chances to pass the promotions test shape retained students’ subsequent progress, including the portability of special education placement, being retained again, or rejoining their age-appropriate classmates? 3. Did retention lead to higher achievement for these students than if they had been promoted to the next grade? (p.310)

They used historical data on student performance assessments to estimate effect using Hierarchical Linear Modeling (HML) and then tested robustness of their findings using a two-stage probit least squares model (2SPLS). They found that third graders who were retained experienced no academic benefit, positive or negative, to their reading achievement. Sixth graders who were retained had negative impacts on academic performance in reading 1 year after the promotional gate and remaining in effect 2 years after the gate, aligning with previous research stating that retention in later grades has significant negative implications for students’ academic performance (Holmes, 1989). They identified some concerning trends in the increase of students being identified with disabilities and receiving special education following the implementation of the promotional gates, noting third grade students retained two years post gate were 3 times

more likely to be identified with a disability and sixth graders retained two years post gate were 6 times more likely to be identified with a disability compared to other low- achieving students not retained (Roderick and Nagaoka, 2005). Greene and Winters (2007) looked at the impact of grade retention on student proficiency in reading one and two years after the retention decision in the state of Florida. Their research centered on Florida’s test-based promotion policy in third grade that was implemented in 2002, the first literacy policy with retention at the state level. The state utilized the Florida Comprehensive Assessment Test (FCAT) to determine promotion requiring a minimum score of Level 2 (the second lowest of five levels). They utilized an instrumental variable approach using a regression discontinuity design.

Greene and Winters (2007) felt the instrumental variable approach was appropriate due to the reduction of subjectivity of retention decisions compared to previous research that was conducted on more subjective criteria, such as observed characteristics. They found that retained students slightly outperformed socially promoted students in reading in the first year following retention. Further, they found that in the second year following the retention students retained performed at an even higher level in comparison to the students socially promoted (2007).

Greene and Winters (2007) identified the new objective retention policy trend as a way for researches to create more meaningful groups with which to compare to retained students in a way that was not previously available. The retention piece to policies of this type may be objective in the sense of retained or not retained, but the exceptions and consistency with other elements are subjective. The exceptions of which students are included or excluded are not specifically addressed in how this impacts their analysis, as well as the lack of consistency is measures or pathways to meet promotion criteria. Greene and Winters (2007) were able to address the academic performance, but did not address or consider the social emotional implications, which are significant and must be considered.

Greene and Winters (2012) investigated the long term impact of retention by extending their research and examining the impact of Florida’s test-based promotion policy on multiple outcomes up to five years after intervention, utilizing a regression discontinuity strategy to provide causal estimates of relationships between a student having been retained under the policy and achievement through seventh grade. They looked at the effect of retention and subsequently being assigned a high-quality teacher during the year of retention, as well as mandatory summer programming to addressing reading deficits. The study noted that upon initial implementation Florida saw an increase in retention rates to 12% from 3% (2012). They found evidence that the students made substantial gains in all subjects in the short term following retention and intervention (2012). When they looked beyond the short term, they found that students’ gains declined with time. Although it is important to note that they found a statistically significant and meaningful difference in student achievement several years after the treatment, they were unable to determine which elements of the treatment were of significance. They did have some evidence that the estimated effect of treatment did not

appear to be driven by teacher assignment (p.307). The researchers also identified that in their analyses they were unable to “disentangle” the impact of retention from other factors due to the existence of alternative exemptions other than performance on an alternative assessment prohibits a way to identify a discontinuity in who is retained or promoted (2012).

Rose and Schimke (2012) identify three elements of TGRG and many other state policies; early identification, intervention that addresses the area of deficit, and retention. The first two elements are critical components to a Response to Intervention (RTI) Model, or the academic side of a Multi-Tiered System of Support. The Response to Intervention model focuses on identifying skills that students need to develop in order to access the curriculum, developing a plan to provide intervention, monitoring progress/response to the intervention, and then making decisions based on the data collected. RTI is foundational to many LEAs’ instructional practices in Ohio, has been implemented prior to these new policies. RTI is recognized in federal law to address the needs of students and is an evidence-based practice shown to increase student learning/skill beyond the one year of anticipated growth (Hattie, 2009). The lack of consideration for effective existing models and methodologies could be considered a flaw in the TGRG policy.

Rose and Schimke (2012) address the lessons learned from Florida and New York implementation. Included in those lessons are early identification, intervention, need for professional development, and strong school leadership. Other considerations for policies of this nature include a jump in expected retention in initial years, which one could argue should not happen if a strong core curriculum is in place. Brozo (2010) addresses the critical need for strong literacy skill development in Tier I, or core instruction in an RTI model, to allow all students to access the curriculum. Shifting the focus from intervention beyond Tier I back to instruction for all students with use of research based strategies and differentiation would be cost effective and likely increase the outcomes for all students.

Franco (2015) conducted a case study, funded by the Ohio Department of Education, on Ohio’s Third Grade Reading Guarantee. The case study investigated the use of an Ohio Department of Education competitive grant to design and implement an early literacy plan to align with compliance to the TGRG policy by a consortium, consisting of 3 Local Education Agencies (LEAs) focusing on the use of Orton- Gillingham (O/G). One of the three LEAs elected not to participate in the case study. The grant was designed to ease the financial impact of compliance with the unfunded policy; however, it was only provided to selected LEAs or consortiums who went through the grant process. The case study utilized mixed methods examining perceptions of the O/G, the successes and challenges around the maintenance of the professional learning community, and on student performance and outcomes, specifically measuring the percentage of students identified as not on track. Franco’s (2015) descriptive statistics identifying students not on track for the two participatory LEAs reveals two trends. The first trend identified is the discrepancy in the assessment utilized

to determine on-track and not-on-track between the two LEAs as illustrated in Table 4.0.

Table 4.0: Diagnostics Utilized to Determine On-Track or Not-On-Track Grade Level LEA 1: Diagnostic LEA 2: Diagnostic K KRAL NWEA (2012) DIBELS (2013) 1 State Screener NWEA (2012) DIBELS (2013) 2 State Screener NWEA (2012) DIBELS (2013) 3 OAA & DIBELS NWEA (2012) DIBELS (2013) (Franco 2015)

All the assessments utilized are approved by the Ohio Department of Education despite the vast difference in the administration and skills assessed. This is problematic when they have significant implication for students. The second trend Franco’s (2015) work reveals is the decline in students assessed as not on track from year 1 to year 2 of implementation for both LEAs. This data also revealed a significant decline in students not-on-track in the fall administration compared to the spring administration for Kindergarten. Both LEAs had less than 5% of students assessed as not on track in spring. Franco suggests this indicates that grade K should not be given the diagnostic assessment until further into the school year (2015). The requirement date for K diagnostics was shifted in the policy for the 2015-2016 school year from September 30, like other grades, to November 30. Franco provides the implications and recommendations for policy makers, including funding the policy to allow for improvements and sustainability of literacy programs and interventions, as well as looking at diagnostic requirements to reduce testing of students.

Ethical Decision Making by School Leaders Education policy, and in particular literacy policy, is laced with an increasing amount of accountability measures impacting the operation of LEAs. Cohen and Moffitt (2009) identify this as a “reform” strategy around public education, focusing on providing a framework through policy, while allowing practitioners to maintain autonomy in their craft in the educating of our nation’s youth. Accountability has evolved over the course of this time manifesting itself in the form of standardized testing, district/state report cards, new graduation requirements, and many other mandates established by state leaders of education. The policies that have instituted these measures of accountability are drafted and made law by policy makers who have limited knowledge on K-12 public education beyond their own experience, which does not equate to expertise on student learning and growth. The leadership in public schools has the challenge of implementing these policies in a meaningful way that leads to positive outcomes for students with little to no funding. Beyond the funding, implementation of policies and mandates requires a great deal of time and energy at the expense and distraction of the core business of educating students. Leaders are tasked with making ethical decisions to align with policy, at times altering systems or practices that are 19

effective, while maintaining the mission of the learning community to the greatest extent possible. The implications of the age of accountability in education have developed from reports of failing schools, A Nation At Risk (Gardner & And Others, 1983), political promises and legislation, Goals 2000 (Kildee, 1994) and No Child Left Behind (2002). The accountability is mandated, changing policy and requiring LEAs and educational leaders to align practices with the new requirements. This causes a shift in practice and leaves leaders making decisions regarding practices and resources. The dissonance between policy and practice increases the pressures of our educational leaders to make decisions to minimize the dissonance and increase the likelihood that the intent will be achieved. Educational leaders, serving in a democracy, are servants to the legislature. Strike (2008) identifies that the legislature in a democracy is tasked with doing the will of the people and that their decisions should therefore be obeyed, even when deemed unwise, as they are legitimate in their authority to make such decisions. Those hired to implement or carry out these decisions, functionaries, must behave in a democratic way and do the work they are assigned in accordance with the decisions. School leaders are functionaries and must comply as their formal authority is gained as servants of the legislature, making it democratic (Strike, 2008). Booher-Jennings (2005) did qualitative research on the impact of Texas’ accountability system on teacher practice. She looked to answer why the accountability measures, high-stakes testing, led to change when so many other reforms have been failures in implementation. Her research identified a form of educational triage, where the teacher identifies safe students (likely to meet proficient ratings on high stake assessments), suitable cases for treatment (bubble kids), and hopeless cases. Strike (2008) identifies this practice as a form of “gaming,” focusing resources on students who are “on the bubble” and neglecting those who are sure to pass or sure to fail is a form of gaming (Kindle Locations 3298-3299). Booher-Jennings (2005) found this classification then dictated instructional practices and resource allocation with the target population being the “bubble kids,” while the hopeless cases were removed or minimized through referral for special education. The classification of a group of students as “bubble kids” marginalizes them as learners and should be cause for great concern. This classification posed by Booher-Jennings (2005) aligns resources and focus, while inadvertently negating the needs of other students not in this classification, students who may well have significant identified needs based on measures of accountability dictated at the state level. The accountability was found to impact teacher-teacher relationships as it increased pressure and the stress of public embarrassment if their students’ performance was not at a high level. Booher-Jennings (2005) illuminates the unintended consequences of accountability measures on K-12 education, specifically how it can perpetuate or even exacerbate the educational experience of marginalized populations including students with disabilities, students living in poverty, and students of color. In Kenneth Strike’s Ethical Leadership in Schools (2008), he provides perspective, shapes aspirations, and enables understanding for leaders in education. He

defines ethics and ethical decision making around the following two questions (1) What is good? and (2) What is right? He provides the following explanation: “What is good?” concerns the fundamental aims of communities. “What is right?” concerns the principles and shared understandings that enable social cooperation. Good communities have worthy aims and a fair basis of cooperation. I will encourage you to view ethical leadership as the art of creating good school communities (2008, Kindle Locations 339-341). Strike provides the language and framework for leaders in education to engage in ethical decision making. Education and the policy governing practice, including accountability, require leaders to actively engage in ethical decision making. Issues of equality in access to opportunity, resources, and ensuring all members of the school community are valued for the differences they bring. Strike reviews historical theories and approaches utilized when discussing ethical decision making, including the golden rule, utilitarianism, and communitarianism. One approach is to consider the ideal, similar to the golden rule, to treat others as you would want to be treated. Strike references Kant in this approach and noted good ends do not justify unethical means. One possible scenario that would related to this approach would be a LEA making the decision to change the cut score for the diagnostic in order to achieve the desired K3 accountability grade. Ethical decision making must consider individuals as having value to the community and being more than mere instruments to achieve a desired end. Utilitarianism looks at making ethical decisions based on the greatest good for the greatest number by considering the range of consequences of an action, determining the desirability of these consequences, and making a judgment as to which set of likely outcomes is most desirable (2008). A plausible scenario for such a decision would be similar to that identified by Booher-Jennings (2005) where a district classifies students and then allocates resources to ensure a favorable outcome for most of the students, the greatest good for the greatest number. Communitarianism looks at ethical decision making from the perspective of community and the understanding that the community has purposes and traditions, while understanding that this is maintained by individuals. This theory strongly looks at ethical decision making in terms of maintaining the community through the trust and loyalty (2008). LEAs grapple with policy compliance, district mission, and best interest of students when engaging in ethical decision making, always striving to align the various factors in order to maintain what they identify as their identity as a learning community. Strike acknowledges his approach in Ethical Leadership for Schools aligns most closely with communitarianism, but clearly identifies that his purpose is not about referencing of theories. He sets out to identify ethical ideals, captured by these theories and central to ethical decision making. Figure 1 identifies the ideals and characteristics associated with ethical decision making. These identified ideals and characteristics provide a framework for educational leaders and will be the conceptual framework guiding this study.

Figure 1: Strike’s Ethical Decision Making Ideals Prominent in Ethical Decision Making

1. View others as ends in themselves and entitled to equal respect. 2. Maximize good outcomes. 3. Create and sustain healthy communities. Four Characteristics of Good Decision Making 1. The decision is supported by evidence. This evidence supports the claim that acting on this decision is more likely to achieve desired ends at an appropriate cost than other courses of action that might be taken. 2. The ends aimed at by the decision are the ends that ought to be aimed at. 3. The decision can be implemented morally. 4. The decision has been legitimately achieved. (Strike, 2008) Heilig and Darling-Hammond (2008) conducted a longitudinal study on the Texas accountability system on student progress and achievement on the elementary, middle, and high school levels in relation to accountability policy incentives. They used mixed methods and identified LEAs faced with policy involving rewards and punishments based on student performance on high-stake assessments created incentives for schools to “game the system.” Heilig and Darling-Hammond (2008) found LEAs gamed the system through various methods including: exclusion of students’ scores through special education; retention; failing to code drop outs; exclusion of low- performing students from open enrollment; and encouragement of transitioning to GED programs. The gaming strategies employed by LEAs were found to reduce educational opportunity for African American and Latino high school students (2008). In one school year, one of the LEAs in the study retained 30% of ninth grade students in order for them to not be tested with only 12% being tested the following year and only 8% meeting proficient level or higher (2008). As mentioned in the previous section on retention literature, Larsen and Akmal (2007) explored retention in terms of ethical decision making and dilemma. In their research, they interviewed 44 participants in varying roles, including building administrators, teachers, counselors, coordinators/directors in central office, and assistant superintendents, over the course of 29 interviews reflecting on retention policy implementation. They utilized documentation of the board policies and procedures, as well as school retention paperwork in their research. They explored this idea that “those who are charged with applying these policies in the school setting often find themselves on the horns of an ethical dilemma as they weigh, on one hand, the circumstances that lie behind an individual student’s academic deficiencies and, on the other, the obligation they feel to comply with policies that appear to ignore either the sources of these deficits or the unbidden, yet foreseeable, consequences of retention” (p. 37).

The literature reviewed guides this research of TGRG. Factors having significant implications for early literacy development and school readiness included prenatal care, prematurity, prenatal exposure to toxins, low socio-economic status (SES), parental education, and early childhood programming. Precursor skills to literacy identified by NELP include: alphabet knowledge, phonological awareness, rapid automatic naming,

rapid automatic naming of objects or colors, writing or writing name, and phonological memory. Children who develop these precursor skills experience more success in developing conventional literacy skills of fluency, comprehension, phonemic awareness, vocabulary, and phonics. This literature provides the foundational guidance for policy makers and practitioners on essential components for creating a framework supporting literacy development for all students.

Retention research related to third grade literacy policies is inconclusive. While there is evidence that students experience some short-term academic gains, the evidence of short-term social emotional gains and/or long-term gains in either academic or social emotional growth is sustainability limited. Trends noted in the research above include students retained experiencing short term academic gains that dissipate over time. More important, the literature of retention separate of third grade policy identifies retention as a predictor of later school failure and a stressful event comparable for some students to the loss of a parent. In addition, not all states have instituted a retention component to their reading policies. Massachusetts has the highest fourth grade reading proficiency, and their state reading policy does not include retention.

The accountability measures associated with Ohio’s Third Grade Reading Guarantee also provide another angle of consideration in the literature. Accountability measures add another level of pressure in the implementation process for LEA leadership. They must contend with how to integrate new requirements into existing systems with limited resources. This process of decision making examined by this study is guided by the work Strike (2008), Booher-Jennings (2005), Heilig and Darling- Hammond (2008), and Larsen and Akmal (2007). The research suggests that LEAs are “gaming” as a result of the pressures of accountability as a result of policy implemented by the legislature. This calls into question the ethics of decision making in policy implementation by educational leaders.

Chapter 3: Methods

A tension exists between policy makers who define problems and develop solutions and the practitioners who are responsible for implementation. Cohen and Moffitt (2009) identify the “the likelihood that policies will be realized in practice depends on the fit between aims and capability” (p. 15). It is critical that when research is conducted regarding policy in public education, it investigates both the aim, policymaker intent, the implementation, and practitioner capability. This study investigates the intent of the policy maker as measured by the accountability components of TGRG and the implementation as experienced by one LEA leadership team.

Standard discourse frames this study utilizing a mixed method design. Quantz (2015) suggests one approach to mixed methods in a standard discourse looks to collect the quantitative data to provide reliable and valid truth claims, but use qualitative techniques to provide depth or “color” the non-human context provided by the quantitative analysis. This aligns with Cohen and Moffitt’s (2009) assessment of policy research in terms of measuring aim and capability. This study used quantitative analysis with statewide data collected by the Ohio Department of Education from LEAs to identify trends in third grade reading proficiency rates prior to the implementation of TGRG and years following as indicated by high-stake state testing, as well as correlations between district demographics and district performance on K3 Literacy accountability measure. The quantitative analysis seeks to determine if a relationship exists in diagnostic performance and reading proficiency as measured by high-stakes state reading assessment for one specific LEA. Qualitative analysis examines the ethical challenges one LEA faced when implementing TGRG. Figure 2 illustrates the concurrent triangulation strategy to data collection and analysis.

The quantitative approach to investigating third grade reading policies is established in the literature. Quantz (2015) identifies rigorous methods of measurement and statistics to describe the precise nature of the relationship of the variables, such as students’ Figure 2: Visual Model of Study reading proficiency relationship to demographic factors. Much of the previous research related to reading policies has been quantitative in nature as an evaluation of the policies, effectiveness in improving literacy outcomes for students, and or the impact retention has on students’ academic performance short and long term (Hernandez, 2011, Greene & Winters 2007, 2012, Coulton, Fischer, & Kim 2015, Roderick & Nagaoka, 2005, Franco, 2015, Figlio, 2016). The research designs have varied from a single case study (Franco, 2015), discontinuity regression (Greene & Winters, 2007, 2012, Roderick & Nagaoka 2005, Figlio, 2016), and cohort longitudinal (Coulton, Fischer, & Kim 2015).

There are mixed methodology studies examining ethical decision making in the context of educational policy implementation (Heilig & Darling-Hammond, 2008, Larsen & Akmal 2007, Booher-Jennings 2005). The qualitative analysis of the decisions made during implementation of Ohio’s Third Grade Reading Guarantee provides depth and human context to the policy. A qualitative lens gives voice to the reality experienced as district leaders took this new policy and implemented the requirements into their existing systems illuminating the unspoken consequences of K-12 policy implementation. Qualitative analysis of the LEA leadership team will offer other district leadership teams a frame of reference based on experience. Identifying the processes, a district navigates, the resources they utilize, and the ultimate change that evolves through ethical decision making can be insightful to other school leaders. The qualitative data was gathered through semi-structured interviews with members of the LEA leadership team including 3 building principals, 1 literacy specialist, and 2 district level administrators.

Meaning is made from analysis using theory from ethical leadership and specifically examining ethical decision making for LEA leadership teams (Strike, 2006). Literature on how educational leaders respond to policy and accountability demands through gaming, a behavior that aims only at reaching a mandated benchmark and that fails to take into account the achievement of the underlying purposes of the activity, provides context for analysis of the qualitative data (Strike, 2006).

Research Questions and Data Sources What are the statewide trends in reading proficiency in third grade in years before and after implementation of third grade reading guarantee? Quantitative data sets retrieved through The Ohio Department of Education (ODE) were evaluated to identify statewide trends in reading proficiency in third grade three years prior to full implementation and two years post full implementation. Data sets for all public-school LEAs (n=609) for SY 11-12, SY 12-13, SY 13-14, SY 14-15, and SY 15-16 are utilized. ODE collects data from LEAs through the Education Management Information System (EMIS). The data reported is then compiled and published through the ODE. The Ohio Schools Report Card page (http://reportcard.education.ohio.gov/Pages/default.aspx) provides information regarding performance on state assessments, graduation rates, and other accountability measures that impact LEA report cards. The assessment utilized to assess reading proficiency during the identified years did change. For third grade in SY 12-13, SY 13-14 ad SY 14-15, students took the reading OAA in the fall, spring, and when needed for promotion opportunity, the summer. The proficiency scaled score of 400 for the reading OAA. Table 5.0 below shows the breakdown of scores and the descriptors for each proficiency level as determined by ODE. Third grade students in SY 15-16 took the English Language Arts AIR assessment. Table 5.1 below shows the performance levels and description for ELA AIR assessment as determined by ODE. A proficient score was 700-724. In addition to the 25

scale score ranges for grade 3 English language arts, Table 5.3 below summarizes the grade 3 reading promotion performance standards for the 2015-2016 school year. As indicated in the table below, the 2015-2016 promotion score for the Third grade Reading Guarantee is 42.

Table 5.0: Performance Level Descriptors Grade 3 Reading Achievement Test Performance Description of Performance Level Level

Limited Students performing at the limited level do not yet have the skills identified at the basic level. 260

Basic Students performing at the Basic Level make limited use of reading comprehension strategies, such as 385 inferencing, predicting, comparing and contrasting and summarizing, to build meaning from text. They can usually respond accurately to literal questions but inconsistently answer inferential and evaluative questions. They are able to decode words and can define some unfamiliar words by using context clues in grade-appropriate reading material. A student at the basic level needs teacher support and prompting to comprehend grade-level texts.

Proficient Students performing at the Proficient Level usually apply reading comprehension strategies to 400 construct meaning. They use their understanding of the elements of literature (e.g., characters, setting and plot) and the author’s use of language to develop an accurate understanding of the text. They use text features, such as titles, subtitles and visual aids, to support their comprehension. They often use organizational text features such as chronology, cause and effect and problem/solution to help them organize and recall information. Typically, these students are able to determine the meanings of unfamiliar at- or above-grade level words by using context clues and structural analysis. These students can read and comprehend grade-level texts with little or no teacher support.

Accelerated Students performing at the Accelerated Level consistently apply comprehension strategies to develop 415 a thorough understanding of what they read. These students demonstrate an ability to use text structures to comprehend and recall what they have read. They can identify and describe various literary elements including plot, theme, character and setting. They respond accurately to inferential and evaluative questions. They consistently determine meanings of unfamiliar at- and above-grade level words by using word structure analysis and context clues. These students can read and comprehend grade-level texts independently and relate what they have read to what they have read and to their own experiences.

Advanced Students performing at the Advanced Level apply comprehension strategies to develop a thorough 432 and cohesive understanding of what they read. These students demonstrate an ability to use text structures to interpret, evaluate and extend what they read. They consistently respond accurately to questions about what they have read. They can infer and evaluate the ways that authors affect texts. They use their knowledge of word structure and context clues to extend their vocabulary. These students can use critical reasoning to evaluate texts and are able to relate their understanding of textual information to other texts or situations. (Adopted by State Board of Education, 2003)

Table 5.1: Grade 3 English Language Arts AIR Performance Descriptors Performance Scale Score Description of Performance Level Level

Advance 752-863 A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Grade 3 English Language Arts. A student at this level has a sophisticated ability to determine the central message, lesson, or moral and explain how it is conveyed through key details in the text, describe the logical connection between particular sentences and paragraphs in a text, and write an opinion or explanatory piece that introduces a topic, provides reasons to support an opinion, uses linking words and phrases, and provides a concluding statement.

Accelerated 725-751 A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Grade 3 English Language Arts. A student at this level has a superior ability to determine the central message, lesson, or moral and explain how it is conveyed through key details in the text, describe the logical connection between particular sentences and paragraphs in a text, and write an opinion or explanatory piece that introduces a topic, provides reasons to support an opinion, uses linking words and phrases, and provides a concluding statement.

Proficient 700-724 A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Grade 3 English Language Arts. A student at this level has a consistent ability to determine the central message, lesson, or moral and explain how it is conveyed through key details in the text, describe the logical connection between particular sentences and paragraphs in a text, and write an opinion or explanatory piece that introduces a topic, provides reasons to support an opinion, uses linking words and phrases, and provides a concluding statement.

Basic 672-699 A student performing at the Basic Level demonstrates partial command of Ohio’s Learning Standards for Grade 3 English Language Arts. A student at this level has a general ability to determine the central message, lesson, or moral and explain how it is conveyed through key details in the text, describe the logical connection between particular sentences and paragraphs in a text, and write an opinion or explanatory piece that introduces a topic, provides reasons to support an opinion, uses linking words and phrases, and provides a concluding statement.

Limited 545-671 A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Grade 3 English Language Arts. A student at this level has an emerging ability to determine the central message, lesson, or moral and explain how it is conveyed through key details in the text; describe the logical connection between particular sentences and paragraphs in a text; and write an opinion or explanatory piece that introduces a topic, provides reasons to support an opinion, uses linking words and phrases, and provides a concluding statement. (Adopted by State Board of Education, 2015)

The Table 5.3: Third Grade Reading Guarantee Level Scale Score Ranges SY Does Not Meet Promotion Standard Meets Promotion Standard Promotion Score

15-16 16-41 42-86 42 16-17 16-43 44-86 44

The reading proficiency trends are evaluated through the promotion percentage, a statistic that reports what percentage of students met a promotion criterion to advance to the fourth grade. The technical guide for this calculation can be found in Appendix B. This variable does not include students who are exempted. Student exemptions can take place through an IEP team decision for a student with a disability, an English Language Learner who has not received instruction in English for two consecutive years, or a student who is previously retained, as well as students with intellectual disabilities, as identified under the Individuals with Disabilities Education Act (IDEA), who are exempted from all provisions of the policy. The percent of exemption is calculated by considering students exempted from the retention clause of the policy to the entire grade 3 student population for given LEA (with exemption of students with significant disabilities). The percent of proficient on the state reading assessment considers the numbers of students meeting the proficient performance indicator to the entire grade level of students who took the assessment. The percent of poverty takes the students meeting free and reduced lunch qualifications to the entire student population for the LEA.

What is the relationship between student demographics and a school’s performance on the k-3 literacy measure? The data sets for all public-school LEAs (n=609) for SY 14-15 and SY 15-16 are utilized to evaluate relationships between student demographics in a LEA to the K-3 Literacy measure assigned on the LEA’s report card. Demographic student information includes the following subgroups: economically disadvantaged (ED), students with disabilities (SWD), and minority (non-white). Economically disadvantaged as defined by the Ohio Department of Education is a measure that indicates the number of students living in poverty in school districts. This is a student based indicator that reflects the portion of a district’s student population that meets any of the following conditions.

1. Students who are known to be eligible to receive free or reduced-price lunches; a program through the United States Department of Agriculture (U.S.D.A) National School Lunch Program. Eligibility for free or reduced-price lunch can be determined through a variety of methods including the electronic direct certification process or completion by a parent or guardian of a free and reduced-price lunch application. A student with an approved application on file for a free or reduced-price lunch is qualified to be reported to ODE as economically disadvantaged.

2. Students who have not submitted an application for free or reduced-price lunch or who have not been directly certified as eligible but reside in a household in which a member (e.g. sibling) is known to be eligible for free or reduced-price lunch via an approved application or through direct certification.

3. Students who are known to be recipients of or whose guardians are known to be recipients of public assistance. A source for determining whether a student’s family is receiving public assistance is the Education Monetary Assistance Distribution (EMAD) system.

4. Students whose parents or guardians have completed a Title I student income form and meet the income guidelines specified. (ODE, 2015)

ODE’s Third Grade Reading Guarantee page (http://education.ohio.gov/Topics/Learning-in-Ohio/Literacy/Third grade-Reading- Guarantee) provides data on LEAs’ performance on the K-3 Literacy measure, on on-task performance on diagnostics, exemption percentage, and retention percentage. Student demographics in the LEA are evaluated to determine any correlations to K-3 Literacy measure utilizing SPSS Statistics, a software package used for statistical analysis. The K3 Literacy Measure, an accountability measure directly tied to TGRG and implemented on the LEA report card for Ohio school LEAs in SY 13-14, aims to account for the improvement of students’ reading in a LEA K-3. The measure gives LEAs a letter grade, A-F, as well as NR, calculated based on the fall diagnostic and the third Figure 3.0: K3 Literacy Calculation grade reading Ohio Achievement Assessment. The measure only considers students enrolled for the entire academic school year. Using the fall diagnostic, students are determined not-on-track or on-track. Students determined not-on- track are required to be placed on a Reading Improvement and Monitoring Plan (RIMP); however, any student can be placed on a RIMP, which has the potential to impact the K-3 Literacy measure. The K-3 Literacy measure focuses on students determined to be not- on-track following progress to the following year fall diagnostic. Figure 3.0 shows the improvement pattern utilized in the calculation. The calculation is the percentage of not-on-track students who improve to on-track or proficient on the next assessment. Similarly, the calculation provides credit for a third grade student who was not-on-track on the third grade diagnostic but improved to proficient on the third grade high-stake state reading assessment (ODE, 2014). A reduction in the score results when a student does not meet proficient level and was not a RIMP. The state average represents the minimum of the “C” range of the A-F report card, which is subject to change yearly based on average. The measure includes data from two school years and follows students within LEAs. Any school LEA that has fewer than five percent of its kindergarteners reading below grade level does not receive a letter grade, but a NR indicates not reported. Finally, the promotion score for SY 13-14 is 392 and for SY 14-15 is 394, while the proficient score for the OAA is a scaled score of 400 (ODE, 2014).

Descriptive statistics provide an overview of results on the accountability measures for SY 13-14, SY 14-15, and SY 15-16. K3 Literacy measure (coded A-4, B-3, C-2, D-1, F-0 and NR-blank), percent of retention, percent of exemptions, and relationship to percent will be evaluated for correlation. Assuming sampling adequacy and Bartlett’s test of sphericity is significant, the data is determined to for factor analysis (Huck, 2011). An ordinary least square regression is the method to be utilized for factor extraction to provide an explanation of the variance of the dependent variables (Huck, 2011). The statistical analysis looks to make a statement in reference to the content validity of the K3 Literacy measure. SPSS Statistics, a software package is used statistical analysis.

What is the relationship between the 3rd grade and 4th grade reading proficiency as measured by the diagnostic measure for students who received reading intervention through a Reading Improvement and Monitoring Plan during their 3rd grade year of schooling? The data collected for analysis is from a LEA in the state of Ohio. The sample in the LEA included all third grade students, except for 12 students exempted from all provision of the policy due to significant intellectual disabilities and any student not enrolled during both school years (n=550) in the LEA third grade during the 2013-2014 school year and as fourth graders during the 2014-2015 school year. The sample was determined based on enrollment data submitted to Ohio Department of Education through EMIS. The data analysis was done on a smaller segment of this population (n=53) of students assessed not on track on the third grade reading diagnostic, AIMSweb’s third grade fall reading curriculum based measured, R-CBM, which is a benchmark assessment of oral reading fluency. The school LEA student diagnostic data assesses students’ literacy skills, specifically fluency for this LEA’s vendor assessment. The diagnostic data utilized was gathered by the LEA for third grade students by September 30 of the school year. AIMSWeb, a vendor approved assessment tool approved by Ohio Department of Education, is the tool selected by the LEA, from those allowed by the Ohio Department of Education, to measure the students’ literacy skills. The Reading Curriculum Based Measure, or R-CBM, is a general outcome measure that assesses students’ ability to reading fluently and accurately. Students assessed reading 43 correct words per minute or higher were identified as On-Track. Student assessed reading below 43 correct words per minute (cwpm) were identified as Not-On-Track, which is the 12th percentile and lower using national norms. AIMSWeb has National Norms and has been shown to be reliable and valid measures of reading skills (NCS Pearson, Inc., 2012). The same measure was utilized to determine the proficiency performance for the same group of students as fourth graders in school year 15.16. The national norms show that fourth grade students in the 50th percentile read 99 cwpm and in the 25th percentile read 73 cwpm. With the use of SPSS Statistics, a software package for statistical analysis, an OLS regression with fourth grade reading CBM as the outcome variable, and student demographics, status of RIMP, and which assessment they passed to complete the TRGR as independent variables. 31

푆푖 = 훽0 + 푆퐸푆훽1+푅푎푐푒훽2+푅퐼푀푃훽3 + 퐷푖푠푎푏푖푙푖푡푦훽4+푅퐼푀푃훽5+퐴푠푠푒푠푠훽6 + 휀푖

The outcome measure S is the fourth grade CBM score for student i. The independent variables are 푆퐸푆훽1 which is a measure of the student socio-economic status (coded 1 if they qualify for free and reduced priced lunch, and 0 if they do not), race (coded 1 if they are non-white, 0 if they are white), disability status (coded 1 if they have a disability, and 0 if they do not), RIMP status (coded 1 RIMP, and 0 if no RIMP), is a variable for level of proficiency performance on assessment (coded 0 if below promotion score, 1 if below proficiency level, and 2 if proficient or above) they used to pass the TRGR and an error term.

How did one school LEA leadership team implement the Ohio’s Third Grade Reading Guarantee?

Semi-structured interviews conducted with 6 district administrators of the LEA, 3 building level leaders, 1 literacy specialist, and 2 district level leaders, charged with guiding the LEA in the implementation and compliance of Ohio’s Third Grade Reading Guarantee, provide the qualitative data set, as well as implementation documentation from the LEA. Implementation documentation is the notation kept by the district detailing the implementation of the policy, as well as related topics during SY 13.14, SY 14.15, and SY 15.16. The semi-structured interview protocol (Appendix C) was used to collect data on the team’s perspective on the implementation process and the process of ethical decision making using the ideals provided by Strike (2006) and reviewed in Chapter 2. The qualitative approach to policy implementation and accountability is supported in research (Booher-Jennings, 2005, Heilig & Darling-Hammond, 2008, Larsen & Akmal, 2007).

The qualitative data collected through documentation was coded to identified key themes in relationship to the implementation based on the literature reviewed. Specific themes coded for include: Accountability (Strike, 2006; Heilig & Darling-Hammond, 2008) Compliance (Strike, 2008), Decision (Heilig and Darling-Hammond, 2008; Strike, 2008; Booher-Jennings, 2005; Larsen and Akmal, 2007), Exemption (Rose & Schimke, 2012), Intervention (Hattie, 2009; Jacob & Lefgren, 2004; Brozo, 2010), Leadership (Strike, 2008; Larsen and Akmal, 2007; Booher-Jennings, 2005), Literacy (Rose & Schimke, 2012; Riley, 2011), Policy (Honig, 2003; Heilig & Darling-Hammond, 2008; Cook & Bornfruend, 2015), Resources (Booher-Jennings, 2005; Cook & Bornfruend, 2015), and Retention (Roderick & Nagaoka, 2005; Workman, 2014; Smith & Shepard, 1987; Shepard, 1989; Rose & Schimke, 2012; Maitland & Moreno, 2012). Additional information regarding implementation of the policy was identified including total meeting sessions and resource distribution related to compliance of the mandate.

Limitation A limitation to quantitative research on Ohio’s Third Grade Reading Guarantee is minimizing the variability in data collected as allowed by the policy. This variation

exists in the tools used for the diagnostic component to determine if a student is “on track” or “not on track,” as well as the way in which a student meets promotion criteria. The Ohio Department of Education has 12 vendor assessments approved for the diagnostic and they have different formats (on-line, paper-pencil), measuring different reading skill sets (fluency, comprehension, decoding, and vocabulary), and varying in length of time. The approved vendor list has changed over the course of the life of the policy. This has been extremely problematic for districts who have developed systems that inform instruction and intervention for students around certain assessments. This poses a significant issue for researchers trying to make statements using the data for analysis because the vast discrepancy that cannot be mitigated.

The other consideration is the change in the assessment tool utilized for reading proficiency data. The first two years of data collections, school year 13-14 and 14-15 referenced the Ohio Achievement Assessment with each year having a different promotion score. In school year 15-16, the data collection referenced the ELA AIR assessment with a reading sub score used for promotion criteria. Not only were different assessments used to determine proficiency, but different scores were used each year to determine promotion.

Promotion data also poses similar challenges for research and for districts. Districts administer the state reading assessment in the fall and spring for all third grade students. If a student does not pass either of these assessments, students are provided another opportunity to take the state assessment in the summer or pass an alternative assessment. ODE recognized three assessments for this purpose during SY 13-14 and SY 14-15: Iowa Assessments Form F Level 9, Northwest Evaluation Association (NWEA)’s Measurement of Academic Progress (MAP) and Terra Nova 3. The use of these assessments is optional. They can be administered twice during the school year and once during the summer. This provides each third grade student up to six opportunities to meet the minimum competency level of reading dictated by ODE. ODE does not track how students meet the promotion criteria, by alternative assessment or high stake assessment. This limits the research opportunities related to correlations between assessment and promotion.

This study is designed to look at the large-scale results of implementation of TGRG considering assessment, exemptions, and retention. The evidence gathered speaks to the effectiveness of the policy identifying the outcome measures dictated by policymakers. The methods are limited by access to data sets related to the assessment and interventions utilized for students to meet the criteria for promotion to fourth grade as identified by the policy. This is to say, if data were retrieved regarding the number of students meeting the criteria as a result of meeting the cut score determined by the state, as well as the number meeting the criteria between the established cut score and the proficient level, and the various alternative assessments, which cut scores are also below a third grade level proficiency, it might provide more evidence of the validity of the policy. In addition, if the research were designed to follow students who met the criteria, but not grade 3 level of proficiency, to determine short-term and long-term implications for student performance and outcomes, specifically looking at additional performance on

high stake tests and graduation rates, it would provide a comprehensive evaluation of the policy. However, other data sets will not have the variation (proficiency rate on high stake assessment) increasing confidence in results. The quantitative data set is robust including all public-schools in Ohio. The qualitative data method would be limited in its ability to make inferences regarding policy implementation. The single subject qualitative research does not allow generalizability. However, it does illuminate the discussion and decisions facing districts in the implementation and continued compliance for TGRG.

Another limitation to be addressed is my personal bias. As an educational administrator in a public-school system in Ohio, I bring a certain lens to the evaluation of this policy, as mentioned in the statement regarding positionality in Chapter 1. I had an essential role in a district team focusing on the implementation TGRG. I’ve had the opportunity to go to various community, regional, and state sessions focusing on the implementation of TGRG. As an educational administrator, I seek opportunities to evaluate and reflect upon the ramifications for policies dictating practice for strengths and weaknesses. The goal is to maximize strengths and develop intentional strategies to build capacity around weakness, as well as advocate through the feedback process at the state level. Chapter 4: Results

Cohen and Moffitt (2009) believe policy is effective when there is a fit between the aims and the capacity to meet those aims. It is critical that when research is conducted regarding policy in public education that it investigates the aim, policymaker intent, the implementation, and the practitioner capability (2009). The results will review the quantitative measures of success dictated by the policy maker, as well as the qualitative data looking at a one LEA and its implementation experience.

What are the statewide proficiency and retention trends? The quantitative data set (see Chart 1) shows a consistent trend for the statewide proficiency rate for third grade students on the high-stake reading assessment, within 3 points, for 7 school years prior to the change in assessment in 2015-2016. In 2015-2016 school year, the assessment tool changed from the Ohio Achievement Assessment (OAA) to the English Language Arts AIR Assessment producing a dramatic decline of 23.6 points in students achieving a proficient level on the high-stakes assessment, see Tables 2.1 and 2.2 for descriptors. However, the statewide trends in retention (see Table 2) as a result of TGRG reveal only a 0.7 % increase in the number of students retained during this same span. This data reveals that a large majority of students met the promotion criteria through an approved alternative assessment or by a promotion score below proficient on the statewide high-stakes assessment during the 2015-2016 school year. In the three years since the full implementation of TGRG, the state of Ohio has retained a total of 19,952

third grade students with an increasing number of students being retained each year, which aligns with the research done on other state implementation of literacy policy (Rose and Schimke 2012).

Figure 4: Ohio's third grade proficiency trends

90%

80%

70%

60%

50%

40%

30%

20%

10%

0% ADV Plus ADV ACC PRO BAS LIM PROF

2009-2010 OAA 2010-2011 OAA 2011-2012 OAA 2012-2013 OAA 2013-2014 OAA 2014-2015 OAA 2015-2016 ELA AIR

ADV Plus=Advanced Plus, ADV=advanced, ACC=accelerated, PRO=proficient, BAS=basic, LIM=limited

Table 6.0: Statewide results of third grade reading guarantee School Year # Third % Exempt # Subject to % Met % Did not # Third graders from TGRG TGRG promotion meet Graders enrolled at promotion promotion threshold promotion retained end of threshold threshold threshold school year 2013-2014 126,531 5.70% 119,304 95.80% 4.20% 5,010 2014-2015 125,623 5.70% 118,496 94.10% 5.90% 6,991 2015-2016 127,486 5.50% 120,481 93.40% 6.60% 7,951

The interviewees all articulated that the goal of their LEA was for students to meet proficient and for no student to be retained. The ethical decisions made by this LEA aligned with goals of proficient readers and no student retained, doing what was right for students. Each leader interviewed shared that they felt retention was not the answer and each spoke to the research that supports this. One stated, “My concern is the whole retention threat that's built into the policy. That's another feature that's not research-based. It seems very punitive and really not very helpful and not student- centered.” Another leader commented, “The negative impact of retention is known and yet the State of Ohio has legislated that students will be retained. I have a hard time understanding the reason or rationale for that being a consequence.” The LEA worked hard to target resources and develop paths to ensure students met the threshold for promotion to the fourth grade. Thus far, this LEA has not had to retain any students based on the requirements of TGRG. The district literacy team notes provide additional evidence supporting the goals of the LEA. The district team continually revisited the compliance requirements of TGRG, specifically the diagnostic and writing of RIMPs. Diagnostics were agenda items a total of 27 times over the course of the three years, including examining the choice of using 1 or two measures to determine on or off track. RIMPs were explored 14 times by the district team. They looked specifically discussed if they should expand beyond the students who were off-track on the diagnostic to students who scored below proficiency on the fall assessment. The team elected to put all students who were performing below proficient expectations as indicated by the diagnostic and the fall administration of the high-stake assessment on a RIMP. The team notes indicated interventions as a specific agenda item 25 times and resources 30 times in three years revealing the district’s commitment to reallocating resources to achieve its goal to improve literacy skills of students from kindergarten through third grade.

Is there a relationship between student demographics and a district’s performance on the K3 literacy measure? The K3 Literacy grade is the accountability measure tied to the Third Grade Reading Guarantee. Every district and school in Ohio educating students from kindergarten through third grade receives a letter grade if more than 5% of their kindergarten students assess off-track by the fall diagnostic. The calculation to determine the grade involves looking at student diagnostic data from one year to the next, RIMPS,

and student performance on the reading state assessment. That state provides an eight page guidance document (available at: http://education.ohio.gov/getattachment/Topics/Data/Report-Card-Resources/K-3- Literacy-Measure/Technical-Documentation-K-3-Literacy-Improvement- Measure.pdf.aspx) detailing the calculation. The distribution of assigned K3 Literacy Grades for districts (see Chart below) shows a decline from 120 (24%) districts receiving an A or B letter designation in the 2013-2014 school year to only 22 (4%) in the 2015- 2016 school year. The calculation formula includes a ranking formula that leads to a normal distribution, limiting districts to earn or receive a high designation if the normal distribution does not allow. Many districts earning a D or F for their K3 Literacy grade had no students retained, which is often misunderstood by the larger community. Figure 5: Statewide district K3 literacy grades

39.5 40 35 31.8 31.6 28.8 28 30 26.2 25.4 25 16.3 16.1 20 14.5 15 11.2 10.7

10 6.1 4.9 3.5 3 5 1.6 0.7 0 13.14 K3 GRADE 14.15 K3 GRADE 15.16 K3 GRADE

F D C B A NR

Districts with fewer than 5% of their kindergarten students scoring not on track on the fall diagnostic do not receive a K3 Literacy Grade. The number of districts receiving an NR (not reported) has fluctuated annually from 16.1% of districts in SY13.14 to a high of 25.4% of districts in SY14.15 and then leveling off in SY 15.16 at 10.7% of districts (see Chart 2). The correlation between student demographics and districts receiving a NR on their report card was analyzed for the first three years of implementation. The correlation (see Table 4) reveals a significant, at the .01 level, negative relationship between districts receiving the NR designation and district student demographics. This means that the higher the percent of the demographic category (SWD/ED/Minority) the less likely the district is to receive the NR designation. This relationship indicates a potential issue with equity in the assignment of the K3 Literacy grade. This accountability measure is calculated based on information submitted by the LEAs and has significant discrepancy. The discrepancy comes from the diagnostic tool utilized and the cut score determined to indicate on or off track. The tool can be chosen from an approved list from ODE. The approved diagnostic tools vary greatly in how they are administered and the information they provided. The cut scores are determined at the

LEA’s discretion, which adds an additional element discounting the validity of this measure. Table 7.0: Statewide correlation between district student demographics and NR % SWD % ED % Minority 13.14 NR 14.15 NR 15.16 NR % SWD 1 .581 ** .238** -.190** -.220** -.196** %ED .581** 1 .260** -.261** -.260** -.193** % Minority .238** .260** 1 -.148** -.111** -.147** 13.14 NR -.190** -.261** -.148** 1 .290** .297** 14.15 NR -.220** -.260** -.111** .290** 1 .325** 15.16 NR -.196** -.193** -.147** .297** .325** 1 ** Correlation is significant at the 0.01 level (2-tailed).

Analysis was done to determine the correlation (see Table 5) between district student demographics and district K3 Literacy Grade for Ohio. District student demographic included the percentage of students with disabilities (SWD), percentage of racial minority students (students not identifying as Caucasian), and percentage of students identified as economically disadvantaged (ED).

A significant negative correlation, to a 0.01 level of significance, was identified across all three school years in K3 Literacy Grade to the percentage of students identified as economically disadvantaged. The other student demographics, SWD and minority, had significant negative correlations, but not across all three school years and not to the same degree of significance. In addition, when student demographics alone are considered, there is a significant positive correlation, to a 0.01 level of significance, between all student demographic, sub-groups.

The LEA leaders, when asked to reflect on the K3 Literacy Grade, noted that they felt it was biased and did not represent the work being done and the reading proficiency achieved by the district. They had a shared opinion that the calculation was flawed and did not provide valid or standardized information, as the measurements used to calculate it have such dramatic discrepancy.

One interviewee commented, “They also have a diagnostic procedure that every district can choose whatever diagnostic and pick whatever way they want to have the cut scores be. It's not at all statistically reliable. It’s measuring apples to oranges and there isn't any statistical reliability between the data that they're getting at all. Trying to explain that to families, if they see your report card, they're not gonna understand that because the state's saying you may have got a C or a D, and that doesn't look good. When you could be doing really great things. All students have passed and yeah, we're just choosing to do really good things, and we're working on explaining it the best we can.”

Another LEA leader noted that this year, for the first year, their district would receive a NR based on decisions made at the district level regarding the diagnostic. When asked if all districts would be able to make a decision resulting in a NR, the leader

clearly indicated that districts with a significant at-risk population would be challenged to do the same. The LEA’s district literacy team’s notes from the first three years of implementation notated a continuous discussion of the diagnostic, specifically looking at the tool used and the cut scores. The team sought to determine whether the cut scores would identify more students or whether it should be more targeted. The team’s decision also looked at the factor of what students were put on RIMPs and when they would be removed. The final decisions were guided by the K3 Literacy Grade calculation. One interviewee noted that the calculation was not valid or standardized and set districts up to “game the system.”

Table 8.0: Correlation between student demographics and K3 literacy grade % SWD % ED % Minority 13.14 K3 14.15 K3 15.15 K3 GRADE GRADE GRADE % SWD 1 .581 ** .238** -.112* -.144** -.068 %ED .581** 1 .260** -.181** -.186** -.128** % Minority .238** .260** 1 -.118** -.180** -.041 ** Correlation is significant at the 0.01 level (2-tailed). *Correlation is significant at the 0.05 level (2-tailed).

The quantitative state data set was also analyzed to determine if correlations existed between district student demographics and the district percentage of students exempted from the consequences of TGRG and the district percentage of students that did not meet the promotion threshold. The results indicate district student demographics, subgroups, have a significant correlation with the district exemption and retention across all three years of initial implementation. Theses correlations suggest that the higher the subgroup, higher the exemption rate and higher the retention rate, with the strongest correlation being %ED. The strong, significant positive correlation of %ED with rates of retention and rates of exemption indicate potential issues of equity. The trend over the three years does imply that the strength of the correlation is declining; however, it is still present and significant in SY15.16 at .332.

Table 9.0: Statewide correlation between student demographics, exemptions, and retention 13.14 13.14 14.15 14.15 15.16 15.16 Exempt Retention Exempt Retention Exempt Retention % SWD .335** .274** .353** .324** .334** .227** % .381** .515** .418** .462** .307** .332** ED % .238** .349** .296** .356** .163** .142** Minority 13.14 1 .316** .419** .270** .314** .218** Exempt 13.14 .316** 1 .381** .483** .219** .378** Retention 14.15 .419** .381** 1 .048 .368** .170** Exempt 14.15 .270** .483** .048 1 .118** .393** Retention 15.16 .314** .219** .368** .118** 1 -.047 Exempt 15.16 .218** .378** .170** .393** -.047 1 Retention **. Correlation is significant at the 0.01 level (2-tailed).

What is the relationship between 3 rd grade and 4th grade reading for students on a Reading Improvement and Monitoring Plan in one LEA?

The relationship between 3rd grade and 4th grade reading diagnostic for students on a Reading Improvement and Monitoring Plan was analyzed used data retrieved from one LEA with the cohort of students being in 3rd grade during the School Year 2013- 2014. Table 7 displays the demographic information for the 53 third grade students identified as off track and placed on a RIMP. The LEA has a student population of 7,650 students with 14% students with disabilities (SWD), 11% minority, and 16% economically disadvantaged (ED).

Table 10: LEA demographic information for third grade cohort Identification Categories Frequency Percentage Category Gender Male 23 43.4 Female 30 56.6 Disability No Disability 19 35.8 Specific Learning Disability 22 41.5 Other Health Impaired (minor) 10 1.9 Autism 1 18.9 Visual Impairment 1 1.9 Initial ETR Not identified 19 35.8 Grade 3 Not initial 18 34.0 Initial 16 30.2 Disadvantaged Not disadvantaged 35 66.0 Disadvantaged 18 34.0 Exemption Not exempt 23 43.4 Exempt 30 57.7

To determine appropriate statistical analysis, normality was determined by running Shapiro-Wilk test of normality for the variables. Shapiro-Wilk showed that all variables were significantly different than the normal distribution as such the Spearman statistic was utilized. The correlation (see Table 8) of rs= .636 between the fall reading diagnostic score in grade 3 and the fall reading diagnostic score in grade 4 for students on a RIMP. The simple product-moment correlation between Fall Grade 3 OAA Reading and Spring Grade 3 OAA Reading is positive and statistically significant (rs=.586). A positive correlation exists for students on RIMPs and their performance on the Fall and Spring Grade 3 Reading OAA. Table 11.0: LEA correlation for diagnostic and high-stake assessment for students on RIMPs

During the 13-14 school year the promotion score for TRGR was determined either by the student’s performance on the Ohio Achievement Assessment (OAA) or through an approved alternative assessment elected to be provided by LEAs at their expense. The LEA where this cohort attended elected to use the NWEA MAP assessment as an alternative assessment. Table 9 identifies how those students at-risk and on RIMPs meet the promotion to the 4th grade. No students were retained during the 13- 14 school year for this LEA. In addition, while all the students were promoted, it is important to note that all the students in this cohort continued to make gains, but did perform below a proficient level on the 4th grade diagnostic. Further consideration will be given to this in the discussion and expanded upon through the qualitative data analysis.

Table 12.0: LEA promotion criteria met for 3rd grade cohort Met Promotion Criteria Number of Percentage of Students Students Met Proficient (400 or above) 35 19.2% Met Under Proficient (392-399) 7 13.5% Alternative Assessment 10 67.3%

How did one LEA approach the implementation of Ohio’s Third Grade Reading Guarantee? The qualitative data collected is intended to analyze the implementation experienced by one LEA leadership team. The data consisted of 5 interviews with 6 individuals using a semi-structured interview protocol and meeting notes from a district level literacy team tasked with guidance for implementation. The interviews were transcribed and then coded utilizing dedoose. The frequency of codes (see Table 10.1) revealed themes. Table 13.1: Frequency of codes from six interviews ACC COM DEC EXE INT LEA LIT POL RES RET Totals INT 1 6 10 17 2 6 17 6 18 17 5 104 INT 2 7 11 39 2 14 39 20 27 21 9 189 INT 3 10 13 17 6 5 17 9 18 8 14 117 INT 4 5 8 15 3 8 12 7 9 9 3 79 INT 5 4 11 14 3 4 15 2 14 2 3 72 Totals 32 53 102 16 37 100 44 86 57 34 INT=Interview; ACC=Accountability, COM=Compliance, DEC=Decision, EXE=Exempt, INT=Intervention, LEA=Leadership, LIT=Literacy, POL=Policy, RES=Resources, RET=retention

The frequency codes from the interviews conducted reveal a focus on the leadership surrounding implementation, the alignment of the district practices to the policy, and the decisions associated. The leaders interviewed frequently noted the amount of time spent by the district to align practice and policy. This was all done through decisions made by the leaders of the district. These decisions required leaders to consider the ethical implementation, doing what was right and good. In addition, the notes were coded using dedoose to identify areas of focus by the district literacy team. The district literacy team was charged with ensuring compliance with TGRG and evolved into a team focused on expanding the focus on early literacy and intervention. Table 11.1 displays the frequency of codes from the meeting notes from SY 13-14, SY 14-15, and SY 15-16. Table 13.2: Frequency of meetings during initial implementation years Implementation Year Frequency of Meetings School Year 13.14 42 School Year 14.15 27 School Year 15.16 19

Table 13.3: Frequency of codes from district literacy team notes ACC COM DEC EXE INT LEA LIT POL RES RET Totals 13.14 NOTES 7 14 11 5 28 8 27 48 20 11 179 14.15 NOTES 14 23 19 4 29 12 19 31 11 7 169 15.16 NOTES 22 44 42 2 32 36 25 12 11 4 230 Totals 43 81 72 11 89 56 71 91 42 22 INT=Interview; ACC=Accountability, COM=Compliance, DEC=Decision, EXE=Exempt, INT=Intervention, LEA=Leadership, LIT=Literacy, POL=Policy, RES=Resources, RET=retention

The coding of the notes shows a slightly different focus of the LEA. The top three focused identified by the meeting notes were policy, interventions, and compliance. The district in its decision making was committed to understanding the policy and being compliant in its implementation. The focus on the interventions aligns with the district’s focus on ensuring students received the necessary instruction to meet proficiency in reading. It’s interesting to note, that retention was not highly noted in either the interviews or the meeting notes. However, it was communicated in both that retention was not a desired outcome for the students based on Third Grade Reading Guarantee.

Policy and Compliance Those interviewed all stated that implementation was very much led at the district level through an engagement process with the intention of being “proactive.” The leaders, acting as functionaries, needed to ensure they understood the policy and requirements to ensure compliance. Through collaboration with district and building leaders information was shared and distributed to teachers and the community. A common response by those interviewed was that the policy was a change in reporting and paperwork, but not in the way they were addressing student need. The LEA had a solid RTI framework, response to intervention, that was already using data to identify need and then addressing through intervention and monitoring. This was further supported by the district literacy team notes. One district level administrator commented, “We had many interventions in place for reading already, and an RTI model in place already. We felt we were doing a pretty good job or already meeting many of the components, but it really did expand our thinking into kind of reaching down to our students that weren't even enrolled yet, and to working with the early childhood providers in our district, and trying to hit the students with as many resources and interventions before they even enrolled.”

All the interviewees noted the focus on early literacy as a result of the policy. The district applied and received a grant offered by the Ohio Department of Education to assist with initial implementation. Early literacy and preschool partnerships were

reoccurring agenda items during the initial implementation year and the focus of the competitive grant. The grant allowed the district to create resources to address literacy prior to kindergarten, including a literacy focused program prior to the start of kindergarten, targeting students who performed below proficient on a kindergarten screening. A literacy tool kit developed by the district targeting early literacy skills was shared with local child care providers, preschools, and families of students enrolled.

Resources The interviewees identified a shift in resources with the implementation of TGRG. The shifts in resources noted by the district leaders aligned with the state education mandates.

A district level administrator commented, “It breathed life into that mindset (early intervention) because frankly, over the years a lot of our resources and energy and effort has been inverted in our district. Meaning when the Ohio Graduation Test came out, the money and resources were being sent to the high school.” The district made an intentional decision following the TGRG to lower class size for grade K-3 from 28-32 students to 20-25 students per class, along with expanding the interventions needed for students at-risk to include after school tutoring. It was noted by the building level administrators after the financial support provided by the grant was gone, that there was a reduction in the access to the afterschool tutoring. The district began to target only 3rd graders at-risk for retention due to limited resources. This decision was made by the district leadership team and took into consideration the risk of retention as a primary rationale for the distribution of resources in this manner. This meant that students who were potentially still reading below grade level and/or met exemption criteria, as indicated by another measure, would not have access to this intervention. The district team evaluated resources and its commitment to having all students meet the promotion criteria to reach this decision. The district also made the decision to financially commit resources to teachers to obtain the new requirements for compliance under the policy. The district paid for the coursework through its local educational service center and also paid the fee for the praxis exam required to prove mastery in reading instruction. The district felt this was critical in supporting teachers through this process.

Retention This LEA had no students, who were educated by the district from kindergarten through third grade, retained as a result of the policy since its implementation. Each interviewee spoke to the goal of ensuring all students met the promotion criteria. They all indicated that professionally they felt that the retention component of the policy was flawed and “punitive.” One district leader commented, “Students being retained, I think some say the only thing that could be worse in a child's life is if a parent dies or I think moving is up there in terms of stress. Aside from a parent dying or maybe a sibling dying, 45

the retention might be one of the most stressful impacts on a child's life at that age.” The LEA clearly felt compelled to protect students from retention and had a shared view of the negative impact on students, academically and socially. The interviews and notes indicated the commitment to promoting students through reallocation of resources and expanding intervention options for students with deficits in their literacy skills. Strike (2008) would identify this as making decisions based on what is “good” for students aligning the means with the desired outcome. Strike (2008) would also indicate that as functionaries, the district leaders need to ensure they are doing what is “right” and carry out the policy and all its compliance indicators.

Leadership and Decisions The interviewees identified three subgroups that caused them the most concern in terms of planning and supporting: students with disabilities, English language learners, and transient students. The interviewees discussed at length the discussion held prior to any decisions related to exemptions and/or at-risk students. A district leader made the following reference to a team response to a child at-risk of being retained, “Yeah, well I think this team of teachers really did struggle with that idea, because we're looking at it from that perspective of the whole child, so we know that of course we want our students to meet those academic targets, but that there can be many other factors, like they're an English language learner or they just transferred into our district and they have had some lack of instruction coming in. We don't want to identify them with a disability to exempt them from the law, because we don't think that's the case.” The district literacy team developed guidance for the building leaders and teachers on making decisions regarding exemptions. Exemptions were something the district literacy team reviewed each year to ensure the guidance was still aligned to the policy and understood by all building leaders involved in those decisions. Exemptions were monitored by the district literacy team. The transient students, those moving in and out of districts, were deemed by the interviewees and district team notes to be at the greatest risk of retention. One building leader noted, “We have had student move in right before the spring test. We had that happen this year. One of our students had moved in a week before the test who was removed from his old school for some discipline issues and was asked to leave. He ended up enrolling with us and had to take the test. He ended up not passing and personally I think it’s unfair. If it’s going to be our kid and we own all our kids, give us some time to work with them.” Another area identified as a concern was ensuring those transient students and/or students who enrolled as 4th graders either met the criteria or enrolled in the appropriate grade. This led to the LEA creating a team that developed systems to ensure this information was collected and options for students who were “retained” as a result of not meeting the promotion criteria.

Each leader discussed the policy from the parent and student perspective. One district leader recalled when a 3rd girl went to share with her mom at the end of the day that she had met the promotion criteria and would go to 4th grade. The girl shared with her mom with the biggest smile that, "I passed." Her mom was crying and I mean, that pressure was so real for them. Another building leader noted, “A little guy after he passed, he came down to the office and said, "I get to go to fourth grade!" We didn't realize that he even knew he could be retained.” All the leaders, accept for one at the district level, noted the anxiety experienced by the students and parents. All leaders did note that the teachers intentional sought balance between “insulating” the students from the “punitive,” while also communicating the seriousness of the assessment. The pressure and anxiety noted for students extended to the teachers. The building leaders shared that many teachers in third grade elected to teach other grades to avoid this. We have seen an exodus of teachers out of the third grade over the years. Teachers didn't want to be a part of that accountability at that grade level. They didn’t want the added pressure. The costs were too high. The administrators all expressed that the goal was to ensure all students met promotion. This decision guided the use of their resources. Many of the programming options were proactive, but they also put targeted options into place for the last-minute push in 3rd grade. The leadership team had problem solved and discussed how to ensure all students were developing proficient literacy skills. The team had discussed changing the diagnostic measure used or the cut score to either cast the net wider or make it more stringent to be more targeted. The team determined the system of response to intervention in place was doing its job and focused on refining the system and ensuring the interventions in place were producing results. The district leaders repeatedly articulated that they were already doing what was best for student learning and literacy development. However, when questioned about the accountability measure, the K3 Literacy grade, they all expressed their discontent of the measure. Each leader spoke of the discrepancy in the information collected making the measure invalid and unstandardized. One leader stated, “You want to really compare apples, I mean the report card is, in a sense, a tool for families to compare districts and to compare buildings. However, the parameters in which the system is built are unstandardized. They're not comparing apples to apples and the state is giving a false impression that they are.” Another leader said, “The K3 Literacy measure frankly on the report card is misleading to the public.” One district leader stated that districts are “gaming the system” when asked a follow up question regarding the NR received by numerous districts who have less than 5% of their kindergarten students assessed as off track on the fall diagnostic. The district leader went on to state that their district would be receiving a NR this year based on a decision made to change the diagnostic measure utilized for kindergarten students. This same leader acknowledged that not all districts would have this option due to their at-risk population, which is supported by the correlations between the NR designation and %SWD (-.190, -.220, -.196), % ED (-.261, -.260, -.193), and % minority (-.148, -.111, -.147). The sentiment shared by all those interviewed was that this measure lacked the standardized measures to make it valid. The concern related to the K3 Literacy grade

was not a focus of the district level team specifically. The notes did identify a review of which students must be on RIMPS and whether they should include more students to potentially improve the outcome of K3 Literacy measure. The team did repeatedly discuss diagnostics and cut scores. They reviewed the options based on their roles as functionaries and doing what was right. In the end, the team determined that the policy allowed for the designation and clearly indicated how it was assigned. They changed the measure, which is allowable by the policy, resulting in the NR designation.

Chapter 5: Discussion

This study reveals the outcomes of Ohio’s Third Grade Reading Guarantee as measured by the proficiency trends and accountability measures outlined by the policymakers and for one LEA at the local level of enactment during the first three full years of implementation, SY 13.14, SY 14.15, and SY 15.16. The policymaker perspective, a macro level of analysis, illustrates the inconsistencies and gaming in the policy, leading to districts grappling with ethical decision making, doing what is right and doing what is good, expanding the existing evidence of outcomes related to literacy policies (National Conference of State Legislatures, 2016; Roderick and Nagaoka, 2005; Greene and Winters, 2007,2012; Franco, 2015). State-level data reveals a strong correlation between lower performance on the district accountability measure, K3 Literacy, and higher percentage of subgroups within the district. Further, the state-level data indicates a strong correlation between higher rates of retentions in districts with higher percentages of subgroups. This data is alarming and cause for policymakers and educators to come together to reimagine the policy.

The data and evidence from the LEA during implementation provides a lens on three initial years of full implementation of TGRG, adding a new dimension to the body of research related to implementation of policy and the impact of decision making (Heilig and Darling-Hammond, 2008; Strike, 2008; Booher-Jennings, 2005; Larsen and Akmal, 2007). The LEA’s initial cohort of third grade students subjected to the consequences of TGRG all met the promotion requirements established by the policy. All the students in the cohort who were determined off-track as evident by the fall third grade diagnostic were found to be off-track as fourth graders as evident by the fall fourth grade diagnostic despite meeting the requirements for promotion. If policy fails to deliver outcomes, despite intent, those implementing are increasingly likely to engage in ethical decision making to try and reconcile the disconnect (Cohen and Moffitt, 2009).

Personal Narrative I had the unique opportunity to interview several administrators from one LEA on their approach to implementing this policy, ensuring compliance and seeking to integrate into existing systems to improve student learning and growth. The access to the LEA, leaders, and notes documenting implementation provide a detailed account bringing context and depth of analysis to the study. Further, I share my personal reflection to add an additional element of insight.

I served on the literacy leadership team for this LEA for two years, SY 14.15 and SY 15.16., in a district level administrative role focused on intervention services. During this time, I maintained and updated the guidance document based on the policy mandates from Ohio Department of Education. I communicated and sought guidance from ODE to guide our team’s ethical decision making. Our focus was doing what was right, based on principles and shared understanding, which was for no students to be retained. Our aim, derived by the evidence of the negative outcomes identified in the research for students retained, guiding decision making.

As a team, which other interviewees referenced, we struggled with decisions around a diagnostic tool, cut scores for on- and off- track determinations, whom to place on RIMPs, how to communicate with parents, and how to allocate of resources to ensure compliance, but more important, we worked together on how to ensure that our students were gaining the critical literacy skills. We worked to ensure the decisions made were supported by evidence, the ends were the right ends to aim for, implementation was done morally achieved legitimately (Strike, 2008).

There were a few specific ethical decisions leading me to this work. During my initial year, SY 14.15, our team discussed whether we would be best served altering our cut score to determine on- and off- track in kindergarten to ensure our interventions began as early as possible. This conversation was driven by flexibility of the guidance by ODE in selecting the diagnostic and cut scores, as well as the K3 Literacy grade from the previous year, which was a C. The K3 Literacy grade was of great concern to our district. We knew our community would adequate this measure with student learning and literacy. However, we felt that this measure was did not provide such an indication and that it merely reflected student performance on two or three single measures. We were perplexed as we had no students retained from SY 13.14 with all students meeting the promotion criteria either through exemption, high-stakes assessment, or alternative assessment. Our focus remained on early intervention and increasing literacy skill development in our students with the use of the grant dollars we were awarded.

Our team spent time reviewing the calculation used for K3 and quickly determined it was a numbers game, and this was the first time the team was aware of the NR, a designation for LEAs with less than 5% of their kindergarten students being assessed off-track. Our team decided it was in our best interest to cast our net wide and provide the most targeted interventions early in kindergarten. We felt this decision was in the best interest of students and had the potential to improve our K3 Literacy grade. However, as we received our K3 Literacy grade for SY 14.15, a D despite no students being retained, we determined the decision to cast our net wide was not of benefit to our students, nor our report card.

The summer between SY 15.16 and SY 16.17, after receiving a D again with no retentions, our team discussion began to consider reviewing the kindergarten diagnostic with the thought of aligning with an NR designation with less than 5% of our kindergarteners being assessed off-track. Through continued discourse and processing, the team made the ethical decision to focus on the early intervention programming to ensure all our students met promotion requirements and not the accountability measure. I categorize the decision as ethical, as it meets the parameters set forth by Strike (2008) in doing what is “good” and what is “right”. The “good” was the decision was student centered, as it would remove the distraction of the K3 Literacy measure, which held no value to the district due to its lack of validity. The “right” was the decision remained in alignment with our duties as functionaries to carry out the policy with fidelity.

I have confidence our nation agrees literacy is vital for success and the intent of Ohio’s TGRG and other states’ literacy policies are spot on in their intent. The research

clearly indicates the significance of reading proficiency in third grade (Hernandez, 2011; Feister & Smith, 2010). The research also indicates the negative, sometimes devastating impact retention has on students (Jimerson & Renshaw, 2012; Anderson, Jimerson, & Whipple, 2005; Roderick, 1994; Bloom, 2012; Hattie, 2009). The fact that this policy incorporates both is paradoxical and truly limiting its ability to create the change intended. During an interview with of one the LEA administrators, she noted that the policy had the potential to disrupt, sometimes to the detriment of students, the response to intervention (RTI) process.

I think the other thing is the state reporting system is a huge challenge. I don't think it is an accurate way to report the third grade guarantee. It documents what kids come on and off of RIMPs, reading improvement plan. The system wants you to take students off the interventions and if you look at a clear RTI process, you shouldn't be rewarded for taking students off if they truly need that. We all know that some students may need an intervention through their whole career and that's okay. They may be on a tier two plan, so trying to ding a school district for taking, you know, for keeping a student on an intervention when that is their need is counterproductive. It's not the RTI model, it's not a research-based model. We've just chosen to, you know, we do what's good for kids, and if the state wants to have a reporting feature that doesn't match research, then that's their choice.

The LEA team agreed they would remain committed to the evidence-based model of Response to Intervention and attempt to find other avenues to minimize the impact on its K3 Literacy grade. The focus would continue to be “what’s good for kids” and not allowing the distraction of the accountability measure. The LEA adjusted during the 16.17 school year and received an NR and truly did not concern themselves with the accountability component. This is a critical issue needing to be resolved. Not all LEAs have a student population allowing for the NR designation, even with the various measures allowable for the diagnostic. Districts with high percentages of students in subgroups at-risk for literacy deficits do not have the ability to game the numbers, even with the variability in tools and control of the cut scores, to meet the criteria for NR.

As a new administrator in K-12, Ohio’s Third Grade Reading Guarantee challenged me, and caused me a high level of frustration. The various compliance factors, including the reading diagnostic, the math and writing diagnostic, and the Reading Improvement and Monitoring Plan, required what I perceived to be tedious, detailed tracking that was not informing instruction or practice leading to student growth in literacy skill development. In fact, it was counter to our systems in place. In addition, the tracking of students who met criteria for promotion, but not proficiency- required the development of mid-year RIMPs, as well as documenting those who were removed from RIMPs because they met proficiency- was additional work and reporting. Each of these components required parent correspondence. Each year the criteria for promotion changed requiring all related correspondence to families to be updated, as well as the guidance document. In the end, compliance was maintained, but we were jumping through hoops.

I do acknowledge and believe in the importance of policy focusing on literacy. TGRG refocused Ohio’s LEAs on early literacy. It inspired me to see the innovative ways we were partnering with community and families to develop literacy awareness and experiences for our students. However, it also caused me great stress when we were meeting about individual students at risk of retention. The stories of students crying or expressing their anxiety over an assessment with such significant implications caused me great concern. We owe it to our children in Ohio and many other states and cities to do better. The following sections address implications and recommendations informed by the findings of this study.

Implications for Policy Makers and Educational Leaders The implications of these findings are critical for both policy makers and educational leaders to consider. TGRG has evolved through minor changes since becoming law and alterations in policy are critical to ensure they are meeting the intended aim. As such, policymakers and educational leaders must take research and feedback on the policy seeking to strengthen, to remove barriers, and increase outcomes for all. More important, as noted earlier, literacy policies are increasing in prevalence nationwide. Seeking a more comprehensive understanding of variance in these policies and the outcomes achieved short term and long-term. Literacy, a civil right, is a science and requires evidence-based direct instruction to reach proficiency. There is vast research on the development of literacy skills in young children and yet we still struggle to ensure all students have reached this proficient level. More must be done to as a nation to crack this code to literacy for all our students, regardless of demographics. It may require us to reimagine our structure and standards for early education.

Another significant implication for educational leaders is the reallocation of resources. The LEA struggled with how to target our interventions. During this decision-making process, they discussed what was in the best interest of students and ultimately circled back to ensuring no students were retained. This decision process aligned with the four characteristics of good decision making outlined by Strike (2008). It involved supporting with evidence (research on impact of retention), ensuring the ends aimed at by the decision were the aims that should be aimed at (preventing retention), it was moral, and achieved through legitimate means (compliance with the policy). As a result, the afterschool interventions focused on students still at-risk for retention, excluding other groups that would benefit, essentially focusing resources on the bubble kids, or the students just below proficient, (Booher-Jennings, 2005) to ensure no student was faced with retention. Similar trends were found in the Texas literacy policy with bubble kids classification dictating instructional practices and resource allocation with this target population, while the hopeless cases were removed or minimized through referral for special education (2005). Another area policymakers and educational leaders must consider is the impact of exemptions on students. The Ohio Department of Education allows for exemptions (see Table 3.0: Exemptions from retention under TGRG) for students at high risk for retention due to factors posing a significant challenge at that time to meet requirements 52

for promotion. While this intent is understandable, the unintended consequences have the potential to be detrimental to these same students. One way this can be detrimental is through an increased number of students being identified with disabilities to avoid retention through exemptions. This has become such a significant concern, it was mentioned by the Ohio Department of Education as an area they are monitoring at the annual Federal Programs Conference and the annual Special Education Leadership Conference. LEAs are faced with the ethical decision of retention or evaluating for an educational disability, when that student may be benefiting from intervention, but not there yet.

Implications for Students The implications of the findings for our students in Ohio is grave, requiring the most in-depth and careful consideration to ensure we are focusing on the right things in the right way, making the most ethical decision for our future generation. To do anything less would be criminal. However, the findings and research suggest that we are not actively engaging in advocacy to ensure all our students have their needs met in a way that maximizes their future opportunities.

During the first three years of full implementation, Ohio retained 19,952 students. We now have 19,952 students who are experiencing one of the most stressful events of their life (Anderson, Jimerson, & Whipple, 2005). We now have 19,952 students who are at an increased risk of future failure (Hernandez, 2011; Feister & Smith, 2010). These 19,952 students are only those retained in Ohio. How many more students in our nation have been retained as a result of literacy policy and now facing future risk? According to the findings of this study, there is also disproportionality in the demographics of those retained, which must be addressed. In addition, based on the findings, despite the small sampling current criteria to meet promotion does not equate to proficiency in grade 3 or later grades. The primary intended consequence of the policy, all students reading proficiently by end of third grade, has not come to fruition.

The building level leaders interviewed identified the need to buffer the students from the potential consequence. However, they also noted that their efforts were not always successful and students experienced anxiety.

It wasn't just six months of anxiety, it really was multiple years for some families of anxiety of what's going to happen when my student gets to third grade.

I think it put a lot of unnecessary stress on some children who were maybe less equipped to deal with that stress. Some little guy after he passed, he came down to the office and said, "I get to go to fourth grade!" We didn't realize that he even knew, but he did. I guess, he must've heard it from his parents or something. He went to the office and said, "Guess what? I get to go to fourth, I passed. Did you hear? I passed the test."

This anxiety (an obvious precursor to our student’s relief in passing), as well as the fact that we are looking at one measure on one day, needs to be addressed through policy revision. Continuing to implement the policy in its current state, while it is increasing anxiety in students, families, and educators is ill-advised. More needs to be done by all stake-holders to improve the policy to reach its aims without undue harm.

Political Gaming Cleary TGRG was not a novel idea to improve outcomes for students. The trend to develop literacy policy is evident nationwide. Currently, 17 states and multiple cities including Washington, D.C., Chicago, and New York, have or have passed literacy policies that require (or will require) retention when the students are unable to demonstrate proficiency in reading by the end of third grade, while an additional 10 states do not require but do allow for retention (National Conference of State Legislatures, 2016). Legislatures have much to gain politically from recommending such policy. Who would argue against universal third grade literacy as an aim for our education system? The aim of the policy has overwhelming support, as literacy is tied to greater opportunities and outcomes for students. The state politicians are able to boast their commitment to literacy and opportunity. They identify these policies as a way to hold the public education system accountability for the develop of literacy skills for all third graders. But their promises of improved literacy rates are often falling short because of their failure to consider the implications of implementation and the unintended consequences for some students.

The dissonance between the political aims and the implementation are great. While all public schools would likely identify literacy as a focus, the compliance elements of such policies often disrupt districts’ practices with little to no guidance. In addition, LEAs are forced to make decisions regarding the allocation of funds. The unintended consequences of many of these literacy policies is often overshadowed by this grand notion of literacy for all third graders. However, the truth is that it is not all third graders. Third graders who receive exemptions due to an educational disability, previous retention, or being an English Language Learner are not encompassed in this grand notion. This accounts for approximately 5.6% of all third graders attending public schools, a total of approximately 21,383 students in the first 3 years of implementation. This 5.6% of students are at-risk for not receiving the same access to resources and instruction. Booher-Jennings (2005) illuminates the unintended consequences of accountability measures on K-12 education, specifically how they can perpetuate or even exacerbate the educational experience of marginalized populations including students with disabilities, students living in poverty, and students of color. In addition, there is an added pressure experienced by teachers to ensure no student is faced with retention. This pressure is felt not only by teachers and administrators, but by families and students. This pressure that can influence ethical decision making. What is good? What is right?

The Aim The aim of this and other literacy policies is for students to learn to read at a proficient level. TGRG shares this aim, but also allows districts to game the accountability measure through imprecise definitions, variability in measurement, and exemptions. Heilig and Darling-Hammond (2008) found LEAs gamed the system through various methods including: Exclusion of students’ scores through special education, retention, failing to code drop outs, exclusion of low-performing students from open enrollment, and encouragement of transitioning to GED program. Districts may engage in this gaming for a variety of reasons, some ethical and some unethical. For the LEA in this study, the decision was based on what was “right” as functionaries carrying out the policy legislated, while still considering what was “good” for the students in terms of taking measures to ensure to minimize or avoid student retention. The LEA had great concern for making a determination of promotion or retention based on 2 or 3 testing opportunities versus looking at the student growth and development in the area of literacy. In addition, the district did what was “right” after several years of focusing on understanding the accountability measure, which they finally deemed to be a distraction and a measure with such discrepancy it miscommunicated the great work being done for students to develop literacy skills. The LEA felt it was right and good to remove this distraction, in an ethical decision that aligned with policy compliance, in order to focus on doing what was “good” for the students of the district. The gaming does not end with the accountability measure. Another area susceptible to gaming is with the exemptions. The criteria for exemptions under ELL are very specific in the language, meant minimize any opportunity for gaming. However, exemptions for previously retained and individuals with educational disabilities have language which is vague. Students previously retained can be exempted if they previously received intensive remediation as well. ▪ A student who has received intensive remediation in reading for two years and was previously retained in any of grades kindergarten through grade 3.** **A student that is promoted under this exemption must continue to receive intensive reading instruction in grade 4. This instruction shall include an altered instructional day that includes specialized, diagnostic information and specific, research-based reading strategies for the student that have been successful in improving reading among low-performing readers (ODE, 2017)

The description or criteria for intense intervention is not provided. Students with disabilities can be exempted through the student’s IEP. ODE offers guidance, but educators, including those interviewed for this study, identify a pressure to avoid student retention due to the negative implications of retention. In addition, districts may evaluate students in the response to intervention (RTI) process, receiving intervention to develop a skill to develop proficiency, differently moving to evaluation in order to exempt the students from retention. Districts are facing these decisions and trying to determine what is ethically good and right.

Policy Recommendations Research on education policy must go beyond the evaluation of the intended outcomes. Evaluating policy through mixed methods has the potential to truly informing policymakers on intended and unintended consequences. This requires educational leaders to be more vocal and vulnerable in order to advocate for the students they have committed to educate. K-12 Educational Leaders cannot be complacent in policy implementation. Their voice and expertise as practitioners is necessary in policy. The following are specific recommendations for TGRG based on the findings of this research and related research cited. These recommendations potentially have relevance to other literacy policies in other states and cities and should be considered.

Remove Retention Why retention? That is the question. Most of the discussion surrounding literacy policy and retention is the readers, noting the research that we cannot allow our students to leave third grade without being proficient readers note the research on the connection between third grade reading proficiency and future success and failure (Hernandez, 2011, Roderick, 1994). While there is evidence that students experience some short-term academic gains when retained, the evidence of short-term social emotional gains and/or long-term gains in either academic or social emotional growth is substantially limited (Greene & Winters, 2007, Greene & Winters, 2012). More important, the literature identifies retention as stressful event comparable for some students to the loss of a parent (Anderson, Jimerson, & Whipple, 2005).

There are other state literacy policies that do not include the consequence of retention. Many of these states have alternative routes to demonstrate literacy skills, including portfolios. As mentioned previously, Massachusetts has the highest fourth grade reading proficiency, and its state reading policy does not include retention (Rose & Schimke, 2012). Massachusetts took another approach to their literacy policy by first focusing on the barriers to the development of literacy skills in their young children (Cook & Bornfruend, 2015). In 2012, Governor Deval Patrick approved a literacy policy focusing on comprehensive language-rich curricula, effective instructional practices, professional development and training, developmentally-appropriate assessment, and family partnership strategies (Cook & Bornfruend, 2015). The state went beyond the language of the policy to also secure funding through grants recognizing the task of ensuring third grade reading proficiency for students required investment (2015). Much of the work being done in Massachusetts begins prior to a child entering kindergarten. We could gain much insight through collaboration of states, sharing and reflecting on literacy policies to guide curriculum and instructional practices focusing on literacy skills. This discussion and reflection needs to focus on subgroups disproportionately facing retention. Through research, reflection, and collaboration we can inform policy and truly meet the aim of having all third grade students reading proficiently in our country.

All Learners Exemptions need to be removed from the literacy policy. Alternative pathways or criteria to demonstrate mastery and/or show growth in literacy skills need to be constructed. This shows the continued focus on all learners, while still acknowledging that the rate of skill development and/or level of mastery is influenced by other factors. This removes the pressure of districts to make determination and/or initial identifications of students with disabilities. Alternative pathways would also align with the RTI process many LEAs implement to ensure students’ individual needs are met. Removing exemptions also increases the likelihood that all students, in all subgroups, receive the benefit of resources allocated to focus on early literacy skill development. Removing retention and exemptions from TGRG would reframe the policy to align with the aim and language of the policy, which is that all students develop literacy skills to proficiency without the potential of the detrimental impact of retention. It would reduce the likelihood that educators have the pressure to identify students with educational disabilities in order to prevent them from being retained as noted by Roderick and Nagaoka (2005).

Targeted Application One of the leaders interviewed suggested the plausibility of the policy only applying to districts struggling with literacy as measured by student performance on the state assessment. The policy could be revised to exempt districts from being held to compliance factors of the policy if they are able to meet a proficiency rate on the state reading assessment. This would honor the LEAs who have solid core instruction on literacy and a system of response to intervention for students who struggle. They would not be held to compliance indicators that have the potential to distract from their effective systems.

Closing Thoughts K-12 Education policy has significant implications and has the potential to improve our public education system and larger community. Engaging in policy research allows the policy-makers to determine if the aims meet the outcomes. However, the research must extend beyond the identified outcomes and explore the potential unintended consequences. This research must be done by evaluating the intended outcomes, typically the quantitative side of research, as well as the unintended outcomes, typically the more micro level analysis through the qualitative approach to research. One example would be to expand upon the work of Larsen and Akmal (2007) to better understand the ethical dissonance experienced by classroom teachers as related to literacy policies with retention as a consequence for third graders. Full analysis allows more informed modifications to the policy and/or the implementation to maximize the outcomes for all learners.

In addition, education leaders must be more active in policy. They serve to inform policymakers on the logistical framework necessary for educational policy to materialize and achieve the intended goal. It is critical for all those informing education policy, to be cognizant of inclusive practices, while acknowledging not all students will demonstrate skill or learning in a similar way. Educational leaders have the challenge of implementing with fidelity the policy mandates from legislators. This task must be guided through ethical decision making and ensuring leaders are acting in a way that is “right” and “good” as identified by Strike (2008).

In an age of accountability where a student’s future is determined by one measure on one day and a teacher’s effectiveness is determined by this same measure, the stakes are high. Education has become more legislated, requiring leaders to be informed and compliant, while negotiating implementation. The challenge is the dissonance of policy mandates and research of what is best for student learning and growth, leaving leaders to engage in this ethical decision making process. Leaders must have a framework to guide this ethical decision making process, always striving to do what is “right” and “good”.

Appendix A: Reading Competencies

Appendix B: 2014-2015 Third Grade Reading Guarantee

Promotion Percentage

2014-2015 Technical Documentation – Third Grade Reading Guarantee Promotion Percent March 10, 2016 Background Ohio's Third Grade Reading Guarantee is a program to identify students from kindergarten through grade 3 that are behind in reading. Schools provide help and support to make sure students are on track for reading success by the end of third grade. Students take diagnostic assessments each year from kindergarten through third grade. Students who score “not on track” with their reading skills in any grade between K-3 are required to be placed on a Reading Improvement and Monitoring Plan (RIMP) where they receive intensive remediation to help improve their reading skills. Schools must create the RIMP within 60 days of when a student is designated not on track with his or her reading skills. A plan must include: Identification of the student’s specific reading deficiency; A description of proposed supplemental instruction services that will target the student’s identified reading deficiencies; Opportunities for the student’s parents or guardians to be involved in the instructional services; A process to monitor the implementation of the student’s instructional services; A reading curriculum during regular school hours that assists students to read at grade level and provides for reliable tests and ongoing analysis of each student’s reading progress; and A statement that unless the student attains the appropriate level of reading competency by the end of grade 3, the student will be retained, unless otherwise exempt. Schools also may electively put students who are identified as on track on reading improvement and monitoring plans if there is evidence of a reading deficiency and the decision is that these students need help with reading. If a school electively places an on- track student on a reading improvement and monitoring plan, all the requirements of a reading improvement and monitoring plan must be adhered to, including reporting requirements. Third Grade Reading Guarantee Promotion Percentage Calculation If a student is not on track with reading by the end of third grade, he or she might be retained in 3rd grade. The Third Grade Reading Guarantee (TGRG) Promotion Percentage is a statistic to report what percentage of students met the bar through some means to be promoted to the fourth grade. The details below will explain how a student can be promoted and how the TGRG Promotion Percentage is calculated. In addition, some students are exempt from retention and they are not included in this calculation. The rules outlined below also will provide details on which students are exempt from retention and how they are removed from the TGRG Promotion calculation. Building the Denominator

In order to determine the percentage of students promoted, we first have to identify who is included in the calculation and who is removed. This pool of “included” students is the denominator from which the promotion percentage comes. The calculation starts with any student who is reported with a grade of 3 for the 2014-15 school year, but then some students are filtered out and are excluded from the calculation. This section of the technical documentation explains how the filters are applied so that a final denominator is created. The next section will explain all of the students who are included in the denominator. 2014-2015 Third Grade Reading Guarantee Promotion Percentage Building the Denominator - Students Excluded from the Calculation There are two initial filters that are used to begin building the denominator. The first step is to use the agency’s general accountability rules (see the Technical Documentation on the Where Kids Count for more information on how accountability is determined) to develop a pool of 3rd grade students for which a school or LEA is held accountable. If a student does not meet the Where Kids Count rules for accountability, they are removed from the calculation. The second step filters students for whom a school or LEA is otherwise accountable but who withdraw before the end of the school year. In the 2014-15 school year, a school or LEA is held accountable for a student who is continuously enrolled from October 10, 2014 through April 13, 2015. Thus, an accountable student who withdraws sometime after April 13th, but before the end of the 2014-15 school year is removed. Once these two ‘general’ filters are applied, some additional filters are used to remove students who are exempt from retention. State law says that some students are exempt from retention even if they are struggling with their reading skills. We remove these students from the denominator of the calculation – so they have no effect on the final promotion percentage. Students who are exempt from retention include: Limited English proficient students enrolled in U.S. schools for less than three full school years and with less than three years of instruction in an English as a second language program. These students are reported with a Retained Status Element (FN070) of “D”; Special education students whose IEPs specifically exempt them from retention under the Third Grade Reading Guarantee. These students are reported with a Retained Status Element (FN070) of “E”; A student whose IEP shows that the student has received intensive remediation in reading for two years, and the student was previously retained in any of grades kindergarten through grade 3. These students are reported with a Retained Status Element (FN070) of “F”; Any student who has received intensive remediation for two years and was previously retained in any of grades kindergarten through grade 3 (note that a student that is promoted under this exemption must continue to receive intensive reading instruction in grade four. This instruction shall include an altered instructional day that includes specialized diagnostic information and specific research-based reading strategies for the student that have been successful in improving reading among low-performing readers). These students are reported with a Retained Status Element (FN070) of “F”. Any student who has a severe cognitive disability. These students are reported with a Retained Status Element (FN070) of “G”.

Another filter removes repeat 3rd graders during their second/subsequent year(s) in the 3rd grade so that each student is counted in only one year’s promotion calculation. For example, if a student was in the 3rd grade for the first time during the 2013-14 school year, he was included in that year’s (2014) promotion calculation. If he remains in the 3rd grade, the student will be removed from any subsequent calculations (2015 and later). A final filter removes from the calculation those students who were not reported with a severe cognitive disability (no Retained Status Element of “G” as noted in the bullet immediately above), but who took the alternate assessment for students with significant cognitive disabilities (AASCD). This test type is noted in the IEP Test Type Option (GE160) of “ALT”. Building the Denominator - Students Included in the Calculation The remaining students, once the filters explained above are applied, are in the denominator. It is important to understand that the codes in the FN070 Retained Status Element are used to identify which students are exempt from retention. If a LEA fails to report one of these elements for a student who is exempt, the agency has no other means to establish exemption eligibility and the student will be included in the denominator. Numerator of the Calculation Once the denominator is created, we then look at which students meet the criteria to be promoted to the 4th grade. These students are in the numerator. Listed below are the ways in which a “not-exempt” student meets the Third Grade Reading Guarantee and is eligible for promotion. Ohio Achievement Assessment (OAA) Promotion Score When students become third graders they take Ohio’s state assessment in reading and the scale score they receive is one criteria that helps determine if the student can be promoted. For the 2014-15 school year the students will take the “old” Ohio Achievement Assessment or OAA. In order to be promoted to the 4th grade they must reach the required “promotion score” on that test. For the 2014-15 school year, third grade students will have the opportunity to take the Ohio Achievement Assessment three times; once in October 2014, once in April/May 2015 and once in the summer of 2015 soon after the school year ended. In order to be promoted to the 4th grade a student must receive a scale score of 394** or higher on this assessment. A student who reaches this threshold on any of the three assessments (fall, spring or summer) will be in the numerator of the third grade reading guarantee promotion percentage. **Note that this is slightly lower than the score needed to be deemed “proficient” on this assessment. A score of 400 is needed to reach the Proficient range on the 3rd grade OAA, but the state board of education set a score of 394 as the minimum needed for promotion to the 4th grade. Alternative Reading Assessments The Alternative Assessment is a standardized assessment(s) for reading determined by the Ohio Department of Education for the Third Grade Reading Guarantee. It allows a third grade student to take and demonstrate an acceptable level of reading performance for promotion to the fourth grade.

The alternative reading assessments are optional for LEAs to administer and are in addition to the administration of the state reading assessment. The LEA is responsible for the cost of the alternative assessment. LEAs may administer these alternative assessments twice during the school year and once during the summer to mimic Ohio’s 3rd grade reading test schedule. LEAs and schools should set dates for the administration of these assessments. For 2014-2015, the following assessments can be used for promotion to the 4th grade. Test Promotion Score

Iowa Assessments 170

Northwest Evaluation Association (NWEA) Measure of Academic 184 Progress (MAP

Terra Nova 3 596

Appendix C: Interview Protocol

1. What was the initial response by the LEA to Ohio’s Third Grade Reading Guarantee? 2. Explain your role in the LEA and specifically in terms of implementation of TGRG? 3. How did your team determine how to utilize resources to support implementation? 4. Did any subgroup of students cause a particular challenge to the decision made when determining how to implement the requirements of TGRG? 5. Has the allocation of resources to support TGRG evolved or changed following the initial financial support of the grant? 6. How did you team communicate your implementation and the implications of TGRG to families? 7. What support did you receive from outside the LEA during implementation? 8. What would you identify as the greatest challenge? 9. What has been the greatest benefit? 10. Have you as an individual professional struggled with any of this work related to TGRG: a. Retention b. Exemptions c. Redistribution of resources 11. What was the response or conversation with various stakeholders regarding TGRG? a. Board of Education b. Principals c. Teachers d. Families e. Students 12. Can you identify any specific cases related to this policy that caused you concern or pause? Please describe.

Appendix D: TGRG and Students with Disabilities

Students with Disabilities MULTI-TIERED SYSTEM OF SUPPORT - FOCUS: STUDENTS WITH DISABILITIES In a multi-tiered system of support, every student has access to Tier 1 universal instruction. A multi-tiered system of support presumes that every student will participate, including those with disabilities, in the core curriculum, which includes small- and large-group instruction. Intensive intervention and supports: Receiving intensive interventions and support does not automatically mean that a student will be evaluated for special education eligibility. Not all students receiving intensive supports are identified as students with disabilities. Additionally, not all students identified with disabilities need intensive supports in all areas.

REQUIREMENTS UNDER THE THIRD GRADE READING GUARANTEE STUDENTS WITH SIGNIFICANT COGNITIVE DISABILITIES: EXEMPT FROM THE PROVISIONS OF THE THIRD GRADE READING GUARANTEE Students with significant cognitive disabilities are exempt from the requirements outlined in the Third Grade Reading Guarantee. Students with the most significant cognitive disabilities are completing a substantially modified curriculum from the general education curriculum, according to an individualized education program (IEP). Students with significant cognitive disabilities are exempt from taking the reading diagnostic. As a result, these students are exempt from all the other provisions of the Third Grade Reading Guarantee such as reading improvement and monitoring plans and retention. A student’s IEP team should discuss and decide on this exemption annually. Each student’s IEP should address foundational skills for literacy. For example, the IEP should address the student’s mode of communication and communicative intent.

EXEMPT ONLY FROM THE RETENTION PROVISION OF THE THIRD GRADE READING GUARANTEE Not every student with an IEP should be exempt from the retention provision of the Third Grade Reading Guarantee; IEP teams may discuss this option. The IEP team should have adequate progress monitoring and reading achievement data to identify students who may be eligible to be exempt from retention before the spring grade 3 English language arts test. Even if a student is exempt from the retention provision, all of the remaining provisions of the Third Grade Reading Guarantee are still required for the student. Districts and schools should document a student’s evaluation and assessment data, including the required reading diagnostic reading results and previous interventions within the student’s Evaluation Team Report. Districts and schools should use caution to make sure that a student with an IEP does not receive less intensive reading interventions and supports than students without IEPs. Considerations for students who are exempt from the retention provision of the Third Grade Reading Guarantee: • The reading improvement and monitoring plan aligns and does not conflict with the child’s IEP throughout the kindergarten through grade 3 years. • Retention in third grade and promotion to grade 4 should be discussed as a part of each child’s yearly reading improvement and monitoring plan, as well as during the IEP meetings. • The IEP team should document supporting rationale within the present levels section of the IEP that clearly describes the data, discussion and progress monitoring through the reading improvement and monitoring plan. The team also should include rationale and decisions related to its consideration of a child’s exemption from retention. • If the IEP team decides to exempt a child from retention in grade 3 and the child will move on to grade 4, the IEP should include clear, targeted reading interventions and progress markers for grade 4

References

2011 State Profiles. (2014). Retrieved December 9, 2014, from

http://nieer.org/publications/2011-state-profiles

Anderson, G. E., Jimerson, S. R., & Whipple, A. D. (2005). Student ratings of stressful

experiences at home and school: Loss of a parent and grade retention as superlative

stressors. Journal of Applied School Psychology, 21(1), 1–20.

Bloom, M. (2012). Why a third gade reading guarantee on its own won’t help Ohio

children. Retrieved from http://stateimpact.npr.org/ohio/2012/04/30/why-the-third

grade-reading-guarantee-on-its-own-wont-help-ohio-children/

Bloom, M. (2012b, June 22). SB 316: Twelve changes coming to schools near you.

Retrieved from http://stateimpact.npr.org/ohio/2012/06/22/sb-316-twelve-

changes-coming-to-schools-near-you/

Booher-Jennings. (2005). Below the bubble: “Educational triage” and the Texas

accountability system. American Educational Research Journal 42(2), pp. 231-

268.

Brown, J., & Amos, D. (2013, June 24). Schools scurrying to meet 3rd-grade reading

guarantee. Retrieved December 9, 2014, from

http://archive.cincinnati.com/article/20130625/NEWS0102/306200010/Schools-

scurrying-meet-3rd-grade-reading-guarantee

Brozo, W. G. (2010). The role of content literacy in an effective RTI program.

Reading Teacher, 64(2), 147–150.

Chester, M. (2001, September 26). Memorandum: Fourth-grade Reading Guarantee.

Ohio Department of Education.

Cook, S., & Bornfruend, L. (2015). Starting young: Massachusetts birth-3rd grade

policies that support children’s literacy development. Retrieved December 5,

2017, from https://static.newamerica.org/attachments/11901-starting-

young/StartingYoung11.13.3ebe6fdcefde4d86b28717e2399119af.pdf

Feister, L. (2010). Early warning! Why reading by the end of third grade matters.

Retrieved December 8, 2014, from http://www.aecf.org/resources/early-warning-

why-reading-by-the-end-of-third grade-matters/

Fields, Reginald. (2012). Ohio Gov. John Kasich signs third grade reading guarantee bill

into law. The Plain Dealer. Available at:

http://www.cleveland.com/open/index.ssf/2012/06/ohio_gov_kasich_signs_third-

gr.html. Retrieved on May 9, 2016.

Gardner, D. P., & And Others. (1983). A nation at risk: The imperative for educational

reform. An open letter to the American people. A report to the nation and the

Secretary of Education. Superintendent of Documents, Government Printing Office,

Washington, DC 20402 (Stock No. Retrieved from

https://eric.ed.gov/?id=ED226006

Gilchrist, Shannon. (2015). 3rd grade reading guarantee paying off. The Columbus

Dispatch. Available at:

http://www.dispatch.com/content/stories/local/2015/09/13/3rd-grade-reading-

guarantee-paying-off.html. Retrieved on May, 3, 2016.

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to

achievement. Routledge.

Heilig, J. V., & Darling-Hammond, L. (2008). Accountability Texas-style: The progress

and learning of urban minority students in a high-stakes testing context. Educational

Evaluation and Policy Analysis, 30(2), 75–110.

Hernandez, Donald. (2011). Double jeopardy: How third grade reading skills and

poverty influence high school graduation. Baltimore, Maryland: The Annie E.

Casey Foundation.

Honig, M.I. (2003). Building policy from practice: District central office administrators’

roles and capacity for implementing collaborative education policy. Educational

Administration Quarterly, 39(3), 292-338.

Honig, M.I. (2009). No small thing: School district central office bureaucracies and the

implementation of new small autonomous schools initiatives. American

Educational Research Journal, 46(2): pp. 387-422.

Huck, S. W. (2011). Reading statistics and research (6 edition). Pearson. Kindle

Edition.

Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A

regression-discontinuity analysis. Review of Economics and Statistics, 86(1), 226–

244. https://doi.org/10.1162/003465304323023778

Jacob, B. A., & Lefgren, L. (2009). The effect of grade retention on high school

completion. American Economic Journal: Applied Economics, 1(3), 33–58.

Jimerson, S. R., Anderson, G. E., & Whipple, A. D. (2002). Winning the battle and

losing the war: Examining the relation between grade retention and dropping out

of high school. Psychology in the Schools, 39(4), 441–57.

Kaplan, Abraham. (1998). The conduct of inquiry: Methodology for behavioral science.

New Brunswick, NJ: Transaction Publishers.

Kildee, D. (1994, March 31). H.R.1804 - 103rd Congress (1993-1994): Goals 2000:

Educate America Act [webpage]. Retrieved December 7, 2017, from

https://www.congress.gov/bill/103rd-congress/house-bill/1804

Larsen, D. E., & Akmal, T. T. (2007). Making decisions in the dark: Disconnects

between retention research and middle-level practice. NASSP Bulletin, 91(1), 33–

56.

Literacy, N. I. for. (2008). Developing early literacy report of the National Early

Literacy Panel. National Institute for Literacy.

No Child Left Behind Act of 2001, P.L. 107-110, 20 U.S.C. § 6319 (2002). O’Donnell, Patrick. (2014). Third grade reading guarantee made just over 4 percent of

Ohio’s third grader repeat third grade. Available at:

www.cleveland.com/metro/index.ssf/2014/11/third_grade_reading_guarantee_2.h

tml. Retrieved on May 9, 2016.

Ohio Department of Education (2014). What’s changing in Ohio education? Available

at: https://education.ohio.gov/getattachment/Media/Press-Kits/TGRG.pdf.aspx

Retrieved on May 3, 2016.

Ohio Department of Education (2014). Component: K-3 literacy. Available at:

http://education.ohio.gov/getattachment/Topics/Data/Report-Card-Resources/K-3-

Literacy-Measure/K-3-Literacy-Measure.pdf.aspx. Retrieved on April 2, 2016.

Ohio Department of Education (2015). Performance level recommendations for Ohio’s

state tests. Available at:

http://education.ohio.gov/getattachment/Topics/Testing/State-Test-Updates-for-

2015_2016/September-2015/Performance-Level-Recommendations-for-Ohios-

State/performancelevels.pdf.aspx. Retrieved on May 3, 2016.

Ohio Department of Education (2016). Grade 3 English language arts performance level

indicators. Available at: http://oh.portal.airast.org/ocba/wp-

content/uploads/Ohio_G3_ELA_PLDs.pdf. Retrieved on May 3, 2016.

Ohio Department of Education (2010). Ohio’s new learning standards: English language

standards. Retrieved at: http://education.ohio.gov/getattachment/Topics/Ohio-s-

New-Learning-Standards/English/ELA-Standards.pdf.aspx

http://education.ohio.gov/getattachment/Topics/Testing/Ohio-Achievement-

Assessments/Ohio-Achievement-Tests-Performance-Level-Descriptors.pdf.aspx

Oliver, C. (1991). Strategic responses to institutional processes. Academy of

Management Review, 16, 145-179.

Partin, Emmy. (2011, November 1). Ohio needs a reading guarantee | The Thomas B.

Fordham Institute. Retrieved October 28, 2017, from

https://edexcellence.net/articles/ohio-needs-a-reading-guarantee

NCS Pearson, Inc. (2012). AIMSweb Technical Manual. Retrieved from

http://www.aimsweb.com/resources/research-articles-and-information

Quantz, R. (2015). Written lecture notes on standard discourse. Oxford, OH.

Quantz, R. (2015). Written lecture on methodology and discourse. Oxford, OH.

Roderick, M., & Nagaoka, J. (2005). Retention under Chicago’s high-stakes testing

program: Helpful, harmful, or harmless? Educational Evaluation and Policy

Analysis, 27(4), 309–340.

Rose, S., & Schimke, K. (2012). Third grade literacy policies: Identification,

intervention, retention. Education of the Commission of the States. Retrieved from

http://www.ecs.org/clearinghouse/01/01/54/10154.pdf

Rose, J. S., & And Others. (1983). A fresh look at the retention-promotion controversy.

Journal of School Psychology, 21(3), 201–11.

Services, U. S. D. of H. and H. (2000). Report of the National Reading Panel: Teaching

children to read: An evidence-based assessment of the scientific research

literature on reading and its implications for reading instruction. New York:

National Institute of Child Health & Human Development.

Strike, K. A. (2006). Ethical leadership in schools: Creating community in an

environment of accountability (1 edition). Thousand Oaks, Calif: Corwin.

Workman, Emily. (2014). Third grade reading policies. Education Commission of

States. Retrieved from http://www.ecs.org/clearinghouse/01/16/44/11644.pdf

Winters, M.A., & Greene, J.P. (2007). Revisiting grade retention: An Evaluation of

Florida’s test-based promotion policy. Education Finance and Policy, 2(4), pp.

319-340.

Winters, M. A., & Greene, J. P. (2012). The medium-run effects of Florida’s test-

based promotion policy. Education Finance and Policy, 7(3), 305–330.