SCHOOLING VOLUME 10, NUMBER 1, 2019 Readability of the STARR Test is Still Misaligned Susan Szabo, EdD Professor Texas A&M University-Commerce Commerce, TX Becky Barton Sinclair, PhD Associate Professor Texas A&M University-Commerce Commerce, TX Abstract This study examined the readability of the State of Texas Assessments of Academic Readiness (STAAR) Reading passages for 2018 for grades 3-8. These results where then compared to the authors’ first study on the topic, which found that the readability of the STAAR Reading passages were one-to-three years higher than the grade level for which they were written to assess (Szabo & Sinclair, 2012). This study found that some characteristics of the STAAR test had changed since 2012, but many of the reading passages were still misaligned for the targeted grade level. The term “accountability” has a plethora of meanings. However, in the context of public education the term has come to represent ideas that were first associated with the No Child Left Behind Act of 2001 (NCLB, 2002) and later with the College and Career Readiness Standards (Conley, 2010; 2012).These legislative policies have focused on student achievement as a way to produce better student learning. Additionally, the Every Student Succeeds Act (U.S. Department of Education, 2015) limited the federal role in defining state accountability (Greene & McShane, 2018).Texas uses the Texas Essential Knowledge Skills (TEKS) to direct student learning and the State of Texas Assessments of Academic Readiness (STAAR) test to determine if students are learning the intended curriculum. Purpose of the Study In this age of accountability, assessments have gained a negative reputation. However, criterion-referenced assessments are a valuable tool to help drive instruction and to help students to be successful learners (Sindelar, 2015). Assessments can be powerful in helping teachers plan instruction and in letting the students gauge if their learning is up to the state standards. As the STAAR test is a criterion-referenced test based on the TEKS, it is important to investigate why the passing rates are not higher. In 2012, it was found that the readability of the STAAR Reading (grades 3-8) passages were written one-to-three grade levels above the grade for which it was intended and that the 1 SCHOOLING 2___________________________________________________________________________________ questions were written at a higher level as they were either think-and-search questions or on-your- own questions (Szabo & Sinclair, 2012). However, this current study only focused on the quantitative dimensions of text complexity as determined by using various readability formulas. The following question guided our study: How has the readability of the STAAR Reading passages and total overall readability average changed over the past seven years? Readability and Readability Formulas Readability is the ease with which a text can be read and understood (Gunning, 1952). Readability determines if any given written text is written clearly and at a comprehensible level. There are both pros and cons toward the use of readability formulas. Both viewpoints have research to support their beliefs (Zmanian & Heydari, 2012). During the last century, many researchers (e.g. Dale & Chall, 1949; Flesch, 1948; Fry, 1968; McLaughlin, 1969) have addressed the issue of readability and how to calculate it as a way to make classrooms, the work place, and public communications more effective. The purpose of readability formulas is to determine the difficulty of the text so that the reader can determine if the reading material can be read without frustration (Begeny & Greene, 2014). This information helps authors convey complex ideas more clearly and more effectively towards their targeted audience. It also gives the reader advanced knowledge about the text, which may help in determining which book to check-out or to purchase (Zamanian & Heydari, 2012). However, readability formulas cannot tell if the target audience will understand the text, as they do not measure the context, the reader’s prior knowledge, or interest level in the topic or the cohesiveness of the text (Bailin & Grafstein, 2001; Bertram & Newman, 1981; Kirkwood & Wolfe, 1980; Zamanian & Heydari, 2012). Additionally, it was found that tinkering with the text to produce acceptable readability levels may make the text more difficult to understand (Rezaei, 2000). Nevertheless, today, various readability formulas are commonly used to determine the readability of government documents, educational materials for students, newspapers, magazines and popular literature (Begeny & Greene, 2014). Readability formulas are mathematical in nature and focus on different text features. These features include the number of words in a sentence, the percentage of high frequency words on predetermined grade level word lists, the number of multisyllabic or “hard” words, and/or the number of letters in the text (Bailin & Grafstein, 2001; Begeny & Greene, 2014). For this reason, several formulas should be used and averaged when determining the readability of a selection of text, to account for the differences in formula design (Szabo & Sinclair, 2012). Methodology Procedure In 2012, the researchers investigated the readability level of the STAAR Reading tests for grades 3-8 as well as the types of questions asked (Szabo & Sinclair, 2012). Another readability study on the STAAR Reading passages was done by Lopez and Pilgrim (2016) who found similar results. And in 2015, HB 743 required the Texas Education Agency (TEA) to modify the STAAR Reading passages reducing, the number of passages that had to be read and the number of questions SUSAN SZABO AND BECKY BARTON SINCLAIR ___________________________________________________________________________________3 that were answered (Huberty, 2015). These changes had third grade students reading four passages and answering 34 questions. Additionally, 2 questions were added to each grade level so that eighth grade students had 44 questions to answer about their reading passages. All students from 4-8 grades read six passages. Because of the previous study and the changes by TEA, this study only focused on the 2018 STAAR Reading passages to investigate what, if any, changes occurred in the readability of the assessment passages. First, all of the released STAAR Reading passages for grades 3-8 were downloaded from the Texas Education Agency website (2007-2019). The pdf documents were converted into word documents. All photos, graphics, directions and item questions were removed. Line by line editing was done to ensure that line numbers and page numbers did not appear and that there was consistent spacing between all words. Additionally, at each grade level of the STAAR Reading tests, one poem for students to read and interpret was included. However, these poetry passages were not included in this readability study, as the variations in format prevent accurate readability calculations (Fry, 1977). The reading passages were entered into three different online readability index calculators that linked the readability levels to grade levels. The first calculator used both the Fry and the Raygor readability formulas (ReadabilityFormulas, 2019a). The second calculator used the Flesch- Kincaid, the Coleman-Liau Index, the SMOG Index, the Automated Readability Index (ARI), and the Linsear Write (ReadabilityFormulas, 2019b). The third calculator used the Dale-Chall Formula (ReadabilityFormulas, 2019c). Thus, eight readability formulas were used in the calculation of readability. This approach was necessary, as more than one readability formula should be used to provide a more accurate grade-level indication (Szabo & Sinclair, 2012). After determining the readability of each of the grade-level reading passage, the results were added and averaged. These scores were then compared to the Reading Consensus Scores (RCS; ReadabilityFormulas, 2019b). It was found that the results were similar. Following are brief descriptions of the readability formulas used in this study. Flesh-Kincaid. Rudolph Flesch’s readability research (1948) made him an early authority in the field and he inspired additional readability formula variations and applications (Kincaid, Fishburne, Rogers, & Chissom, 1975). The formula looks at the number of words and sentence length per 100 words to determine a grade-level reading score. This readability formula was first used by the Department of Defense to determine the difficulty level of technical manuals and today is a standard function on all Microsoft Word products (Zamanian & Heydari, 2012). Coleman-Liau Index. This formula examines the number of letters in a word and the number of sentences per 300 words in the text. It was created for the U.S. Department of Education to calculate the readability of textbooks for schools (Coleman & Liau, 1975). SMOG Index. The SMOG, which was created by McLaughlin (1969), was first used to evaluate healthcare material. The formula counts every word with three or more syllables within 30 sentences. This formula is appropriate for fourth grade to college age readers. The Automated Readability Index (ARI). This formula was created to use computers to calculate the readability of text. The formula uses ratio representing the number of letters per word and the number of words per sentence (Kincaid et al., 1975). SCHOOLING 4___________________________________________________________________________________ Linsear Write Formula. This formula was developed