Popular Measurement 2
Total Page：16
File Type：pdf, Size：1020Kb
One Fish, Two Fish Ranch Measures Reading Best Benjamin D. Wright and A. Jackson Stenner Think of reading as the tree in Figure l . It has roots up nine different reading tests to prove the separate identities like oral comprehension and phonological awareness. As read- of his nine kinds. He gave his nine tests to hundreds of stu- ing ability grows, a trunk extends through grade school, high dents, analyzed their responses to prove his thesis, and reported school, and college branching at the top into specialized vo- that he had established nine kinds of reading . But when Louis cabularies. That single trunk is longer than many realize. It Thurstone reanalyzed Davis' data (1946), Thurstone showed grows quite straight and conclusively that Davis singular from first grade Figure 1 had no evidence of more through college. than one dimension of Reading has al- The Reading Tree reading . ways been the most re- searched topic in educa- Anchor Study - tion. There have been 1970s many studies of reading In the 1970s, ability, large and small, worry about national lit- local and national. When One eracy moved the U.S. gov the results of these stud- Dominant ernment to finance a na- ies are reviewed, one clear Factor tional Anchor Study Uae- picture emerges. Despite Defines get, 1973) . Fourteen dif- the 97 ways to test read- the Trunk ferent reading tests were ing ability, many decades administered to a great of empirical data docu- many children in order to ment definitively that no uncover the relationships Carrot, 1971 researcher has been able Bashaw-Rentz, 1975 among the 14 different to measure more than one DavWrhurstone, 1948 test scores Bormuth, 1988 . Millions of kind of reading ability KosNn-Zeno-Kosan,1973 dollars were spent. Thou- Rins1and, 1945 (Mitchell, 1985). This has Thorndike, 1952 sands of responses were proven true in spite ofin- Stenner, 1988 analyzed. The final report Zeno, 1995 tense interest in discover- Zwick, 1987 required 15,000 pages in ing diversity. Consider 30 volumes - just the three examples: the 1940s kind of document one Davis Study, the 1970s Anchor Study, and six 1980s and 1990s reads overnight, takes to school the next day and applies to ETS studies. teaching (Loret et al., 1974) . In reaction to this futility, and Davis - 1940s against a great deal ofproprietary resistance, Bashaw and Rentz Fred Davis went to a great deal of trouble to define were able to obtain a small grant to reanalyze the Anchor Study and operationalize nine kinds ofreading ability (1944). He made data (1975, 1977) . By applying new methods for constructing 34 POPULAR MEASUREMENT SPRING 1999 objective measurement (Wright and Stone, 1979), Bashaw and Later, when the various sets of ETS data were reana- Rentz were able to show that all 14 tests used in the Anchor lyzed by independent researchers, no evidence for three kinds Study - with all their different kinds of items, item authors, of reading measures could be found (Bernstein, & Teng, 1989; and publishers - could all be calibrated onto one linear "Na- Reder, Rock and Yamamoto, 1994; 1996; Salganik and Tal, tional Reference Scale" of reading ability. 1989; Zwick, 1987) . The correlations among ETS prose, docu- The essence of the Bashaw and Rentz results can be ment, and quantitative reading measures ranged from 0.89 to summarized on one easy-to-read page (1977) - a bit more 0.96. Thus, once again and in spite of strong proprietary and useful than 15,000 pages. Their one-page summary shows how theoretical interests in proving otherwise, nobody had suc- every raw score from the 14 Anchor Study reading tests can be ceeded in measuring more than one kind of reading ability. equated to one linear National Reference Scale. Their page Lexiles also shows that the scores of all 14 tests can be understood as Figure 2 is a reading ruler. Its Lexile units work just measuring the same kind ofreading on one common scale. The like the inches. The Lexile ruler is built out ofreadability theory, Bashaw and Rentz National Reference Scale is additional evi- school practice, and educational science. The Lexile scale is an dence that, so far, no more than one kind of reading ability has interval scale. It comes from a theoretical specification of a ever been measured. Unfortunately, their work had little effect readability unit that corresponds to the empirical calibrations on the course of U.S. education . The experts went right on ofreading test items. It is a readability ruler. And it is a reading claiming there must be more than one kind of reading - and ability ruler. sending teachers confusing messages as to what they were sup- Readability formulas are built out of abstract charac- posed to teach and how to do it. teristics of language. No attempt is made to identify what a ETS Studies - 1980s and 1990s word or sentence means. The idea is not new. The Athenian In the 1980s and Bar Association used read- 1990s, the Educational ability calculations to teach Figure 2 Testing Service (ETS) did lawyers to write briefs in a series of studies for the Educational Status 400 B.C. (Chall, 1988 ; U.S. government. ETS by Average Lexile Zakaluk and Samuels, (1990) insisted on three 1988) . According to the kinds of reading : prose 1400 L Athenians, the ability to reading, document reading, Senior read a passage was not the and quantitative reading. 1300 L , unior ability to interpretwhat the They built a separate test to Freshman passage was about . The measure each of these three 1200 L 12th ability to read was just the 11th kinds of reading -greatly 1100 L 10th ability to read. Talmudic increasing costs. Versions of 9th teachers who wanted to these tests were adminis- 1000 L School regularize their students' tered to samples of school Lexiles 7th studies, used readability children, prisoners, young 900 L stn Level measures to divide the To- adults, mature adults, and 5th rah readings intoequal por- senior citizens. ETS re- 800 L tions ofreading difficulty in ported three reading mea- 700 A.D. (Lorge) . Like the L 4th sures for each person and 700 Athenians, their concern claimed to have measured 600 L in doing this was not with three kinds of reading 3rd what a particular Torah (Kirsch &Jungeblut, 1986) . 500 L passage was about, but But reviewers noted that, rather the extent to which 2nd no matter which kind of 400 L passage readability bur- reading was chosen, there dened readers. were no differences in the 300 L In the twentieth century, results (Kirsch &Jungeblut, 200L "- 1 st every imaginable structural 1993, 1994 ; Reder, 1996; characteristic of a passage Zwick, 1987) . When the re- has been tested as a poten lationships among reading and age and ethnicity were analyzed, tial source for a readability measure: the number of letters and whether for prose, document, or quantitative reading, all con- syllables in a word; the number of sentences in a passage; sen- clusions came out the same. tence length; balances between pronouns and nouns, verbs and SPRING 1999 POPULAR MEASUREMENT 35 prepositions (Stenner, 1997) . The Lexile readability measure ing on student reading ability. Table 2 lists the correlations uses word familiarity and sentence length. between Lexile Readability and Basal Reader Order for the Lexile Accuracies eleven basal readers most used in the United States. Each se- Table 1 lists the correlations between readability mea- ries is built to mark out successive units of increasing reading sures from the ten most studied readability equations and stu- difficulty. Ginn has 53 units - from book 1 at the easiest to dent responses to different types of reading test items. The col- book 53 at the hardest . HBJ Eagle has 70 units. Teachers work umns ofTable 1 report on five item types: their students through these series from start to finish. Table 2 Lexile Slices ; Table 2 SRA Passages ; Correlations between Battery Test Sentences ; Mastery Test Cloze Gaps; Basal Reader Order & Lexile Readability Peabody Test Pictures. Ba sa I R eade r Ba a l s r R R' The item types span the range ofreading comprehen- Se rues units sion items. The numbers in the table show the correlations Ginn 53 .93 .98 1 .00 between theoretical readability measures ofitem text and em HBJEagle 70 .93 .98 1 .00 pirical item calibrations calculated from students' test responses. S F F ocus 92 .84 .99 1 .00 Consider the top row. The Lexile readability equation predicted Riverside 67 .87 .97 1 .00 HM (1983 ) 33 .88 .96 .99 Table 1 Econom y 67 .86 .96 .99 Correlations between SF Amer Trad 88 .85 .97 .99 Empirical & Theoretical HBJOdyssey 38 .79 .97 .99 Holt 54 .87 .96 .98 Item Difficulties H M (1986 j 46 .81 .95 .97 Open Court 52 .54 .94 .97 Ten Five Test Item Types Readability Adapted from Sterner, 1997 Ledle SRA Battery Mastery Peabody r =raw R =corrected for attenuation R' = corrected for attenuation and range restriction Equations Slice Passage Sentence Cloze Picture Ladle .90 .92 .85 .74 .94 shows that the correlations between Lexile measures ofthe texts Fksch .85 .94 .85 .70 .85 of these basal readers and their sequential positions from easy ARI .85 .93 .85 .71 .85 FOG .85 .92 .73 .75 .85 to hard are extraordinarily high. In fact, when corrected for Powers .82 .93 .83 .65 .74 attenuation and range restriction, these correlations approach Holquist .81 .91 .81 .84 .86 perfection (Stenner, 1997) Fees ch-1 .79 .92 .81 .61 .69 Each designer of a basal reader series used their own Fles ch-2 .75 .87 .70 .52 .71 Coleman .74 .87 .75 .75 .83 ideas, consultants, and theory to decide what was easy and what Dale-C ha ll .76 .92 .82 .73 .67 was hard.