DOCUMENT RESUME

ED 398 570 CS 012 591

AUTHOR Reinking, David; Watkins, Janet TITLE A Formative Experiment Investigating the Use of Multimedia Book Reviews To Increase Elementary Students' Independent Reading. Reading Research Report No. 55. INSTITUTION National Reading Research Center, Athens, GA.; National Reading Research Center, College Park, MD. SPONS AGENCY Office of Educational Research and Improvement (ED), Washington, DC. PUB DATE 96 CONTRACT 117A20007 NOTE 100p. PUB TYPE Reports Research/Technical (143)

EDRS PRICE MF01/PC04 Plus Postage. DESCRIPTORS *Book Reviews; *Classroom Environment; Classroom Research; Grade 4; Grade 5; *Independent Reading; *Instructional Effectiveness; Intermediate Grades; *Multimedia Materials; Reading Research; *Teacher Role

ABSTRACT Using an approach to classroom research that D. Newman (1990) has termed a formative experiment, a study investigated the effects of engaging elementary school students in creating computer-based multimedia reviews of books they read independently. Formative experiments are designed to investigate how an instructional intervention can be implemented to achievea pedagogical goal in a particular educational environment. Creating multimedia book reviews was the intervention; increasing the amount and diversity of students' independent reading was the pedagogical goal. Diverse quantitative and qualitative data were gathered during 2 academic years in 9 4th-grade and 5th-grade classroomsacross 3 schools. Consistent with the intent of formative experiments, results are presented guided by the following questions:(1) What factors in the educational environment enhance or inhibit the intervention's effectiveness in achieving the pedagogical goal?;(2) How can the intervention and its implementation be modified during the experiment to achieve more effectively the pedagogical goal?;(3) What unanticipated positive or negative effects does the intervention produce?; and (4) Has the educational environment changed as a result of the intervention? Results indicated that the multimedia book review activity contributed to achieving the pedagogical goal of increasing the amount of children's independent reading; and school environments and teachers' roles to some extent shaped the effects of the activity. Findings suggest that formative experiments can address the limitations of conventional research methods previously used to study computer-based literacy activities in classrooms. (Contains 64 references, and 4 tables and 11 figures of data. Appendixes presents parent and teacher questionnaires.) (Author/RS) A Formative Experiment Investigating the Use of Multimedia Book Reviews to increase Elementary Students' Independent Reading

David Reinking Janet Watkins University of Georgia

U.S. DEPARTMENT OFEDUCATION Office of Educational Research andImprovement EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC) This document has been reproducedas received from the person ororganization originating it. Minor changes have been madeto improve reproduction quality.

Points of view or opinions stated inthis document do not necessarilyrepresent official OERI position or policy.

National Reading Research Center

READING RESEARCH REPORT NO. 55 Summer 1996 2 BEST COPY AVAILABLE NRRC National Reading Research Center

A Formative Experiment Investigating the Use of Multimedia Book Reviews to Increase Elementary Students' Independent Reading

David Reinking Janet Watkins University of Georgia

READING RESEARCH REPORT NO. 55 Summer 1996

The work reported herein is a National Reading Research Project of the University of Georgia and University of Maryland.It was supported under the Educational Research and Development Centers Program (PR/AWARD NO. 117A20007) as administered by the Office of Educational Research and Improvement, U.S. Department of Education. The findings and opinions expressed here do not necessarily reflect the position or policies of the National Reading Research Center, the Office of Educational Research and Improvement, or the U.S. Department of Education.

3 NRRC National Reading Research Center

Executive Committee National Advisory Board Donna E. Alvermann, Co-Director Phyllis W. Aldrich University of Georgia Saratoga Warren Board of Cooperative Educational John T. Guthrie, Co-Director Services, Saratoga Springs, New York University of Maryland College Park Arthur N. Applebee James F. Baumann, Associate Director State University of New York, Albany University of Georgia Ronald S. Brandt Patricia S. Koskinen, Associate Director Association for Supervision and Curriculum University of Maryland College Park Development Jamie Lynn Metsala, Associate Director Marsha T. DeLain University of Maryland College Park Nancy B. Mizelle, Assistant Director Delaware Department of Public Instruction University of Georgia Carl A. Grant Penny Oldfather University of Wisconsin-Madison University of Georgia Barbara McCombs John F. O'Flahavan Mid-Continent Regional Educational Laboratory (MCREL) University of Maryland College Park Luis C. Moll James V. Hoffman University of Arizona University of Texas at Austin Carol M. Santa Cynthia R. Hynd School District No. 5 University of Georgia Kalispell, Montana Robert Serpell Anne P. Sweet University of Maryland Baltimore County Office of Educational Research and Improvement, Betty Shockley U.S. Department of Education Clarke County School District, Athens, Georgia Louise Cherry Wilkinson Linda DeGroff Rutgers University University of Georgia Peter Winograd Publications Editors University of Kentucky Research Reports and Perspectives Production Editor Linda DeGroff, Editor Katherine P. Hutchison University of Georgia University of Georgia James V. Hoffman, Associate Editor University of Texas at Austin Dissemination Coordinator Mariam Jean Dreher, Associate Editor Jordana E. Rich University of Maryland College Park University of Georgia Instructional Resources Lee Galda, University of Georgia Text Formatter Research Highlights Angela R. Wilson William G. Holliday University of Georgia University of Maryland College Park Policy Briefs James V. Hoffman NRRC - University of Georgia University of Texas at Austin 318 Aderhold Videos University of Georgia Shawn M. Glynn, University of Georgia Athens, Georgia 30602-7125 (706) 542-3674 Fax: (706) 542-3678 NRRC Staff INTERNET: [email protected] Barbara F. Howard, Office Manager Kathy B. Davis, Senior Secretary NRRC - University of Maryland College Park University of Georgia 3216 J. M. Patterson Building University of Maryland Barbara A. Neitzey, Administrative Assistant College Park, Maryland 20742 Valerie Tyra, Accountant (301) 405-8035 Fax: (301) 314-9625 University of Maryland College Park INTERNET: [email protected] About the National Reading Research Center

The National Reading Research Center (NRRC) is Dissemination is an important feature of NRRC funded by the Office of Educational Research and activities. Information on NRRC research appears in Improvement of the U.S. Department of Education to several formats. Research Reports communicate the conduct research on reading and reading instruction. results of original research or synthesize the findings of The NRRC is operated by a consortium of the Univer- several lines of inquiry. They are written primarily for sity of Georgia and the University of Maryland College researchers studying various areas of reading and Park in collaboration with researchers at several institu- reading instruction. The Perspective Series presents a tions nationwide. wide range of publications, from calls for research and The NRRC's mission is to discover and document commentary on research and practice to first-person those conditions in homes, schools, and communities accountsof experiencesinschools.Instructional that encourage children to become skilled, enthusiastic, Resources include curriculum materials, instructional lifelong readers. NRRC researchers are committed to guides, and materials for professional growth, designed advancing the development of instructional programs primarily for teachers. sensitive to the cognitive, sociocultural, and motiva- For more information about the NRRC's research tional factors that affect children's success in reading. projects and other activities, or to have your name NRRC researchers from a variety of disciplines conduct added to the mailing list, please contact: studies with teachers and students from widely diverse cultural and socioeconomic backgrounds in pre-kinder- Donna E. Alvermann, Co-Director garten through grade 12 classrooms. Research projects National Reading Research Center deal with the influence of family and family-school 318 Aderhold Hall interactions on the development of literacy; the interac- University of Georgia tion of sociocultural factors and motivation to read; the Athens, GA 30602-7125 impact of literature-based reading programs on reading (706) 542-3674 achievement; the effects of reading strategies instruction on comprehension and critical thinking in literature, John T. Guthrie, Co-Director science, and history; the influence of innovative group National Reading Research Center participation structures on motivation and learning; the 3216 J. M. Patterson Building potential of computer technology to enhance literacy; University of Maryland and the development of methods and standards for College Park, MD 20742 alternative literacy assessments. (301) 405-8035 The NRRC is further committed to the participation of teachers as full partners in its research. A better understanding of how teachers view the development of literacy, how they use knowledge from research, and how they approach change in the classroom is crucial to improving instruction. To further this understanding, the NRRC conducts school-based research in which teachers explore their own philosophical and pedagogi- cal orientations and trace their professional growth. NRRCEditorial Review Board

Peter AMerbach Suzanne Clewell Mary Graham University of Maryland College Park Montgomery County Public Schools McLean, Virginia Rockville, Maryland Jane Agee Rachel Grant University of Georgia Joan Coley University of Maryland College Park Western Maryland College JoBeth Allen Barbara Guzzetti Michelle Conuneyras Arizona State University University of Georgia University of Georgia Frances Hancock Janice F. Ahnasi Linda Cooper Concordia College of Saint Paul, University of Buffalo-SUIVY Shaker Heights City Schools Minnesota Shaker Heights, Ohio Patty Anders Kathleen Heubach University of Arizona Karen Costello University of Georgia Connecticut Department of Education Harriette Arrington Hartford, Connecticut Sally Hudson-Ross University of Kentucky University of Georgia Jim Cunningham Marta Banning Gibsonville, North Carolina Cynthia Hynd University of Utah University of Georgia Karin Dahl Jill Bartoli Ohio State University Gay Ivey Elizabethtown College University of Georgia Marcia Delany Eurydice Bauer Wilkes County Public Schools David Jardine University of Georgia Washington, Georgia University of Calgary

Janet Benton Lynne Diaz-Rico Robert Jimenez Bowling Green, Kentucky California State University-San University of Oregon Bernardino Irene Blum Michelle Kelly Pine Springs Elementary School Mark Dressman University of Utah Falls Church, Virginia New Mexico State University James King David Bloome Ann Duffy University of South Florida Amherst College University of Georgia Kate Kirby John Borkowski Ann Egan-Robertson Gwinnett County Public Schools Notre Dame University Amherst College Lawrenceville, Georgia

Fenice Boyd Jim Flood Linda Labbo University of Georgia San Diego State University University of Georgia

Karen Bromley Dana Fox Michael Law Binghamton University University of Arizona University of Georgia

Martha Carr Linda Gambrell Donald T. Leu University of Georgia University of Maryland College Park Syracuse University Susan Lytle Barbara M. Palmer Bernard Spodek University of Pennsylvania Mount Saint Mary's College University of Illinois

Bert Mangino Stephen Phelps Bettie St. Pierre Las Vegas, Nevada Buffalo State College University of Georgia

Susan Mazzoni Mike Pickle Steve Stahl Baltimore, Maryland Georgia Southern University University of Georgia

Ann Dacey McCann Amber T. Prince Roger Stewart University of Maryland College Park Berry College University of Wyoming

Sarah McCarthey Gaoyin Qian Anne P. Sweet University of Texas at Austin Lehman College -CUNY Office of Educational Research and Improvement Veda McClain Tom Reeves University of Georgia University of Georgia Louise Tomlinson University of Georgia Lisa McFalls Lenore Ringler University of Georgia New York University Bruce VanSledright University of Maryland College Park Randy McGinnis Mary Roe University of Maryland University of Delaware Barbara Walker Mike McKenna Nadeen T. Ruiz Eastern Montana University-Billings Georgia Southern University California State University- Sacramento Louise Waynant Barbara Michalove Prince George's County Schools Fowler Drive Elementary School Olivia Saracho Upper Marlboro, Maryland Athens, Georgia University of Maryland College Park Dera Weaver Elizabeth B. Moje Paula Schwanenflugel Athens Academy University of Utah University of Georgia Athens, Georgia

Lesley Morrow Robert Serpell Jane West Rutgers University University of Maryland Baltimore Agnes Scott College County Bruce Murray Renee Weisburg University of Georgia Betty Shockley Elkins Park, Pennsylvania Fowler Drive Elementary School Susan Neuman Athens, Georgia Allan Wigfield Temple University University of Maryland College Park Wayne H. Slater John O'Flahavan University of Maryland College Park Shelley Wong University of Maryland College Park University of Maryland College Park Margaret Smith Marilyn Ohlhausen-McKinney Las Vegas, Nevada Josephine Peyton Young University of Nevada University of Georgia Susan Sonnenschein Penny Oldfather University of Maryland Baltimore Hallic Yopp University of Georgia County California State University About the Authors

David Reinking is Professor of Education at the Janet Watkins is a doctoral student in the Depart- University of Georgia where he serves as head of ment of Reading Education at the University of the Department of Reading Education. He is also a Georgia. Her research interests include investigat- principal investigator with the National Reading ing how technology can enhance literacy instruc- Research Center and is Editor of the Journal of tion in schools. In addition to her work on the Literacy Research. Professor Reinking's primary present study, she has assisted in developing research interest is the connection between tech- software for beginning reading instruction and in nology and literacy. His research publications in conducting research on its effectiveness. She has this area have appeared in Reading Research also developed an electronic portfolio application Quarterly, Journal of Reading Behavior, and for the Department of Early Childhood and Read- Handbook of Reading Research. He edited Reading ing at Georgia Southern University, where she is and Computers: Issues for Theory and Practice an instructor. Ms. Watkins taught English and (published by Teachers College Press), and has reading at the middle and secondary levels for 12 written a commercial software program for in- years. creasing reading comprehension, which has been distributed nationally to elementary and secondary schools. Dr. Reinking taught reading and math at the elementary school level for 8 years. National Reading Research Center Universities of Georgia and Maryland Reading Research Report No. 55 Summer 1996

A Formative Experiment Investigating the Use of Multimedia Book Reviews to Increase Elementary Students' Independent Reading

David Reinking Janet Watkins University of Georgia

Abstract. Using an approach to classroom result of the intervention? In addition to describing research that Newman (1990) has termed a the study and reporting results, the authors discuss formative experiment, this study investigated the emerging understandings of formative experiments. effects of engaging elementary school students in They argue that formative experiments can address creating computer-based multimedia reviews of the limitations of conventional research methods books they read independently. Formative experi- previously used to study computer-based literacy ments are designed to investigate how an instruc- activities in classrooms. tional intervention can be implemented to achieve a Diverse approaches to research have been valued pedagogical goal in a particular educational used to study whether computer-based instruc- environment. In the present study, creating multime- tionalactivities can enhance literacyin dia book reviews was the intervention; increasing the amount and diversity of students' independent classrooms. However, most studies have reading was the pedagogical goal. Diverse quantita- usedconventionalexperimentaldesigns tive and qualitative data were gathered during 2 aimed at comparing the effectiveness of academic years in 9 fourth- and fifth-grade class- interventions with and without computer- rooms across 3 schools. Consistent with the intent of based activities (cf. Means et al., 1993; Rein- formative experiments, the authors present results king. & Pickle, 1993). Despite the prevalence guided by the following questions: (a) What factors of experimental studies, reviews of classroom in the educational environment enhance or inhibit research (e.g., Reinking & Bridwell:Bowles, the intervention's effectiveness in achieving the 1991) as well as the observations and findings pedagogical goal?; (b) How can the intervention and its implementation be modified during the of some researchers (Bruce & Rubin, 1993; experiment to achieve more effectively the pedagogi- Dickinson, 1986; Mehan, 1989, Michaels & cal goal?; (c) What unanticipated positive or nega- Bruce, 1989; Reinking & Pickle, 1993; Rubin tive effects does the intervention produce ?; and & Bruce, 1990) suggest that conventional (d) Has the educational environment changed as a experiments alone do not provide adequate

1 2 Reinking & Watkins

information about how computer-based activi- The limited scope of conventional experi- ties might effectively contribute to the acquisi- mental research involving computers in class- tion of literacy in schools. rooms may explain in part why that research Conventional experiments do not adequately has not produced clear recommendations for deal with the many interacting variables that educators interested in how technology might influence the effectiveness of computer-based enhance literacy instruction in schools. Indeed, interventions in schools; nor do they typically that research does not consistently support the deal with how a particular computer-based use of computer-based activities over other activity might produce unique effects depending interventions not using computers (Reinking & on how it is integrated into a particular instruc- Bridwell-Bowles, 1991; Roblyer, Castine, & tionalenvironment.Instead,conventional King, 1988). As Becker (1992) states in his experiments require researchers (a) to control review of the computer-based integrated learn- the influence of most situational factors; (b) to ing systems, implement a well-defined intervention that remains unchanged during the experiment, ... the widely varying effects sizes and often in the face of changing conditions in the the modestly positive effect sizes that are classroom; and (c) to focus primarily on post- typical suggest that... results differ as intervention outcomes. These requirements much based on the different conditions of may perpetuate a point of view that restricts the study as on the different software the influence of technology in schools. As packages in use (and on the different Emihovich and Wager (1992) state, methodologies used to design and con- duct the analysis). (p. 38)

...media are still perceived as add-ons to the educational process ... [and] this The relatively few qualitative and ethno- perception will not change until educators graphic studies investigating computers and begin to realize that media or technology literacy counter some of these limitations by use in schools should be examined from focusing on the influence of situational factors a holistic cultural perspective. That is, over time within a particular classroom or the introduction of any new technology school (Dickinson, 1986; Friedman, 1990; should be considered in relation to its McGee, 1987; Mehan, 1989; Riel,1989; effect on the school culture as a whole, Turner & Dipinto,1992). However, such including the way students and teachers perceive the new technology, how admin- studies typically do not focus on the most istrators view it in relation to the organi- instructionallyrelevantquestions:What zational climate of the school, and the factors add to or detract from an intervention's expectations parents and the broader successin accomplishing a pedagogically community share concerning the effects valued goal and, more importantly, how might of new technologies have, or should the intervention be adapted in response to have, on learning. (pp. 435-36) those factors?

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

f, 10 A Formative Experiment 3

To address the inherent limitations of they have strong potentialto transform conventional approaches to classroom re- positively the standard modes of teaching search involving computer-based activities and learning in schools (e.g., Newman, and to address these instructionally relevant 1990; Papert, 1993; Sheingold, 1991). On questions, we adopted an approach to class- the other hand, it is clear from a variety of room research that Newman (1990) has sources that simply introducing innovative, termed a "formative experiment." Formative powerful, computer-based activities into a experiments, as we will describe more fully classroom is typically not enough to achieve in a subsequent section, are aimed at investi- this potential (Means et al., 1993). Empiri- gating how instructional interventions can be cal studies of how computers are used in adapted in response to factors that enhance schools support this conclusion. Such studies or inhibit their effectiveness in achieving a have documented consistently that the num- valued pedagogical goal. And, they allow ber of computers for instruction in schools researchers to examine a broad range of has increased geometrically since the early interacting factors and events that influence 1980s, but the amount of time they are used an intervention's effectiveness as well as its and their level of integration into the curric- unanticipated consequences. The instruc- ulum has not kept pace with availability tional intervention reported here involved (c.f., Becker, 1990, 1992a; Congressional fourth- and fifth-grade students and their Office of Technology Assessment, 1988; teachers in creating multimedia book reviews Martinez & Mead, 1988). These findings related to their independent reading as an reflect what many writers argue is an inordi- alternative to the conventional required book nate concern for acquiring technology with- report. That is, we were interested in sys- out concomitant attention to establishing tematically examining how we might engage clear pedagogical goals that it's acquisition elementary school students and their teachers will promote (see Reeves, 1992). Even when in creating multimedia book reviews with the schools and teachers attempt to integrate goal of increasing the amount and diversity computers into instruction to achieve clearly of students' independent reading. Our study defined goals, they often face many obsta- encompassed seven classroomsinthree cles (Gustafson, 1993; Hadley & Sheingold, schools during two academic years. 1993). These obstacles may be technologi- Formative experiments are especially cal, logistical, curricular, financial, interper- applicable to conducting classroom research sonal, and so forth. aimed at investigating computer-based inter- Consequently, there has been an increas- ventions (Newman, 1990) because the ex- ingly clear realization based on research and pected advantages of such interventions have supported by the experiences of many educa- been difficult to achieve. On one hand, it is tors in schools that simply placing computers clear that much of the interest in educational into a school or classroom rarely produces uses of computers is related to the belief that notable changes in teaching and learning

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

A 11 4 Reinking & Watkins

(see Miller & Olson, 1994). As Bresler and For example, some teachers and students Walker (1990) state, "Even when an innova- might use it to develop a classroom news- tion meets people's expressed needs, it may paper while others might focus on writing still not succeed unless it fits the patterns by about academic topics. In short, Bruce and which they run their lives as students and his colleagues designed QUILL to effect teachers" (p. 66). Yet, many writers and changes in the way reading and writing educators remain steadfast in their belief that were typically carried out in individual computers have strong potential to do so. classrooms. As Michaels and Bruce (1989) Even when computer-based activities are state, they hoped that it "would alter the specifically designed to transform or reorient writing systems, and hence students' access teaching and learning, they may be imple- to writing and reading opportunities" (p.12). mented in such a way as to maintain the However, they found that "rather than the status quo. A well-documented example is new technologyradicallyreshapingthe the work of Bruce and his colleagues (Bruce learning environment, the computers them- & Peyton, 1990; Bruce & Rubin, 1993; selves were shaped to fit the already estab- Michaels & Bruce, 1989) who have con- lished patterns of social organization" (p. ducted extensive investigations of a computer- 12). Based on their analysis of QUILL in based intervention called QUILL. QUILL is two classrooms over two years they con- a comprehensive computer-based tool to cluded that: support writing and reading. It is comprised of distinct components such as the "mailbag" ...it becomes clear that no innova- (to facilitate on-line communication among tion, no matter how well conceived, students and teachers), the "planner" (for and no application of innovations, no getting ideas for writing and for organizing matter how well intended and exe- ideasrespective to various genres), the cuted, can in and of themselves be "library" (a textual information base created assured of achieving positive change in by and shared among students and teachers), instruction. Instead, one must continu- and the "writers assistant" (a word process- ally reexamine what is happening and ask critical questions: What isthe ing program designed to be compatible with innovation we are using? What are the the other components). different ways that students are en- Bruce and his colleagues developed gaged in learning? What is the effect QUILL specifically to promote valued peda- of my actions on students' learning? gogical goals in literacy such as critical What are the goals of instruction, of thinking, integration of reading and writing, each activity? And so on. (Michaels & meaningful communication, collaboration, Bruce, 1989, p. 35) and revision. QUILL was also developed to be open ended with the intent that it In a similar vein, Bruce and Rubin (1993) would be used differently in each situation. state that their "detailed, self-critical ap-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

12 A Formative Experiment 5

praisal of the evidence [about QUILL] yields based on our experiences in the present study; surprises and reveals a richness in what see Reinking & Pickle, 1993; Baumann, Dil- students and teachers do that belies both lon, Shockley, Alvermann, & Reinking, 1996). optimistic and pessimistic visions of technol- Second, the concept of a formative experiment ogy in relation to educational change" (p. is not yet clearly distinguishable from related 1). Our decision to use a formative experi- concepts such as "situated evaluation" (Bruce ment as an approach to research in the pres- & Rubin,1993),"designexperiments" ent study was based on our concurrence with (Brown, 1992), "formative evaluation" (see their position and with the position of re- Flagg, 1990), and "rapid prototyping" (Tripp searchers who have discussed the limitations & Bichelmeyer, 1990). While supporting the of conventional experiments in conducting fact that classroom research is evolving, Eisen- classroom research (e.g., Reinking & Pickle, hart and Borko (1993) state that 1993). the standards for using ... alternative Formative Experiments methodologies in educational research are not routinized in the same way they are for established methodologies; thus Dissatisfaction with conventional experiments their use demands more thought and as an approach to classroom research as ex- explanation than might be necessary if pressed by many writers interested in technol- conventional procedures were used. ogy (e.g., Bruce & Rubin 1993; Newman, (p. 11) 1990; Papert, 1987; Reinking & Bridwell- Bowles,1991; Reinking & Pickle,1993; To address this issue, we provide here an Venezky, 1983) reflects the broader concern overview of our evolving understanding of thatresearchers must seek out alternative formative experiments based on Newman's approaches to research that address the com- (1990) description and our own experience plexity of classrooms (see Eisenhart & Borko, in the present research project. Thus, the 1993; Jackson, 1990). The concept of a forma- present report has a dual purpose. Foremost, tive experiment as described by Newman it provides a description of a computer-based (1990) is one proposed alternative to conven- intervention and how it was implemented to tional experiments. accomplish the goal of increasing the amount However, it is important to note that the and diversity of students' independent reading. concept of a formative experiment is evolving. However, it also reports our expanded under- This characterization is evidenced first by the standing of formative experiments and their fact that there are few published sources of potential to contribute to literacy research. information discussing explicitly the concept of Newman (1990) has described a formative a formative experiment (to our knowledge experiment as follows: "In a formative experi- limited to Newman's 1990 article in the Educa- ment, the researcher sets a pedagogical goal tional Researcher and our own published work and finds out what it takes in terms of materials,

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

13 6 Reinking & Watkins

Qualitative or Ethonographic Conventional Experiments Studies Formative Experiments

How does the use of multimedia In a classroom using multimedia Given a pedagogical rationale for book reviews compare to some book reviews (perhaps when believing that multimedia book other classroom activity aimed at compared to classrooms using reviews have potential to meet the increasing independent reading? alternative activities), what are goal of increasing independent the dominant factors that explain reading, how must this activity be why some students read more and implemented in a particular class- some read less? room to achieve its stated goal?

Figure 1. Matching related research questions and approaches to research.

organization, or changes in...technology to 3. As the intervention is implemented, what reach the goal" (p. 10). Thus, formative exper- factors enhance or inhibit its effectiveness iments address a different category of questions in achieving the pedagogical goal? when compared to conventional experiments or qualitative approaches to research. Figure 1 4.How can the intervention and its imple- compares how three approaches to research mentation be modified to achieve more can be matched to three related but different effectively the pedagogical goal? research questions that might guide an investi- gation of multimedia book reviews as a class- 5.What unanticipated positive or negative room intervention. effects does the intervention produce? Drawing on Newman's (1990) explication of formative experiments and our own experi- 6. Has the instructional environment changed ence in conducting the present investigation, as a result of the intervention? we have proposed six questions as a framework for designing and conducting formative experi- These questions serve as a guide for ments (see Baumann et al., 1996): reporting our investigation. Questions 1 and 2 are related and are analogous to the rationale 1. What is the pedagogical goal of the for the experiment and the literature review experiment and what pedagogical theory included in conventional research reports. establishes its value? These questions will be addressed in the next two sections of this report. Questions 3-6 2. What is an instructional intervention that involve the collection of baseline data prior to has potential to achieve the identified implementing the intervention and the collec- pedagogical goal? tion of on-going data during the intervention.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

14 A Formative Experiment 7

Issues pertaining to data collection and analysis and school administrators (see O'Flahavan et in a formative experiment will be addressed al., 1992). specifically in a subsequent section and as we Despite the substantiated view that the present our methods and findings. As in a amount of students' independent reading fig- conventional research report, we conclude with ures highly in their reading ability and their a summary and general discussion of our propensity to read, there is strong evidence that findings. many children engage in independent reading infrequently, especially outside of school. For Our Pedagogical Goal and Its Value example, Anderson, Wilson, and Fielding (1988) found that reading ranked well behind Our pedagogical goal in this formative other outside-school activities such as watching experiment was to increase the amount and television and talking on the telephone. A diversity of students' independent reading. In comparison of two administrations of the addition to being a longstanding, widely ac- National Assessment of Educational Progress cepted, and intuitively valued instructional goal (NAEP) in 1988 and 1990 (see Foertsch, 1992) in reading instruction, research has reinforced suggests that there may be a trend toward the centrality of independent reading as a further decreases in the amount of students' powerful means to increase students' reading independent reading. For example, that com- competency. Theorists and reviewers of exist- parison found decreases in library use and in ing research such as Stanovich (1986) have reading for fun, which was accompanied by an argued that differences in reading ability are increase in the percentage of students reporting due in large measure to differences in the that they read only fiction or only nonfiction. amount of children's reading. And, studies Likewise, approximately half of the students in involving large samples of students nationally the eighth grade (no data were available for the indicate a strong positive correlation between fourth-grade students) reported reading the reading out of school and reading achievement same author either weekly or monthly, which (see Foertsch, 1992). Similarly, there is evi- perhaps indicates a lack of diversity in their dence that valued correlates of reading ability reading. such as vocabulary knowledge are also related Thus, the value of setting a pedagogical to how much children read (see Freebody & goal to increase the amount and diversity of Anderson, 1983). In addition, independent students' independent reading is justified be- readingisa defining attribute of engaged cause (a) research indicates that independent reading, which has been identified as a domi- reading plays a central role in determining nant theme of the National Reading Research reading ability; (b) evidence suggests that Center (see Alvermann & Guthrie, 1993). many children do relatively little independent Increasing children's interest in reading inde- reading, especially outside of school, and that pendently has also been identified as having the there is a trend toward further decreases in the highest priority for research among teachers amount and diversity of independent reading;

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

15 8 Reinking & Watkins

and (c) surveys of educators identify increasing ity aimed at encouraging students' independent interest in reading as a priority for reading reading. Despite its widespread use, many research. authors have questioned its effectiveness in accomplishing that goal. Although no empirical The Instructional Intervention evidence can be cited, writers such as Spiegel (1981) have argued that the required book The instructional intervention in a forma- report may actually subvert teachers' intentions tive experiment may be one found in the exist- by encouraging some students to read less (by ing literature, selected primarily to testits selecting shorter books) and more narrowly (by relation to a pedagogical theory, or one the avoiding genres or topics that may prove investigator designs specifically to address the unappealing). Likewise, the required book pedagogical goal, thus testing an alternative report is frequently cited by our undergraduate means to accomplish the goal (Baumann et al., and graduate students as one of the most nega- 1996). The instructional intervention in the tive memories of elementary school reading present experiment falls within the latter cate- instruction. In contrast, the opportunity to gory. That is, the only previous literature work on a computer has been consistently related to having children create multimedia identified as a highly motivating instructional book reviews is our own work carried out activity in schools (Becker, 1990; Reinking & within a conventional experimental paradigm Bridwell-Bowles, 1991). (see Reinking & Pickle, 1993). In that work, Although there are many classroom activi- we report only limited findings related to an ties to encourage independent reading and intervention that used computers but did not authentic responses to books, there is no evi- employ multimedia. In the remainder of this dence that such activities have replaced the section, we elaborate our rationale for design- wide spread use of conventional book reports. ing the instructional intervention and we pro- We reasoned that using technology to trans- vide an overview of its components as they form an activity familiar to both students and evolved during the 2 years of implementation teachers had an advantage over other alterna- in three elementary schools. tives not so firmly entrenched in the dominant instructional culture of elementary schools. Rationale for the Intervention That is, multimedia book reviews are concep-

Our rationale for selecting multimedia 'We purposefully used the termbook reviewinstead of book reviews' as an intervention to accomplish book reportwhen discussing this intervention among the goal of increasing the amount and diversity ourselves and among the teachers and students with whom we worked. This decision was based on our desire of reading was based primarily on its potential to highlight the differences we perceived between the as an alternative to the conventional required purposes of the conventional book report and the multi- book report. In the middle grades, the required media activities with which we wished to engage teachers book report is a ubiquitous instructional activ- and students.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

16 A Formative Experiment 9

tually similar to book reports and therefore are writing (e.g., Bruce & Rubin, 1993; Means et not radically different from a familiar school al., 1993; Reinking, 1986). There is also some activity. This familiarity might enhance the evidence that engaging students in creating potential for integration into existing instruc- multimedia activities can imbue school reading tional activities, while at the same time the and writing activities with the characteristics of unique characteristics of multimedia book out-of-school reading and writing. For exam- reviews might counter some of the negative ple, Turner and Dipinto (1992) concluded in aspects of conventional book reports. This their qualitative study of students who became perspective is consistent with one of the ques- hypermedia authors that, tions addressed by formative experiments as an approach to classroom research. That is, for- Their strong sense of audience motivated mative experiments address whether an inter- the students to present the information so vention has been fully appropriated by teachers thattheir peers could understandit and students (see Newman, 1990) or the degree better. Technology didn't just enhance to which the educational environment has the appearance of students' reports, it changed as the result of the intervention (see also encouraged them to rethink how to question 6 in the framework for designing a present information to communicate it formative experiment presented in a previous more effectively. (p. 198) section). The required book report can also be Extrinsic awards are also often associated criticized because as typically implemented in with conventional book report activities. For classrooms it is inconsistent with the meaning- example, a popular restaurant chain provides ful communicative activities that have been free pizzas for students who can document shown to enhance students' reading and writ- independent reading of books, and many ing. For example, Kirby and Kirby (1985) in schools participate in such programs. Simi- their analysis of the research on assigned larly, a popular commercial computer program school-related and unassigned out-of-school awards points based on the difficulty and tasks concluded that students who complete length of various books when students can pass assigned school tasks take fewer risks in read- an on-line test of factual information in the ing and writing. As typically implemented in books. Teachers will frequently use the points many classrooms, students write book reports as a basis for distributing prizes for reading primarily for the teacher in order to satisfy a independently. Our pedagogical theory deval- requirement or to obtain extra credit. One of ues these approaches in favor of more intrinsi- the advantages of computer technology that has cally motivating activities, a point of view been frequently cited in the literature is its substantiated in part by consistent findings that potential to create opportunities for students to tangible rewards simply for completing tasks engage in meaningful communicative experi- has a somewhat negative effect on intrinsic ences that frequently involve reading and motivation (see Cameron & Pierce, 1994).

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

17 10 Reinking & Watkins

FOXTROT

IbLIVE ALL MY TEACHER ITS 60946 To BE THE You Look SMILES OVER GAVE ME GREATEST Book REPORT e So WHAT'S EXCITED. I AM. I'm A Book PERMISSION EVER. IT LL HAVE SouND, 1 THE BooK ? I FoRbET. WORKING REPORT ? / To Do MINE ANIMATION, DIGITIZED WANNA I I oN A Book % AS A VIDEO COMMENTARY AND SEE MY REPORT. MULTIMEDIA I HAVE AN IDEA FOR A r"--\, FloWcHART? . sv -' CoMPuTER REALLY Cool PoINT-AND- , I4 ; 01 ..,. ) / 0) PRESENTATIoN, CLICK HIERARCHICAL ,, , . / INTERFACE. Is 'i & lllVS ,,iik, ig,1 It' 1 ) NAND4.18

FOXTROT

DOING A RIGHT NoW I WANT MY MULTIMEDIA HAVE You EVER BEEN WHAT Beak REPORT oN rM SURE EVENTuAU.Y, MY BIGGEST Book REPORT To BE THE HIT OVER THE HEAD Do 1:1) "oLD YELLER' AND ELL GET PRIORITY IS GREATEST AND COOLEST WITH A SUGGESTION? THINK You HAVEN'T EVEN ARouND GETTING THRoobil AND MoST AMAZING ABOUT READ IT ?! TO IT ALL THESE Book REPORT MY SCHOOL owe THE EVENTuAuT COMP VIER HAS EVER WITNESSED. MUSIC FROM 0 MANUALS. )7FoARRWATTlitt cREDITs? 1110 k?.- sjilL maw raionsomt 449

FOXTROT 0111041h1AoraWOKIN 11...IP..eft** IT'S TAKEN TWo DAYS You Mow, MAYBE of Norl-STOP WORK _ea I'VE IGot THE THERM A REASool AND A couPLE HUNDRED FINALLY BookTITLE MOST KIDS Dori PHEW. PAGES OF READING... DONE IT. TODANCE Do MULTIMEDIA \ % \ ACROSSTHE BookREPORTS. -- \ SCREEN. 1 _...., le-ww

e_ =1 il zisiramliikt... 0 6.,..' ..,..!...% ii iil ,:? -*...,. 1 '

Figure 2. Comic strip representation of a potential limitation of multimedia book reviews.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

18 A Formative Experiment 11

Thus, there is reason to believe that Overview of the Intervention multimedia book reviews, as an alternative and Computer Programs to conventional required book reports, have potential for increasing the amount and In a formative experiment the intervention diversity of students' independent reading by and the way it is implemented may change as engaging them in personally meaningful parties involved in the research respond to on- responses to what they read, by sharing going data collection aimed at determining the those responses through their multimedia factors that enhance or inhibit the accomplish- presentations, and by capitalizing on the ment of the pedagogical goal. Nonetheless, it is possible to describe the intervention in intrinsic motivation of using a computer to general terms, at least as it is initially con- share information about what one has been ceived prior to the experiment. In the present reading. experiment, our initial broad conception of the At the same time, we recognized a intervention remained intact throughout the 2 priori that there may be potential problems years we worked in three elementary schools; associated with multimedia book reviews, although, many details related to its implemen- which may undermine their effectiveness in tation evolved and were adapted in response to accomplishing the pedagogical goal. For our on-going data collection and analysis. Here example, one potential problem isthat we provide an overview of the intervention and students might have more interest in creating describe the most recent version of the com- multimedia products than in reading books, puter programs in use at the completion of the and thus the intervention may actually inter- experiment. fere with students propensity to read. This As we have indicated previously in this limitation is reflected poignantly in a recent report, we conceptualized multimedia book comic strip presented in Figure 2. As this reviews as an alternative to conventional book concern illustrates, formative experiments, reports. Like a conventional book report, a unlike conventional experiments, encourage multimedia book review was to be completed by students independently after they read a researchers to acknowledge that virtually all book. Unlike a conventional book report, instructional interventions entail some po- multimedia book reviews were to be created tentially negative instructional by-products. with the aid of a computer that allowed graph- However, formative experiments are designed ics and sound to accompany textual informa- to identify specifically what factors inhibit tion. Another key difference was our intention the accomplishment of the pedagogical goal that the multimedia book reviews would be and to adapt the intervention or its imple- compiled into a searchable database available mentation to address those factors that have for use by students, teachers, parents, and a negative affect on achieving the pedagogi- other interested individuals. We envisioned cal goal. that this database of students' multimedia book

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

1 9' 12 Reinking & Watkins

Master Stack

Books I have Read 11 I Where the Red Fern Muggie Maggie Bridge To Terabithia Grows II I Kissing May Sarah, Plain and Song of the Trees Tall

STUDENTS NAME II What's Cooking, Here's Hermione A RosVs Romance Jenny Archer Rosy Cole ProductionII Jenny Archer to the I Have A Friend There's a Wocket in Rescue my Pocket

'=!:Class lEACIAERS About fiaiNi My [11 More *bout 'MFetch virg.< List pripi Me! e_./:' Books / his. hook

Figure 3. Template card 1: Main menu screen showing the books a student has read and reviewed.

reviews would be available in an easily accessi- school-related task. However, given the pur- ble location such as a school's media center. pose of the database, the audience for multi- As discussed previously, we reasoned that the media book reviews would be much broader, use of the computer to create multimedia and the purpose would be more functional and products had potential to increase students' therefore authentic. In addition, the database motivation and engagement when compared to would provide an appealing mechanism for conventional book reports. students to explore each others' reading and to Likewise, we reasoned that the database find books they might like to read. Similarly, was an important extension of the conventional although each multimedia book review would book report. The audience for conventional be an individual student's product, we envi- book reports is typically the teacher, and book sioned much collaborative work among stu- reports are usually completed as a required dents as they assisted each other in dealing

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

29 A Formative Experiment 13

Master Stack copy 1

Title:Bridge To Terabithia

Author:Katherine Paterson Category:Fiction, friendship, death Reviewer:STUDENTS NAME Audio: Summary: Jess had always wanted to be the fastest runner in his grade. So he could run all summer trying to get fast He would have been if it hadn't been for Leslie Burke. Later in the story Jess and Leslie become best friends. They have their own secert place called Terabithia. They gather every day Jess is Kina Leslie is Queen until one terible day when Leslie aets killed to find Review: I liked this book even though it was sad. You'd better be prepared to cry if you read this book because it is so sad.I don't know what I'd do if my best friend died.I liked the secret place they had to get away to.

Last summer my friends and me had a secret hiding place too. I auess that's why I liked this book so muchl TEACHERS MZ Class More about OM.< List Books this book

Figure 4. Template card 2: Review screen completed by students for each book reviewed. with the new technological skills needed to appropriated fully into the instructional envi- create multimedia book reviews. Taken to- ronment), we decided to teach teachers and gether, we believed that these differences students how to use HyperCard (Version 2.1). between conventional book reports and multi- HyperCard is an authoring system for the media book reviews had potential to transform Macintosh computer, and it can be used to a common school activity in a way that would create multimedia presentations. It is a power- have a positive affect on the amount and diver- ful, open-ended application for creating com- sity of their independent reading. puter programs, but it permits a user to create To provide students and teachers with relatively sophisticated computer applications much flexibility in creating multimedia book with only basic knowledge of its operation. We reviews and to increase their involvement in planned to acquaint teachers and students this alternative (as well as its potential to be with the rudimentary tools necessary to create

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

21 14 Reinking & Watkins

In other words, the database of multimedia book reviews that we envisioned required that information be embedded into established categories. However, if we required students to enter information about the books they read into predetermined categories, we might subvert their creativity and their personal responses to creating each book review. Our successive attempts during the experi- ment to resolve this dilemma and to notice what effects it had on students and teachers is one example of why this approach to research is described as formative. That is, the solution to our dilemma emerged through a series of adaptations and compromises in response to what we observed during the experiment. Some of these adaptations include: (1) seeking teachers' and students' input into what catego- ries of information about the books they read that they thought most relevant; (2) seeking Figure 5. A student's drawing (using clip art and their input about the layout of these categories drawing tools) from the "More about this book" on a standard card used by all students, which option on the review card (see Figure 4). came to be referred to as the book review "template"; and (3) creating nonsearchable but personalized sections of the template (e.g., HyperCard programs referred to as "stacks," boxes for pictures and photographs, an audio which is an extension of the HyperCard meta- button that played a student's comments about phor of screens as "cards" that are linked by the book, and a card that provided an autobio- clicking on "buttons." We taught them how to graphical sketch of each student reviewer). In use all of the basic HyperCard tools short of addition, we eventually created a button that scripting, which is a more advanced level of led to a separate nonsearchable stack that HyperCard programming. students could choose to create as their per- One dilemma that we faced early in our sonal supplement to the basic template. planning and implementation of the interven- The most recent book review template, tion was how to design the multimedia book which is similar in many respects to earlier reviews to be open-ended and personal while versions,iscomprisedof threescreens accommodating the standardization necessary ("cards" in HyperCard) presented in Figures 3, for the database to be searched systematically. 4, and 6. The screen shown in Figure 3 is a

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

22 A Formative Experiment 15

Master Stack calm 1

Hi, my name is STUDENT'S NAME. I am in fifth grade at Borders school. My birthday is April 19, 1 9831 My favorite food is pizza with lots of cheeze. I like to play soccer and watch TV. My favorite show is Home Improvement. In 20 years I'll be a vetinarian because I like animals

so much. My favorite subject is social studies. I like learning about other countries. Maybe someday fly all over the world. I'll like to read books that keep you' interested especially about kids my age. STUDENTS NAME

ty:tu: Class 7EA CHEWS About a--i-.1 my Lai Mare about "RICFetch &at'List ri ri P Me! d-.--' Books this book

Figure 6. Template card 3: Student-written biographical sketch. template card 1 that serves as a main menu navigating among the book reviews and the comprised of the student's picture and 12 icons database functions. The addition of this menu representing the books that a student has read bar to earlier versions of the program illus- (additional screens can be added when a student trates the formative aspects of creating a com- has entered more than 12 books). A student's puter applicationthatoperated efficiently multimedia review of a particular book can be within the larger goals of the experiment. accessed by clicking on the appropriate book Template card 2, which serves as the main icon. Other components of the program can be review screen, is shown in Figure 4. In addition accessed by clicking one of the buttons on the to the menu bar, it is comprised of (a) textual memo bar at the bottom of the screen. This fields into which students can enter descriptive menu bar appears at the bottom of all template information about the book as well as a sum- cards and serves as the primary means for mary and their comments about the book, (b) an

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

23 16 Reinking & Watkins

[Clicking the 'Fetch Fide button initiates a search through all of the students' book reviews for books authored by E. B. White, as specified in the search criteria above. The results of the search vould be displayed in this box. The listing vould include title, author, and reviewer for any review meeting the search criteria. A user can then click on any book listed to see how a student reviewed ill.]

Figure 7. "Fido Fetch" screen used to direct the search of the data base containing all students' reviews.

audio button that plays students' comments book's authors or characters (see Figure 5). or sound effects, and (c) and navigation The textual fields are scrolling fields so that buttons, including one ("more about this a student has unlimited space to enter text book") that allows students to add additional despite the small area displayed initially on cards to their own specifications. For exam- this screen. All of the textual fields can be ple, on these additional cards, many students searched when using the database application chose to include pictures from clip art files, described momentarily. often modified or supplemented by their own Figure 6 shows the third template card, art work created on-line with the drawing which is the screen that students used to tools, and additional information about the enter biographical information. Their bio-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55 A Formative Experiment 17

This project

Corning Soon!

:Teaoher::- 4...,N§.,y,' ";% 7E4CHERS About ,,,,. '.....?!.,.. ' " Class My LLA recent. zav,-..-. Me ! Options.. _ .r, , List PPP Books

Figure 8. Screen showing teachers whose students' reviews can be accessed by clicking on a teacher's picture.

graphical sketches can be accessed by click- ing a key word or phrase in the textual ing the button on the menu bar labeled field on the upper part of the screen and "About Me!" clicking on the fido fetch button, the data- The database application, which we base application would find all reviews called "Fido Fetch," was operated from the containing the key word or phrase speci- card shown in Figure 7. To search the data- fied. These reviews would be displayed by base, a user would select one of the catego- title, author, and reviewer in the scrolling ries to be searched, each category corre- textual field on the lower part of the sponding to the textual fields on the main screen. Clicking on one of these reviews review template (see Figure 4). After enter- would display the full review.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

25 18 Reinking & Watkins

bide'

OtIthErt4e, *IWO studentifte en.

ef1,014.1 Class 6.11;f.t List Options...

Figure 9. Screen showing students in a class whose reviews can be viewed by clicking on a student's name.

Initial entry into the reviews and the Method database application was achieved from the cards shown in Figures 8 and 9. The Research Sites screen in Figure 8 displays pictures of teachers whose students have entered book This formative experiment was conducted in two elementary schools during the 1992-93 reviews. A user could click on a teacher to school year and in one elementary school dur- access a list of students in that teacher's ing the 1993-94 school year. The schools were class (Figure 9) and then access a particular selected from among six schools contacted that student's main book review menu (Figure 3) had expressed an interest in the project and by clicking on the student's name in the list. that had or were willing to purchase the mini-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

26 A Formative Experiment 19

mal hardware needed to implement the inter- School, we worked with 1 fifth-grade and 1 vention. The final selection of research sites fourth-grade class, their respective teachers, was made by the university researchers based and a Chapter 1 teacher who also managed the on the schools' location, equipment and facili- computer lab. ties, enthusiasm for and commitment to the Hartwig and Borders schools each had a project, student populations, and so forth. Of small computer lab with 12 and 10 Macintosh the two schools that participated during the LC II computers, respectively. Both schools 1992-93 school year, one school (hereafter had access to a color scanner for digitizing referredto with the pseudonym "Collins pictures and several printers. Borders school School") was located in a small town within also had a black and white LCD panel for commuting distance of a large metropolitan displaying Macintosh output on a standard area. The school was relatively large, accom- overhead projector. In comparing the imple- modating 3-4 classrooms of approximately 30 mentation of the intervention at the three students at each grade level K-5. Students and schools, we found that the LCD panel was an teachers reflected the community's homoge- invaluable tool in teaching students Hyper- nous population consisting of predominantly Card.It greatly streamlined that phase of White, middle- and upper-middle-class fami- implementing the intervention. Collins school lies, many of whom commuted to work in the had a computer lab of Apple IIe computers, but nearby large metropolitan area. We worked purchased two Macintosh LC II computers and with two fourth-grade classes and their teachers a scanner to be shared by two classes involved at Collins School. The second school (hereafter in the experiment. During the first year of the referred to with the pseudonym "Hartwig project, we were interested in discovering the School") was located in a rural area, although logistical and pedagogical implications of its students came from blue-collar and pro- implementing the intervention in a lab setting fessional homes in a small town as well as where students worked for approximately 1-2 from agricultural areas. Approximately 15 % hr per week in the computer lab when com- of the Hartwig student population was Afri- pared to a setting in which a single computer can American. The school was moderately was available constantly in a classroom. Based large, accommodating approximately 2-3 on our experiences, these two alternatives classrooms of 25-30 students at each grade represent typical patterns of availability in level (K-5). We worked with 3 fifth-grade elementary schools (see also Becker, 1990, classes, their respective teachers, a Chapter 1 1992a, 1992b). Likewise, our goal at Collins teacher, and a gifted teacher in Hartwig School was to minimize the amount of knowl- School. The third school (hereafter referred edge students would need about HyperCard in to with the pseudonym "Borders School") order to enter their book reviews, relying was in the same school district as Hartwig instead on training teachers and parent volun- School and had a similar, although some- teers to assist with HyperCard applications what smaller, student population. At Borders developed by the university researchers.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

27 20 Reinking & Watkins

Participants university researcher. In addition, teachers with whom we worked were co-authors on The composition of the university research papers presented at two national professional team varied during the course of the project, meetings and the teachers participating in the but included the principal investigator (the first project at Borders school appeared without the author of this report), three consultants, and university researchers on the program of a state five doctoral students who provided assistance professional meeting where they made a pre- at various stages of the project in areas such as sentation related to their involvement in the solving technological difficulties, collecting project. and analyzing data, teaching HyperCard to Nonetheless, it may not be accurate to teachers and students, and writing the final consider the present formative experiment as report. The three consultants included Thomas an example of collaborative research given Reeves, a Professor of Instructional Tech- current definitions (cf.Allen, Buchanan, nology at the University of Georgia (UGA); Edelsky, & Norton, 1992; Anders, 1996; Mary Jo Brown, an Assistant Professor at Jervis, Carr, Lockhart, & Rogers, 1996). We UGA who teaches courses in qualitative re- did not involve the teachers in selecting the search; and Valerie Garfield, a middle-grades pedagogical goal (although they clearly agreed teacher who has had much successful experi- that it was an important one), nor did we formally involve them in analyzing data. Nei- ence in integrating computers into her teaching. Although we are indebted to them for their ther did we involve them directly in preparing this report. Also, we observed and analyzed input and assistance, this report reflects only teachers' reactions and behaviors during the our own summary and interpretations. time we were working in the schools as part of Consistent with the goals of the National the experiment without systematically sharing Reading Research Center (see Alvermann & our conclusions with them until after the exper- Guthrie, 1993) and the nature of formative iment. Thus, in this report we use pseudonyms experiments, we considered the classroom to refer to the teachers with whom we worked teachers with whom we worked on this project closely. to be integral members of our research team. We wish to note that our decision to not We continually sought their interpretations of involve teachers fully as equal partners in the what was occurring in their classrooms relative research process was not based on our dis- to the experiment, and we continually solicited agreement with the more stringent standards their suggestions for modifying the interven- outlined in the emerging literature on collabo- tion. Members of the university research team rative research. Our stance in this project was met regularly with the teachers todiscuss their consistent with that literature, which suggests observations of what was occurring and to plan that truly collaborative research in which future strategies and modifications. Teachers teachers define themselves as researchers tends also kept written logs of their observations, to evolve over an extended time and across a which were discussed periodically with a number of collaborative experiences. Given the

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

28 A ForMative Experiment 21

separate professional cultures of researchers sionally receive instruction from other teachers and teachers that is prevalent today, it may not who taught subjects such as Art, Physical be possible to generate the trust, confidence, Education, and Band, and these times often and understanding necessary for a true collabo- provided an opportunity for teachers to have a ration without an extended period of interaction planning period. A relatively small percentage (see Anders, 1996; Jervis et al., 1996). As we of the students in each classroom left periodi- write this report, further collaborations be- cally for special instruction in the Chapter 1 or tween some of the teachers and the university gifted programs. A commercial basal reading researchers are being planned. We are confi- series figured prominently in reading instruc- dent that these future collaborations will move tion. Although two of the teachers in Collins increasingly towards the ideal of truly collabo- school described themselves as having a whole rative research and that the teachers involved language orientation, we did not observe their will increasingly define themselves as research- practices to deviate greatly from the other ers. teachers who used basals extensively. Prior to In general, the participants in this experi- their involvement in this experiment, all of the ment did not deviate in significant ways from teachers devoted some time to classroom the researchers, teachers, and students one activities aimed at encouraging independent would expect to find in their respective envi- reading. Such activities included requiring ronments. The researchers from the university students to write conventional book reports, had the orientations and values one would reading sections of books that they thought expect of other doctoral students and faculty in students might want to read, and putting up a major research-oriented university, albeit we bulletin boards showing the number of books believe with a heightened awareness of collabo- read by each student in the class. For the most rative research and of the value of teachers as part, the teachers made available books for professional colleagues. From the outset of this independent reading through a classroom project, we also were excited about the poten- library rather than through regular trips to the tial of formative experiments as a new ap- school library. proach to classroom intervention research, and As determined through a structured inter- we were eager to explore the new insights it view (see Appendix A), all of the teachers had might produce about computer technology and some background with computers in their literacy and about our own understandings of teaching, although the amount and type of that research. experience varied somewhat. Through observ- In general, the schools and classrooms in ing their classrooms and daily routines during this investigation reflect the organization and a baseline period before the multimedia book operation typical of many elementary schools review activity was introduced, we found that serving the community populations described all of the teachers were actively using the previously. Classrooms were self-contained for computer in conjunction with their teaching most of the day, although students did occa- activities, but they did so in ways that were

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

29 22 Reinking & Watkins

May June-August Sept.-mid Oct. Oct.-January February-May May

Phase 1 Phase 2 Phase 3 Phase 4 Phases 5 and 6 Phase 7 Preliminary Teacher and Gather baseline Teach students Students create Gather meetings support staff data HyperCard and multimedia postexperiment training develop book reviews data templates and enter them into database

Figure 10. Phases of the experiment and approximate time line of events.

largely perfunctory and not fully integrated into big majority that was terrified. They had their instructional program. For example, the no experience, they didn't even want to following comment by one of the teachers was touch it. But after the workshop, our typical. staff has [begun] to feel like its not going to bite if they touch it. I use it [the computer] mostly for when they finish their work and enrichment Beyond these similarities, there were clearly type [activities]. The games that were on important individual differences among teach- [the computer] were just to reinforce ers, students, and classrooms relevant for this skills. But not everyone had access to it study, but these differences will be highlighted because there were always the ones who subsequently in our presentation of findings never go to it. and interpretations. The teachers were all enthusiastic about discov- ering more ways to use the computer in their Procedures instruction, which was one of our criterion for choosing to work with them in this project; The research project proceeded through however, they all acknowledged some degree several phases during the first year in Collins of trepidation about technology, which they felt and Hartwig Schools, and these phases were was typical of but not as extreme as their peers. repeated with minor variations during the For example, in an interview one teacher second year in Borders School. The phases of stated, the project and the events associated with each phase are described in this section and are We had a workshop this summer on summarized in Figure 10. using the computer as an instructional Phase 1. Phase 1 took place in the spring tool. For anyone who wanted to sign up. prior to the school year during which the ... They did a survey, and there was a intervention was to be implemented. At that

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

30 A Formative Experiment 23

time, representatives of the university research attitude toward reading in school and out of team met with teachers, the school principal, school. Teachers also administered a student and in some cases district office personnel to questionnaire designed by the university discuss the project, what expectations there research team to determine particularly the would be for the various participants, needed diversity of students' reading (see Appendix hardware, and so forth. B). A parent questionnaire was also designed Phase 2. When expectations were clearly by the university researchers to determine understood and supported by all participants, parents' perceptions of students' independent we proceeded to Phase 2, which took place reading outside of school (see Appendix C). during the summer. During several summer This questionnaire was sent home with a meetings, members of the university research letter signed by the principal investigator team trained teachers and in some instances and the classroom teacher explaining the parent volunteers and teacher aids to use the school activities that would be taking place Macintosh LC II computer as well as the basic in conjunction with the project and encour- tools and operations of HyperCard. We also aging parentsto communicate questions, discussed with the teachers our conceptions of concerns, and comments to us. The principal the multimedia book reviews, how we thought investigator also attended a fall meeting of the they might be implemented, the nature of Parent Teacher Organization in each partici- formative experiments, logistical problems, pating school to explain the project and data collection, and so forth. Together with the answer questions. Also during Phase 3, teachers, we developed an initial plan and teachers completed the DeFord Teacher schedule for implementing the multimedia book Orientation to Reading Profile (TORP; see review project beginning early in the fall. Deford, 1985) so that we might obtain infor- Phase 3. Phase 3 occurred during approx- mation about teachers' views of reading and imately the first 6 weeks of the school year. the teaching of reading. During that time, members of the university All of these quantitative and qualitative research team gathered baseline data including sources of baseline data were also employed in qualitative observational and interview data three comparison-classroom classroomsin aimed at creating a rich description of the Collins School. These classrooms were se- school and classroom environment as well as of lected because they represented similar popula- the teachers and students. We also gathered tions of students and because the teachers in quantitative data aimed specifically at determin- these classrooms were using an alternative, ing the amount and diversity of students' inde- computer-based approach to enhancing stu- pendent reading and their attitudes toward dents' independent reading. This approach was reading. Toward that end, teachers adminis- centered in the use of a commercial software tered the Elementary Reading Attitude Survey program that automatically tested students' (McKenna & Kear,1990), a standardized factual knowledge of a book they had read and instrument designed to measure students' awarded points based on their performance.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

31 24 Reinking & Watkins

Teachers rewarded students in various ways HyperCard where they worked individually or based on the number of points they received. in pairs at a computer. The instruction was One development during Phase 3 at Bor- provided by a member of the university re- ders School is noteworthy because it clearly search team, particularly one graduate student. illustrates the differences between formative The teachers brought their students to the lab and conventional experiments. Approximately and typically assisted students in carrying out 1 month into the school year, one of the teach- HyperCard tasks as directed by the graduate ers participating in the project left the school to student. Other members of the university take another position. Her replacement knew research team along with a Chapter 1 teacher little of the project when she arrived at the and parent volunteers who had been trained in school to take over the class. This development HyperCard were also typically available to would be seen as a severe threat to internal assist students. validity in a conventional experiment. How- The weekly lessons and activities for ever, from the perspective of a formative students were typically designed and modified experiment, we saw this as a new opportunity from week to week based on an analysis of the to learn about how the project activities could previous week's lesson and observations of accommodate the unforeseen variations that are students' reactions and understandings as we characteristic of dynamic educational contexts proceeded to teach them HyperCard. The and that can affect instructional interventions in weekly lessons were also specifically designed schools. to prepare students to create multimedia book Phase 4. In Phase 4, students were taught reviews (e.g., examples and sample activities how to program in HyperCard up to the level related to information about books). Between of scripting. They were informed that the the weekly lessons, the Chapter 1teacher ultimate purpose for learning HyperCard was worked with students who had been absent or so that they could create multimedia book who needed additional help. Phase 4 required reviews. That is, they learned how to use all of approximately 18 weeks in Hartwig School the HyperCard tools for drawing, copying and during the first year of the project. By the end pasting graphics, creating buttons and text of Phase 4, all students were minimally compe- fields, linking cards, and so forth. Phase 4 was tent in using HyperCard, as indicated by an carried out in Hartwig and Borders Schools but informal assessment task requiring students to not in Collins School where a computer lab use all of the HyperCard skills they had was unavailable to teach HyperCard. There learned. During all of the weekly lessons, were important differences between Phase 4 in observational data were recorded by a member Hartwig School during Year 1 and in Borders of the university research team. School during Year 2, and these differences Based on our experiences and analysis of reflect the formative aspects of this approach to data during Year 1at Hartwig School, the research. In Hartwig School, students came to university research team developed a series of the computer lab once a week for instruction in lesson plans aimed at teaching students how to

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

32 A Formative Experiment 25

program in HyperCard.' These lessons were into the template. However, the template and presented to the teachers involved with the its use varied across the three schools. In project at Borders School during two 4-hr Collins School, the university research team sessions during the summer (Phase 2) before created the template based on the specifications the start of Year 2. Several of the teachers decided by students and teachers, because from Hartwig also participated in these ses- students were not taught how to program in sions. Our intent was to discover whether HyperCard. Thus, students at Collins school teachers with relatively little background in were limited to the categories of information using computers and who had no knowledge of available on the template. At Hartwig School, HyperCard could be trained in a reasonable students not only created the template them- time to teach their students HyperCard with the selves (with some guidance from teachers and lesson plans, materials, and activities we had the university researchers), but they also were developed. At Borders School, the Chapter 1 able to use their HyperCard skills to go beyond teacher presented the content of the lessons to the standard review template. We found that whole classes for 1 hr a week in the school's this approach had advantages but also pre- media center using an LCD panel. She fol- sented problems. For example, when students lowed up this whole-group presentation on created their own version of the standard Monday with groups of 8-12 students on template, there were frequently small varia- Tuesday through Friday. In the small groups, tions, omissions, or programming errors in students reviewed the content of the lesson and their HyperCard stacks that made the possibil- completed activities on computers in the lab for ity of a database search problematic. At Bor- approximately 40 min each. Under these condi- ders School, we compromised these extremes tions, students were able to master HyperCard by developing a standard working template for programming in approximately10 weeks. students while adding a button to the template During Phase 4 at Borders School, teachers that would enable them to use their HyperCard critiqued the effectiveness of the lesson plans in skills to enter personalized information about a writing and made suggestions for further im- book they had read. provements. Members of the university re- Phase 6. Phase 6 involved compiling search team were also collecting and analyzing students' book reviews into a database and observational, interview, and video data during making the database available for use in a that time. central location. A number of factors prevented Phase 5. In Phase 5, the book review us from reaching this phase during the first template was finalized and students entered year in Collins and Hartwig Schools. Many of information about the books they were reading the same factors explained why we did not reach Phase 6 until the end of April in Borders School during the second year of the project. 2The lesson plans and related materials are available as an Mainly, we had underestimated the interact- instructional resource from the NRRC (Reinking & Bonham, 1996). ing logistical, pedagogical, and technological

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

33 26 Reinking & Watkins

problems associated with teaching teachers and for reporting our results in the following sec- students HyperCard, creating workable pro- tion. grams, and consolidating the book reviews into Quantitative data were gathered during the database so that they could easily be Phases 3 and 7 of the project, as described in searched. Although the purpose of a formative the previous sections. Although we chose to approach is to discover necessary adaptations collect these data, we do not believe that and adjustments, we underestimated the num- quantitative data are essential to conducting ber and extent of these revisions. Thus, a a formative experiment. We see formative major component of the intervention aimed at experiments existing within the domain of addressing the pedagogical goal was imple- research that Salomon (1991) has described as mented only marginally during the second year systemic, a descriptor that he argues transcends of the project. However, we have data indicat- qualitative and quantitative paradigms (see also ing that creating multimedia book reviews, Eisenhart & Borko, 1993). Neither is a pre-/ even if they are not entered and searched into post-intervention comparison,as we have a database, effects changes relevant to our done, essential. The goal of formative experi- pedagogical goal. ments to modify instructional interventions to Phase 7. In Phase 7, we had students, accomplish a pedagogical goal necessitates that teachers, and parents complete again the writ- they be more open ended than conventional ten surveys administered in Phase 3. These experiments. Newman (1990) provides a basis data were gathered during the last few weeks of for this open-endedness from a sociohistorical the school year in May and allowed us to perspective of cognitive change that draws compare any differences that may have oc- heavily on the Vygotskian concept of the zone curred in responses on these instruments since of proximal development (ZPD; see also New- the baseline data were collected at the begin- man, Griffin, & Cole, 1989). He sees com- ning of the school year. Data were collected in puter technology as a means for bringing the classrooms using the multimedia book teachers' and students' interactions into the review intervention as well as the three com- ZPD because it amplifies teachers' capability parison classrooms using the computer-based to organize the educational environment. Yet, point system. this sociohistorical perspective leads him to reject the pedagogical goal as a static endpoint Data Collection and Analysis for all students. Many different endpoints are possible, some of which may transcend the In this section, we present the types of original goals. Thus, the goal identified in a data we gathered and how we gathered it. We formativeexperimentimpliesnospecific also outline our perspectives on how we view standard for measuring achievement of the data collection within our emerging under- pedagogical goal, and the goal can never be standing of a formative experiment. Our origi- achieved in an absolute sense. A formative nal and current perspectives provide a context experiment is not completed when a predeter-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO 55

34 A Formative Experiment 27

mined criterion has been reached but instead From the outset of this investigation, we stops at some point because of practical con- had a conceptual understanding that formative straints (e.g., a school year ends). experiments entailed collecting data continu- Nonetheless, formative experiments must ously to guide iterative modifications of the provide a substantiated baseline description of instructional intervention in order to enhance studentsand theeducational environment its effectiveness in accomplishing the pedagogi- relative to the pedagogical goal as a reference cal goal. However, the absence of detailed point from which progress can be gauged. Data models of formative experiments in the litera- supporting the description may be quantitative, ture prevented us from knowing exactly how qualitative, or both. Beyond baseline data, a this might be accomplished in practice. Origi- formative experiment requires on-going, con- nally, we imagined that data collection and tinuous data collection aimed at determining modifications to the intervention would pro- what factors enhance or inhibit the interven- ceed through well-defined cycles. In practice, tion's success in accomplishing the pedagogical we found that such distinct cycles did not goal, what the unanticipated by-products of the occur. Instead, the relation between data col- intervention are, and how the degree to which lection and modification of the intervention the intervention is being appropriated into the tended to be fluid, even at times ad hoc in the educational context. sense that adaptations were based on the intu- Thus, in addition to the quantitative data itive demands of the moment rather than a gathered in Phases 3 and 7, qualitative data rigorous analysis of data. were gathered during Phases 2-6. Sources of In our reflection on this investigation as qualitative data included (a) taped, semistruc- an example of a formative experiment, we tured interviews with teachers (e.g., about their have considered several explanations for this experiences with computers, see Appendix A); relation. First, we discovered early during (b) log books in which teachers recorded their the first year of the project that we had been observations about events related to the project; too ambitious in deciding to work in two (c) focus-group discussions with students; (d) schools simultaneously. Likewise, we under- observational field notes; and (e) videotapes of estimated the complexity of implementing various events during the project. In addition, the intervention in two schools where the we examined students' book reviews on the technologicalinfrastructurenecessary to computer, both qualitatively (e.g., to examine support the intervention was not firmly in the types of books they were reading) as well place. Consequently, during the first year, as quantitatively (e.g., the number of books we were not able to engage in the level of they were entering). These data were analyzed systematic data collection and detached for the purpose of understanding what effects reflection and analysis that we had hoped to the intervention was having on the pedagogical achieve. Neither were we able to implement goal and on other related aspects of the educa- the database activity during Year 1 and only tional environment. partially in Year 2.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

35 28 Reinking & Watkins

Another explanation was our discovery Taking a formative stance, we became that our research activities during the experi- aware of the unanticipated challenges and ment were formative in several different complexities we were encountering; we made areassimultaneously.For example, we several adjustments that affected data collection continuously made adjustments that were and analysis. For example, we decided to work responses to factors in the following areas: in one school during the second year of the logistical (e.g., How could we create more project, where we would also have the added time in the daily schedule for students to benefit of our first-year experiences. work on the computer), methodological In addition, we decided to gather more (e.g., Our observations of students working in-depth data on 4 focal students in each in the lab on one day might suggest that we class. These 4 focal students were identified needed to interview the teachers about some- by the teacher as being representative of thing we observed on the following day), students in each of the following categories pedagogical (e.g., How might we adjust based on combinations of reading ability and implementation to encourage more poor interest in reading independently: (a) above readers to create book reviews?), technologi- average in ability and in interest, (b) above cal (e.g., The search speed on the database average in ability but below average in program is too slow. How 'can we make it interest, (c) below average in ability but run faster?), interpersonal (e.g., How can above average in interest, and (d) below we as university researchers establish good average in ability and interest. To insure the rapport with the teachers; maintain teacher validity of selecting students to represent morale; take into account the needs of teach- these categories, a member of the university ers, administrators, parents, and support research team who had collected baseline personnel; etc.), and ethical (e.g., Should data in the classroom and the students' we adjust the way we implement the inter- teachers from the previous school year also vention based on our values or the teachers' independently categorized students along values?). In addition, adjustments made in these dimensions before being aware of the response to a concern in one area would fre- teachers' selections. There was no disagree- quently have implications requiring adjustments ment that each focal student fell within the in another area. The interacting effects of categories identified. Teachers and members adjustments in multiple areas required that we of the research team took steps to insure that be less structured than we anticipated in our focal students were not aware that they had approach to conducting a formative experi- been selected for special attention during ment. However, at the same time this real- data collection. For example, we observed ization reinforced our belief that formative and recorded our observations about focal experiments deal realistically with the inher- students inconspicuously, and we noted their ent complexities of implementing instructional responses to specific questions that were interventions and researching their effects. addressed to a group of students or, when

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

36 A Formative Experiment 29

addressed to them individually, were done so initial choice as an approach to data collection. informally as opposed to a formal interview. As we transcribed and annotated field notes, Our formative stance toward data collec- we determined that this approach created too tion, which led to on-going adjustments in what many gaps in our understanding of students' data we gathered and how we analyzed it, is independent reading. Thus, we expanded our consistent with current thinking in qualitative data collection and analysis to include focused and ethnographic approaches to research (see group discussions with teachers and students LaCompte & Preissle, 1993). As is accepted using techniques and theoretical perspectives practice in qualitative studies, the data deemed described in Morgan (1993). most useful emerged from our on-going analy- Another exampleillustrates how our sis and emerging interpretive theories (Glaser evolving understanding of formative experi- & Strauss, 1967), but also from our evolving ments affected our selection of theoretical understanding of formative experiments. The frameworks for data collection and analysis. concept of a formative experiment provided a As stated previously in this section, we discov- broad theoretical framework for collecting and ered during the project that a formative stance analyzing data. However, currently, formative toward the instructional intervention applies to experiments are methodologically and theoreti- several distinct but interacting areas. We real- cally ill-defined. At the same time, the general ized that a formative approach to classroom concept of a formative experiment implies research and the emergence of these distinct adaptability in collecting data (Baumann et al., areas were consistent with Patton's (1990) 1996). Thus, theoretical frameworks for data description of using qualitative research meth- collection and analysis were varied in response ods, specifically a process/outcomes matrix, to to on-going developments within the project, evaluate programs: our emerging theories based on data collected earlier in the project, and our evolving under- The linkage between processes and out- standing of formative experiments. comes is a fundamental issue in many For example, we began gathering obser- programevaluations.An evaluation vational data in classrooms using Glaser and research design based on qualitative Strauss' (1967) "frame work of local concepts" methods can be particularly appropriate as an approach to determining the amount and where either program processes or program impacts, or both, are largely diversity of students' independent, reading, unspecified, for whatever reasons. Some- which was critical to gauging progress toward times the reason is because outcomes are the pedagogical goal of this formative experi- meant to be individualized; sometimes ment. Our local framework consisted of our the program is simply uncertain about experiential knowledge of classrooms, teach- what the outcomes will be; and in many ers, and students in general. This knowledge programs neither processes nor impacts suggested that observation of the regular class- have been carefully articulated. Under rooms and the computer lab would be a logical such conditions, one purpose of the

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

t.? 37 30 Reinking & Watkins

evaluation may be to help articulate accomplish our pedagogical goal. The key program processes, program impacts, events are organized into themes indicated by and the linkages between the two. This the headings in this section. Despite our real- task can be facilitated by constructing a ization that the formative adjustments during process/outcomes matrix to organize the the experiment occurred in response to factors data. (p. 415) in a variety of areas as we discussed previ- ously, we report here only those adjustments Although we did not set out to create a related to these key events. process/outcomes matrix, our concept of a In the discussion section, we will summa- formative experiment guided us to look for rize and discuss the results within the frame- linkages between processes and outcomes work of the questions guiding a formative under the conditions that Patton describes. experiment as presented earlier in this report. Interestingly, our efforts led us to formulate six Before presenting key events, we provide distinct areas that could become the foundation baseline data concerning the amount and diver- of such a matrix (i.e., logistical, methodologi- sity of students' independent reading as a cal, pedagogical, technological, interpersonal, reference point for considering progress toward and ethical). Thus, we conclude that Patton's this experiment's pedagogical goal. theoretical framework for program evaluation may be a useful one for conducting formative Baseline Data experiments. Baseline data were gathered during Phase Results 3 of the experiment, which extended from about the second week of the school year and As with any approach to research involv- continued through mid-October. Data collec- ing inductive methods over an extended period tion during this period focused on (a) deter- of time, it is not feasible to report in detail all mining the extent and diversity of students' of the data collected in this formative experi- independent reading and on (b) generating a ment. Of necessity, the data reported represent general description of the school and classroom the researchers' synthesis and interpretation of environments, including teachers' orientations events, and it may be organized and presented toward instruction and the use of computers. in various ways. We have chosen to use a Sources of data included the quantitative instru- format that Patton (1990) refers to as "key ments described in the procedures section of events," which he defines as critical incidents this report, observational data, and focused or major events, not necessarily presented in group interviews with students and individual their order of occurrence. To present our interviews with teachers. Because we have results, we chose events that illustrate our already briefly described the school and class- major findings or that were pivotal in determin- room contexts and will continue to explicate ing how the intervention was implemented to them in conjunction with presenting our results

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

88 A Formative Experiment 31

in this report, we will focus here on data in Table 2. Of the variables listed, "range of related to the amount and diversity of inde- reading" (see Table 2) may be considered a pendent reading and to teachers' orientation broad indicator of students' levels of inde- toward reading. pendent reading outside of school. The Through our baseline observations and means reflect the average values across all focused group interviews with students at each students in each class whose parents rated school, we determined that there was already a them on 5-point likert scales associated with good deal of independent reading occurring in a variety of reading activities outside of the classrooms with which we worked in this school (item 9 on the questionnaire; see investigation. We frequently observed students Appendix C). The mean values for each reading during the school day; many had books class range from 2.73 to 3.15 indicating in their desks; they knew where the local moderate levels of reading outside of school, library was; they read at least occasionally at as perceived by parents, with little variation home, and so forth. Our observational data across the 9 classrooms. indicated that although there were a few stu- Teachers' views toward teaching read- dents in every class who did very little, if any, ing were determined in part through their independent reading, the majority of students at responses on the TORP. Results of their leastoccasionally engaged in independent responses to 5-point likert scale items are reading. The variations between classrooms shown in Table 3, which lists means and and across schools in this regard was surpris- standard deviations by individual teacher ingly small. Similarly, the mean raw scores on across the TORP's three orientations to Elementary Reading Attitude Survey (ERAS; teaching reading: phonics, skills, and whole McKenna & Kear, 1990) indicated that students language. The range of scores across all in these classes had relatively positive attitudes teachers and orientation scales on the TORP about in-school and out-of-school reading (see completed before the project began was 2.3 Table 1). On that instrument, all but two of the to 3.8 and the maximum difference among classes (Ms. Andrews at Collins and Ms. the scales for an individual teacher was 1.4. Sievers at Borders School) had mean raw These values do not indicate strong differ- scores that fell above the 50th percentile. ences in orientations to teaching reading An indicationof students'baseline among the teachers nor an especially strong levels of independent reading outside of commitment to any one of the orientations school from the perspective of their parents by a particular teacher. These results are can be found in a questionnaire completed consistent with our observations of and discus- by parents before the project began. The sions with the teachers. A notable exception questionnaire is included as Appendix C; a is Ms. Andrews who expressed a strong key to scoring the questionnaire is shown in commitment to whole language but whose Figure 11; and, the means and standard responses on the TORP were weighted more deviations for selected variables are shown heavily toward phonics and skills.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

39 32 Reinking & Watkins

Variable Name (Range) Items Scoring

1.Free Time 1-2 0 = reading not checked (0-2) 1 = reading checked 2 = reading circled

2.Reading/TV 2-3 ratio of hours (nearest 1) (x/y) reading (x) to hours viewing TV (y)

3.Reading for enjoyment 4 0 = rarely (04) 4 = all the time

4.Ability estimate 6 well above = 4 (0-4) well below = 0

5.Children's books 7 none = 0 (0-5) fewer than 5 = 1 more than 100 = 5

6.Range of reading 9 sum of all responses on 1-5 scale (x) (x/22) divided by 22 (scores for 3 items reversed)

7.Library card 11 no = 1; yes = 2 (1-2)

8.Trips to library 12 > 5 = 6 (0-6)

9.Books checked out 13 none = 0 (0-3) 1 to 3 = 1 4 to 10 = 2 > 10 = 3

10.Change in reading 19* 0 = much less (0-4) 4 = much more

11.Change in interest 20* 0 = much less 4 = much more

*Items included only on postexperimental questionnaire

Figure 11. Key to variables on parent's questionnaire.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

49 Table 1. Means (Standard Deviations) for Pre- and Post-Experimental Raw Scores on Subscales of the Elementary Reading yz Attitudes SurveySchool/Teacher (ERAS) pre Recreational Reading gain/loss' post pre Academic Reading gain/loss° post pre gain/loss' Total post tt AndrewsCollins School (n = 25) ( 5.37)29.7429.36 .96 ( 4.72)29.2430.32 ( 6.27)29.1923.08 .00 24.1923.08 5.33 (10.28)58.9352.44 .96 ( 8.49)53.4353.40 tr1 BurtonHartwigBroward (n (n = =16) 21) School ( 5.40)29.00 (.50) .25 ( 4.63)29.25 ( 6.19)26.88 (5.00)** 1.13 ( 4.17)28.00 (10.74)55.88 (5.50)** 1.37 ( 7.43)57.25 7z1tri Palmer (n = 21) ( 6.61)5.50)29.76 (.14) ( 6.40)4.95)29.62 ( 8.40)4.58)28.0525.24 3.62* ( 4.89)6.47)28.8631.00 (14.15)( 8.92)59.6055.00 3.48*5.00* (10.87)( 8.67)58.48 MorrisBordersPearson School(n = 20) (n = 26) ( 3.35)27.3331.55 (2.05)*(1.91) ( 5.19)25.4333.60 ( 5.19)25.50 (2.52)(2.95) ( 3.85)23.25 ( 6.58)52.83 (4.15) ( 7.69)48.6864.60 trl SieversComparison (tt = 20) Classes ( 5.06)7.97)27.00 1.15 (10.07)( 6.96)28.15 ( 5.59)7.29)24.96 .32 ( 8.23)6.69)25.28 (14.44)( 8.96)51.96 1.47 (12.83)(17.33)53.43 :$ an*p = gain(loss) < .05. **pTeacher < .01.1 2 (n =***p 22) < .001. (n = 18) ( 6.38)4.78)31.0930.50 (3.73)**(2.06) ( 6.92)4.62)27.3628.44 ( 7.37)5.89)27.8629.06 (5.04)*(8.34)*** ( 5.80)5.04)22.8220.72 (12.68)( 9.03)58.9559.56 (8.77)**10.40 ( 9.17)11.3550.1849.16 41 BEST COPY MAILABLE 42 Table 2. Means (Standard Deviations) for Selected Variables on Pre- and Post-Experimental Parents' Questionnaire' FREE TIME READING/TV ABILITY ESTIMATE CHILDREN'S BOOKS RANGE OF READING CollinsSchool/Teacher School pre gain/loss° post pre gain/loss° post pre gain/loss° post pre gain/loss° post pre gain/loss° post BrowardAndrews (n = (n = 14) 11) (.47)(.40) .29.18 .14.28 (.51)(.52) .43.46 (.42)(.36).31.49 .11.05 (.33)(.39) .42.44 (.73)(.93)1.772.55 .38** .19 (.80)2.15(.67)2.36 (.80)3.21(.98)3.82 .14.09 (1.07)3.07(.83)3.91 (.45)2.82(.44)2.85 .11.24 (.47)2.93(.37)3.09 PalmerBurtonHartwig (nSchool = 19) (n = 16) (.45) .90.25 .53*.31 (.81) .37.56 (.36) .52.31 .43*.13 (.50) .95.44 2.63(.76)2.43 .06.07 2.69(.76)2.50 (1.24)3.162.75 .10.06 (1.20)3.262.69 (.79)3.102.99 (.16).25* 3.35(.63)2.83 BordersPearson School (n = 21) (.67)(.81) .41 .18 (.59)(.60).59 (.51)(.56) .43 .17 (1.11)(.57) .60 (.70)2.47(.81) .05 (.61)2.42(.79) (1.49)(1.34)3.14 .24 (1.26)(1.33)2.91 (.64)(.69)3.15 .04 (.51)(.52)3.19 SieversMorris (n =(n = 13)14) (.78)(.41) .54.20 .08.13 (.52)(.62).46.33 (.54)(.82) .51.43 1.51*.96* (1.37)(2.55) 1.471.94 (1.18)2.31(.91)2.53 .23.06 (1.22)2.54(.83)2.47 (1.17)3.23(.99)3.13 .51*.34 (1.13)3.74(.52)3.47 (.72)(.69)3.092.73 .14.05 (.79)3.23(.65)2.78 TeacherComparison 21 (n Teachers = 22)17) (.80) .29.46 .26.09 (.86) .51.55 (.70) .18.60 .14.00 (.53) .32.60 2.43(.96)2.71 .14.10 (.97)2.62 .32 (1.05)3.003.41 .46.09 (1.13)2.543.32 2.97(.79)2.95 .21.03 (.81)2.762.97 43 *p'See < Appendix .05. **p C < for .01. complete questionnaire= gain(loss) and Figure 11 for explanation of variables. (.49) (.79) (.12) (.31) (.54) (.79) (1.29) (1.13) (.51) .70 44 Table 2. (Continued) Means (Standard Deviations) for Selected Variables on Pre- and Post-Experimental Parents' Questionnaires LIBRARY CARD TRIPS TO LIBRARY BOOKS CHECKED OUT INTERESTCHANGE IN AMOUNTCHANGE IN READINGENJOYMENT FOR School/TeachersCollins School pre gain/loss' post pre gain/loss" post pre gain/loss' post post post pre gain/loss" post AndrewsBroward (in = 11) (n = 14) (.47)1.361.73 .43**.27* (.05)2.001.79 (2.10) 1.291.80 .89*.00 (.33)2.18(.92)1.80 (.30)1.211.09 .08.18 (.47)1.291.27 (1.04) 3.142.46 (1.13) 3.072.55 (1.08) 1.691.82 .08.00 (1.08) 1.771.82.93 HartwigBurton School (n = 16) (.51)(.47)1.56 .07 (.50)(.51)1.63 (.92)(.42)1.13 .20 (1.45) 1.33 (.34)(.73)1.13 .19 (.57)(.80) .94 (1.09)(.80)2.44 (1.09)(1.07) 2.38 (1.06)(.75)1.53 .84* (1.06)2.37 PearsonPalmer (n (n = = 19) 21) (.48)(.81)1.681.74 .14.00 (.40)(.60)1.821.74 (1.80)(.56)2.001.67 .09.06 (1.58)(1.11)1.761.94 (.70)(.81)1.141.11 .09.00 (.61)(.79)1.051.11 (.87)(.86)2.772.68 (1.59)(.71)2.862.38 (1.12)2.10(.69)1.94 .56*.05 (.93)2.15(.52)3.35 BordersSieversMorris School (n =(n = 13)14) (.47)1.851.71 .07.00 (.47)1.921.71 (1.88)2.501.77 .00.62 (1.56) 2.502.39 (.47)1.081.29 .07.08 (.43)1.151.21 (.63)2.79NA 2.71(.65)2.50 (1.08) 1.771.64 .23.15 (1.05)2.001.79 TeacherComparison 1 Teachers (n = (.38)1.73 .18 (.28)1.91 (1.00) 1.57 .38 (1.93) 1.95 (.28)1.65 .35 (.56)1.30 (1.31)2.35 (1.27)2.35 (1.17)1.96 .18 (1.16) 2.14 *pbe,aSee <= Appendixgain(loss).05. **pTeacher C< for .01.2 complete questionnaire and Figure 11 for explanation of variables. (n = 22)17) (1.25)(.46)1.71 .00 (.49)(.29)1.71 (2.41)(1.67)2.14 .00 (1.77)(1.60) 2.14 (1.88)(.49)1.29 .15 (.38)(.47)1.14 (.58)2.00(.81) (.69)2.14(.86) (1.29) 1.14.69 .43 (1.13) 1.57.79 45 BEST COPY AVAILABLE 46 36 Reinking & Watkins

Technology and Students' Interactions February 23 With Peers and Teachers I saw the book Where the Red Fern Grows Analysis of our field notes and videotapes on Dee's desk and asked her if she was read- of students and teachers in the computer lab ing it. She said that she was and that she documented consistently that peer interaction liked it. When I asked her why she picked it was greater during times devoted to the multi- to read, she said that she didn't know why media book review activity than during times and didn't seem to remember the incident devoted to other academic activities. This with Aaron. finding is consistent with Dickinson's (1986) ethnographic study of collaborative writing Similarly, Ms. Pearson at Hartwig School involving computers in primary-grade class- wrote in her log in January a comment that rooms. However, because formative experi- coincidentally refers to the same book: "Beth ments focus on how an instructional interven- Rollins read 'Where the Read Fern Grows.' tion contributes to the accomplishment of a Most of the people in that section of the com- pedagogical goal, we were interested in how puter lab have now read that book." the increased peer reaction might relate to As represented by these incidents, we found students' reading. Further analysis revealed that contending with the technological skills that increased peer interactions mediated the necessary to create multimedia book reviews effect of multimedia book reviews on students' led to increased peer interactions, which often independent reading. This mediation is illus- led in turn to incidental sharing of information trated by the following event transcribed from about books, and, for some students, to more tape recorded observations at Collins School: reading. This finding is important because it was not anticipated to occur in the early stages February 16 of implementing the intervention. Originally, I worked with Aaron, Dee, and Tyrone' to we were interested in moving quickly through show them how to add audio to their basic the necessary technological training so that template [on the computer]. I asked them students could enter book reviews into a data- who had a book we could use as an example. base. Our prediction was that the database Aaron said he had Where the Red Fern Grows would provide a mechanism for students to and got it from his desk. He told some of the exchange information about their reading and story into the microphone and I showed how his comments could be recorded while the thus promote more interest in reading a wide others observed. Dee asked Aaron if she variety of books. Despite the fact that we were could look at the book after we had finished. not able to implement fully the database com- ponent as planned, we unexpectedly found 3All students' names are pseudonyms throughout this evidence that learning technological skills led report. The three students in this incident were focal to incidental sharing of information about students. books.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

47 A Formative Experiment 37

In addition, the interactions we observed of HyperCard tools were often cause for during the times students were working on special attention and sharing. For example, at multimedia book reviews in the computer lab Hartwig School, Betty, a student who had a were different from their interactions with Macintosh computer she could use at home, peers and with teachers in the classroom. Two created a special presentation on the computer related differences are illustrated in an event at as a Valentine to Ms. Pearson, her teacher. Borders School. We were working with the The whole class gathered around a computer to teachers there to select some video clips to use enjoy her presentation, several students asking, in a presentation they were developing for a "How did you do that?" Helpful interactions state language arts conference. Ms. Ellers tended to occur spontaneously; but at Hartwig selected a clip of two girls, one who performed School, we (the teachers and the university well academically and one who performed researchers) tried to enhance this positive effect poorly, working on their multimedia reviews in by asking students who questioned us about the computer lab on adjoining computers. some aspect of HyperCard to consult with Because the girls were clearly interacting about another student who could help them. Across some aspect of entering their book reviews, all of the classrooms we observed, negative this clip was chosen to illustrate student collab- interactions occurred less frequently in the oration and cooperation. The audio was of poor computer lab and they tended to be conflicts quality and our first assumption was that the related to using the technology such as a dis- high achieving girl was assisting the low pute about whose turn it was to type at the achieving girl. However, after watching the keyboard. clip several times and listening carefully to the Another difference illustrated by the inci- audio, Ms. Ellers, with enthusiastic surprise, dent on the videotape was that, in the computer pointed out that the low achieving girl was lab, helpful interactions originated from the actually offering the assistance. lower achieving students as often as from the As illustrated by the previous event, stu- higher achieving students. Students who had dents' interactions related to entering multi- academic difficulties in the classroom some- media book reviews were more cooperative and times became experts in the computer lab collaborative in the computer lab. Dealing with where they might be called upon to assist their the challenges of learning the technological higher achieving peers. For example, our field skills necessary to enter multimedia book notes taken while observing Shawn, a focal reviews seemed to generate a heightened sense student classified as low reading ability and of camaraderie and helpfulness among stu- interest, state, dents. In the computer lab, they seemed genu- inely interested in the achievements of their Shawn has been ready to assist others in classmates and in the products they were reading what was on their screens in order developing. Discovering special effects on to figure out what should be done next; the computer screen or creative applications and he was not at all inhibited about

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

48 38 Reinking & Watkins

helping someone new to the class with classroom stated, "It doesn't bother me at all to what needed to be done. say to one of my children to tell me how you did that (on the computer)." As this example indicates, the assistance of- We did not find any evidence that the num- fered by students having low reading ability ber of computers had any noticeable effect on frequently involved reading and interpreting the level of peer interactions. At Collins school texts. where there was a single computer in each Interactions between teachers and students classroom, peer interactions related to the were also affected by participation in the multi- multimedia book reviews occurred over an media book review activities. For example, it extended period of time as small groups of was clear that teachers did not define them- students took turns using the computer during selves as experts when presenting information free time. At Collins and Hartwig Schools, two related to the multimedia book review activi- or three students often worked together at the ties. Neither did students hesitate to offer their computer. However, at Borders School, where expertise to teachers. In fact, all of the teachers students typically worked at their own com- involved in this project seemed to enjoy the puter, there was still considerable interaction idea that many students exceeded their own among the students working in the computer technological understandings and that students lab. Students would be curious about each were often able to teach their teachers. The other's work or seek assistance, which led to following statement by Ms. Ellers' from an considerable interaction. October meeting of the teachers and research The connection between increased peer team illustrates this viewpoint: interaction and incidental sharing of informa- tion about books illustrates how findings that . .. and they love to show me because emerge during a formative experiment can even with the demonstration this week of guide adjustments in implementing the inter- the new fields, you know (gave rapid vention. For example, at Hartwig School we directions on how to do fields), well then tried to enhance the effect of increased interac- I said you need to do that each time. Well tions as they related to books by pairing stu- then today, and this was the first time that I even knew it, they said 'oh you don't dents to debug HyperCard stacks created by have to do that for a new field, you just go students in another class. Students completed up to new field and automatically go over this activity periodically using a guide sheet there on the tool palette.' Like wow that we provided for them. We reasoned that in (laughing). I didn't know that. But hey, the process of debugging the programs, paired they really get a kick out of that. students might also interact with each other about the books being reviewed by their peers Likewise, Ms. Palmer at Hartwig School ina in other classes. Interestingly, we did not find semistructured interview about her experiences that to be the case. In fact, the guide sheet with and attitudes about computers in the tended to encourage students to take on the role

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

49 A Formative Experiment 39

of highly critical editors who looked for sur- is easy books anyway, so why put them on face level mistakes in spelling and punctuation. the computer." Likewise, when we purposefully used students' We hypothesized that some of the poor work from Collins School as examples to readers, in Ms. Broward's class may feel the students in the Hartwig School, the Hartwig same way; that is, perhaps they were embar- students often focused on surface level mistakes rassed to enter books below grade level, thus in the examples. We did find some evidence publicizing their reading problems. We met that this critical stance did enhance students' with Ms. Broward to discuss this possibility. concern for the accuracy of their own work. She agreed that it might explain the lack of However, because students' reviewing of each interest among the poor readers, and we dis- others' work seemed to distract students from cussed possible solutions. Ms. Broward sug- activities directly related to the experiment's gested that since we had discussed eventually pedagogical goal, we did not introduce this making the database containing her students' activity at Borders School during Year 2. reviews available for students in lower grades to find books they might like to read, it would Effects Related to Reading Ability be necessary to have easier books in the data- base. From our data, especially related to focal This observation evolved into a relatively students, we noted that reading ability often minor but effective formative adjustment in the figured prominently in determining and under- way the intervention was implemented in Ms. standing the effects of the multimedia book Broward's class. Soon after our meeting, Ms. review activity, although these effects were Broward announced to the class that she was complex and often varied across classrooms. concerned about the lack of multimedia book For example, Ms. Broward's class at Collins reviews of easier books. She explained that School had a disproportionately high number of eventually some second- and third-grade stu- readers reading well below grade level when dents would be interested in searching the compared to other classes in her school. She multimedia book reviews for books they might often expressed concern about, students' lack of like to read. She expressed hope that some achievement and communicated her displeasure students would enter easier books for these to students when she felt they were not meeting students. Following this announcement, we their potential. In her log, Ms. Broward noted noted an increase in the number of book re- that many of the poor readers in her class views entered by poor readers. In fact, within did not seem interested in entering book several weeks, James had entered more books reviews. In a focused group interview with than any other student in the class. Because he her students where we were seeking students' tended to be a leader, at least among his poor- reactions to entering their books on the com- reading peers, his sanctioning of the activity puter, James, one of the focal students classi- seemed to encourage other poor readers to fied as a poor reader, stated, "All I can read enter easy books too.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

50 40 Reinking & Watkins

On the other hand, in Hartwig and Borders issues related to creating multimedia book Schools where we taught students basic Hyper- reviews tended to obscure visible differences Card skills, the low achieving students seemed in reading ability, which were more obvious to gain confidence and self-esteem immediately when students were engaged in other aca- from working on the computers. For example, demic activities. For example, we attempted Robert, a focal student in special education at to validate independently through our obser- Hartwig school, one day presented a member vations teachers' selections of focal students of the university research team with an illus- who varied on reading ability and interest. It trated poster printout he had made on the was not possible to do so by only observing computer that stated,"I love computers." students working with the computers, as Follow-up discussions with his classroom indicated by these notes from our October teacher and special education teacher revealed field notes while observing in the computer that Robert was enjoying his new status as a lab: computer expert not only in the lab but also showing others how to use the computer for Q (to myself) What about 4 selected word processing in his classroom. Ms. Pear- students? I can't tell who they might be in son, his teacher, wrote in her log, "When we here. .. Q (to myself) Does anyone got a word processor to use, Roger was the one appear to have a problem with reading? who showed the rest of the class how to use it. Doesn't look like it in here. He was proud of himself, and so was I." There is some evidence that Roger's atti- On the other hand, it was possible to do so tudes toward reading improved as well. His by observing students in the classroom, as overall score on the Elementary Reading Atti- indicated again from our October field notes: tude Survey (ERAS) increased from the 62nd "Aide [in classroom] assisting students who are percentile in September to the 78th percentile having trouble with their work. Teacher moni- in the following May, although there was little toring/checking students comprehensionesp. evidence that he was reading more or more poor students." The teachers' comments and divergently as measured by the Parent's Ques- our own observations repeatedly converged to tionnaire and the Choosing Things to Read indicate that distinctions in reading ability were instrument. Interestingly, his higher overall less noticeable in the computer lab than in the score on the ERAS was the result of a substan- classroom. tial increase on his score for the school reading Shane, one of the focal students classified subtest that compensated for a slight decrease as low ability and low interest in reading, was in his home reading score. observed to have many difficulties reading in Related to our finding that dealing with his classroom, but had no problems reading in technological issues increased peer interac- the computer lab. We observed him avoiding tions across academic ability, we found that reading and answering questions in the class- dealing with the more complex technological room but not in the computer lab when work-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

51 A Formative Experiment 41

ing on the multimedia book review activities. E:I told them all the things that we had been In an interview, Ms. Ellers stated: doing that day in the computer lab, and how I couldn't wait until the next time we go in Shane ... [was] really loving the initial there. It's real fun. stuff where I would show the whole group what we would be doing on Monday. With (discussion about the previous HyperCard their little group, I would show them on lesson on audio buttons) the overhead and they would see what we were doing that day. And he would re- member itall. He was very successful E:That was fun. I did mine. I did all my voice when we were just creating their stacks impressions. I did my Zsa Zsa Gabor impres- and doing the fields and buttons and copy- sions, I did a real sophisticated lady impres- ing from Art Bits (clip art available for sion, and Mrs. Morris was saying, Go on, students to use in their reviews) and what- keep going. not. Likewise, in a February interview Ms. Ellers Lower ability students who were not especially commented on the fact that Elizabeth has adept at using the technology could distinguish always been a good reader, while citing an themselves in other ways, too. For example, example of how working on the multimedia Ms. Burton states in her log, "Even my slow book reviews provided an opportunity for ones are doing a good job. Lee and David recommending a book to her teacher. (special ed.) have some good ideas for creating stacks. They need more help, but other stu- Elizabeth is a very top-notch student. She dents are there to help out." reads a lot anyway. She even brought me The multimedia book review activities also a book before Christmas and said you tended to stimulate creativity among many of need to read this. I said I would try to the high ability readers; in our estimation, read this over the holidays. She was in my more so than would be likely with conventional third-grade classroom and we've been real book reports. And, it seemed to reinforce their close, and so she is comfortable with me. typically strong interest in reading. For exam- Of course she is doing well and I can see ple, Elizabeth (E), a focal student identified as an increase in reading. She has always having high ability and interest in reading been a top reader. responded to a researcher's (R) interview questions as follows: Transcriptions of a tape-recorded April R:Have you told anyone about this HyperCard meeting of the university research team and the project? project teachers aimed at discussing how to increase students' sharing information about E:I've told basically friends and family. books revealed that good readers were already R:What have you told them? doing this:

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

52 42 Reinking & Watkins

R:Could we have the students to exchange book where his work page is. Shane answers

reviews ... ? that he has thrown away his work. Teacher

says to add the work on to his paper. ... Ms. E:They've already done that. One'll hear I haven't been looking, but Shane appears what another says, or they will tell me to be done again (no one else is). about their booksbut these are the top students. (notes from computer lab observation, two days later) Similarly, Ms. Palmer at Hartwig School wrote in her journal, "I have found that students that Shane passes out the template guide at the I find that read consistently were the ones that beginning of class....All students work- had several books to put into the files after ing now; no one off task. A new students Christmas." gasps. Shane (in a neighboring seat) says, "What's wrong?" Antoinette says,"I don't know." Shane immediately scoots Increased Student Engagement over to help her. ... Our data collected in various ways and (later that class period) across various contexts contain repeated exam- ples indicating that students were more engaged Antionette: "You left out your `E'." in learning and using the technology related to Shane: "Where?" creating multimedia book reviews than in other Another illustration comes from our video- academic activities in the classroom. A typical tapes of students working in the lab. Ms. Ellers example comes from a comparison of our field and a member of the research team discovered notes while observing Shane in the classroom that Jason, a student whom teachers considered and in the computer lab: to be hyperactive because of his extreme dis- tractibility in the classroom, was sitting almost (notes from classroom observation) motionless staring at the computer screen for Shane [5th grade focal studentlow abil- more than a minute working on a multimedia ity/low interest] does not appear to be book review activity. Most of the obvious

working at all on his math problems.... examples of increased attention were related to Shane is turned around in his seat to talk students who were poor readers. For example,

to his neighbors.. .. Shane is the first in discussing a poor reader at an October

one done with his math. .. . Shane is meeting between the teachers and the univer- sitting and tapping some pencils. Shane sity research team, Ms. Ellers stated, is looking at what a neighbor is drawing. ... Shane is playing with two blue high- Ms. E:I know this group of fifth graders real

lighters....Shane gets up to check the well. Some of them have just really ex-

lunch menu. . ..Teacher asks Shane celled, 'cause I bragged on one of them in about whether he has shown his work, and [Mrs. Morris] 's room the other day be-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

53 A Formative Experiment 43

cause this child is probably reading at least multimedia book reviews as indicated from the two years behind grade level; but when he following reflection from our field notes. comes in here, he's totally tuned in to what I'm doing, and he probably cannot read. I find it interesting that, having watched, It's probably accurate to say he cannot that they WILL ask questions. Because read, you know, the pull down menus and there are some students, and I'm sure you what it says. But he is focusing exactly on know it, that if they don't understand, what I'm doing, he's seeing where I'm they won't even ask questions. But in dragging down to, and I mean, he's going here, it seems that nobody's afraid to ask to town on it.... He copied five pictures something, you know if they don't see when everybody else, you know the most something right. I have no idea of their was two. abilities, because I'm not there every single day to see who's the best and who's Beyond increased attention and involve- the worst, or not even to use those terms. ment in tasks related to the multimedia book review activities, increased student engagement The technological challenges of using was a label that captured other categories of HyperCard seemed to enhance students' en- our observations as well. For example, many gagement as opposed to frustrating them. As students seemed to acquire a different persona Ms. Sievers stated in a focused group meeting when involved with project activities, often with teachers, "...I haven't seen anybody get becoming less inhibited, more verbal, and really frustrated. They do in the classroom on more cooperative. In an audiotaped focused other things, but not on this." In fact, students' group discussion with the teachers, Ms. Morris frustration surfaced mainly if something pre- observed: vented them from working on the computers. Ms. Sievers observed in the same meeting, And a lot of the ones that answer ques- tions in the media center, like when we're [the students] just threw a fit if they didn't doing the whole group; a lot of those that get to come to HyperCard. I mean, ... I've seen, you know they raise their hands we had thought it might be a little late, and they wanna answer are those that you was it yesterday? (looking at Ms. Ellers) wouldn't, they're not real verbal in class and I mean we fell to pieces on the floor as far as answering something that we've in the classroom. They really got upset discussed. You know, they're the ones because they didn't think they were gonna that remember that you (several voices, come, and when I tried to explain, I mean unintelligible) edit and go down to this, they were just so [upset]. and what to punch to do this. When we talked to students about their Students who were shy and reserved in the participation in the multimedia book review classroom seemed more willing to take risks activities, their responses were almost always and ask questions when involved in creating positive reflecting their motivation and engage-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55.

54 44 Reinking & Watkins

ment. However, their positive responses were Our discussions of formative adjustments almost always related to using the computer, to implementing the multimedia book review particularly HyperCard, as opposed to specific activities often centered on how to channel comments about book reviews. Typical com- students'increased engagement relatedto ments were, "It gives you a chance to do more HyperCard and the computer toward accom- activities and funner stuff," " ..it's like; you plishing our pedagogical goal. At times there can draw ... you canpick pictures to draw were tradeoffs, sometimes due to technological and stuff. You can do your own voice and it considerations. For example, the decision to (HyperCard) has all sorts of stuff on it," use a standard template for the book reviews to "'cause you get to draw and don't do no work be searched in the database, decreased the need for students to exercise their HyperCard skills. . . . (Researcher asks:HyperCard's not work9) Nope." In fact, as students mas- We compromised by adding an open-ended tered HyperCard skills and began to focus section to the template where students could more on entering book reviews, some of the use their HyperCard skills to create additional individualized information about each book motivation and enthusiasm waned. In a Febru- they reviewed. At Hartwig School, the teachers ary meeting with teachers aimed at discussing decided to insist that students enter new infor- formative adjustments in the way we were mation about the books they were reading implementing project activities, Ms. Ellers during the first 20 min of the weekly hour-long stated, session in the computer lab; during the remain- ing time, students were free to incorporate ... they are coming in there quite regu- multimedia presentations into existing book larly (students who have read books to put reviews if they chose to do so. However, it is on the computer), so they're really pro- important to note that we did not find any gressing whereas some kind of get left evidence that students disliked entering book behind. We haven't decided on how we're reviews or that the intriguing aspects of using going to handle getting those others to get multimedia on the computer distracted students to that stateyou know, the less motivat- from reading books. edthat's something we need to sit down and discuss... Effects Related to Particular Schools and Teachers At times, the lure of nonacademic games and drawing programs also overshadowed students' As expected, the specific effects of the interest in entering book reviews. For example, multimedia book reviewactivitiesvaried Ms. Palmer at Hartwig School wrote in her among the schools and classrooms participating journal that one of her students suggested "a in the project. Working in seven classrooms in nonstructured day in the lab, just to have fun three schools with seven classroom teachers

.. . as abreak. " and two key adjunct teachers allowed us to

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

55 A Formative Experiment 45

discover general effects of the intervention that Collins and Hartwig Schools during Year 1. cut across the different instructional contexts, Teachers at both schools were enthusiastic but also allowed us to speculate about varia- about the project in the early stages, as were tions that intensified or mitigated those effects. members of the university research team. For In this section, we discuss these variations. Our example, as the project began, teachers in both findings in this area are limited by the fact that schools wrote enthusiastic comments in their the intervention was modified when imple- logs indicating that they had high expectations mented during Year 2 in Borders School based for the project and were looking forward to on what occurred during Year 1 in Collins and their participation. During an interview early Hartwig Schools. In a more general sense, any in the school year before the computer activi- formative experiment is limited by conditions ties were introduced, teachers indicated that that are not amenable to adjustment in particu- they thought the multimedia book review lar situations, but noting the effects of these activities would "inspire students to read," conditions may enhance understanding of their "allow students to use the computer in an importance in achieving the pedagogical goal. interesting way," and "give me [the teacher] an Examples of conditions that were not easily opportunity to learn more about the computer." modified in the present formative experiment As the year progressed, however, the clear include the impracticality of creating multiple enthusiasm and morale evidenced early in the versions of the software, the availability of year deteriorated at Collins School but not at space and equipment in a particular school, Hartwig School. By spring, Ms. Andrews at scheduling students, and so forth. Beyond such Collins School, who initially had greater com- practical, technological, and logistical con- petence and parental assistance for using com- straints, there are the expected differences in puters and higher achieving students in her teaching styles, educational values, instruc- classroom than did her colleague Ms. Broward, tional approaches, and so forth that vary among had fallen far behind her, almost abandoning teachers and schools. Even if it were possible entirely the project activities. At Hartwig to modify these naturally occurring differences School (and later at Borders School), project in educational orientation, any attempts to do activities extended into other areas of the so are fraught with ethical problems. Thus, the curriculum, but this was not the case at Collins differences we discuss in this section are aimed School. at understanding more about the effects of the Drawing on our observational and inter- present intervention, but our specific examples view data, the following factors seem to be may also be generically useful in understanding involved in explaining the difference between issues researchers may face when conducting these two schools: formative experiments. 1. The professional climate of the two We devoted considerableattentionto schools was distinctly different. For example, searching our data and speculating about causes although principals and central office personnel for the clear differences we observed between were highly supportive of the project in both

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

56 46 Reinking & Watkins

schools, the administrative style in Collins necessary. For example, a visit from the Dis- School was more top down, and administrators' trict Language Arts Coordinator at Hartwig interest in the project seemed related more to School was punctuated by many comments to its value for public relations in the community members of the university research team and its implications for test scores than for its about the project's potential to promote the potential to enhance curricular goals or to District's goal of moving toward a more an further students' and teachers' development. integratedcurriculumrichinauthentic Teachers seemed constrained, pressured, and reading and the goal of enhancing teachers' sometimes even intimidated by this administra- technological expertise. Although adminis- tive stance. We conducted interviews with six trators associated with Collins School occa- teachers at Collins School early in the school sionally referred to similar goals, they more year. When asked about possible negative often mentioned how the project would outcomes of the project, virtually all of them promote the school's reputation such as expressed concern that they might not have achieving a state-wide designation asa time to work the project activities into their set school of excellence. schedules. For example, Ms. Andrews justified 2. The Collins teachers perceived that they her preference for using computers in a lab were not getting as much attention from the setting by stating that "the lab guarantees university research team as were the Hartwig everybody the right to 45 minutes." (Although teachers. Furthermore, they perceived that teachers at the other schools also frequently when compared to the Hartwig teachers they discussed the pressures they were under to were at a disadvantage because they only had cover the established curriculum, they were one computer in their classroom instead of a willing to seek out ways to work the computer computer lab. The event that clearly initiated activities into their schedules and commented these perceptions was a joint presentation on how worthwhile it was to do so, and they involving teachers from both schools at a seemed to expect cooperation from their ad- national conference. During the presentation, ministrators.) Additionally, toward the end of Collins teachers saw that they had not accom- the school year, to our great discomfort, it plished as much as the Hartwig teachers, and became evident that administrators at Collins from that point seemed dissatisfied with their School were not honoring teachers' requests own situation as evidenced by frequent refer- for future teaching assignments because of the ences to Hartwig's superior resources during new assignments' potential effect on the proj- our subsequent interactions with them. To a ect.In contrast, administrators at Hartwig certain degree their perceptions were accurate. School seemed focused more on student and The research team did spend more time at teacherdevelopment,respectingteachers' Hartwig because more time was necessary to judgement in altering their schedules when teach students how to use HyperCard, whereas necessary and encouraging them to work as an we delivered more finished products to Collins. team, seeking administrative support when The substantially greater distance to Collins

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

57 A Formative Experiment 47

School also limited the number of visits by the However, the Collins teachers seemed to be university research team. As the year pro- more concerned about how the data were being gressed, we found it increasingly necessary to collected, whether we thought the project was give teachers at Collins encouragement that we being successful in our eyes, and whether we were not disappointed with their progress, yet approved of their involvement. Their concern this encouragement became less convincing to and doubt may have reflected their decreasing the teachers, and importantly to ourselves too morale and likewise exacerbated it. Conse- as the year progressed. Nonetheless, we be- quently, the Collins teachers seemed to rely lieve the differences in morale are due more to heavily on our direction and support. Ideas for the teachers' perceptions than differences in extending the project were discussed with resources or support. Neither are we convinced enthusiasm but were not often implemented that the multimedia book review activity faces without direct support and followup from the insurmountable problems in a classroom with university research team. only a single computer. 4. Although teachersin both schools expressed that they felt pressured to cover 3. Teachers at Collins School seemed to be curriculum, Ms. Andrews and Ms. Broward at more conscious of whether the research was Collins School were under greater stress from being conducted properly and whether the other sources than were the teachers at Hartwig activities were successfully meeting our goals. School. In addition to greater administrative We explained the concept of a formative exper- pressures to conform and achieve, both of their iment to teachers at both schools early in the families experienced serious medical problems project emphasizing that we did not expect the during the school year. In addition, at Collins project activities to succeed uniformly in every we did not observe a strong sense of teamwork context or without formative adjustments based and mutual support among the teachers at the on our data collection. Regardless of the school. Ms. Andrews and Ms. Broward origi- school, teachers on the whole did not often nally volunteered to become involved in the seem comfortable with pointing out difficulties project because they had developed an affinity with project activities or with suggesting ideas based on their interest in whole language, for improvement with members of the university openness to new ideas, and desire to learn research team, to whom they often deferred. more about technology. Throughout the project For example, during January, Ms. Pearson at they clearly collaborated well on project activi- Hartwig School wrote in her journal: ties under the direction of the university re- search team, but they did not fully integrate I had my interview with Dr. R today. I their efforts, at least to the extent we observed was very relieved to find that they [the at Hartwig School. There was little sharing of university research team] were pleased expertise and resources that proceeded from with our progress. A lot of apprehension joint problem solving. In our interviews and was relieved. informal interactions with other teachers at the

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

58 48 Reinking & Watkins

school, we often heard of criticisms, jealousies, to use the computer, we observed the follow- and disagreements between and among teach- ing: ".. .there were several students who

ers, some of which extended to incidents or were not paying much attention. . . it was situations outside of school in the community. snack time and some of them were exchanging In addition, there was noticeable turmoil in potato chips with each other and others were Collins School precipitated by the fact that the preoccupied with an art project." In contrast, principal was running for election as school our notes observing in the computer labs sug- superintendent, there was uncertainty about gested that students were highly attentive to who would be her successor as principal when presentations unless distracted by their desire she was elected in November, and which to work on the computer. teachers would be assigned to a new school that Another difference between Collins School was to be opened the following year. Collins and the other two schools relates to variations School was also preparing for an extensive among teachers and their roles in implementing remodeling that disrupted activities, especially the intervention. The effects of the multimedia during the last month of the school year. book review activity were clearly influenced by 5. The fact that the project activities were the characteristics of individual teachers work- carried out with a computer in the classroom at ing together with their colleagues and the Collins as compared to a computer lab in the research team on the project, although the other schools may have been a factor that effects of teacher variations and school varia- figured in the differences we observed. Sched- tions clearly interact. All of the teachers who ules had to be arranged to use the computer participated in the project were enthusiastic labs, thus there needed to be a fixed time in the about their participation at the outset, and none teachers' schedules for engaging in book re- had more than minimal acquaintance with view activities, and that time had to be coordi- using computers personally and in the class- nated with the other teachers involved in the room prior to this project as evidenced by their project. Having the flexibility to work the responses to a structured interview (see Appen- activities into one's schedule on an on-going dix A). However as the project proceeded, we basis, as was the case at Collins, may have led found that teachers tended. to fulfill several to fewer collaborations between the teachers identifiable roles. To a certain degree, teachers and the temptation to forego the activities when shifted roles throughout the project, although they felt pressures to complete mandated in- they tended to gravitate toward one role. Each struction. Additionally, students at Collins of the following roles seem salient in under- seemed less enthused during group presenta- standing the effects of the multimedia book tions, perhaps because these presentations review activity and how it might be effectively occurred in their classroom as opposed to implemented': making a special trip outside the classroom to the computer lab. For example, in our field 4Our categories might be compared to those suggested by notes recorded during a presentation about how Hadley and Sheingold (1993).

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

59 A Formative Experiment 49

The technology expert. A teacher at each of dents how to create a display for the school the three schools acquired this role early in the science fair. Likewise, Ms. Pace, a Chapter 1 project by virtue of her greater interest and teacher assisting the teachersat Hartwig, quicker success in mastering the hardware and quickly became a technology expert helping software, accompanied by a greater commit- students with make-up activities and debugging ment to work on the computer beyond the their work. minimal requirements for the project. We The emerging or marginal technology conducted training sessions for all of the teach- expert. Another role was characterized by ers prior to the beginning of the school year teachers whose early involvement in the project and the role of technology expert emerged at activities was enthusiastic but passive, defer- that time. The other teachers not only acknowl- ring almost entirely to members of the research edged this role (e.g., comments such as "Ms. team and to the teacher(s) in the role of tech- Ellers will help us do that. "), but they seemed nology expert. Gradually, however, teachers to need the technology expert to sustain their in this category seemed to become more com- own efforts (e.g., "I know I couldn't have done fortable using the technology and more enthu- that without Ms. Burton's help."), sometimes siastic about the intervention's effects. Ms. in a way that seemed to prevent them from Pearson's involvement during the project at extending their own technological expertise Hartwig School is illustrative. Transcriptions (e.g., "I couldn't do this next year myself, but of our tape recorded field notes early in the maybe I could with Ms. Burton's and Ms. school year state, Pace's help."). The negative effects implied by this latter comment indicating dependency can [Ms. Pearson] sat at the table away from be seen by Ms. Andrews progressive abdica- the computers in the lab today counting money and filling out book orders. She tion of the role of technology expert at Collins seemed to put us [members of the univer- School. As Ms. Andrews became less involved sity research team] in charge with no in the project activities, Ms. Broward was intention of participating in learning how forced to deal more with the technology herself to use the HyperCard stacks or of working and by the end of the school year had increased with students. her expertise beyond Ms. Andrews. On the other hand, it was the teacher in the role of As the year progressed, however, Ms. Pear- technology expert at each school who was son's involvement increased greatly as she instrumental in extending the project activities became more comfortable with the technology into other areas of the curriculum or other and saw the enthusiastic response of her stu- school activities, and this served as a model dents. Later in the year, on her own, she and resource for the other teachers. For exam- decided to extend the multimedia book review ple, Ms. Burton, who became a technology activity into her social studies unit on the Civil expert at Hartwig School, used her new Hyper- War. After all of her students had read a novel Card skills to create a tutorial directing stu- set at the time of the Civil War, she divided

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

60 50 Reinking & Watkins

her students into groups to develop a multime- tempts to facilitate the positive effects she was dia presentation on some aspect of the book, seeing on her students in the classroom. Ms. including a class videotape in which students in Sievers at Borders School also could be classi- costume acted out various episodes in the book. fied as a facilitator because she did not indicate As the year progressed, Ms. Pearson became more than passing interest in the computer, but more actively involved in the activities in the she attempted to connect classroom reading and lab by offering help to students or letting them writing activities with students' multimedia teach her skills. Ms. Pearson did not seem to book review activities in the computer lab. define herself as a technology expert, but did Likewise, teachers who focused on identifying move far in that direction over the year. This and solving practical and logistical problems growth is apparent from our field notes at the were assuming the role of facilitator. end of the school year: The passive participant. Teachers in this role may be enthusiastic about the project and I told [Ms. Pearson] that next year we its potential benefits, but they rely primarily on would be moving on to another school and others (their fellow teachers or members of the that Hartwig teachers would be one their research team) for explicit direction and guid- own. Of all the teachers at the beginning ance. Their personal investment in the project of the year, she is the one I thought would is low in terms of independent effort to engage be most concerned with that. I thought it was significant that now she didn't seem in creative problem solving to address logisti- that concerned. She kind of made a little cal, practical, technological, and pedagogical face when I told her, but it wasn't at all a problems. They do not contemplate possibili- negative reaction... more like "I'm not ties for extending or adapting the multimedia sure" but maybe I can. book review activities or coordinating them with other curricular areas; or, if they do, such Another example, was Ms. Morris at Borders extensions must not create too much of a dis- School who in a teasing manner chastised us ruption to their current instructional routines. for not observing the day that she took over for There is little evidence among teachers assum- Ms. Ellers and taught her students a lesson on ing this role of attempts to master the computer using HyperCard. outside of school hours including formal train- The facilitator. Some teachers, at least for ing sessions led by the members of the univer- a time, seemed to assume the role of a facilita- sity research team after school. They have a tor; that is, a teacher who is not especially relatively low tolerance for dealing with devel- intrigued with the technology, which was in the opments that prevent the activity from meeting domain of the technology experts, but who is their expectations of success. This role tended interested in discovering and enhancing the to be assumed more often by the teachers in nontechnological effects of the program. For Collins School, especially as the project pro- example, Ms. Pearson's movement toward gressed. However, we wish to emphasize that technology seemed to originate with her at- this is not a generalization that extends to other

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

61 A Formative Experiment 51

instructional activities in their classrooms; nor, activities seemed embedded within a conven- does it do justice to their more laudatory teach- tional schema for what students ought to be ing practices. Additionally, we believe that the doing and learning in school, while simulta- professional climate of their school and district neously they engaged enthusiastically in an may have predisposed them to assume this role, activity that to some extent subverted that as did their declining morale explained previ- schema.Teachers' concerns thatstudents ously in this section. acquire particular skills and be held account- able for their work co-existed with their enthu- The Effects of the Project on the siasm for activities that were clearly addressing Educational Environment less tangible goals (such as increasing students' self-direction,confidence,creativity,and Our interest in the effects of the project on ability to master the technology). This dual the educational environment was motivated by focus can be seen in Ms. Burton's log where Newman's (1990) explication of formative she states, experiments. He states, I felt they needed to see a stack in prog- Whatever the pedagogical theory moti- ress to understand more about what is vating the experiment, the outcome to be required. I want them to be creative!... observed must include how the environ- Once they got into creating their own ment becomes organized differently as it cards, the creative juices began to flow. appropriates the technology and other [And one page later] I am having students resources. ...If the environment, rather record pages read each week and having than the technology, is the unit of analy- parents sign. sis, changes in the instructional interac- tions, changes in teacher roles, and other On one hand, teachers tended to see the ways that the educational environment is project activities in conventional terms. For changed are observed. (p. 10) example, despite the research teams' explana- tions that we preferred the term "book re- We found that teachers' involvement with views" instead of "book reports" (because of project activities was somewhat paradoxical. the negative connotations of book reports) and Often they tended to view multimedia book despite our consistent use of this terminology, reviews and their potential benefits in relation the teachers almost always referred to what to conventional academic activities and skills. students were doing as "multimedia book At the same time, particularly at Hartwig and reports" or "book reports on the computer." Borders Schools, teachers were willing to give ..As described more fully in a subsequent sec- up time devoted to teaching conventional lion,teachers were especially enthusiastic activities and skills so that students had time to about how the project activities were benefiting work on the book review activities. In the the technical aspects of students' writing. Addi- teachers' views, the multimedia book review tionally, they seemed more satisfied than did

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

62 52 Reinking & Watkins

members of the research team with conven- their own plan for catching up students who tional responses to books such as writing had missed a lab activity. summaries as long as they were technically As the year progressed, at Hartwig School correct. For example, teachers were often especially, we saw evidence that the teachers reluctant to provide the research team with were appropriating computers and the book disks containing students' work until they had review activity into their teaching beyond the been proofread and corrected, often by one of specific project activities.Contrary to the the teachers. In her log, Ms. Pearson stated, beginning of the year when computers in the "They are not proficient writers. Their writing classrooms were used almost exclusively for skills are very poor. They refuse to use dictio- drill and practice or game programs for stu- naries to check their spelling. I am embarrassed dents who had completed their work, the to let some of them take their work to the classroom computers began to be used for computer lab." Some of the teachers sent home other purposes. Ms. Burton, who initially reports of students' progress on the book acquired the role of technology expert at Hart- review activity to parents and they were con- wig, used her new HyperCard skills to develop cerned that students not enter certain kinds of a tutorial directing students how to prepare reading material such as magazine articles or their science fair displays. In our field notes we joke books on the computer. We see these also observed that, findings as consistent with previous research (e.g., Bruce & Rubin, 1993) indicating that [Ms. Pearson] was excited to show me teachers tend to conceptualize and use innova- that the students were working on a word tive activities involving technology in a way processing station in her classroom.... that conforms to conventional academic values She also told me that she had ordered a and experiences regardless of the activities' word processing program on the Apple potential to transform standard practice. [computer]. Apparently she did not have a word processing program in the class- On the other hand, we saw some evidence room before. I think this may be an that teachers' enthusiasm for the benefits of indication that she is starting to use the the project led them to forego, displace, or computer more in her classroom. extend more conventional activities in their classrooms.For example,although the Later in the year, one of us noted that, "I teachers at Hartwig and Borders Schools thought it was very significant that [Ms. Pear- often emphasized the many expectations son] told me in passing that she skips spelling placed upon them for covering content and to do word processing in her classroom." becoming involved in special activities such as essay contests, they rarely opted out of From the standpoint of the project's peda- the project activities even though we invited gogical goal, the most notable example of how them to do so if necessary. In addition, the project activities could extend and trans- teachers in both of these schools created form conventional instruction occurred in Ms.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

63 A Formative Experiment 53

Pearson's class. In February, Ms. Pearson history buff. She had actually visited and decided to break students into groups to create studied the Shanandoah Valley where the a class review on the computer of a book she story takes place. had decided to have students read in conjunc- tion with a social studies unit on the Civil War. Through this activity Ms. Pearson's students She wrote in her log, also participated in a more in-depth response to a book than they would have in writing a Today I gave all of my students a copy of conventional book review. For example, she Shades of Gray. They are going to do a wrote in her log that one of the groups in- book report on the computer. They were volved in this class project dealt with "prob- very excited and immediately began to lems of main characterHow (problems) they read. I am going to let them group to- were dealt with/book/how same problem gether to do different parts of the book should be dealt with by us if were faced with report. They are going to make the deci- same challenges." sions on the way the book report should be As the project progressed, teachers also done. Almost everyone uses any spare noticed connections between classroom activi- moment to read this book. ties and students' work on the multimedia book Along with the computer-based activities de- reviews in the computer lab. At an October signed by each group, the students with mini- meeting, Ms. Ellers remarked, mal teacher direction and support produced a video of episodes from the book. Students got Well, these things that [John, a member of involved in all aspects of the production includ- the university research team offering technologicalassistance]observed the ing elaborate costumes and staging and accord- other day though.... [W]hen after I ing to Ms. Pearson, "They were willing to give show them what to do, I tend to go over up breaks to organize the video." The multime- here to my right and work with this dia book review project clearly led the teacher group. And this group back herethere's to conceive of this activity and for the students about three students. Well, they created to carry it out with enthusiasm while becoming their fields, but they started just typing a involved in a positive experience related to story in the fields. You know, they did their reading. The activity also led students to their own little thing. So, you know, and explore independently aspects of books that he [John] said, "Well, where do you think they would have unlikely encountered prior to they got that from?" And I said, "Well, the intervention. For example, as Ms. Pearson they do a lot of creative writing in [Ms. states in her log, Sievers] class. So that has some [carry over from one situation to another]" .... One group talked about the author and her personallife.They did research and Nonetheless, the general enthusiasm for discovered that the author was a Civil War and commitment to the project activities among

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

64 54 Reinking & Watkins

both teachers and students tended to balance, going to handle getting those others to get sometimes precipitously, on itsrelation to to that stateyou know, the less moti- conventional instruction. Teachers often ex- vatedthat's something we need to sit pressed the benefits of the project in terms of down and discuss... conventional instructional goals while enjoying its less academically oriented effects, occasion- Technology, particularly as manifested in ally linking the two as when we pointedly learning and using HyperCard, seemed to play asked the Borders teachers in a meeting: "Does a pivotal role in tilting the balance away from the need to make the book review present- conventional instruction. As documented previ- ableunderstandableplay any role in damp- ously in this report, teachers were less inclined ening students' enthusiasm?" Both answered to see themselves as experts in the computer "no" loudly in unison. Students too were lab, often deferring to students' expertise. In affected by this balance. If the multimedia book addition, students' and teachers' work on review activities began to take on the character- HyperCard involved highly engaging and istics of conventional classroom activities, interesting activities in a nonthreatening aca- enthusiasm waned. For example, In a February demic environment that was separate from the interview Ms. Ellers observed, classroom where participants carried out most of their daily routine. Students who had aca- But several of the fifth graders when they demic problems in the classroom frequently come in here into the classroom [lab], excelled in using the technology, and students' they're ready. They just need to plug in increased interactions with their peers and with the information. That's one biggie I would their teachers were frequently supportive and say. The fifth grade really likes doing it. positive. The cumulative effect of these charac- I hear a lot of positive comments from teristics seemed to override teachers' concerns them, from certain groups of them, be- associated with conventional instruction. How- cause there are several that don't come in ever, entering information about books, as a there as much anymore because they know more conventional activity, seemed to remind that is the requirement now: if they don't participants of more academic concerns, which have their book review ready or written it, in turn evoked more conventional reactions. then they're not ready to go to the lab. As the emphasis on learning and using They're going to miss their time whereas HyperCard decreased, concerns related to if someone else is ready that can go in conventional academic goals and achievement their place since they now [each] have their own individual disk. And so some of increased, thus perhaps negating some of the them are in there two, three, or sometimes potential influence of the multimedia book even four times a week. They are coming review activity. At Hartwig School this effect in there quite regularly, so they're really was mitigated by the fact that the intervention progressing; whereas some kind of get left clearly extended into other classroom reading behind. We haven't decided on how we're and writing activities and indeed into other

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

65 A Formative Experiment 55

areas of the curriculum. At Collins School, requiring minimal amounts of reading. How- where students were not systematically taught ever, these effects are mitigated when teachers HyperCard and where students completed more fully appropriate computers into their project activities in their classroom, the tech- classroom teaching and extend the multimedia nology remained firmly entrenched within the book review activities into other areas of the context of conventional classroom activities. curriculum. These effects might also be miti- This too, may account for the relative lack of gated by moving quickly to the database phase involvement by teachers and students at Col- of the intervention; although this is speculative since we were not able to implement fully this lins. We see Borders School as representing a feature. In fact, the delay in implementing the middle ground where interest in technology database may have contributed greatly to the was high, which sustained the activity much of shift toward a more conventional perspective of the year, but where there was no clear evidence the activity among some of the teachers. that the intervention was influencing instruc- Quantitative data from the TORP com- tional activities beyond the specific book re- pleted by the teachers at the beginning and view activities. Consequently, some students again and the end of the school year are rele- seemed to lose interest in the book review vant to these issues. Pre- and postexperimental activity later in the school year. means and standard deviations on the TORP Comparing our experiences within and are shown by teacher in Table 3. Two statisti- among schools, we hypothesize that the initial cally significant changes in orientation to contribution of the intervention toward accom- teaching were noted. Ms. Burton at Hartwig plishing the pedagogical goal is based on how School became less oriented toward whole involvementwithtechnology,particularly language as can be seen by comparing her learning HyperCard in a computer lab, estab- means on that subscale at the beginning and the lishes a positive context that clearly transcends end of the school year. Ms. Ellers at Borders conventional academic activities,concerns, School became less oriented toward phonics demands, responses, and so forth. Multimedia as can be seen by comparing her means on that subscale at the beginning and the end of book reviews are associated positively with this the school year. These data do not provide environment and contribute to the pedagogical any clear evidence that the intervention goal as mediated by factors such as increased changed teachers orientation toward teaching interaction. Gradually, however, as emphasis reading. Ms. Ellers' move away from pho- shifts away from learning and using the tech- nics may have been associated with the nology toward completing book reviews, a university course work she was taking as more conventional mindset emerges. Teachers part of a degree program. Ms. Burton's begin to associate the project's benefits with move away from whole language may have conventional academic skills such as writing been due to the stress on skills associated skills and conventional practices such as with end-of-year testing. However, we have no

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

66 56 Reinking & Watkins

Table 3. Means (Standard Deviations) for Teachers' Pre- and Postexperimental Scores on Subscales of the TORP

Subscales

Phonics (10 items) Skills (10 items) Whole Language (10 items)

School/Teacher pre post pre post pre post

Collins School

Andrews 3.8a 3.8 3.8 4.1 2.4 2.3 (1.08) (1.02) (.84) (.90) (.71) (.75)

Broward 2.8 2.8 2.9 2.3 2.4 1.8 (.64) (.81) (.91) (.80) (.81) (.75)

Hartwig School

Burton 2.9 2.8 2.5 2.5 3.0 2.5* (1.14) (1.01) (.95) (.88) (.50) (.62)

Palmer 2.9 3.0 2.3 2.4 2.6 2.4 (1.30) (1.11) (.90) (.85) (.48) (.54)

Pearson 3.0 2.9 2.9 2.6 3.6 3.2 (.40) (.48) (.75) (.87) (.63) (.72)

Borders School

Ellers` 3.2 2.5* 2.4 2.3 2.5 2.6 (.80) (.60) (.56) (.62) (.63) (.78)

Sievers NAb 3.1 NA 2.4 NA 3.9 (.88) (.62) (1.05)

Morris 2.6 2.5 3.1 3.1 2.7 2.8 (.72) (.64) (.54) (.66) (1.12) (1.30)

'Means are based on a 5-point Liken scale where 1 = strong agreement; lower values indicate stronger orientation. bPretest scores not available for Ms. Sievers because she replaced original teacher after baseline data were collected. `Though Sievers took over Ellers' class, Ellers was the teacher most directly involved in working with all students in the computer lab with each teacher. *p < .05.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

67 A Formative Experiment 57

supporting evidence from our qualitative data was highlighted in applications for a state-wide to clearly link the two statistically significant "school of excellence" award, and members of changes to the multimedia book review activities. the university research team were invited to make presentations about the project at meet- Changes in the Amount and Diversity ings of the parent-teacher organization and the of Students' Reading local school board. Parents expressed their support for the project in terms of their belief The pedagogical goal of this formative that a familiarity with technology was impor- experiment was to increase the amount and tant to their children, and students were excited diversity of students' independent reading. about being able to use the computers. The Before presenting our findings concerning the effects of this general enthusiasm and sup- effects of the intervention on students' indepen- port can be seen in the following transcript dent reading, we discuss three factors that of tape recorded reactions one of us dictated limited our ability to collect data aimed at after attending a meeting of the Parent determining whether theintervention was Teacher Organization early in the school accomplishing that goal. year: "Several parents commented to me First limitation: The Hawthorne effect. It how supportive they were of the project was difficult to separate a Hawthorne effect being at the school. One father said that his from other effects due to specific aspects of the son had brought a book home in anticipation intervention. From the outset, all of the stake- of the beginning of the project." Also, holders in each of the three schools expressed teachers, perhaps because of the high enthu- unmitigated enthusiasm for the project, and siasm and expectations, suggested that they they were clearly excited by the facts that their were apprehensive about the project's suc- schools had been selected to participate in a cess. One of the teachers recorded in her project supported by a major federal grant. In log: "I had my interview with Dr. [member addition, these schools, like many other schools of the university research team] today. I was in our experience, had a high commitment to very relieved to find that they [research integrating technology into instruction and they team] were pleased with our progress. A lot were seeking guidance to accomplish that goal. of my apprehension was relieved." This commitment was heightened by a state- The enthusiasm, high expectations, and wide initiative to infuse technology into schools apprehension created by the project clearly had backed by substantial new funding from the potential to affect the pedagogical goal. In fact, state lottery earnings. as discussed earlier in this report, we believe School and central office administrators these factors figure prominently in explaining were clearly interested in capitalizing on the the decline of morale at Collins School. How- public relations value of the project as evi- ever, other considerations place the limitation denced by the facts that articles about the implied by the Hawthorne effect in perspec- project appeared in local papers, the project tive. First, we did collect data in several con-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

68 58 Reinking & Watkins

trol classrooms that were using a computer pro- reading to be subtle and to emerge over an gram designed to promote independent reading extended time. Likewise, we expected that by awarding points when students demonstrated detecting such changes and connecting them adequate knowledge of a book's content. In unequivocally to the intervention would be addition, the project acquired a lower profile as difficult. Not only was it difficult to connect the year progressed and teachers became more the intervention to observed movement toward accustomed to the idea of a formative experi- the pedagogical goal, it was also difficult to ment in which the intervention could be determine specific aspects of the intervention's adapted in response to negative outcomes or implementation that enhanced or inhibited problems. The fact that we worked in the progress toward the goal. participating schools for an entire school year To address this limitation, we collected a also mitigates over time against the Hawthorne variety of quantitative and qualitative data effect. Lastly, we believe that the Hawthorne aimed at determining the amount and diversity effect is to some degree inextricably linked to of students' independent reading throughout the interventions involving computer technology in school year. Quantitative data included having schools, at least currently. That is, given the students complete the baseline and postexperi- present interest in using technology in schools mental ERAS (McKenna & Kear, 1990) and a and the relative novelty and unconventionality questionnaire concerning the diversity of their of computer-based activities, the Hawthorne independent reading.Additionally, parents effect is not just a nuisance variable but one completed a questionnaire concerning students' that merits study in its own right as a factor reading outside of school. These data are that may influence the effectiveness of com- reported in Table 2. Qualitative data included puter-based interventions. observational field notes, semistructured and Second limitation: Detecting subtle changes focused group interviews with teachers and in students' independent reading. From the students, videotapes, and teachers' logs. None- outset of this project, we believed that the theless, our difficulties in collecting data were effects of multimedia book reviews on students' evident in the following excerpt from our field independent reading were likely to be less notes. direct and less immediate than other interven- I worried that I was just not seeing tions such as providing extrinsic rewards for what I was "supposed" to be seeing. I the number of books read. However, our saw the good readers sneaking books to pedagogical theory values more highly those read under their desks and relying on activities designed to develop long-term, intrin- themselves or close friends for book sic motivation to read as included in what suggestions. I don't know that with the Alvermann and Guthrie (1993) describe as the time I spent in the classroom that I engagement perspective. Given that the inter- would ever see the students diversify vention proceeded from this perspective, we their own reading or increase it.I was expected changes in students' independent simply not there all the time, and I was

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

69 A Formative Experiment 59

not their teacher. If I taught these stu- presentation and interpretation of the ERAS dents every day, I would see what was data. in their bookbags, I would hear their Findings from the quantitative data. Quan- stories, I would see their interactions in titative data were obtained from (a) the ERAS the classrooms,I would see what is (McKenna & Kear, 1990), (b) a questionnaire done with their free or extra moments. designed to measure the diversity of students' The short time segments that I spent in reading, (c) a questionnaire designed to mea- the classroom just didn't let me see sure parents' perceptions of students' reading what needed to be seen. outside of school, and (d) the Deford Theoreti- cal Orientation to Reading Profile (TORP; Consequently, we tried to make our class- DeFord, 1985). Students completed the ERAS room visits at times we thought students would and the diversity questionnaire; their parents have some free time to read. We also relied completed the parent's questionnaire; and, heavily on teachers' comments during inter- teachers completed the TORP. Participants views and in their logs to monitor the effects of completed these instruments during the first the project on students' independent reading. month of the school year prior to beginning the Third limitation: Baseline levels of inde- intervention and again during the last few pendent reading. As noted in the section weeks before the end of the school year. reporting baseline data, the majority of the Tables 1-4 show pre- and postexperimental students involved in this project already results including the gain or loss by various engaged in a good deal of independent reading categories. Also shown, are those pre- to post- and had positive attitudes about in-school and experimental differences that are statistically out-of-school reading. Under such conditions it significant when compared using a t-test for is less likely that we would be able to deter- correlated samples. mine whether the multimedia book review The results of the ERAS shown in Table 1 activity was increasing independent reading indicate that for the intervention classes, statis- than if there were many students who did little tically significant gains in mean raw scores independent reading. Determining the effects of were evident in one class on the recreational the intervention on independent reading was reading subscale, in two classes on the aca- further complicated by the documented finding demic reading subscale, and three classes on that independent reading tends to decrease as the total across both subscales. In the two students advance through school (see Foertsch, comparison classes where a computer program 1992; McKenna, Kear, & Ellsworth, 1995). was used to award points for reading books, From this point of view, maintaining students' statistically significant decreases in raw scores level of independent reading over an extended were evident in one of the classes on the recre- period of time can be seen as movement in a ational reading subscale and in both classes on positive direction relative to this trend. Indeed, the academic subscale and on the total scores this phenomenon enters into our subsequent across both subscales. Changes in scores on the

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

70 60 Reinking & Watkins

Table 4. Means (Standard Deviations) for Students' Scores on the "Choosing Things to Read" Questionnaire

Time of Administration

School/Teacher Fall Spring Gain/Loss'

Collins School Andrews 56.00 53.64 (2.36) (12.64) (12.16) Broward 61.61 49.83 (11.78)* (13.09) (11.05)

Hartwig School Burton 52.55 44.45 (8.09)* (14.43) (14.89) Palmer 31.27 44.91 13.64** (11.88) (10.68) Pearson 54.35 51.39 (2.96) (17.49) (17.57)

Borders School Morris 52.51 56.00 3.79 (13.40) (18.45) Sievers 47.33 53.81 6.48 (14.45) (16.41)

Comparison Classes Teacher 1 51.25 52.00 .75 (11.51) (13.86) Teacher 2 58.00 46.68 (11.32) (12.47) ( 9.27) aA = gain(loss) *p < .05. p < .001.

ERAS from the beginning to the end of the gest that students attitudes toward academic school year must be interpreted in light of the and recreational reading tend to increase (or at tendency of elementary school students' raw least not to decrease at expected levels) more scores to decrease over time (Foertsch, 1992; in the classes involved in the multimedia book review activities than in two classes using an McKenna et al., 1995). For example, a raw alternative computer-based activity aimed at score of 52 on the ERAS results in percentiles increasing independent reading. of 35, 42, and 49 for fourth-, fifth-, and sixth- Means and standard deviations by class for grade students, respectively. These data sug- the "Choosing Things to Read" questionnaire

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

71 A Formative Experiment 61

are shown in Table 4 (see Appendix B for the Borders School). No statistically significant complete questionnaire). Pre- and postexperi- increases or decreases on the variables identi- mental means for four classes involved in the fied in the Parents' questionnaire were ob- intervention decreased, two of which were served in Ms. Pearson's class at Hartwig statistically significant, and means for three School or in either of the comparison classes. classes increased, one of which was statistically However, the statistical significance of these significant. The mean for one of the compari- findings must be interpreted in light of the son classes increased and one decreased; nei- many comparisons conducted across class- ther of which was statistically significant. rooms and variables. These data do not clearly support or contradict Findings from qualitative data. Teachers an assertion that the intervention had any effect and parents clearly observed increases in some on the diversity of children's reading across students' independent reading, which they schools. connected to the project activities. Their com- Means and standard deviations by class on selected variables from the Parents' question- ments in interviews, audiotaped and videotaped naire are shown in Table 2 (see Appendix C for meetings, project logs, and off-hand comments the complete questionnaire and Figure 11 for recorded in our field notes, consistently refer an explanation of each variable). Among the to positive changes in students' independent seven classes involved in creating multimedia reading, which was manifested in various book reviews, statistically significant gains on ways. variables included on the parents' questionnaire Our data contain frequent reports of chan- were as follows: free time reading (1 class); ges in individual student's reading as reported ratio of time spent watching TV and reading (2 by parents and teachers. For example, our field classes); number of children's books in the notes from Hartwig school include the follow- home (1 class); range of reading materials (1 ing notation: class); number of trips to the library (1 class); possession of a library card (1 class); and I was talking [with a parent who helped in reading for enjoyment at home (2 classes). the lab] and she said she didn't know if it None of the decreases in means were statisti- was due to his project, but she had noticed cally significant. Among the seven classes a very significant, noticeable improvement involved in the study over 2 years, one class ... in her daughter's reading at home. had 4 statistically significant increases (Ms. She had said that her daughter has many Palmer at Hartwig), two classes had 2 statisti- books at home before and would occasion- cally significant increases (Ms. Broward at ally read parts of them but would never Collins School and Ms. Sievers at Borders seem to finish them ... whereas now she School), three classes had 1 statistically signifi- observed her daughter doing much more cant increase (Ms. Andrews at Collins School, reading, finishing the books and talking Ms. Burton at Hartwig, and Ms. Morris at more about them. ...

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

72 62 Reinking & Watkins

Likewise, Ms. Pearson recorded the following aided her in choosing to read and she said in her log: she liked putting her work on the com- puter. Several parents told me throughout the year how pleased they were that we were During a visit to observe in Ms. Broward's involved in this research project. They class at Collins School, she pointed out that said their children had always had plenty Mitch, a poor reader, was unexpectedly bring- of books at home, but they never seemed ing abridged classics to school to read. We to completely read their books. They (the discovered that he was asking his mother to parents) were able to see drastic changes buy these books when he accompanied her to in their children throughout the year. the supermarket. He was anxious to show us Their children were now completely read- his new book each week and explained that he ing books and asking for more. The par- wanted to get them to enter into the computer. ents were very excited about the changes. Some of the teachers observed positive And, on another occasion she wrote again in changes across all of the students in their her log: classes, which they attributed to the project activities. For example, Ms. Burton wrote in I tutor a third grader. One afternoon when her log, "I saw a lot of growth in my class in I took him home from tutoring, his many ways. Toward the end of school, I saw mother wanted to know if we would be kids through with work reading books. I saw doing the research project with the them completing work in order to read their computers when [her younger son] got book." As discussed in a previous section, to 5th grade. She wanted us to because other teachers observed that the effects were Karen, her daughter in Ms. Palmer's more obvious with readers of high or low class, had greatly benefitted from this ability. For example, Ms. Pearson stated in an year's project. She said that Karen was interview that she thought the project "encour- reading at home all the time. She also said ages students to read, especially those with that Karen's writing skills had greatly lower reading abilities. . .." Ms. Ellers, on improved this year due to using comput- ers to do book reports. the other hand, saw more of an effect on the better readers as evidenced from her response Ms. Burton wrote in her log in February, when we asked her what she saw happening relative to the pedagogical goal: "With the top Today I noticed that Candace had a col- readers, yes, this [project] is a hit. With the lection of books on her desk. I asked her high ability/high interest readers it amplifies if she was reading now and did she enjoy that effect." it. She showed enthusiasm about reading Teachers frequently observed connections and told me about her books. I asked her between project activities and classroom events if she felt the computer book reports had that they perceived as positive changes related

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

73 A Formative Experiment 63

to the amount of children's reading. For exam- that he could do it on that one. To actually ple, Ms. Pearson stated in a videotaped inter- take the time to get it and go and do itit view, was a real surprise; a pleasing surprise.

Today, two low-average readers turned in The qualitative data that we gathered Troll book orders. What impressed me provides little evidence of an increase in the most was that these 2 students ordered diversity of students' independent reading, at Troll [book club] at home for the summer. least in terms of books. Teachers rarely made It is now February. In 13 years of teach- comments related to diversity in their logs or ing 5th grade, I have never had any stu- brought up this aspect of the pedagogical goal. dents to order Troll at home even closer to In fact, when asked directly about the diversity the summer time. of students' reading, the teachers indicated Ms. Palmer recorded in her journal: "I'm that they were not seeing any positive or noticing more students are asking for additional negative changes. For example, Ms. Ellers pages to record books." Ms. Sievers at Borders responded as follows: School related the following incident during an interview. Now as far as diversity, I'm not seeing a whole lot of diversity because they're I know we finished reading the book [into] series books and you know some Summer of the Monkeys and I was real little girls like the Sweet Valley Twins or surprisedlast week I guess it was, one of whatever ...they're coming in here with the students, Paul, came up to me and a notebook or sheets of papers, so amount asked if he could borrow it.It was one I've seen more, but not diversity. that was real special to me. I didn't know what he was going to do with it at that In the same discussion, Ms. Sievers agreed, time. I said why are you going to borrow adding "I'm seeing not much diversity also it when we've already read it, and he said (either). They kind of read the same things: he was going to do HyperCard, do it on fourth-grade books, chapter books." the book report, so he took it with him. Nonetheless, teachers occasionally saw Then the next daythe next time when he changes that might have indirect effects on came back and asked me if he could bor- diversity. For example, Ms. Palmer commented row it again and he could not find it, he in her log that as a result of the project, got very disturbed that he couldn't find it. students were beginning to realize "that this So he said maybe Andy had apparently gotten it to go and do his book report on it [activity] was not about book size (number of also. It is a thick booka fairly hard book pages)but to read whatever interests them just to read by yourself. But that is some- and use this reading to relate or pass on to thing that Andy, not being a good reader other students their opinions of various litera- for him to take the initiative and either do ture." We also observed that focal students it on his own or have someone tell him read diverse materials. For example, we re-

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

74 64 Reinking & Watkins

corded the following in our field notes: "Eliza- pertaining to the pedagogical goal of increasing beth (hi ability/hi interest) was observed having independent reading, they were also encour- different paperback books in her desk on three aged to note other effects of the project activi- occasions." Likewise, when we received per- ties. In this regard, they often focused on the mission to look at what books were in students' benefits of the program in helping students desks and when we observed them during free become more proficient in the technical aspects reading times during February, we saw evi- of writing and of editing. Students increased dence that many individual students were attention to technical aspects of their writing reading diverse materials such as nonfiction, were not only quite visible but also seemed to fantasy, and football souvenir programs. How- be especially noteworthy in teachers' explana- ever, this diversity did not seem to be con- tions for the value of the activity. In their logs, nected directly to the multimedia book reviews teachers repeatedly made comments such as the or influenced greatly by this activity. Effects in following. this area seemed to be indirect at best, as proposed in a previous section discussing how ...it seems the kids are being more careful dealing with some of the technological compo- about word usage, spelling, and punctuation. nents of the activity had secondary effects on the amount and diversity of students' reading. It is interesting to watch how much easier the Several possible explanations may account kids are catching mistakes (spelling & usage) for this finding. First, teachers seemed to relate of each other. There is a lot of revising going on when these kids pair up to create their more directly to the goal of increasing indepen- stacks. dent reading than to increasing diversity. This orientation may have subtlety affected the way Their essays [in a writing contest entered by all they implemented or reinforced project activi- of the fifth-grade students] were much im- ties and/or selectively influenced their percep- proved over prior students. We feel like this tions. Also, the project activities focused on project has helped students' writing skills as books, not other reading materials such as well as reading skills. magazines, which may have given an overly narrow view of students' reading. Likewise, We are working in [the lab] reviewing one because the database activity was not fully another's workediting, etc. Amazing how implemented, students did not participate for an students can see mistakes of other students and extended period of time in a part of the project not recognize their own. with greater potential to increase diversity. I do feel that our students' exposure in com- puter program helped them to be better editors. Students' Increased Concern for Writing

...this experience has made them [students] Although the teachers were aware that we aware of the writing/proof reading/editing were most interested in their observations process.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

75 A Formative Experiment 65

Clearly, the involvement in creating Miscellaneous Preliminary Findings multimedia book reviews had an effect on and Unanticipated Effects students' writing as we also noted indepen- dently in our field notes in comments such We report here several preliminary find- as the following. ings and observations that may be of interest to those who wish to conduct related formative I've noticed ...that students are much experiments. These preliminary findings were more careful with their writing. They not central to our stated pedagogical goal nor ask how to spell words when writing. to our interest in determining the degree to [Ms. Burton] notice too that students which computer-based activities were fully working in the lab are more concerned about their spelling.[She feltthat] appropriated by teachers and students. Neither students feel some ownership over their are these findings based on extensive, system- book reviews. atic data collection and analysis. 1. Typing was not a major obstacle for This increased care in writing seemed students in this activity. Although none of the related to students viewing the multimedia students had highly developed typing skills, book reviews as public documents intended for they were content to type slowly when entering use by teachers, parents, the university re- textual information. Students virtually never search team, and other students. Jason, one of complained about having to type information, the focal students at Hartwig school, stated, "I despite that it was a laborious task for most of have to fix that [spelling error] because I don't them. Moreover, students working in pairs at want anybody to think I'm dumb." Students a computer were usually content to wait on also seemed to gain satisfaction from finding their partners when typing information. errors in spelling, grammar, and punctuation 2. Participation in this project had a on those occasions when they read each others' notableeffect on teachers'professional reviews. Their attention to others' work may development that extended beyond the project. have increased attention to their own, perhaps Although this project did not have all of the to avoid criticism from their peers. Teachers marks of a true collaborative research project also emphasized technical correctness in stu- between classroom teachers and university dents' reviews, which may have heightened researchers (see Allen et al., 1992; Anders, students' concern for technical correctness. 1996; Jervis et al., 1996), teachers seemed to Several of the teachers insisted that students benefit professionally from their involvement write out a technically Correct version of the in the project. All of the teachers presented at text of their reviews before they were allowed to enter it on the computer. However, for the least once at a professional conference (to most part they did so in an off-hand, informal our knowledge, none had done so before); way rather than badgering students about their one teacher began an advanced degree pro- writing. gram in conjunction with the project while

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

76 66 Reinking & Watkins

another incorporated it into an on-going pro- effectively. They learned specific skills related gram. Most notably, three teachers submitted to HyperCard such as being able to create proposals to a state conference on their own in buttons and to link cards, which can be trans- the year following the completion of the pro- ferred to other applications. They also learned ject. Several of the teachers noted in their logs general information about hardware and soft- that these activities were somewhat intimidating ware such as how to hook up and use an LCD but at the same time rewarding and profession- panel or how to move files from a disk to the ally meaningful. hard drive. Also as a result of their participa- 3. Parental involvement in the classroom tion in the project, they learned how texts and school increased in Collins and Hartwig might be incorporated with other media to Schools where a concerted attempt was made to create electronic documents. Long-term experi- recruit and train parents to assist with the ences with such activities have demonstrated HyperCard programming. Increased parental that students change their conceptions of writ- involvement was also a by-product of the video ing and reading (see Tierney et al., 1992) created by Ms. Pearson's class because parents were enlisted to make costumes, find props, Discussion and donate video equipment. Even in Borders School where parents were not recruited to assist with the day-to-day multimedia book The primary purpose of the present study review activities, many parents attended a was to investigate how a multimedia book school technology fair where the multimedia review activity might be implemented in mid- book review project was among the displays. dle-grade classrooms to effect increases in the Many parents seem to have an inherent interest amount and diversity of students' independent in technology on behalf of their children and reading. That purpose was derived from our enthusiastically support such efforts. Their interest in an approach to classroom research involvement seemed to be valued equally by that Newman (1990) has described as a forma- teachers, administrators, and students, although tive experiment. Formative experiments ad- for reasons that are likely to be somewhat dif- dress a domain of questions and issues that are ferent for each group. Parents seemed to react particularly relevant to the use of computer positively to participating in learning about technology in schools and that have not been technology in a nonthreatening way with teach- addressed by conventional experiments, nor ers and their children. Our experience in this typically by qualitative approaches to research. project fits well within and reinforces the A secondary purpose was to expand and to model for creating a community of learners refine our knowledge about the concept of a through the implementation of computers as formative experiment as a an approach to proposed by Keeler and Alexander (1994). classroom research and about the practicality of 4. The project clearly increased the ability conducting such research. In this section, we of teachers and students to use the computer discuss each of these purposes separately.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

77 A Formative Experiment 67

Discussion of Results ers' involvement with the project are leading them to emphasize independent Both the quantitative and qualitative data reading more which filters down to the indicate that the multimedia book review activ- students so that some of the effects we're ity contributed to achieving the pedagogical seeing maybe during the project are not goal of increasing the amount of children's directly due to the students entering book reviews but the total impact that the proj- reading. Although, as one might expect to ect is having on the school or the teachers discover using a formative experiment, the such that independent reading is empha- manner in which the activity contributed to the sized more. pedagogical goal was not necessarily antici- pated at the outset of the project. The interven- Put another way, our pedagogical goal of tion's effect on achieving the pedagogical goal increasing the amount of independent reading was clearly mediated by students' and teachers' was furthered more by connecting books to an responses to the challenges of working with the engaging, challenging use of the computer than technology, particularly HyperCard, not pri- by the capability of the technology to encour- marily by the creation of multimedia book age sharing of information about books. This reviews and sharing them with others as antici- realization gradually led us to see our invest- pated. The introduction of the multimedia book ment of considerable time and effort into review activities represented a novel intrusion having teachers and students learn HyperCard into normal classroom routines, which was not has a frustrating distraction from our in- greeted with much enthusiasm by teachers and tended goal but as an important way to enhance students as well as administrators and parents. it. Thus, in the project's second year, we felt That climate combined with a change in the justified in refining the activities designed for social dynamics of instruction and the increased teachers to learn HyperCard who would then engagement of students and teachers led to teach it to students. increased interactions about and enthusiasm for We believe this finding to be important books, which in turn led to more independent because it could be argued that the intrigue of reading. Thus, the increase in independent the technology would be so distracting as to ready was more a by-product of students' and work against the pedagogical goal, which teachers' primary focus on mastering the would call into question the utility of the entire technology,particularly.HyperCard.This activity for accomplishing the identified goal. conclusion is reflected in the following tran- Also, we believe the fact that involvement with scription from one of our tape-recorded field technology tended to mediate the accomplish- notes at Hartwig School during February: ment of a valued pedagogical goal is consistent with the perception of many educators that [Ms. Pearson's comment] leads me to technology can be valued intrinsically for that think that maybe some of the effects of the purpose. Itis not trivial to determine that project are indirect in the sense that teach- technology can act as a catalyst for changing

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

78 68 Reinking & Watkins

the typical interaction patterns among teach- especially in the early stages of the project. For ers and students. Such changes are the basis example, in the early stages of the project, for arguments that technology offers strong integration rarely extended beyond occasionally potential for bringing about school reform rescheduling previously-planned activities to (see Means, 1994). Thus, the results of this work extra time on multimedia book review study have implications that extend beyond activities. As the year progressed in several of the particular pedagogical goal and the the classrooms, there was some evidence that instructional intervention investigated. Not project activities were being extended and only may the use of technology in schools integrated into other classroom activities, most positively alter social interaction patterns in notably in Ms. Pearson's class with increased schools, it may further curricular goals in word processing activities and her implementa- various subject areas by embedding relevant tion of the group book review activity related content within challenging and engaging to her social studies unit. This minimal exten- computer-based tasks such as using Hyper- sion and integration coupled with the teachers' Card. In that regard, the findings of the connection of the project activities with con- current investigation are consistent with ventionalcurricular goals(e.g.,technical aspects of writing) are consistent with much other studies indicating that challenging uses previous research findings on the effects of of technology can over time further the goals introducing technology into classrooms (see of literacyinstructioninschools(e.g., Bruce & Rubin, 1993; Miller & Olson, 1994). Tierney et al.,1992; Turner & Dipinto, It is also consistent with a defining attribute of 1992). formative experiments as described by New- On the other hand, we found that the extent man (1990) who argues that many endpoints to which the technology was fully appropriated are possible in a particular experiment and that by teachers was decidedly mixed. Factors that "[t]he environment may transcend its initial seem to account for differences include the role goals. It may also retain goals and organization that teachers assumed in relation to the activi- in spite of the technology designer's concerted ties,the logistical arrangements that were efforts to support alternative models" (p. 10). necessary to carry out the activities, and the Thus, although computer-based activities degree to which the overall climate of a school such as the one in this study seem to have and classroom led teachers to be flexible and strong potential to alter positively the social independent. Although there was considerable dynamics between teachers and students, this variation from classroom to classroom, we effect in itself does not seem to be adequate found that teachers readily accommodated the to transcend teachers' commitment to con- multimedia book review activities into their ventional instructional goals and activities. instructional program, but the level of integra- It seems to us, however, that teachers may tion into their teaching beyond the project be much more receptive to transforming activities tended to be minimal and superficial, their instruction within the context created

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

79 A Formative Experiment 69

by a computer-based activity that points in to deviate from set schedules and established that direction, although the data in this curriculum. The availability of equipment such investigation only support that contention as an LCD panel for group presentations indirectly. One development that does sup- changed considerably not only the options for port such an interpretation is that the project implementing the activities but the type of activities continued to be implemented in social interactions that occurred during the varying degrees at Hartwig and Borders activities. Given our experience, we are con- Schools into a subsequent school year despite vinced that it would be difficult, if not impossi- the fact that the project had concluded in both ble, to carry out the multimedia book review schools. project without at least one teacher who as- We found little evidence that the activity, sumed the role of "technology expert" as as implemented, had an effect on the goal of discussed in the results section of this report. increasing the diversity of students' indepen- These differences had to be taken into account dent reading. One explanation for the lack of as we assisted teachers who were to implement results in this area is that we did not fully the project in each school, and which led to implement the database activity, which was variations within the overall framework of designed specifically to expand students' aware- the multimedia book review activity. How- ness of a variety of books being read by their ever, when using a formative experiment as classmates. Another factor may be that teachers an approach to classroom research, these seemed less conscious of and concerned about differences become opportunities to extend this goal. It is also possible that the multimedia understanding of the relation between the book review activity may not be adequate in intervention and the pedagogical goal. itself to overcome students' propensity to read Through the use of a formative experi- a narrow range of topics, genres, and authors. ment, we were also attuned to unanticipated Broadening students reading may depend upon outcomes of the intervention not directly re- coordinating and supplementing the multimedia lated to our pedagogical goal but that might book review activity with other classroom provide the basis of further study. That is, it is activities in the areas such as literature re- possible that the same or a similar intervention sponse, thematic units, and reading diverse might be studied in relation to other pedagogi- texts aloud to students. cal goals. For example, the multimedia book Not surprisingly, we found that the school review activity as carried out in this investiga- environments and teachers'roles to some tion seemed to have an effect on students' extent shaped the effects of the multimedia writing and on teachers' perceptions of their book review activity. For example, the positive writing. Likewise, other outcomes that might effects of the project seemed enhanced by a be investigatedinclude increased parental supportive, nonthreatening school environment involvement, increases in computer skills and in which teachers felt supported and trusted to uses, attitudes toward computers, and teachers' make their own decisions and had the freedom professional development.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

80 70 Reinking & Watkins

Discussion of the Present Investigation time in response to it. We discovered that the as a Formative Experiment multiple levels on which formative adjustments needed to be made almost on a daily basis did The concept of a formative experiment not permit the detached reflection and carefully guided this investigation. To our knowledge, reasoned decisions that we had hoped. Retro- the explication of what a formative experi- spectively, we do think, however, that future ment is and how it might be conducted is formative experiments might consider develop- limited to (a) a single article by Newman ing and using a process/product matrix that (1990) who proposed the concept and pro- Patton (1990) has proposed as useful in evalu- vided a single example of a study from his ating programs. General components of that own work, and (b) our own speculations matrix might include concerns in the following about itsutility as a form of classroom areas: logistical (in terms of implementing the research and about a framework for designing intervention), methodological (i.e., modes of and conducting such research (see Reinking & data collection), pedagogical, technological, Pickle, 1993; Baumann et al., 1996). Thus, it interpersonal, and ethical. Such a matrix may seems warranted that we comment briefly here not eliminate the need for more fluid decision on how our experience in using this approach making, but may aid in documenting more compared to our abstract musings about its clearly the relationship between variables and potential merit. adjustments that were made in response to Our most poignant discovery is that the them. formative cycle of data collection and resulting Another issue we were able to address in adjustments to the intervention must be consid- our exploration of using a formative experi- ered fluid as opposed to a discreet and orderly ment was how to write a report of our investi- progression of events. An advantage of ap- gation. We found that it was possible to use the proaching classroom research from the stand- conventional structure reporting a quantitative point of a formative experiment is that the study to be suitable for reporting our formative effects of instructional interventions are ac- experiment. That is, we began with a rationale knowledged to be the product of a complex for our investigation, including a literature array of interacting variables within the school review. However, unlike the introduction to a and classroom environment. We believed at the typical research report, we also included our outset of our investigation that through the pedagogical goal, the pedagogical theory that application of rigorous research methods, we justified it, and a description of the instruc- could isolate these variables and their role in tional intervention we believed addressed our enhancing orinhibitingtheintervention's goal. The remainder of the report followed effectiveness toward accomplishing the peda- convention closely with sections pertaining to gogical goal. Moreover, we thought that when method, results, and a discussion of findings. a single variable was identified, the interven- Although we are not convinced that this is the tion could be modified at a definite point in best or only reasonable way to report formative

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

Si A Formative Experiment 71

experiments, to us it seems advantageous that there unqualified instructional recommenda- a familiar structure works. tions that can assure the accomplishments of The concept of a formative experiment those goals. Nonetheless, formative experi- needs more careful scrutiny and thought before ments do not exclude the possibility of general- it can be clearly articulated as a genre of class- izations. Factors affecting movement toward room research. If it is to be widely accepted the pedagogical goal may be found to cut and used, it will need not only a coherent across different even distinctly different envi- rationale, but also general guidelines for con- ronments that could lead to general recom- ceptualizing investigations; for planning their mendations. implementation; for gathering, analyzing, and In the end, we agree with Eisenhart and interpreting data; and for reporting results. It Borko (1993) that useful classroom research will need a well-articulated epistemological must transcend any particular paradigm and rationale as well. We suspect that such a ration- that researchers need to be creative and eclectic ale may be found within pragmatic points of in their approach to investigating teaching and view as expressed by writers such as Cherry- learning in schools. Based on our experience, holmes (1993). we think that formative experiments have Despite the need for more elaboration and strong potential as a framework for providing examples of formative experiments, we believe such flexibility and eclecticism while conform- through our experience that they have strong ing to the principles they propose should guide potential as a useful, systematic alternative to any form of classroom research. In that regard, existing approaches to classroom research. we believe formative experiments merit Among the advantages of formative experi- attention by a broad spectrum of classroom ments is that they more readily acknowledge researchers, not just those interested in and address the inherent complexities of class- technology. Thus, we hope that this investiga- rooms while seeking ways to address that tion will generate interest not only in our complexity toward the attainment of well- findings relative to using multimedia book defined pedagogical goals. As such, they reviews to enhance the amount and diversity of reflect, albeit more systematically, the often independent reading, but also in the concept of trial-and-error aspects of good teaching. This a formative experiment as a form of classroom similarity to teaching may increase the recep- research. tiveness of classroom teachers to the research enterprise and to defining themselves as an References integral part of that enterprise as teacher/ researchers. This perspective may also appeal Allen, J., Buchanan, J., Edelsky, C., & Norton, G. to educational policy makers and parents who (1992). Teachers as "they" at NRC: An invita- can relate to the fact that formative experiments tion to enter the dialogue on the ethics of collab- involve clear-cut pedagogical goals while at the orative and non-collaborative classroom re- same time impressing upon them that rarely are search. In C. K. Kinzer & D. J. Leu (Eds.),

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

82 72 Reinking & Watkins

Literacy research, theory, and practice: Views Bresler, L., & Walker, D. (1990). Implementation from many perspectives (forty-first yearbook of of computer-based innovation: A case study. the National Reading Conference). Chicago: Journal of Computer-Based Instruction, 17, National Reading Conference. 66-72. Alvermann, D. E., & Guthrie, J. (1993). The Brown, A. L. (1992). Design experiments: Theo- national reading research center. In A. Sweet & retical and methodological challenges in creating J. I. Anderson (Eds.), Reading research into the complex interventions in classroom settings. year 2000 (pp. 129-150). Hillsdale, NJ: Erl- Journal of the Learning Sciences, 2,141-178. baum. Bruce, B. C., & Peyton, J. K. (1990). A new Anders, P. (1996). The ethics of collaborative writing environment and an old culture: A educational research. In L. Baker, P. Affler- situated evaluation of computer networking to bach, & D. Reinking (Eds.), Developing en- teach writing. Interactive Learning Environ- gaged readers in school and home communities ments, 1, 171-191. (pp. 271-285). Mahwah, NJ: Erlbaum. Bruce, B. C., & Rubin, A. (1993). Electronic Anderson, R. C., Wilson, P. T., & Fielding, L. G. quills: A situated evaluation of using computers (1988). Growth in reading and how children for classroom writing. Hillsdale, NJ: Erlbaum. spend their time outside of school. Reading Cameron, J., & Pierce, W. D. (1994). Reinforce- Research Quarterly, 3, 285-303. Baumann, J. F., Dillon, D. R., Shockley, B. B., ment, reward, and intrinsic motivation: A meta- Alvermann, D. E., & Reinking, D. (1996). analysis. Review of Educational Research, 64, 363-423. Perspectives for literacy research. In L. Baker, P. Afflerbach, & D. Reinking (Eds.), Develop- Cherryholmes, C. H. (1993). Reading research. ing engaged readers in school and home com- Journal of Curriculum Studies, 25, 1-32. munities (pp. 217-245). Mahwah, NJ: Erlbaum. Congressional Office of Technology Assessment. Becker, H. J. (1990, April). Computer use in (1988). Power on!: New tools for teaching and United States schools: 1989: An initial report of learning (Publication No. 052-003-01125-5). the U.S. participation in the I. E.A. computers in Washington, DC: U.S. Government Printing education survey. Paper presented at the meeting Office. of the American Educational Research Associa- DeFord D. E. (1985). Validating the construct of tion, Boston, MA. theoretical orientation in reading instruction. Becker, H. J. (1992a, April). Top-Down versus Reading Research Quarterly, 20, 351-367. grass roots decision making about computer Dickinson, D. K. (1986). Cooperation, collabora- acquisition and use in American schools. Paper tion, and a computer: Integrating a computer presented at the meeting of the American Edu- into a first-second grade writing program. cational Research Association, San Francisco, Research in the Teaching of English, 20, 357- CA. 378. Becker, H. J. (1992b). Computer-based integrated Eisenhart, M., & Borko, H. (1993). Designing learning systems in the elementary and middle classroom research: Themes, issues, and strug- grades: A critical review and synthesis of evalu- gles. Boston, MA: Allyn & Bacon. ation reports. Journal of Educational Computing Emihovich, C., & Wager, W. (1992). Media cul- Research, 8,1-41. ture/school culture: An introduction to the

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

83 A Formative Experiment 73

issues. Education and Urban Society, 24, 435- Paper presented at the meeting of the American 439. Educational Research Association, New Or- Flagg, G. N. (1990). Formative evaluation for leans, LA. educational technologies. Hillsdale, NJ: Erl- Kirby, D. R., & Kirby, K. (1985). The reading- baum. writing connection. In L. W. Searfoss & J. E. Foertsch, M. A. (1992). Reading in and out of Readence (Eds.), Helping children learn to read school. Washington, DC: Office of Educational (pp. 338-353). Englewood Cliffs, NJ: Prentice- Research and Improvement, US Department of Hall. Education. LaCompte, M. D., & Preissle, J. (1993). Eth- Friedman, B. (1990, April). Societal issues and nography and qualitative design in educational school practices: An ethnographic investigation research (2nd ed.). New York: Academic Press. of the social context of school computer use. Martinez, M. E., & Mead, N. A. (1988). Computer Paper presented at the meeting of the American competence: The first national assessment (Re- Educational Research Association, Boston, MA. search Rep. No. 17-CC-01). Princeton, NJ: Freebody, P., & Anderson, R. C. (1983). Effects Educational Testing Service. on text comprehension of differing proportions McGee, G. W. (1987). Social context variables and locations of difficult vocabulary. Journal of affecting the implementation of microcomput- Reading Behavior, 15, 19-39. ers. Journal of Educational Computing Re- Glaser, B. G., & Strauss, A. (1967). The discovery search, 3, 189-206. of grounded theory: Strategies for qualitative McKenna, M. C., & Kear, D. J. (1990). Measur- research. Chicago, IL: Aldine. ing attitude toward reading: A new tool for Gustafson, K. L. (1993, April). Introducing teach- teachers. The Reading Teacher, 43, 626-639. ers to computer technology: A case study. Paper McKenna, M. C., Kear, D. J., & Ellsworth, R. A. presented at the meeting of the American Edu- (1995). Children's attitudes toward reading: A cational Research Association, Atlanta, GA. national survey. Reading Research Quarterly, Hadley, M., & Sheingold, K. (1993). Common- 30, 934-956. alities and distinctive patterns in teachers' Means, B. (Ed.). (1994). Technology and education integration of computers. American Journal reform: The reality behind the promise. San of Education, 101, 261-315. Francisco, CA: Jossey-Bass. Jackson, P. W. (1990). Life in classrooms. New Means, B., Blando, J., Olson, K., Morocco, C. C., York: Teachers College Press. Remz, A. R., & Zorfass, J. (1993). Using Jervis, K., Carr, E., Lockhart, P., & Rogers, J. technology to support educational reform. (1996). Multiple entries into inquiry: Dissolving Washington, DC: U.S. Department of Educa- the boundaries between research and teaching. tion. In L. Baker, P. Afflerbach, & D. Reinking Mehan, H. (1989). Microcomputers in classrooms: (Eds.), Developing engaged readers in school Educational technology and social practice. and home communities (pp. 247-270). Mahwah, Anthropology & Education Quarterly, 20, 4-21. NJ: Erlbaum. Michaels, S., & Bruce, B. (1989). Classroom Keeler, C. M., & Alexander, G. C. (1994, April). contexts and literacy development: How writing Implementing computers in the classroom: A systems shape the teaching and learning of model for creating a community of learners. composition (Tech. Report No. 4876). Urbana,

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

84 74 Reinking & Watkins

IL: University of Illinois, Center for the Study M. L. Kamil, P. B. Mosenthal, & P. D. Pear- of Reading. son (Eds.), Handbook of reading research (Vol. Miller, L., & Olson, J. (1994). Putting the computer 2, pp. 310-340). New York: Longman. in its place: A study of teaching with technology. Reinking, D., & Pickle, M. (1993). Using a forma- Journal of Curriculum Studies, 26,121-141. tive experiment to study how computers affect Morgan, D. (1993). Successful focus groups: Ad- reading and writing in classrooms. In D. J. Leu vancing the state of the art. Newbury, CA: & C. K. Kinzer (Eds.), Examining central Sage. issues in literacy research, theory, and practice Newman, D. (1990). Opportunities for research on (pp. 263-270). Chicago, IL: National Reading the organizational impact of school computers. Conference. Educational Researcher, 19(3), 8-13. Riel, M. (1989). The impact of computers in class- Newman, D., Griffin, P., & Cole, M. (1989). The rooms. Journal of Research on Computing construction zone: Working for cognitive change Education, 22(2), 180-189. in school. Cambridge: Cambridge University Roblyer, M. D., Castine, W. H., & King, F. J. Press. (1988). Assessing the impact of computer-based O'Flahavan, JGambrell, L. B., Guthrie, J., instruction. Computers in the Schools, 5(3-4), Stahl, S., Baumann, J. B., & Alvermann, D. 1-149. E. (1992, August/September). Poll results Rubin, A. D., & Bruce, B. (1990). Alternate guide activities of research center. Reading realizations of purpose in computer-supported Today, 12. writing. Theory into Practice, 29,256-263. Papert, S. (1987). Computer criticism vs. techno- Salomon, G. (1991). Transcending the qualitative- centric thinking. Educational Researcher, 16(1), quantitative debate: The analytic and systemic 22-30. approaches to educational research. Educational Papert, S. (1993). The children's machine: Rethink- Researcher, 20(6), 10-18. ing school in the age of the computer. New Sheingold, K. (1991). Restructuring for learning York: Basic Books. with technology: The potential for synergy. Phi Patton, M. Q. (1990). Qualitative evaluation and Delta Kappan, 73, 17-27. research methods (2nd ed.). Newbury Park, Spiegel, D. L. (1981). Reading for pleasure: Guide- CA: Sage lines. Newark DE: International Reading Asso- Reeves, T. C. (1992). Evaluating schools infused ciation. with technology. Education and Urban Society, Stanovich, K. (1986). Matthew effects in reading: 24, 519-534. Some consequences of individual differences in Reinking, D. (1986). Six advantages of computer- the acquisition of literacy. Reading Research mediated text for reading and writing instruc- Quarterly, 21, 360-407. tion. The Reading Instruction Journal, 29,8-16. Tierney, R. J., Kieffer, R. D., Stowell, L., Desai, Reinking, D., & Bonham, S. (1996). Multimedia L. E., Whalin, K., & Moss, A. G. (1992). book reviews (Instructional Resource No. 40). Computer acquisition: A longitudinal study of Athens, GA: Universities of Georgia and Mary- the influence of high computer access on stu- land College Park. dents' thinking, learning, and interaction Reinking, D., & Bridwell-Bowles, L. (1991). (Apple Classrooms of Tomorrow Report Computers in reading and writing. In R. Barr, #16). Cupertino, CA: Apple Computer.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

85 A Formative Experiment 75

Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31-44. Turner, S. V., & Dipinto, V. M. (1992). Students as hypermedia authors: Themes emerging from a qualitative study. Journal of Research on Computing in Education, 25(2), 187-199. Venezky, R. L. (1983). Evaluating computer- assisted instruction on its own terms. In A. C. Wilkinson (Ed.), Classroom computers and cognitive science (pp. 31-49). New York: Academic Press.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

86 A Formative Experiment 77

Appendix A

Semistructured Interview with Teachers Before Project Began

Experience With and Attitudes About Computers

Structured Interview With Project Teachers

1. What was your experience with computers prior to this point? (Outside of school/in school)

Do you own a computer? Would you buy one if you had the resources? What would you use the computer for?

2. What were your attitudes toward computers prior to this project? What are your attitudes toward computers at this point in time?

3. Have you had any opportunities to observe students using computers? If so, could you describe the situations in which you have observed them? What were some of your observations?

What do you think are the students' view of computers?

4. How do you think other teachers in general view computers? The teachers in your school? Administrators? Parents?

5. What positive outcomes do you hope for in this project? What negative outcomes are you concerned with?

6. Imagine you are in a school that has as many computers as there are teachers. If you were responsible for making the decision about whether the computers would all be put in a lab or one in each classroom what would you do?

7. Can you remember when you first thought about the possibility of using a computer for teaching? Describe what you remember.

8. Has the computer affected your teaching in any way?

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

87 78 Reinking & Watkins

Appendix B

Questionnaire on Diversity of Reading Completed by Students Before the Project Began and at the End of the School Year

CHOOSING THINGS TO READ

Name Teacher Date

Part One

Directions: Suppose that your teacher says you must choose a book to read from the library. Which kinds of books might you choose to take home to read? Check any kind of bookyou might want to read. But, don't check any you probably wouldn't want to read.

a book about how to play a sport a book about an event in history a story that is a mystery a make-believe story a story about a pet an encyclopedia a book about cars or trucks a book about a person in history a science fiction story a book about a movie or TV star a story about Indians a book of beautiful poems a story about kids my own age a story about people in other a book of fairy tales or myths countries a book on science experiments a story about scary things a book about animals a story about a horse a book about planes a story that makes me laugh a book about dinosaurs a book of jokes a book of science experiments a book of funny rhymes and riddles a book about how to cook something a book of cartoons a book about trains a story about someone in a war a story about strange creatures a story that takes place in the a story about an adventure future a book about how to take care of animals a book about volcanoes

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

88 A Formative Experiment 79

a story about someone my age a story about boys and girls getting a book with maps along a story about faraway places a story about someone who likes to a story about monsters or play sports strange creatures a book about what I would like to a book about things that are be when I grow up strange but true

What are some other kinds of books that you like to read?

What are your favorite kinds of books?

Part Two

When you pick a book to read, what do you think about?

Directions: Answer each question by putting an X in one of the boxes.

1. When you pick a book, how often do you think about how the cover of the book looks?

once in never a while often always

2. When you pick a book, how often do you think about how many pictures the book has?

once in never a while often always

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

89 80 Reinking & Watkins

3. When you pick a book, how often do you think about how long the book is?

once in never a while often always

4. When you pick a book, how often do you think about what your friends say about the book?

once in never a while often always

5. When you pick a book, how often do you think about what your teacher says about the book?

once in never a while often always

6. When you pick a book, how often do you think about who the author of the book is?

once in never a while often always

7. When you pick a book, how often do you think about whether you will like the characters or not?

once in never a while often always

8. When you pick a book, how often do you think about whether you will learn something from the book?

once in never a while often always

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

90 A Formative Experiment 81

Part Three

Directions: Answer each question by putting an X in one of the boxes.

1. When you read on your own, how often do you read books?

once in never a while often always

2.When you read on your own, how often do you read magazines?

once in never a while often always

3.When you read on your own, how often do you read newspapers?

once in never a while often always

4.When you read on your own, how often do you look for information in books like an encyclopedia, dictionary, atlas?

once in never a while often always

5.When you read on your own, how often do you read stories that have characters made up by the author?

once in never a while often always

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

91 82 Reinking & Watkins

6.When you read on your own, how often do you read books that give information?

once in never a while often always

7.When your read on your own, how often do you read about how to do something?

once in never a while often always

8.When you read on your own, how often do you read about people who really lived and things that really happened?

once in never a while often always

9.When you read on your own, how often do you read books that are funny?

once in never a while often always

10. When you read on your own, how often do you not finish a book?

once in never a while often always

11. When you read on your own, how often do you read poetry?

once in never a while often always

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

92 A Formative Experiment 83

12. When you read on your own, how often do you read about sports?

once in never a while often always

13. When you read on your own, how often do you read about animals?

once in never a while often always

14. When you read on your own, how often do you read about science?

once in never a while often always

15. When you read on your own, how often do you read about adventure?

once in never a while often always

16. When you read on your own, how often do you read stories that are make believe?

once in never a while often always

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

93 84 Reinking & Watkins

Appendix C

Parent Questionnaire Completed Before Project Began and at the End of the School Year

Parent Questionnaire

We appreciate your cooperation in taking 10-15 minutes to complete this questionnaire, which should be returned to your child's teacher. This information is important to participation in the special computer project. If you have your child deliver the completed questionnaire to her/his teacher, you may want to seal it in the attached envelope. Thank you for your assistance.

Your child's name:

1. Of the following activities, check the three that your child is most likely to do during free time at home.

play outside talk on the telephone work on arts and crafts listen to music read a book or magazine play video games work on a hobby watch TV play a musical instrument play with toys dramatic or pretend play other (describe) (skits, playing house outerspace, etc.)

2. Circle the activity in number one that your child does most during free time.

3. On average, how long does your child spend reading for enjoyment each day? hours minutes

4. On average, how long does your child spend watching TV each day? hours minutes

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

94 A Formative Experiment 85

5. Of the following statements, check the one that best applies to your child:

My child rarely, if ever, reads anything for enjoyment. My child reads for enjoyment once in a while. My child reads regularly for enjoyment but not a lot compared to other activities. My child reads regularly and often for enjoyment. My child reads for enjoyment almost all the time he/she has an opportunity to do so.

6. How would you rate your child's reading ability? (Check one.)

Well above average Below average Above average Well below average Average I'm not sure

7. Are there children's books in your home for your child to read? yes no If so, about how many? (Circle one.)

fewer than 5 5-20 20-50 50-100 more than 100

8. Does your child use a computer at home? yes no If yes, what kind?

9. Rate on a five-point scale how often your child does the following? (Circle a number for each statement.) Never Very Often

Order books from book clubs. 1 2 3 4 5

Look at books/magazines in stores. 1 2 3 4 5

Ask for books /magazines as gifts. 1 2 3 4 5

Complain about having to read a book for school. 1 2 3 4 5

Bring home something from school to read for enjoyment. 1 2 3 4 5

Talk about something he/she has read. 1 2 3 4 5

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

95 86 Reinking & Watkins

Never Very Often

Ask to go to the library to find something to read. 1 2 3 4 5

Look up information in the dictionary or encyclopedia. 1 2 3 4 5

Read something to you. 1 2 3 4 5

Read something in the newspaper. 1 2 3 4 5

Give someone reading material as a gift. 1 2 3 4 5

Read to another child. 1 2 3 4 5

Say he/she doesn't like reading. 1 2 3 4 5

Recognize or talk about a particular author. 1 2 3 4 5

Go to movies or watch TV programs about books he/she reads. 1 2 3 4 5

Write her/his own "books" or stories for enjoyment. 1 2 3 4 5

Read about places he/she will visit/is visiting on a trip. 1 2 3 4 5

Say that reading is boring. 1 2 3 4 5

Read cereal boxes or other materials while eating. 1 2 3 4 5

Read in front of the TV. 1 2 3 4 5

Choose Halloween costumes based on book characters. 1 2 3 4 5

Use their own money to buy a book. 1 2 3 4 5

10. Have you observed other behaviors (positive or negative, like the ones above) that indicate how much your child reads and how he/she feels about reading? If so, please describe them. (Use the back of this questionnaire, if necessary.)

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

96 A Formative Experiment 87

11. Does your child have a library card? yes no

12. About how many times a month does your child go to the library? (Circle one.)

0 1 2 3 4 5 more than 5

13. If your child went to the library, about how many books would you expect her/him to check out?

0 1-34-10more than 10

14. How much does each of these statements sound like your child? (Circle a number.)

a. He/she always reads the same kind of reading material on the same topic (for example, just comic books about the same character or just books about horses).

Not at all Very much like my child 1 2 3 4 5 like my child

b. He/she reads one type of reading material (for example, just books or just newspapers), but reads about a variety of topics.

Not at all Very much like my child 1 2 3 4 5 like my child

c. He/she reads many types of reading materials (for example, books, magazines, newspapers, encyclopedias, etc.), but reads mostly on one topic.

Not at all Very much like my child 1 2 3 4 5 like my child

d. He/she reads many types of reading materials on many different topics.

Not at all Very much like my child 1 2 3 4 5 like my child

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

97 88 Reinking & Watkins

15. Do you subscribe to any magazines for children? yes no If so, how many? (Circle one.)

1 2 3 more than 3

16. Most of the adults that my child knows outside of school (Check one.)

hardly ever read for enjoyment. read very often for enjoyment. read once in a while for enjoyment. I don't know. read often for enjoyment.

17. Most of the children that my child plays with outside of school (Check one.)

hardly ever read for enjoyment. read very often for enjoyment. read once in a while for enjoyment. I don't know. read often for enjoyment.

*18. During the past school year, how often has your child talked about using the computer for reading at school? (Check one.)

never occasionally often almost everyday

*19. During the past school year, how has your child's out-of-school reading changed? (Check one.)

reading much less than before reading a little less than before reading about the same amount as before reading a little more than before reading a lot more than before

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

98 A Formative Experiment 89

*20. During the past year, how has your child's interest in reading changed? (Check one.)

much less interested in reading a little less interested in reading about the same interest in reading a little more interested in reading much more interested in reading

*21. Comment, if you wish, on your child's reading, use of the computer, and so forth. We are especially interested if you believe you have seen any connection between reading activities on the computer at school and reading-related activities outside of school.

*These items were on the end-of-year survey only.

NATIONAL READING RESEARCH CENTER, READING RESEARCH REPORT NO. 55

99 MC National Reading Research Center 318 Aderhold, University of Georgia, Athens, Georgia 30602-7125 3216J. M. Patterson Building, University of Maryland, College Park MD 20742

100 U.S. DEPARTMENT OF EDUCATION Office of Educational Research and improvement (OERO Educational Resources information Center (ERIC) ERIC

NOTICE

REPRODUCTION BASIS

This document is covered by a signed "Reproduction Release (Blanket)" form (on file within the ERIC system), encompassing all or classes of documents from its source organization and, therefore, does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission to reproduce, or is otherwise in the public domain and, therefore, may be reproduced by ERIC without a signed Reproduction Release form (either "Specific Document" or "Blanket").

(9/92)