UNLV Retrospective Theses & Dissertations 1-1-2008 The effects of automated essay scoring as a high school classroom intervention Kathie L Frost University of Nevada, Las Vegas Follow this and additional works at: https://digitalscholarship.unlv.edu/rtds Repository Citation Frost, Kathie L, "The effects of automated essay scoring as a high school classroom intervention" (2008). UNLV Retrospective Theses & Dissertations. 2839. http://dx.doi.org/10.25669/37p4-kdh7 This Dissertation is protected by copyright and/or related rights. It has been brought to you by Digital Scholarship@UNLV with permission from the rights-holder(s). You are free to use this Dissertation in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s) directly, unless additional rights are indicated by a Creative Commons license in the record and/or on the work itself. This Dissertation has been accepted for inclusion in UNLV Retrospective Theses & Dissertations by an authorized administrator of Digital Scholarship@UNLV. For more information, please contact [email protected]. THE EFFECTS OF AUTOMATED ESSAY SCORING AS A HIGH SCHOOL CLASSROOM INTERVENTION by Kathie L. Frost Bachelor of Science University of Arizona Tucson, Arizona Master of Business Administration University of Nevada, Las Vegas A dissertation submitted in partial fulfillment of the requirements for the Doctor of Philosophy Degree in Curriculum and Instruction Department of Curriculum and Instruction College of Education Graduate College University of Nevada, Las Vegas December 2008 UMI Number: 3352171 Copyright 2009 by Frost, Kathie L. All rights reserved. INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. UMI UMI Microform 3352171 Copyright 2009 by ProQuest LLC. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest LLC 789 E. Eisenhower Parkway PO Box 1346 Ann Arbor, Ml 48106-1346 Copyright by Kathie L. Frost 2009 All Rights Reserved Dissertation Approval The Graduate College University of Nevada, Las Vegas November 17 20®^ The Dissertation prepared by Kathie L. Frost Entitled The Effects of Automated Essay Scoring as a High School Classroom Intervention is approved in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Curriculum and Instruction Examination Committee Chair Dean of the Graduate College Examinaèim Committee Member Examination Committee Member Examination Committee Metnber Graduate College-tacuity Representative 1017-52 11 ABSTRACT The Effects of Automated Essay Scoring as a High School Classroom Intervention by Kathie L. Frost Dr. Randall Boone, Examination Committee Chair Professor of Education University of Nevada Las Vegas This quasi-experimental, mixed methods study investigated whether students writing development and proficiency, in combination with teacher-led instruction, are significantly affected by the use of an automated essay scoring (AES) system. The ninth grade standard and honors English students were divided into control and treatment groups at a large, urban high school. Student writing was examined for any changes in proficiency, measured by human- and AES-scored holistic measures. A developmental writing index was used to analyze the rate of change in pre- and post-essays. The AES system was further researched by comparing thetreatment and control groups’ trait score categories. Finally, treatment students were interviewed and surveyed to identify their degree of satisfaction with the AES system. Ill Automated essay scoring systems have moved from their original purpose of rapidly and reliably scoring high stakes testing into the classroom as an instructional tool providing holistic and trait scoring. One area of potential AES usefulness is to provide students with more writing opportunities that include feedback. While supporting research findings that student writing improves if more writing opportunities with feedback are provided, this also supports the iterative process of writing and revision. To support teachers’ optimum classroom technology integration of an AES system to supplement teacher-led instruction, an access ratio of one Internet-connected computer for each student, (i.e., 1:1) needs to be provided. System-provided or teacher- provided writing prompts (i.e., topics) can be selected to provide students with AES simulations of the summative score of high stakes testing, in concert with formative trait scoring, which gives specific recommendations to improve writing. No gender difference was shown for the treatment participants from the AES- scored measures. The human-scored writing proficiency and development measures were inconclusive for gender and class levels due to the small sample size. By class levels, treatment honors students performed significantly better on the AES-scored proficiency measure, but the results were not supported by the human-scored measure. The other AES-scored measures analyzed by class levels, the development and trait category measures, did not show significance. However, the treatment participants expressed a high degree of satisfaction with the use of the AES system. IV TABLE OF CONTENTS APPROVAL PAGE................................................................................................................... ü ABSTRACT............................................................................................................................... iii LIST OF TABLES.................................................................................................................. viii CHAPTER 1 INTRODUCTION............................................................................................. 1 Purpose of the Study.............................................................................................................. 1 Background ............................................................................................................................. 3 Holistic Scoring................................................................................................................. 4 Computer Processing Tools..............................................................................................7 Automated Essay Scoring’s Beginnings............................................................... 8 Program Essay Grade’s Beginnings.................................................................................8 Writers Workbench .....................................................................................................10 Program Essay Grade in the 1990s................................................................................11 Current Automates Essay Scoring Systems...................................................................... 12 The Current Research..........................................................................................................13 CHAPTER 2 REVIEW OF RELATED LITERATURE....................................................15 Technology in the Classroom............................ 17 Internet Access..................................................................................................................17 Classroom Computer Use...............................................................................................20 Feedback ............................................................................................................................... 27 Computer Assisted Instruction....................................................................................... 27 Teacher Feedback .................................................. 29 Automated Essay Scoring Feedback..............................................................................39 Summary........................................................................................................................... 50 CHAPTER 3 METHODOLOGY..........................................................................................52 Research Design.................................................................................................................. 53 Quasi-experimental..........................................................................................................53 Potential Threats to Validity...........................................................................................55 Educational Scientific Research..................................................................................... 56 Participants....................................................................................................................... 57 Setting............................................................................................................................... 57 Teachers............................................................................................................................ 58 Students............................................................................................................................