An Intelligent Tutoring System
Total Page：16
File Type：pdf, Size：1020Kb
SUMMARY STREET: AN INTELLIGENT TUTORING SYSTEM FOR IMPROVING STUDENT WRITING THROUGH THE USE OF LATENT SEMANTIC ANALYSIS BY DAVID J. STEINHART B.S., UNIVERSITY OF CALIFORNIA, SANTA BARBARA, 1986 M.S., UNIVERSITY OF CALIFORNIA, SANTA BARBARA, 1989 M.A., UNIVERSITY OF COLORADO, BOULDER, 1996 A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF COLORADO IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF PSYCHOLOGY 2001 This thesis entitled: Summary Street: An Intelligent Tutoring System for Improving Student Writing through the use of Latent Semantic Analysis written by David J. Steinhart has been approved for the Department of Psychology __________________________ Walter Kintsch __________________________ Thomas K Landauer __________________________ Gerhard Fischer __________________________ Mitchell Nathan March 21, 2001 The final copy of this thesis has been examined by the signatories, and we find that both the content and the form meet acceptable presentation standards of scholarly work in the above mentioned discipline. HRC protocol #1198.19 Steinhart, David J. (Ph.D., Psychology) Summary Street: An Intelligent Tutoring System for Improving Student Writing through the use of Latent Semantic Analysis Thesis directed by Professor Walter Kintsch This dissertation describes the design, evolution, and testing of Summary Street, an intelligent tutoring system which uses Latent Semantic Analysis (LSA) to support writing and revision activities. Summary Street provides various kinds of automatic feedback, primarily whether a summary adequately covers important source content and fulfills other requirements, such as length. The feedback allows students to engage in extensive, independent practice in writing and revising without placing excessive demands on teachers for feedback. The efficacy of this system was examined in three classroom studies in a Boulder County school. In the first study, students read texts about three Mesoamerican civilizations and then composed summaries of those texts. One summary was produced using Summary Street, while the other two were produced using a traditional word processor. The students who used Summary Street to summarize the most difficult text produced better summaries than those students who used a word processor. In the second study, students learned about the human circulatory system and summarized two texts about the heart and lungs. The same pattern from the first study was observed—namely, the students who summarized the more difficult lung text using Summary Street produced better summaries than those students who used a word processor. iii The third and final study produced the clearest results. Ten different texts were used in the study, and there was a correlation between text difficulty and the value of the feedback from Summary Street—the more difficult the text, the more Summary Street helped students write better summaries. In addition to the aforementioned results, individual differences and issues regarding transfer are discussed. Finally, intelligent tutors and the role of technology in the classroom are examined, and Summary Street is compared to existing intelligent tutors. iv DEDICATION This dissertation is dedicated to all children who struggle in their journey to becoming proficient readers and writers. ACKNOWLEDGEMENTS Even though a dissertation has only one author, a dissertation is never the work of just one individual. This work was influenced by countless individuals whom I was fortunate enough to meet during my graduate career. While space (and poor memory) does not permit me to acknowledge them all, I would be remiss if I did not acknowledge the following individuals whose guidance, support, and wisdom so greatly influenced this body of work. First and foremost, this dissertation would not have been what it is if it weren’t for the unique background and perspective each of my committee members brought to the project. Walter Kintsch, my advisor, has been an unwavering source of support and a genuine inspiration. In addition to being the luminary in the field of text comprehension, perhaps more importantly, he believed in me, even when I didn’t believe in myself. Tom Landauer is Latent Semantic Analysis—he brought LSA to the University of Colorado, and was the catalyst for many new and exciting LSA-based projects in addition to this one. He also has an uncanny knack for explaining the most difficult concepts in a way that anyone can understand. Dick Olson is a leading expert on reading disabilities, and his insight into the minds of both children and researchers was invaluable. Gerhard Fischer’s expertise in collaborative design and educational technology contributed greatly, and he too possesses vi the ability to vastly simplify the complex for all to understand. Mitch Nathan’s expertise in the field of education and educational technology was right at home in this project, and his work on intelligent tutors was instrumental. Finally, I acknowledge Gerry Stahl, for his expertise in collaborative design, and perhaps more importantly, he was one of the primary architects of State the Essence, the predecessor of Summary Street. The teachers at Platt Middle School—Lisa Kihn, Ron Lamb, and Cindy Matthews—were enthusiastic about this project from the start, and they provided many wonderful suggestions and a great deal of insight into the teaching process. Somehow, they managed to remain upbeat through the difficult early trials of Summary Street and State the Essence. Eileen Kintsch provided invaluable organizational and logistic help. I dropped the ball on numerous occasions while running studies and analyzing data, and she was always there to pick it it up! She also provided help with editing and proofreading, two of many tasks at which she greatly excels. The CLIPR folks—Jon, Ernie, Linda, and Taki—all provided help with technical aspects of this dissertation, often at the 11th hour. They cheerfully provided solutions to many problems that had me in a panic. Chick Judd and Bill Oliver provided much-needed statistical help, and Corder Lewis coined the name “Summary Street.” I must also gratefully acknowledge Missy Schreiner for help and support above and beyond the call, and for writing a dissertation that was similar enough to mine that I could cite it! vii Most of all, I must acknowledge my wife, Eileen. She suffered through missed deadlines, rotten moods (mine, not hers), and many long hours of being apart, and still managed to love and support me in this endeavor. I would not have succeeded without her. In closing, I would also like to acknowledge the following foods for their help in preparing this dissertation: the unbeatable Butter Paneer from Taj Mahal III in Louisville, York Peppermint Patties, Cinnamon Altoids, and the malted milk balls sold in bulk at the Alferd Packer Grill in the University Memorial Center. viii TABLE OF CONTENTS CHAPTER 1: INTRODUCTION...........................................................................................1 Theoretical Motivation: Why Summarization?.......................................3 Latent Semantic Analysis: An Introduction............................................6 Pre-history of Summary Street: State the Essence..................................9 How Does Summary Street Work?.........................................................12 Summary Street in Action........................................................................15 The Evolution of Computer-Based Instruction.....................................20 The ITS Architecture and Summary Street............................................24 Comparison with other Intelligent Tutoring Systems.........................26 The Importance of Assessment ...............................................................27 CHAPTER 2: THE DETAILED INNER WORKINGS OF SUMMARY STREET.......................30 Introductory Page .....................................................................................32 Saving Changes/Checking Spelling.......................................................36 Feedback on My Summary......................................................................37 Threshold Computation...........................................................................39 Format for Printing ...................................................................................42 Redundancy Check...................................................................................44 Relevance Check........................................................................................47 CHAPTER 3: METHODS, RESULTS, AND POST HOC ANALYSES...................................49 Introduction ...............................................................................................49 Study 1: Ancient Civilizations.................................................................49 Study 2: The Human Circulatory System..............................................51 Study 3: Sources of Energy ......................................................................52 Time on task.............................................................................................54 ix Content versus quality scores..................................................................55 Text Difficulty .........................................................................................57 Transfer....................................................................................................61 Individual