A Prototype Reading Coach that Listens: Summary of Project LISTEN Alex Hauptmann, Jack Mostow, Steven F. Roth, Matthew Kane, and Adam Swift Project LISTEN 1, 215 Cyert Hall, 4910 Forbes Avenue Carnegie Mellon University Robotics Institute, Pittsburgh, PA 15213-3890

What: Project LISTEN is developing a novel weapon against REFERENCES illiteracy: an automated reading coach that displays a story on a computer screen, listens to a child read it aloud, and helps where 1. J. Mostow, S. Roth, A. Hauptmann,M. Kane, A. Swift, L. Chase, needed. The coach provides a combination of reading and listen- and B. Weide, "A Reading Coach that Listens (6-minutevideo)", ing, in which the child reads wherever possible, and the coach VMeo Track of the Twelfth National Conference on Artificial helps wherever necessary. We demonstrated a prototype of this Intelligence (AAA194), American Association for Artificial Intel- coach at the ARPA Workshop on Human Language Technology ligence, Seattle, WA, August 1994. in March 1994. A short video shows the coach in action [1]. 2. E. Herrick, "Literacy Questions and Answers." Pamphlet, Con- Who: The intended users of the coach include children in grades tact Center, Inc., 1990. 1-3, where oral reading is emphasized. Its developers include 3. National Center for Education Statistics, "Adult Literacy in experts on speech technology, reading, and human-computer in- America", Tech. report GPO 065-000-00588-3, U.S. Department teraction. Its testers include approximately 100 second graders in of Education,September 1993. Pittsburgh public schools who have difficulty in reading. 4. X. D. Huang, F. AUeva, H. W. Hon, M. Y. Hwang, K. F. Lee, and Why: Illiteracy costs the United States over $225 billion dollars R. Rosenfeld, "The SPHINX-II system: An annually in corporate retraining, lost competitiveness, and in- overview", Computer Speech and Language, Vol. 7, No. 2, April 1993, pp. 137-148. dustrial accidents [2]. Individuals with low reading proficiency are much likelier to be unemployed, poor, or incarcerated [3]. 5. J. Mostow, A. G. Hauptmann, L. L. Chase, and S. Roth, Automated literacy tutoring is an important, real, and challenging "Towards a Reading Coach that Listens: AutomatedDetection of task requiring multimodal interaction and real-time response. Oral Reading Errors", Proceedings of the Eleventh National Conference on (AAAI93), American As- How: Project LISTEN is made possible by years of government- sociation for Artificial Intelligence, Washington, DC, July 1993, funded research that produced CMU's Sphinx-I/ speech pp. 392-397. recognizer [4], which we adapted to detect errors in oral reading 6. A. G. Hauptmann, L. L. Chase, and J. Mostow, "Speech Recog- of known text [5, 6, 7]. We modelled the coach after expert nition Applied to Reading Assistance for Children: A Baseline reading teachers and refined it based on experimental use [8]. Language Model", Proceedings of the 3rd European Conference on Speech Communication and Technology (EUROSPEECH93), Evaluation: Pilot experiments reported in [7] tested both the Berlin, September 1993, pp. 2255-2258. coach's accuracy in detecting reading errors, and its potential effectiveness in helping children read. 7. J. Mostow, S. Roth, A. G. Hauptmann, and M. Kane, "A Prototy~ Reading Coach that Listens", Proceedings of the Future work: Our primary goal is to extend and deploy the Twelfth National Conference on Artificial Intelligence (AAAI94), coach in order to help children read better over time. Possible American Association for Artificial Intelligence, Seattle, WA, spinoff applications include adult literacy, English as a second August 1994. language, foreign language learning, interactive entertainment, 8. J. Mostow, S. Roth, A. Hauptmann,M. Kane, A. Swift, L. Chase, and computer-assisted writing. We envision a new generation of and B. Weide, "Getting Computers to Listen to ChildrenRead: A intelligent tutoring systems that listen to their students, providing New Way to Combat Illiteracy (7-minute video)", Overview and individualized attention that conventional classrooms cannot. research methodologyof Project LISTEN as of July 1993.

ffThis research was supported primarily by the National Science Foundation under Grant Number MDR-9154059 and by the Advanced Research Projects Agency, DoD, through DARPA Order 5167, monitored by the Air Force Avionics Laboratory under contract N00039-85-C-0163. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsors or of the United States Government.

We thank our principal reading consultant Leslie Thyberg; and the rest of the CMU Speech Group (FiUeno AUeva, Bob Brennan, Lin Chase, Xuedong Huang, Mei-Yuh Hwang, Sunil Issar, Fu-hua Liu, Chenxiang Lu, Pedro Moreno, Ravi Mosur, Yoshiaki Ohshima, Paul Placeway, Roni Rosenfeld, Alex Rudnicky, Matt Siegler, Rich Stern, Eric Thayer, Wayne Ward, and Bob Weide) for Sphinx-H; Paige Angstadt, Morgan Hankins, and Cindy Neelan for transcription; Maxine Eskenazi for transcript analysis; Lee Ann Kane for her voice; Jim Kocher for video production; CTB Macmillan/McGraw-Hill for permission to use copyrighted reading materials from George Spache's Diagnostic Reading Scales; the students and educators at Colfax Elementary School, East Hills Elementary School, Turner School, and Winchester Thurston School for participating in our experiments; and many friends for their assistance.

237