Research Report Cognitive Models of Writing: Writing Proficiency as a Complex Integrated Skill Paul Deane Nora Odendahl Thomas Quinlan Mary Fowles Cyndi Welsh Jennifer Bivens-Tatum October 2008 ETS RR-08-55 Listening. Learning. Leading.® Cognitive Models of Writing: Writing Proficiency as a Complex Integrated Skill Paul Deane, Nora Odendahl, Thomas Quinlan, Mary Fowles, Cyndi Welsh, and Jennifer Bivens-Tatum ETS, Princeton, NJ October 2008 As part of its nonprofit mission, ETS conducts and disseminates the results of research to advance quality and equity in education and assessment for the benefit of ETS’s constituents and the field. ETS Research Reports provide preliminary and limited dissemination of ETS research prior to publication. To obtain a PDF or a print copy of a report, please visit: http://www.ets.org/research/contact.html Copyright © 2008 by Educational Testing Service. All rights reserved. E-RATER, ETS, the ETS logo, LISTENING. LEARNING. LEADING., GRADUATE RECORD EXAMINATIONS, GRE, and TOEFL are registered trademarks of Educational Testing Service (ETS). TEST OF ENGLISH AS A FOREIGN LANGUAGE is a trademark of ETS. Abstract This paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. The model developed is part of the Cognitively Based Assessments of, for, and as Learning (CBAL) initiative, an ongoing research project at ETS intended to develop a new form of kindergarten through Grade 12 (K–12) assessment that is based on modern cognitive understandings; built around integrated, foundational, constructed- response tasks that are equally useful for assessment and for instruction; and structured to allow multiple measurements over the course of the school year. The model that emerges from a review of the literature on writing places a strong emphasis on writing as an integrated, socially situated skill that cannot be assessed properly without taking into account the fact that most writing tasks involve management of a complex array of skills over the course of a writing project, including language and literacy skills, document-creation and document-management skills, and critical-thinking skills. As such, the model makes strong connections with emerging conceptions of reading and literacy, suggesting an assessment approach in which writing is viewed as calling upon a broader construct than is usually tested in assessments that focus on relatively simple, on-demand writing tasks. Key words: Formative assessment, writing instruction, literacy, critical thinking, reading, K-12, literature review, constructed-response i Table of Contents Page Modeling the Cognitive Basis for Writing Skill............................................................................. 1 Introduction.............................................................................................................................. 1 An Intersection of Fields ......................................................................................................... 2 Cognitive Models of Writing................................................................................................... 3 Transcription Automaticity .............................................................................................. 8 Reading Automaticity....................................................................................................... 8 Strategies to Manage the Writing Process........................................................................ 9 Planning................................................................................................................... 10 Revising................................................................................................................... 11 Writing as Social Cognition................................................................................................... 12 Differential Recruitment of Cognitive Abilities in Writing .................................................. 16 The Role of the Audience............................................................................................... 17 Organization, Domain Knowledge, and Working Memory ........................................... 19 Mastery of Textual Cues and Other Genre Conventions................................................ 21 Critical Thinking and Reasoning.................................................................................... 22 Planning and Rhetorical Structure.................................................................................. 24 Document-Structure Plans, Rhetorical Structure, and Discourse Cues................... 26 The Translation Process: Differences Across Types of Writing............................. 28 Revision and Reflection: Critical Thinking and Reading in Different Types of Writing ..................................................................................................... 30 Prospectus.............................................................................................................................. 31 Writing Proficiency, Writing Instruction, and the Characterization of Writing Skill .................. 31 Cognitive Models and Pedagogy........................................................................................... 31 The Transition From Novice to Skilled Writer...................................................................... 35 Interference Effects ........................................................................................................ 36 Lack of Strategic Writing Skills..................................................................................... 36 Insufficient Topic-Specific Knowledge ......................................................................... 37 Weak Content-Reasoning and Research Skills .............................................................. 37 Unformed or Rudimentary Rhetorical Goals ................................................................. 38 ii Schemas, Scaffolding, and Instruction .................................................................................. 39 Toulmin Logic................................................................................................................ 39 Scaffolding Through Argument Mapping...................................................................... 43 Scaffolding Expository Writing: Rhetorical Topoi, Concept Mapping, and Related Concepts.............................................................................................. 44 Towards an Inventory of Skills Differentially Involved in Different Types of Writing ....... 45 Background Knowledge ................................................................................................. 47 Background-Knowledge Skills Related to Exposition and Narrative..................... 48 Background-Knowledge Skills Related to Argumentation..................................... 50 Verbal-Reasoning Skills................................................................................................. 51 Reasoning Skills Connected to Narrative................................................................ 52 Reasoning Skills Connected to Exposition ............................................................. 52 Reasoning Skills Connected to Argumentation ...................................................... 53 Social and Evaluative Skills ........................................................................................... 55 Social and Evaluative Skills Relevant to Narrative ................................................ 55 Social and Evaluative Skills Relevant to Exposition .............................................. 55 Social and Evaluative Skills Relevant to Argumentation ....................................... 58 Linguistic and Rhetorical Skills ..................................................................................... 62 Building a New Kind of Writing Assessment: Design Consideration.......................................... 64 Breadth of Construct, Narrowness of Measurement.............................................................. 64 A Competency Model to Inform Assessment Design ........................................................... 67 Assessment Principles and Strategies: Outline of a Writing Framework.............................. 75 Task and Test Design ..................................................................................................... 75 Sampling and Aggregation of Information..................................................................... 78 Scoring Methods Based on the CBAL Competency Model........................................... 79 Prospects for Automated Scoring................................................................................... 82 A Comparison to Other Writing-Assessment Frameworks................................................... 83 General Frameworks ...................................................................................................... 83 Certain Other Writing Assessments ............................................................................... 90 Conclusions—and Future Directions............................................................................................ 94 iii References.............................................................................................................................. 97 Notes...................................................................................................................................