DOCSLIB.ORG

A Rhetorical Approach to Examining Writing Assessment Validity Claims

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Rhetorical Approach to Examining Writing Assessment Validity Claims A RHETORICAL APPROACH TO EXAMINING WRITING ASSESSMENT VALIDITY CLAIMS A dissertation submitted To Kent State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy by William Alan Morris May 2021 © Copyright All rights reserved Except for previously published materials Dissertation written by William Alan Morris B.A. Humboldt State University 2006 M.A. Humboldt State University 2009 Ph.D. Kent State University 2021 Approved by _______________________________, Chair, Doctoral Dissertation Committee Dr. Brian Huot _______________________________, Members, Doctoral Dissertation Committee Dr. Pamela Takayoshi _______________________________ Dr. Sara Newman _______________________________ Dr. Lucretia (Penny) Pence _______________________________ Dr. Brian Baer Accepted by ______________________________, Chair, Department of English Dr. Babacar M’baye ______________________________, Interim Dean, College of Arts and Sciences Dr. Mandy Munor-Stasiuk TABLE OF CONTENTS ............................................................................................................. iii LIST OF FIGURES ....................................................................................................................... vi ACKNOWLEDGEMENTS ......................................................................................................... vii Chapter 1 ......................................................................................................................................... 1 Exactly What is Validity? ........................................................................................................... 4 Antecedents for Rhetorical Reparation for Writing Assessment Theory ................................... 8 Educational Measurement Method for Validity Arguments ..................................................... 13 Topoi, Argument and Research Questions ............................................................................... 17 Methodology to Inform My Method for Writing Assessment Validity .................................... 19 Significance of the Issue ........................................................................................................... 22 Concluding Remarks ................................................................................................................. 24 Chapter 2 ....................................................................................................................................... 26 Overview of the Topoi .............................................................................................................. 28 Definitional Accounts as Tools to Discover Discrepancies ...................................................... 34 Classic Validity: A Statistical Character in the Test ................................................................. 38 Validity in Three Parts .............................................................................................................. 46 Validity as Argument ................................................................................................................ 49 Two Step Validity: Interpretive and Analytic Argumentation ................................................. 55 Critiques of Toulmin Model of Argument ................................................................................ 60 iii Concluding Remarks ................................................................................................................. 65 Chapter 3 ....................................................................................................................................... 67 The Documents and Their Contexts ......................................................................................... 68 The DSP Assessment Reports ................................................................................................... 71 From Full Text to Toulminian Representation ......................................................................... 72 Interpretive Arguments ............................................................................................................. 75 First Category ....................................................................................................................... 77 Second Category ................................................................................................................... 84 Third Category ...................................................................................................................... 94 Fourth Category.................................................................................................................. 102 Fifth Category ..................................................................................................................... 107 Validity Arguments ................................................................................................................. 112 Second Category ................................................................................................................. 117 Third Category .................................................................................................................... 121 Fourth Category.................................................................................................................. 124 Fifth Category ..................................................................................................................... 128 Summary of Arguments .......................................................................................................... 130 Chapter 4 ..................................................................................................................................... 131 Towards a Topical Approach to Making Validity Arguments ............................................... 131 A Method for Identifying the Topoi ....................................................................................... 134 iv Topical Analysis of NIIA 3a ................................................................................................... 136 NIIA 3a Warrant as If from the Consequence .................................................................... 138 NII 3a Warrant as If from Contrasting Opposites .............................................................. 141 Concluding Remarks about the Topoi for NIIA 3a ............................................................. 146 Topical analysis of NIIA 3b .................................................................................................... 146 NIIA 3b as If from Correlatives .......................................................................................... 148 NIIA 3b as If Identifying Purpose with Cause .................................................................... 151 Summary of Chapter ............................................................................................................... 153 Chapter 5 ..................................................................................................................................... 155 The Larger Upshot of My Work ............................................................................................. 160 Limitations .............................................................................................................................. 162 Future Possibilities .................................................................................................................. 163 References ................................................................................................................................... 166 v List of Figures Figure 1. Toulmin’s Model of Argument ..................................................................................... 14 Figure 2. Mislevy’s Toulmin Model ............................................................................................. 15 Figure 3. My Toulminian Model .................................................................................................. 73 Figure 4. NIIA 1a .......................................................................................................................... 77 Figure 5. NIIA 1b .......................................................................................................................... 78 Figure 6. NIIA 1c .......................................................................................................................... 79 Figure 7. NIIA 2a .......................................................................................................................... 85 Figure 8. NIIA 2b .......................................................................................................................... 86 Figure 9. NIIA 2c .......................................................................................................................... 87 Figure 10. NIIA 3a ........................................................................................................................ 95 Figure 11. NIIA 3b ........................................................................................................................ 96 Figure 12. NIIA 4 ........................................................................................................................ 103 Figure 13. NIIA 5 .......................................................................................................................
Load more

Recommended publications
  • Taking Tests (ACT Version)
    UNIT 2 TAKING TESTS (ACT version) Lesson Descriptions Taking Tests 1: Introduction to ACT/SAT (ACT Version) Why should I take a college entrance exam (ACT or SAT), and how can I prepare? NOTE: This lesson requires coordination with your school counselor. It also requires making arrangements to use the computer lab. Taking Tests 2: ACT Practice Questions What kinds of questions will be on the ACT? Taking Tests 3: ACT Registration How do I register for the ACT? What do I need to know for the test day? NOTE: This lesson requires coordination with your school counselor. It also requires making arrangements to use the computer lab. Taking Tests 4: Workforce and Adult Education Tests What kinds of tests might I face when applying for jobs, the military, or vocational training? © 2010 Roads to Success. For information on re-use under our Creative Commons Attribution Non-Commercial Share Alike license, visit www.roadstosuccess.org. 119 PLANNING PYRAMID GRADE 11, Unit 2, Taking Tests (ACT version) Some Students Will: • Prepare to take the ACT using resources identified in class. Most Students Will: • Identify strategies for approaching ACT questions. • Register to take the ACT. • Decide whether or not to take the writing portion of the ACT. • Understand the mechanics of preparing for test day (For example, make sure your calculator has fresh batteries.) and know what to do after they arrive at the testing site. All Students Will: • Understand the purpose of the ACT. • Understand the types of questions included on the ACT. • Recognize that since there is no penalty for guessing on the ACT, it’s in their best interest to answer every question.
    [Show full text]
  • University of Nevada, Reno the Rhetoric* of Writing Assessment A
    University of Nevada, Reno The Rhetoric* of Writing Assessment A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in English by Katrina Love Miller Dr. Catherine Chaput / Dissertation Advisor May, 2016 Copyright by Katrina Love Miller 2016 All Rights Reserved THE GRADUATE SCHOOL We recommend that the dissertation prepared under our supervision by KATRINA LOVE MILLER Entitled The Rhetoric* of Writing Assessment be accepted in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Dr. Catherine Chaput, Advisor Dr. Norbert Elliot, Committee Member Dr. James Webber., Committee Member Dr. William J. Macauley, Jr, Committee Member Dr. Dianna Townsend, Committee Member Dr. Amy Pason, Graduate School Representative David W. Zeh, Ph. D., Dean, Graduate School May, 2016 i Abstract The Rhetoric* of Writing Assessment uses strategic variations of rhetorical theory to reimagine writing assessment’s complicated disciplinary past and rethink its foundational concepts through its multifaceted rhetorical situatedness. Although scholars such as Bob Broad, Brian Huot, Patricia Lynne, Norbert Elliot, Edward M. White, and Kathleen Blake Yancey have long understood assessment as a rhetorically rich concept, this dissertation extends previous conceptions of writing assessment as a rhetorical act by adopting a generative theoretical framework that approaches writing assessment as a dynamic and relational rhetorical process influenced by a whole host of suasive forces. Situated at the theoretical crossroads of rhetoric and writing studies, this study relies on generative use of rhetorical theory and interpretive forecasting to frame the integrated analyses in each body chapter. In examining the rhetorical history of writing assessment as a field of study, the rhetorical circulation of writing assessment discourse, and the rhetorical relations between individuals and institutions, it articulates the complex rhetoricity of writing assessment as a theoretical construct and field of study.
    [Show full text]
  • Standards for the Assessment of Reading and Writing
    ST A he International Reading Association is a nonprofit, ND Tglobal network of individuals and institutions AR committed to worldwide literacy. The Association provides D a wide range of resources, advocacy efforts, volunteerism, S Revised Edition for the and professional development activities to support literacy professionals as they work to improve the quality of reading Assessment instruction, disseminate research and information about reading, and encourage the lifetime reading habit. Learn STANDARDS more at www.reading.org. for the of Reading Assessment he National Council of Teachers of English (NCTE) of is dedicated to improving the teaching and learning Reading T of English and the language arts at all levels of education. and Writing Since 1911, NCTE has provided a forum for the profession, and an array of opportunities for teachers to continue their Writing professional growth throughout their careers, and a Revised Edition Revised framework for cooperation to deal with issues that affect the teaching of English. For more information, please visit www.ncte.org. ISBN 978-0-87207-776-8 www.reading.org 90000 Prepared by Book Club selection December 2009 the Joint Task Force on Assessment of the International Reading Association 9 780872 077768 www.ncte.org and the National Council of Teachers of English INTERNAT I ONAL Reading Association ® 800 BARKSDALE ROAD , PO BOX 8139 NEWARK , DE 19714-8139, USA www.reading.org IRA BOARD OF DIRECTORS Kathryn Au, SchoolRise LLC, Honolulu, Hawaii, President • Patricia A. Edwards, Michigan State University, East Lansing, Michigan, President-elect • Victoria J. Risko, Peabody College of Vanderbilt University, Nashville, Tennessee, Vice President • Donald J.
    [Show full text]
  • What Teachers Say About Different Kinds of Mandated State Writing Tests
    81 Journal of Writing Assessment Vol. 2, No. 2, pp. 81-108 © Hampton Press Inc All rights of reproduction in any form reserved What Teachers Say About Different Kinds of Mandated State Writing Tests PEGGY O’NEILL Loyola College in Maryland SANDRA MURPHY University of California-Davis BRIAN HUOT Kent State University MICHAEL M. WILLIAMSON Indiana University of Pennsylvania This article reports on a study of the impact of the form of high-stakes, state-man- dated writing tests on high school curricula and teaching practices. Through surveys and focus group interviews of high school English teachers, we addressed two main PeggyO‘Neill is associate professor of writing, directs the composition program, and teaches writing in the Department of Writing at Loyola College in Maryland. Her scholarship focuses on writing assessment, pedagogy, and writing program administration and disciplinarity. She is cur- rently at work, with Brian Huot and Cindy Moore, on the Handbook of College Writing Assessment. Sandra Murphy is a professor who teaches graduate-level courses on research on writing, read- ing, and assessment in the School of Education at the University of California-Davis. She has writ- ten several articles on the teaching and learning of writing and reading and has co-authored sev- eral books on writing assessment. Her research interests include writing assessment and its impact on teachers and curriculum, reading comprehension, and critical perspectives on literacy. Brian Huot is professor of English and Writing Program Coordinator at Kent State University. His work on writing assessment has appeared in various articles, book chapters, and in the monograph (Re)Articulatng Writing Assessment for Teaching and Learning.
    [Show full text]
  • Rhetorical Writing Assessment the Practice and Theory of Complementarity
    7 Journal of Writing Assessment Vol. 2, No. 1, pp. 7-20 © Hampton Press Inc All rights of reproduction in any form reserved Rhetorical Writing Assessment The Practice and Theory of Complementarity BOB BROAD Illinois State University MICHAEL BOYD Morton High School, Morton, IL Writing portfolio assessment and communal (shared, dialogical) assessment are two of our field’s most creative, courageous, and influential innovations. Because they are also relatively expensive innovations, however, they remain vulnerable to cost-cut- ting by university administrators and to attacks from testing corporations. This arti- cle lays a theoretical foundation for those two powerful and valuable practices in teaching and assessing writing. Building on the concept of “complementarity” as Bob Broad is associate professor of English at Illinois State University, where he teaches gradu- ate courses in writing assessment and undergraduate courses in composition pedagogy. In 2003, Utah University Press published his book What We Really Value: Beyond Rubrics in Teaching and Assessing Writing. Bob’s articles have appeared in the journals Research in the Teaching of English, Assessing Writing, Works and Days, African American Review, and Illinois English Bulletin.He con- tributed a chapter to a forthcoming book collection on computerized writing assessment as well as to Preparing a Nation’s Teachers: Model Programs for English and Foreign Languages (MLA, 1999) and New Directions in Portfolio Assessment (Boynton/Cook, 1994). Michael Boyd teaches composition and literature at Morton High School in central Illinois. He recently completed his dissertation, Discourse Community Pedagogy: Opening Doors for Students of Composition, and received his PhD in English studies from Illinois State University.
    [Show full text]
  • Writing Skill Assessment: Problems and Prospects. Educational Testing
    DOCUMENT RESUME ED 401 317, TM 025 888 AUTHOR Breland, Hunter M. TITLE Writing Skill Assessment: Problems and Prospects. Policy Issue Perspective Series. INSTITUTION Educational Testing Service, Princeton, NJ. Policy Information Center. PUB DATE Apr 96 NOTE 29p. PUB TYPE Reports Evaluative/Feasibility (142) EDRS PRICE MF01/PCO2 Plus Postage. DESCRIPTORS College Entrance Examinations; Computer Uses in Education; Cost Effectiveness; *Educational Technology; *Essay Tests; High Schools; *Performance Based Assessment; Prediction; *Test Construction; Test Reliability; Test Use; Test Validity; Writing Evaluation; *Writing Skills; Writing Tests IDENTIFIERS *Authentic Assessment; *Free Response Test Items ABSTRACT Recent trends in writing skill assessment suggest a movement toward the use of free response writing tasks and away from the traditional multiple-choice test. A number of national examinations, including major college admissions tests, have included free-response components. Most of the arguments in support of this trend relate to the hypothesized effects of testing on curriculum and instruction, but others center around systemic validity and authenticity. There are questions in these areas, however, beginning with the question of what the content of a writing assessment should be. The reliability of free-response writing tests is often reported in terms of interrater reliability, but correlations of scores assigned by different raters can inflate the estimate of reliability. Combining assessment types, essay and multiple choice, is a way to improve reliability that is proving workable. The predictive effectiveness of writing skill assessments is related to reliability. Issues of fairness, comparability, cognitive complexity, and cost and efficiency must be addressed in the construction of free-response writing skill assessments.
    [Show full text]
  • Writing Assessment in Admissions to Higher Education
    College Board Report No. 99-3 GRE Board Research Report No. 96-12R Writing Assessment in Admission to Higher Education: Review and Framework HUNTER M. BRELAND, BRENT BRIDGEMAN, and MARY E. FOWLES College Board Report No. 99-3 GRE Board Research Report No. 96-12R ETS RR No. 99-3 Writing Assessment in Admission to Higher Education: Review and Framework HUNTER M. BRELAND, BRENT BRIDGEMAN, and MARY E. FOWLES This report presents the findings of a research project funded by and carried out under the auspices of the College Board and the Graduate Record Examinations Board. College Entrance Examination Board, New York, 1999 Acknowledgments The authors thank Peter Cooper, Claudia Gentile, Donald Powers, and Lawrence Stricker for helpful comments on an earlier draft of this report. Hunter M. Breland is senior research scientist at ETS. Brent Bridgeman is principal research scientist at ETS. Mary E. Fowles is assessment specialist II at ETS. Researchers are encouraged to freely express their professional judgment. Therefore, points of view or opinions stated in Copyright © 1999 by College Entrance Examination Board College Board Reports or Graduate Record Examinations and Educational Testing Service. All rights reserved. College Board Reports do not necessarily represent official College Board, Advanced Placement Program, AP, CLEP, SAT, and Board or Graduate Record Examinations Board position the acorn logo are registered trademarks of the College or policy. Entrance Examination Board. Educational Testing Service, ETS, the ETS logo, Graduate Record Examinations, GRE, The PRAXIS Series: Professional Assessments for Beginning Founded in 1900, the College Board is a not-for-profit Teachers, TOEFL, and TWE are registered trademarks of educational association that supports academic preparation and Educational Testing Service.
    [Show full text]
  • Rethinking K-12 Writing Assessment to Support Best Instructional Practices
    CHAPTER 5. RETHINKING K-12 WRITING ASSESSMENT TO SUPPORT BEST INSTRUCTIONAL PRACTICES Paul Deane, John Sabatini, and Mary Fowles Educational Testing Service The work described in this chapter arises in a specific current United States context: one in which there is tension between best practices in writing instruc- tion and standard approaches to writing assessment, particularly in the context of standardized tests, such as state accountability examinations. On the one hand, the instructional literature emphasizes the importance of teaching de- liberate, well-developed writing processes; indicates the value of peer review; and strongly supports an instructional approach in which explicit strategies are taught in meaningful contexts where content matters (Perin, 2009). On the other hand, the requirements of standardized testing often favor creation of relatively decontextualized writing tasks in which students produce essays under timed conditions with little access to external sources of information—a state of affairs that may have deleterious effects on writing instruction, since the pre- dominance of high-stakes assessments often forces instructors to focus on test preparation (Hillocks, 2002, 2008). Not surprisingly, current reform efforts such as the Race to the Top pro- gram partially conceive of assessments as interventions, and various scholars have argued that such reforms should be guided by modern theories of learning and cognition (Pellegrino, 2009). This point has been advanced in particular by Bennett and Gitomer (2009), who propose a research program that they term CBAL (Cognitively-Based Assessment of, for and as Learning). The goal of this research program is to conceptualize and try out the components of an integrated system in which summative assessments, formative assessments, and teacher professional support combine to encourage and enhance effective teach- ing and learning.
    [Show full text]
  • Learning from Writing Center Assessment: Regular Use Can Mitigate Students’ Challenges
    Learning from Writing Center Assessment: Regular Use Can Mitigate Students’ Challenges Carol Trosset Kathy Evertz Renata Fitzpatrick Carleton College Author Note Carol Trosset, Institutional Research and Assessment, Carleton College; Kathy Evertz, Academic Support Center, Carleton College; Renata Fitzpatrick, Academic Support Center, Carleton College. Carol Trosset is now an Independent Consultant. Correspondence concerning this article should be addressed to trosset.edresearch@ gmail.com. Abstract To demonstrate effectiveness, writing centers collect quantitative and/or qualitative information about and from the students who use their services. A broader understanding of effectiveness requires us to consider both direct measures of writing quality and why some students do not use the writing center. This mixed-methods research followed one entry cohort for two years and found that regular use of our Writing Center was correlated with higher grades in writing- rich courses, regardless of student background. Findings regarding one-time visitors indicate that adopting more flexible pedagogies is key in encouraging them to return. Keywords: writing centers, assessment, administrators, learning centers, tutor training, learning assistance, consultant preparation 30 | TLAR, Volume 24, Number 2 Introduction Like most writing centers, Carleton College’s collects a host of data from our visitors before or at the start of every conference: class year, major (if they have one), the kind of writing project they’re working on, the class for which they’re writing, the writer’s hoped- for focus of the session, the date when the final draft is due, etc. We also ask writers to complete a short post-conference questionnaire that poses two questions: “What did you learn today?” and “Would you return to work with the same consultant (and why or why not)?” Writers deposit their forms in a locked box, our office assistant records the comments on an Excel spreadsheet, and we return the feedback, without the writer’s name, to the consultant.
    [Show full text]
  • Testing, Assessment, and the Teaching of Writing Gregory Shafer Mott Community College
    Language Arts Journal of Michigan Volume 31 | Issue 1 Article 7 2015 Testing, Assessment, and the Teaching of Writing Gregory Shafer Mott Community College Follow this and additional works at: https://scholarworks.gvsu.edu/lajm Recommended Citation Shafer, Gregory (2015) "Testing, Assessment, and the Teaching of Writing," Language Arts Journal of Michigan: Vol. 31: Iss. 1, Article 7. Available at: https://doi.org/10.9707/2168-149X.2094 This Article is brought to you for free and open access by ScholarWorks@GVSU. It has been accepted for inclusion in Language Arts Journal of Michigan by an authorized editor of ScholarWorks@GVSU. For more information, please contact [email protected]. CRITICAL PEDAGOGY Testing, Assessment, and the Teaching of Writing GREGORY SHAFER “As standardized testing has swallowed up public education in the U.S. in the A Quick History of the Fight twenty-first century, its ravenous hunger intensifying yearly since the federal man- for Composition Classrooms date inaugurated by President Bush’s No Child Left Behind and perpetuated by President Obama’s Race to the Top, students have largely become test takers.” In the 1980s, when I was in col- —Robert Hatch lege and learning to teach English, the goal was to transcend narrow tests, to omewhere in our journey to elevate the progress and educational move beyond the constricting peda- achievement of our most challenged students, many of us who teach gogy that centered on numbers and composition have watched with consternation as our most revered theo- multiple choice exams. It was a time of ries of writing have melted away into yet another reason for testing and exciting research, much of it galvanized excessive accountability.
    [Show full text]
  • New SAT Writing Prompt Study
    Research Report No. 2004-1 New SAT® Writing Prompt Study: Analyses of Group Impact and Reliability Hunter Breland Melvin Kubota Kristine Nickerson Catherine Trapani Michael Walker College Board Research Report No. 2004-1 ETS RR-04-03 New SAT® Writing Prompt Study: Analyses of Group Impact and Reliability Hunter Breland, Melvin Kubota, Kristine Nickerson, Catherine Trapani, and Michael Walker College Entrance Examination Board, New York, 2004 Hunter Breland is a principal research scientist at Acknowledgments HunterBreland.com. Melvin Kubota is an assessment specialist III at This project was funded by the College Board. In addi- Educational Testing Service. tion to the authors, a number of staff members at Kristine Nickerson is a program administrator at Educational Testing Service assisted with the data col- Educational Testing Service. lection, administration, and report preparation. Betty Catherine Trapani is an associate research data analyst Lou Estelle, Amy Dabbs, and Regina Mercadante pro- at Educational Testing Service. vided administrative assistance; Jennifer Zanna and Michael Walker is a measurement statistician–lead at Patti Baron conducted statistical analyses associated Educational Testing Service. with creating the data file; Nancy Glazer helped prepare the new persuasive prompts and select the training Researchers are encouraged to freely express their papers for the essay reading, Barbara Hames provided professional judgment. Therefore, points of view or editorial assistance, and Christin Noble created the test opinions stated in College Board Reports do not necessarily booklet design. Rose Ann Morgan and Marilyn Collins represent official College Board position or policy. led the essay reading. Yong-Won Lee provided useful statistical advice. Helpful reviews of report drafts were provided by Brent Bridgeman, Ida Lawrence, Yong- The College Board: Expanding College Opportunity Won Lee, Larry Stricker, and Dan Eignor.
    [Show full text]
  • Best Practices in Writing Assessment for Instruction Robert C
    Chapman University Chapman University Digital Commons Education Faculty Books and Book Chapters College of Educational Studies 2013 Best Practices in Writing Assessment for Instruction Robert C. Calfee University of California - Riverside Roxanne Greitz Miller Chapman University, [email protected] Follow this and additional works at: http://digitalcommons.chapman.edu/education_books Part of the Educational Assessment, Evaluation, and Research Commons Recommended Citation Calfee, R. C., & Miller, R. G. (2013). Best practices in writing assessment for instruction. In S. Graham, C. MacArthur, and J. Fitzgerald (Eds.), Best practices in writing instruction (2nd Edition) (pp. 351-380). New York: Guilford. This Book is brought to you for free and open access by the College of Educational Studies at Chapman University Digital Commons. It has been accepted for inclusion in Education Faculty Books and Book Chapters by an authorized administrator of Chapman University Digital Commons. For more information, please contact [email protected]. Best Practices in Writing Assessment for Instruction ROBERT C. CALFEE ROXANNE GREITZ MILLER When we prepared this chapter several years ago for the first edition of Best Practices, we felt that we were pushing the envelope. We proposed (1) that teachers give high priority to writing as an essential part of a balanced literacy program, (2) that writing be linked to the content areas to "make thinking visible" (Miller & Calfee, 2004; Richart, Church, & Morrison, 2011), and (3) that teachers develop classroom-based writing assessments tailored to their specific settings, their students, and their own learning priorities. We set the stage with three portraits drawn from observations of excellent teachers, illustrating the flow of formative writing assessments across the developmental span from kindergarten to high school.
    [Show full text]