National Assessment Governing Board Developing Achievement Levels on the National Assessment of Educational Progress in Writing Grades 8 and 12 in 2011 Submitted to: National Assessment Governing Board 800 North Capitol Street, NW, Suite 825 Washington, DC 02002-4233 Phone: 202.357.6938 This study was funded by the National Assessment Governing Board under Contract ED-NAG-10-C-0003. Submitted by: Measured Progress 100 Education Way Dover, NH 03820 Phone: 603.749.9102 ALS Writing Process Report September 2012 PANELIST NAMES REDACTED BY GOVERNING BOARD Developing Achievement Levels on the 2011 National Assessment of Educational Progress in Grades 8 and 12 Writing Process Report Luz Bay with Chris Clough Jennifer Dunn Wonsuk Kim Leah McGuire Tia Sukin September 2012 ALS Writing Process Report Measured Progress ii National Assessment Governing Board BOARD MEMBERSHIP (2011–2012) Honorable David P. Driscoll, Chair Former Commissioner of Education Melrose, Massachusetts Mary Frances Taymans, SND, Vice Chair Sisters of Notre Dame National Education Office Bethesda, Maryland Andrés Alonso Shannon Garrison Chief Executive Officer Fourth-Grade Teacher Baltimore City Public Schools Solano Avenue Elementary School Baltimore, Maryland Los Angeles, California David J. Alukonis Doris R. Hicks Former Chairman Principal and Chief Executive Officer Hudson School Board Dr. Martin Luther King, Jr. Charter School Hudson, New Hampshire for Science and Technology New Orleans, Louisiana Louis M. Fabrizio Data, Research and Federal Policy Director Honorable Terry Holliday North Carolina Department of Public Commissioner of Education Instruction Kentucky Department of Education Raleigh, North Carolina Lexington, Kentucky Honorable Anitere Flores Richard Brent Houston Senator Principal Florida State Senate Shawnee Middle School Miami, Florida Shawnee, Oklahoma Alan J. Friedman Hector Ibarra Consultant Eighth-Grade Teacher Museum Development and Science Belin Blank International Center and Talent Communication Development New York, New York Iowa City, Iowa ALS Writing Process Report Measured Progress iii National Assessment Governing Board BOARD MEMBERSHIP (2011–2012) Honorable Tom Luna B. Fielding Rolston Idaho Superintendent of Public Instruction Chairman Boise, Idaho Tennessee State Board of Education Kingsport, Tennessee Honorable Jack Markell Governor of Delaware Cary Sneider Wilmington, Delaware Associate Research Professor Portland State University Tonya Miles Portland, Oregon General Public Representative Mitchellville, Maryland Blair Taylor President and Chief Executive Officer Dale Nowlin Los Angeles Urban League Twelfth-Grade Teacher Los Angeles, California Columbus North High School Columbus, Indiana Honorable Leticia Van de Putte Senator Honorable Sonny Perdue Texas State Senate Former Governor of Georgia San Antonio, Texas Atlanta, Georgia Eileen L. Weiser Susan Pimentel General Public Representative Educational Consultant Ann Arbor, Michigan Hanover, New Hampshire Ex-officio Member W. James Popham John Q. Easton Professor Emeritus Director University of California, Los Angeles Institute of Education Sciences Wilsonville, Oregon U.S. Department of Education Washington, DC Andrew C. Porter Dean Graduate School of Education University of Pennsylvania Philadelphia, Pennsylvania ALS Writing Process Report Measured Progress iv Table of Contents Executive Summary ................................................................................................................................. xii Chapter 1— Introduction ............................................................................................................................ 1 1.1 The Governing Board and the NAEP Achievement Levels for Writing ........................................... 2 1.2 Purpose of This Document .............................................................................................................. 5 1.3 Organization of This Document ...................................................................................................... 6 Chapter 2— Project Overview .................................................................................................................... 8 2.1 Technical Advisory Committee on Standard Setting ...................................................................... 8 2.2 Technical Assistance from the NAEP Alliance .............................................................................. 10 2.3 Project Staff .................................................................................................................................... 11 2.4 Achievement Levels–Setting Staff ................................................................................................. 12 2.4.1 Process Facilitators ................................................................................................................. 13 2.4.2 Content Facilitators ................................................................................................................ 14 2.5 Panelist Nomination and Selection ................................................................................................ 14 2.5.1 Multistage Selection Process ................................................................................................... 15 2.5.1.1 Stage 1: Select Districts and Identify Nominators ......................................................... 15 2.5.1.2 Stage 2: Contact Nominators and Request Nominations ............................................ 20 2.5.1.3 Stage 3: Notify Nominees and Request Acceptance of Nomination .............................21 2.5.1.4 Stage 4: Select and Recruit Panelists ............................................................................ 25 2.5.2 Recruitment for Field Trial.................................................................................................... 30 2.5.3 Recruitment for Field Trial 2 ................................................................................................. 32 2.6 Advanced Materials ....................................................................................................................... 32 2.6.1 Advanced Materials for Pilot Study ....................................................................................... 33 2.6.2 Advanced Materials for Field Trial 2 ..................................................................................... 34 2.6.3 Advanced Materials for Operational ALS Meeting ............................................................... 34 2.7 Incentives and Reimbursements ................................................................................................... 36 2.7.1 Incentives and Reimbursements for Pilot Study ................................................................... 37 2.7.2 Incentives and Reimbursements for Field Trial 2 ................................................................. 37 2.7.3 Incentives and Reimbursements for Operational ALS Meeting ........................................... 38 2.8 Registration ................................................................................................................................... 38 2.8.1 Registration for the Pilot Study ............................................................................................. 39 2.8.2 Registration for Field Trial 2 ................................................................................................. 39 2.8.3 Registration for Operational ALS Meeting ........................................................................... 39 2.9 Achievement Levels–Setting Process ............................................................................................ 40 2.9.1 Development of Achievement Levels Descriptions ............................................................... 40 2.9.2 Body of Work Method ............................................................................................................ 48 2.9.3 Body of Work Technological Integration and Enhancements .............................................. 52 2.9.4 Replicate Panels ..................................................................................................................... 56 2.9.5 Task Pool Division ................................................................................................................. 60 2.9.6 Pseudo-NAEP Scales ............................................................................................................. 63 2.9.7 Facilitator Training ................................................................................................................ 63 2.9.8 Panelist Training.................................................................................................................... 67 2.9.8.1 General Orientation ....................................................................................................... 68 2.9.8.2 Taking a NAEP Exam .................................................................................................... 68 ALS Writing Process Report Measured Progress v 2.9.8.3 Review of Writing Tasks and Scoring Rubrics ............................................................ 70 2.9.8.4 Orientation to the Method .............................................................................................. 71 2.9.8.5 Presentation of NAEP Writing Framework and Achievement Levels Descriptions .... 71 2.9.8.6 Achievement Level Training by Response Classification Exercise ...............................72