Making Sense of Test-Based Accountability in Education Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein, editors Prepared for the National Science Foundation The material in this book is based upon work supported by the National Science Foundation under Grant ESI-9978416. Library of Congress Cataloging-in-Publication Data Hamilton, Laura S. Making sense of test-based accountability in education / Laura S. Hamilton, Brian M. Stecher, Stephen P. Klein. p. cm. “MR-1554.” Includes bibliographical references. ISBN 0-8330-3161-9 1. Educational tests and measurements—United States. 2. Educational accountability—United States. I. Stecher, Brian M. II. Klein, Stephen P. III.Title. LB3051 .H3193 2002 371.26'4—dc21 2002069685 RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND® is a registered trademark. RAND’s publications do not necessarily reflect the opinions or policies of its research sponsors. Cover design by Eileen Delson La Russo © Copyright 2002 RAND All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND. Published 2002 by RAND 1700 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 201 North Craig Street, Suite 202, Pittsburgh, PA 15213 RAND URL: http://www.rand.org/ To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: [email protected] PREFACE Test-based accountability systems are based on the belief that public education can be improved through a simple strategy: require all students to take standardized achievement tests and attach high stakes to the tests in the form of rewards when test scores improve and sanctions when they do not. Test-based accountability has achieved broad support as a strategy for improving public education. Standardized achievement tests have been used to measure students’ educational progress for nearly a century, but the prevalence of tests, and the number of purposes they are being asked to serve, have grown substantially during the past two decades. In addition to the measurement function for which they were originally designed, large-scale achievement tests have be- come an essential component of efforts to reform education more broadly. Test-based accountability systems are in place in nearly ev- ery state, and advocates of these systems believe that the use of high- stakes tests will spur positive change in schools and classrooms. But the emphasis on test-based accountability raises a number of important questions: Do these high-stakes tests measure student achievement accurately? How can policymakers and educators select the right tests, evaluate the test scores correctly, and attach the right consequences to the results of these tests to make accountability systems work as intended? What are the costs of developing and administering these tests? And what kinds of trade-offs do these policies introduce? There is an extensive literature on the psychometric properties of achievement test scores and a much smaller, but still substantial, lit- iii iv Making Sense of Test-Based Accountability in Education erature on the ways in which tests affect students, teachers, and other stakeholders in the education system. Yet for practitioners and policymakers who are responsible for making decisions about as- sessment and accountability systems, it is often difficult to find accu- rate and accessible information to inform these decisions. This book is intended to address that need. With a grant from the National Science Foundation, RAND and other researchers have combined their knowledge to create this book, which should be of interest to practitioners, policymakers, and oth- ers who are involved in some way in test-based accountability sys- tems but who are not technical experts in measurement. This book provides an overview of the major issues faced by those who develop, implement, and use test-based accountability systems, including a summary of the key technical considerations and a brief description of what is known about the impact of these systems. This material is based upon work supported by the National Science Foundation. The research was carried out under the auspices of RAND Education. Any opinions, findings, conclusions, or recom- mendations expressed in this report are those of the authors and do not necessarily reflect the views of the National Science Foundation. CONTENTS Preface ........................................................................................... iii Figures ........................................................................................... ix Tables............................................................................................. xi Summary ..................................................................................... xiii Acknowledgments .......................................................................xxi Chapter One INTRODUCTION .....................................................................1 Defining the Terms ..................................................................2 Tests Are Used Widely and for Several Purposes ..................4 New Federal Legislation Makes Testing Mandatory .............5 The Rationale for Test-Based Accountability Systems Is Compelling....................................................................... 7 Test-Based Accountability Commands Substantial Resources .........................................................................9 About This Book.................................................................... 10 Chapter Two TESTS AND THEIR USE IN TEST-BASED ACCOUNTABILITY SYSTEMS.............................................. 13 A Brief History of Recent Trends in Large-Scale Assessment.................................................................... 14 Using Assessments to Monitor Aggregate Academic Achievement ................................................................. 16 Evolving from Minimum-Competency Testing to Measurement-Driven Instruction............................... 17 v vi Making Sense of Test-Based Accountability in Education The Education Reform Movement and Increased Use of Standardized Tests ................................................... 17 The Second Wave of Reform: Testing Higher-Order Skills ............................................................................... 18 How Large-Scale Tests Are Used Today.............................. 20 Content and Performance Standards .............................. 21 Norm-Referenced Score Reporting ................................. 22 Criterion-Referenced Score Reporting............................ 24 Comparing the Two Types of Score Reporting ............... 25 Disadvantages to Reporting Performance Solely as Norm or Criterion Referenced..................................... 27 Aligning Tests and Standards............................................... 29 Types of Assessments ....................................................... 31 Reporting the Results of Large-Scale Assessments......... 34 Setting Targets for Performance .......................................... 39 Measuring Status............................................................... 40 Measuring Change ............................................................ 42 Rewards and Sanctions ........................................................ 43 School-Level Incentives .................................................... 44 Student-Level Incentives .................................................. 45 Summary ............................................................................... 48 Chapter Three TECHNICAL CRITERIA FOR EVALUATING TESTS............ 51 First, Some Background ....................................................... 51 Score Reliability..................................................................... 52 Factors That Affect Score Reliability ................................ 53 Quantifying Score Reliability............................................ 55 Score Reliability and Decisionmaking............................. 58 Validity ................................................................................... 60 Forms of Validity Evidence............................................... 61 Validity of Scores and Gains Under High-Stakes Conditions ..................................................................... 66 Fairness.................................................................................. 68 Bias-Free Items and Tests................................................. 68 Comparable Opportunities to Demonstrate Knowledge ..................................................................... 70 Trade-Offs: Testing Time, Costs, and Other Factors.......... 74 Conclusion ............................................................................ 76 Contents vii Chapter Four CONSEQUENCES OF LARGE-SCALE, HIGH-STAKES TESTING ON SCHOOL AND CLASSROOM PRACTICE ..... 79 The Importance of the Effects of Testing ............................ 79 The Effects of Testing and Test Validity .......................... 80 Broadened Use of High-Stakes Testing to Promote Changes in School Practice.......................................... 81 Gathering Evidence About the Effects of High-Stakes Testing ..........................................................................