MEASURING WHAT MATTERS Using Assessment and Accountability to Improve Student Learning

A STATEMENT BY THE RESEARCH AND POLICY COMMITTEE OF THE COMMITTEE FOR ECONOMIC DEVELOPMENT MEASURING WHAT MATTERS Using Assessment and Accountability to Improve Student Learning

A STATEMENT BY THE RESEARCH AND POLICY COMMITTEE OF THE COMMITTEE FOR ECONOMIC DEVELOPMENT Library of Congress Cataloging-in-Publication Data

Committee for Economic Development. Research and Policy Committee. Measuring what matters : using assessment and accountability to improve student learning / a statement by the Research and Policy Committee of the Committee for Economic Development p. cm. “December 15, 2000.” Includes bibliographical references. ISBN 0-87186-139-9 1. Educational tests and measurements — United States. 2. Educational accountability — United States. 3. School improvement programs — United States. I. Title.

LB3051 .C632 2001 371.26'0973 — dc21 2001017186

First printing in bound-book form: 2001 Paperback: $15.00 Printed in the United States of America Design: Rowe Design Group

COMMITTEE FOR ECONOMIC DEVELOPMENT 477 Madison Avenue, New York, N.Y. 10022 (212) 688-2063 2000 L Street, N.W., Suite 700, Washington, D.C. 20036 (202) 296-5860 www.ced.org CONTENTS

RESPONSIBILITY FOR CED STATEMENTS ON NATIONAL POLICY iv

PURPOSE OF THIS STATEMENT viii

CHAPTER 1: INTRODUCTION AND SUMMARY 1

CHAPTER 2: MEASURING STUDENT ACHIEVEMENT 5 Issues in designing direct measures of student learning 5 Setting standards for learning 5 Designing assessments 7 Reporting scores 9 Challenges to measurement practices 10 Appropriate test use 10 Testing all students 12 Cost implications 13 Limitations of tests as measures of student learning 14

CHAPTER 3: USING ACHIEVEMENT MEASURES TO IMPROVE TEACHING AND LEARNING 15 Measurement that enhances instruction 15 Teacher capacity 17

CHAPTER 4: USING ACHIEVEMENT MEASURES TO HOLD STUDENTS AND EDUCATORS ACCOUNTABLE 19 Using performance measures to reward and sanction students 21 Using multiple measures 21 Availability of appropriate instruction and assistance 23 Motivating students to do their best 24 Using performance measures to reward and sanction educators 25 Accountability for individual teachers 25 Accountability for schools 28 Guidelines for good practice 30

CHAPTER 5: MONITORING EDUCATIONAL PROGRESS AND THE MEASUREMENT SYSTEM ITSELF 32 National and international assessments 32 Reporting to the public 33 Monitoring the measurement system 35

ENDNOTES 37

REFERENCES 39

MEMORANDA OF COMMENT, RESERVATION, OR DISSENT 42

OBJECTIVES OF THE COMMITTEE FOR ECONOMIC DEVELOPMENT 45

iii RESPONSIBILITY FOR CED STATEMENTS ON NATIONAL POLICY

The Committee for Economic Develop- ing specific legislative proposals; its purpose is ment is an independent research and policy to urge careful consideration of the objectives organization of some 250 business leaders set forth in this statement and of the best means and educators. CED is nonprofit, nonparti- of accomplishing those objectives. san, and nonpolitical. Its purpose is to pro- Each statement is preceded by extensive pose policies that bring about steady eco- discussions, meetings, and exchange of memo- nomic growth at high employment and randa. The research is undertaken by a sub- reasonably stable prices, increased productiv- committee, assisted by advisors chosen for their ity and living standards, greater and more competence in the field under study. equal opportunity for every citizen, and an The full Research and Policy Committee improved quality of life for all. participates in the drafting of recommenda- All CED policy recommendations must tions. Likewise, the trustees on the drafting have the approval of trustees on the Research subcommittee vote to approve or disapprove a and Policy Committee. This committee is di- policy statement, and they share with the rected under the bylaws, which emphasize Research and Policy Committee the privilege that “all research is to be thoroughly objec- of submitting individual comments for publi- tive in character, and the approach in each cation. instance is to be from the standpoint of the general welfare and not from that of any The recommendations presented herein are special political or economic group.” The those of the trustee members of the Research and committee is aided by a Research Advisory Policy Committee and the responsible subcom- Board of leading social scientists and by a mittee. They are not necessarily endorsed by other small permanent professional staff. trustees or by non-trustee subcommittee members, The Research and Policy Committee does advisors, contributors, staff members, or others not attempt to pass judgment on any pend- associated with CED.

iv RESEARCH AND POLICY COMMITTEE

Co-Chairmen PATRICK W. GROSS, Founder and RONALD R. DAVENPORT, Chairman of ALONZO L. MCDONALD, Chairman Chairman, Executive Committee the Board and Chief Executive Officer American Management Systems, Inc. Sheridan Broadcasting Corporation Avenir Group, Inc. BRUCE K. MACLAURY, President JOHN DIEBOLD, Chairman NICHOLAS G. MOORE, Chairman Emeritus John Diebold Incorporated PricewaterhouseCoopers The Brookings Institution FRANK P. DOYLE, Retired Executive JOHN D. ONG, Chairman Emeritus Vice President The BFGoodrich Company Vice Chairmen GE STEFFEN E. PALKO, Vice Chairman IAN ARNOF, Retired Chairman T.J. DERMOT DUNPHY, Retired Bank One, Louisiana, N.A. and President Chairman Cross Timbers Oil Company ROY J. BOSTOCK, Chairman Sealed Air Corporation B/com3 Group, Inc. * CAROL J. PARRY, Retired Executive CHRISTOPHER D. EARL, Managing Vice President CLIFTON R. WHARTON, JR., Former Director The Chase Manhattan Corporation Chairman and Chief Executive Officer Perseus Capital LLC VICTOR A. PELSON, Senior Advisor TIAA-CREF W. D. EBERLE, Chairman Warburg Dillon Read LLC Manchester Associates, Ltd. PETER G. PETERSON, Chairman EDMUND B. FITZGERALD, Managing The Blackstone Group Director Woodmont Associates S. LAWRENCE PRENDERGAST, REX D. ADAMS, Dean Executive Vice President of Finance The Fuqua School of Business HARRY L. FREEMAN, Chair LaBranche & Co. The Mark Twain Institute Duke University NED REGAN, President ALAN BELZER, Retired President and RAYMOND V. GILMARTIN, Chairman, Baruch College Chief Operating Officer President and Chief Executive Officer Merck & Co., Inc. JAMES Q. RIORDAN AlliedSignal Inc. Quentin Partners Inc. PETER A. BENOLIEL, Chairman, BARBARA B. GROGAN, President Western Industrial Contractors MICHAEL I. ROTH, Chairman and Executive Committee Chief Executive Officer Quaker Chemical Corporation RICHARD W. HANSELMAN, Chairman The MONY Group Inc. Health Net Inc. FLETCHER L. BYROM, President and LANDON H. ROWLAND, Chairman, Chief Executive Officer RODERICK M. HILLS, President President and Chief Executive Officer MICASU Corporation Hills Enterprises, Ltd. Stillwell Financial Inc. DONALD R. CALDWELL, Chairman and MATINA S. HORNER, Executive GEORGE RUPP, President Chief Executive Officer Vice President Columbia University Cross Atlantic Capital Partners TIAA-CREF HENRY B. SCHACHT, Director and ROBERT B. CATELL, Chairman and Chief H.V. JONES, Office Managing Director Senior Advisor Executive Officer Korn/Ferry International, Inc. E.M. Warburg, Pincus & Co., Inc. LLC KeySpan Corporation EDWARD A. KANGAS, Chairman, ROCCO C. SICILIANO JOHN B. CAVE, Principal Retired Beverly Hills, California Avenir Group, Inc. Deloitte Touche Tohmatsu MATTHEW J. STOVER, President CAROLYN CHIN, Chief Executive Officer JOSEPH E. KASPUTYS, Chairman, edu.com Cebiz President and Chief Executive Officer Thomson Financial Primark ALAIR A. TOWNSEND, Publisher A. W. CLAUSEN, Retired Chairman and Crain's New York Business Chief Executive Officer CHARLES E.M. KOLB, President BankAmerica Corporation Committee for Economic Development ARNOLD R. WEBER, President Emeritus Northwestern University JOHN L. CLENDENIN, Retired Chairman ALLEN J. KROWE, Retired BellSouth Corporation Vice Chairman JOSH S. WESTON, Honorary Chairman Automatic Data Processing, Inc. GEORGE H. CONRADES, Chairman and Texaco Inc. Chief Executive Officer CHARLES R. LEE, Chairman and DOLORES D. WHARTON, Chairman and Akamai Technologies, Inc. Co-Chief Executive Officer Chief Executive Officer The Fund for Corporate Initiatives, Inc. KATHLEEN B. COOPER, Chief Economist Verizon Communications and Manager, Economics & Energy MARTIN B. ZIMMERMAN, Vice Division President, Governmental Affairs Exxon Mobil Corporation Ford Motor Company

*Voted to approve the policy statement but submitted memoranda of comment, reservation, or dissent. See page 42.

v SUBCOMMITTEE ON EDUCATION POLICY

Co-Chairmen ROY J. BOSTOCK JEROME H. GROSSMAN HOWARD M. ROSENKRANTZ Chairman Chairman and Chief Executive Chief Executive Officer B/com3 Group, Inc. Officer Grey Flannel Auctions EDWARD B. RUST, JR. Lion Gate Management Corporation MICHAEL I. ROTH Chairman and Chief Executive Officer JUDITH H. HAMILTON Chairman and Chief Executive Officer State Farm Insurance Companies President and Chief Executive Officer The MONY Group Inc. Classroom Connect NEIL L. RUDENSTINE WILLIAM A. HASELTINE President Trustees Chairman and Chief Executive Officer Harvard University HENRY P. BECTON Human Genome Sciences, Inc. WILLIAM RUDER President and General Manager HEATHER R. HIGGINS President WGBH Educational Foundation President William Ruder Incorporated PETER A. BENOLIEL Randolph Foundation ALAN G. SPOON Chairman, Executive Committee DAVID KEARNS Managing General Partner Quaker Chemical Corporation Chairman Emeritus Polaris Venture Partners JON A. BOSCIA New American School DONALD M. STEWART President and Chief Executive Officer COLETTE MAHONEY President Lincoln National Corporation President Emeritus The Chicago Community Trust JOHN BRADEMAS Marymount Manhattan College † ROBERT C. WAGGONER President Emeritus ELLEN R. MARRAM Chief Executive Officer New York University Partner Burrelle’s Information Services LLC * WILLIAM E. BROCK North Castle Partners MICHAEL W. WICKHAM Chairman DIANA S. NATALICIO Chairman and Chief Executive Officer Bridges LearningSystems, Inc. President Roadway Express, Inc. THOMAS J. BUCKHOLTZ The University of Texas at El Paso KURT YEAGER Executive Vice President JOHN D. ONG President and Chief Executive Officer Beyond Insight Corporation Chairman Emeritus Electric Power Research Institute JOHN B. CAVE The BFGoodrich Company Principal STEFFEN E. PALKO Avenir Group, Inc. Vice Chairman and President FERDINAND COLLOREDO- Cross Timbers Oil Company Ex-Officio Members MANSFELD ARNOLD B. POLLARD FRANK P. DOYLE Chairman and Chief Executive Officer President and Chief Executive Officer Retired Executive Vice President Cabot Industrial Trust Chief Executive Group GE KATHLEEN FELDSTEIN S. LAWRENCE PRENDERGAST PATRICK W. GROSS President Executive Vice President of Finance Founder & Chairman, Executive Economics Studies, Inc. LaBranche & Co. Committee JOSEPH GANTZ HUGH B. PRICE American Management Systems, Inc. Partner President and Chief Executive Officer CHARLES E.M. KOLB GG Capital, LLC National Urban League President CAROL R. GOLDBERG E. B. ROBINSON, JR. Committee for Economic Development President Chairman Emeritus BRUCE K. MACLAURY The Avcar Group, Ltd. Deposit Guaranty Corporation President Emeritus ROSEMARIE B. GRECO ROY ROMER The Brookings Institution Principal Former Governor of Colorado JOSH S. WESTON Greco Ventures Superintendent of the Los Angeles Honorary Chairman BARBARA B. GROGAN Unified School District Automatic Data Processing, Inc. President Western Industrial Contractors

*Voted to approve the policy statement but submitted memoranda of comment, reservation, or dissent. See page 42. †Disapproved publication of this statement. See page 42.

vi SPECIAL GUESTS AND ADVISORS MYLES A. CANE Chairman Emeritus Skidmore College SAUL COOPERMAN President Citizens for Better Schools SEYMOUR FLIEGEL President Center for Educational Innovation HELEN F. LADD Professor of Public Policy Studies and Economics Sanford Institute of Public Policy Duke University CHRISTINA LAURIA Vice President, Pricing Administration Roadway Express, Inc. LAWRENCE T. PISTELL Summit, New Jersey ANDREW C. PORTER Professor of Educational Psychology University of Wisconsin at Madison DIANE RAVITCH Research Professor New York University Nonresident Senior Fellow The Brookings Institution

PROJECT DIRECTOR JANET HANSEN Vice President and Director of Education Studies Committee for Economic Development

PROJECT ASSOCIATE SETH TUROFF Research Associate Committee for Economic Development

vii PURPOSE OF THIS STATEMENT Education has always been important to the mid-1980s, the development and educa- Americans, but for the last 20 years questions tion of all children from the earliest stages of about the quality of the nation’s schools have their lives must be a national priority. This been debated with particular urgency. Policy requires better preparing children for school, makers, business leaders, parents, and many providing a rich curriculum and ensuring that educators have argued that major changes are every classroom has a teacher with the prepa- necessary to ensure that our children are pre- ration and materials to teach it, adopting school pared for the civic, economic, and social world governance and management structures that that lies ahead of them. Key among these highlight performance and encourage account- changes is Putting Learning First (as CED titled ability, and giving school-age children the so- its last report on education): setting high aca- cial and health supports that will help them demic standards and holding schools account- learn in the classroom. able for helping students reach these standards. Educational measurement by itself won’t Schools cannot be re-oriented toward per- improve America’s schools. Without it, how- formance, however, unless they know what they ever, we cannot know how far we’ve come or are trying to accomplish and can measure how far we have to go to give our children the progress toward these goals. As this statement education they deserve. argues, measuring student achievement is es- sential for effective school reform. CED Trust- ees, accustomed to managing complex organi- ACKNOWLEDGMENTS zations, are convinced that we cannot improve We would like to thank the dedicated group what we do not measure. Thus we strongly of CED Trustees, advisors, and guests who support efforts underway in virtually every state served on the subcommittee that prepared this to specify academic standards and measure report (see page vi). Special thanks go to the improvements in student learning. We wel- subcommittee co-chairs, Roy J. Bostock, Chair- come the spotlight these efforts shine on how man of B/com3 Group, Inc., and Edward B. well schools are serving all students, including Rust, Jr., Chairman and CEO of State Farm students whose educational needs were too Insurance Companies, who guided the project often neglected in the past. with skill and insight. We are grateful to At the same time, we recognize that assess- B/com3 Group, Inc., the GE Fund, The George ment and accountability systems capable of Gund Foundation, the Keyspan Foundation, driving school improvement are still in their the Smith Richardson Foundation, Inc., and formative stages. They are not perfect, and State Farm Insurance Companies for their they sometimes have unintended consequences support of this project. We are also grateful that rightly concern parents and educators. to project director Janet S. Hansen, Vice There is more to learn about how best to de- President and Director of Education Studies at sign and implement such systems. This report CED, and Van Doorn Ooms, CED’s Senior aims both to show why testing and account- Vice President and Director of Research, for ability are indispensable and to explore their their contributions, and to Seth Turoff for responsible use. research assistance. Measuring educational performance is not the same as improving it. The latter requires Patrick W. Gross effort on a host of fronts, many of which are Bruce K. MacLaury beyond the scope of this statement. As CED Co-Chairs has noted in a series of reports dating back to CED Research and Policy Committee viii Chapter 1 INTRODUCTION AND SUMMARY

America cannot get the schools it needs with the rapid pace of change in the 21st century. The without solid measures of academic achieve- debate over testing should not be about whether to rely ment. The public—vitally interested in the qual- on tests but about how best to improve and use them ity of education available to its children to enhance educational outcomes. —strongly supports testing. (See Figure 1.) At In our 1994 policy statement Putting Learn- the same time, there is debate over the in- ing First: Governing and Managing the Schools for creased emphasis being placed on tests and High Achievement,2 CED argued that the pri- test results. At one extreme, critics allege that mary mission of the public schools should be tests are harming rather than help- ing students; some would be Figure 1 happy to eliminate tests. At the other extreme, test advocates The American Public Stands Firmly Behind Tests sometimes seem too ready to rely Percent of People Who Feel There is Not Enough, Too Much, or Just uncritically on test scores in mak- the Right Amount of Standardized Testing in Their Community* ing important educational deci- sions. Our purpose in this policy statement is to show that tests are 21 important and that there are re- Elementary sponsible ways to use them. and Middle 50 School Public scrutiny of testing is 12 healthy and contributes to im- I Not Enough proved policies and practices. I Right Amount Amidst “the wonderful cacophony I of a free people disagreeing,”1 Too Much however, we must not lose sight of 32 a key fact: measuring student achieve- ment is an essential element of effec- High School 44 tive school reform. As business leaders, 9 we know that we can’t improve what we don’t measure. Tests are vital tools for managing and evaluating efforts 0% 20% 40% 60% 80% 100% to ensure that all children receive a high-quality education that prepares * The remaining percentage of people either said they don’t know or refused to them for college, for the workplace, for respond to the question. participation in the nation’s civic life, SOURCE: Belden Russonello & Stewart and Research/Strategy/Management and for lifelong learning to keep up (2000:34).

1 Measuring What Matters learning and achievement. We pointed out based accountability systems most effectively to that there can be no lasting improvement in spur school improvement. Public discussion educational achievement until those who gov- about tests and test use can be hampered by ern the system change the way schools are or- the fact that educational measurement is com- ganized and managed to focus on this mission. plex and sometimes highly technical. This Instead of emphasizing compliance with regu- policy statement aims to help the public par- lations, there should be greater reliance on ticipate in efforts to improve measurement by creating effective incentives for principals, presenting key issues, identifying the benefi- teachers, and students to raise academic cial purposes measurement can serve and the achievement. Incentives (with consequences for unintended consequences that can occur, and both successful and unsuccessful performance) exploring how undesirable consequences might should be accompanied by greater flexibility be avoided. for principals and teachers to choose how they Our central recommendation is that tests will pursue improved learning. In short, educa- should be used and improved now—rather than tion needs to be transformed into a perfor- resisted until they are perfect—because they mance-oriented enterprise, rather than one provide the best means of charting our progress focused on inputs and rules. toward the goal of improved academic achieve- Transforming schools to focus on perfor- ment.* mance requires the ability to measure what We recognize that tests are imperfect and students know and how well schools are suc- incomplete measures of student learning. We ceeding in meeting the goal of improving aca- also acknowledge that they do not measure all demic achievement. Schools should solidly important aspects of schools’ educational mis- 3 ground all students in language and mathemati- sions. In earlier CED reports, we have pointed cal skills and provide them with a broad base of out that both the regular and the “invisible” knowledge in subjects such as literature, sci- curriculum of schools should foster traits in ence, foreign languages, history, social sciences, addition to achievement that are needed for and the arts. Students should be able to use success in the adult world: good work habits, teamwork, perseverance, honesty, self-reliance, and apply this knowledge. Without standards and consideration for others. that articulate expectations for students and We also recognize that tests are not them- measures of their performance on the stan- selves a strategy for improving education. In- dards, we have no way of gauging the success or stead, with thoughtful attention to their failure of our educational system. strengths and weaknesses, tests serve as vital Reorienting schools toward performance is tools for measuring the paramount outcome occurring through standards-based reform as of education—academic achievement. They well as through new governance arrangements provide information that permits those respon- like charter and contract schools that require sible for governing and managing schools to performance data for purposes of public ac- reflect on the results of the daily interaction countability and parental choice. These devel- between teachers and students in classrooms— opments have caused measurement and test where the real work of learning occurs—and use to receive unprecedented attention. Many ask whether the results are satisfactory, what states and school districts have implemented things need changing, and whether changes or are working on new assessment and account- once made are having their intended effects. ability systems relying on more challenging tests Specifically, educational tests along with linked to academic standards. other measures of performance can contrib- There is still more to learn, though, about ute to raising educational achievement in three how to design and use assessment and test- ways:

*See memorandum by CAROL J. PARRY (page 43). 2 Chapter 1: Introduction and Summary

• They can help improve teaching and learning, ties to some groups of children (minorities, both by guiding day-to-day instruction and students with disabilities, and children who are by enabling researchers to evaluate the ef- not native English speakers, for example). fects of different educational reforms, so While public support for tests remains that effective ones can be further imple- strong, it could be undermined by unintended mented and ineffective ones discarded. negative consequences of new assessment poli- • They can provide a means of holding students cies: for example, if test-based accountability and educators accountable for improving educa- systems lead to narrow test-based coaching tional outcomes, relying on incentives rather rather than rich instruction, if they fail to cre- than rules to spur achievement. They can ate incentives for improving the performance contribute to equalizing educational oppor- of all students, or if they rely too heavily on tunity, by focusing attention on children tests to make important decisions about indi- who have for too long been left behind by vidual students and teachers. Parents and teach- America’s schools and by creating incen- ers could lose faith in education reform itself if tives for finding better ways to teach chil- new performance measures identify students dren for whom traditional approaches have who need additional help to succeed academi- failed. They can counter the tendency to- cally, but then policy makers fail to provide this ward complacency about the performance assistance. of the nation’s best students, by providing Measuring student achievement will, we believe, information on whether these students are contribute importantly to improved learning if policy standing still or improving and on how they makers, educators, and the public keep the following measure up against the top students in other in mind: countries. • Tests are a means, not an end, in school reform. • They can provide a means for monitoring the Real educational improvement requires progress of the educational system and for report- changing what goes on in classrooms. Policy ing to the public. They help parents and the makers must do more than just identify what public become better “consumers” of their students know and can do. They have to educational systems by helping them deter- tackle the much tougher job of helping edu- mine how well students are learning and cators address inadequacies in student learn- whether reforms are making a difference in ing and overcome conditions that stand in improving the quality of America’s schools. the way of high academic achievement. Despite these benefits of educational mea- • Assessment and accountability systems are works surement, new assessment and accountability in progress and must be continuously reviewed systems have proven controversial. To some and improved. Educational standards on extent, this is inevitable. New assessment sys- which assessment systems are based are not tems deliver painful information about perfor- yet uniformly rigorous and substantive. mance to some teachers and students. Parents There is more work to do in designing as- don’t always welcome the news that their sessment instruments that can measure the children’s academic achievement is weak. More- rich array of knowledge and skills embed- over, standardized testing, a key (but not the ded in rigorous and substantive standards only) element of educational measurement, and that can accurately portray the perfor- has long been a political lightning rod for a mance of students with special educational small but vocal anti-testing community. Its mem- needs, such as bilingual students and stu- bers argue that tests may be (and indeed in the dents with disabilities. As accountability re- past sometimes have been) used in ways having quirements focus schools’ attention on the effect of denying educational opportuni- measurable student progress, they can have

3 Measuring What Matters

unintended consequences such as teachers data to improve instructional practices, and emphasizing test preparation instead of the assistance for students and schools whom broader curriculum. We need to know more tests show to be poor performers. Despite about how to design accountability systems the attention to assessment and account- and about the impact of specific design fea- ability in recent years, most states still have a tures. Standards, assessment, and account- long way to go in implementing standards- ability provisions need to be regularly based testing systems in all grades and ma- reviewed, using independent evaluators to jor subjects. Few have either the assessment help identify problems and best practices or data systems that would allow individual and to monitor the intended and unin- students’ performance to be tracked over tended consequences of policy changes. time and their progress measured from year • A performance-based educational system built on to year. Without greater investments in such measuring student achievement can’t be con- systems policy makers will find that state structed on the cheap. Such a system requires testing systems are not as useful as they good measurements and test administration could be in helping teachers improve their procedures, information systems that make instructional performance and in holding results available to educators in useful for- educators accountable for the performance mats, training in how to use performance of students in their charge.*

*See memorandum by CAROL J. PARRY (page 43).

4 Chapter 2 MEASURING STUDENT ACHIEVEMENT

Tests are not the only way of obtaining in- students, educators, and the public to monitor formation on student and school performance. performance against the standards. Holding In the past, student performance was often schools accountable for meeting the standards judged mainly by indirect measures that served should create incentives for schools to make (or were thought to serve) as general indica- instructional changes to boost student perfor- tors of academic accomplishment: high school mance, while giving educators flexibility to de- graduation, for example, or enrollment in cide for themselves what instructional and advanced courses and in college. Similarly, the structural changes are needed (rather than performance of schools or districts was fre- imposing one-size-fits-all solutions from above). quently indicated by indirect measures such as As background for considering how tests graduation and dropout rates and teacher and can and should be used to foster educational student attendance rates. improvement (as we do in Chapters 3, 4, and Transforming education to a performance- 5), it is helpful to identify basic issues in design- oriented system emphasizing academic achieve- ing measures of student achievement and to ment, however, has vastly increased the interest outline challenges posed by the new emphasis and attention being given to direct measures being given to tests as a reform tool. of student learning: i.e., tests. Tests, and the accountability systems based on them, are not new and indeed have figured prominently in ISSUES IN DESIGNING DIRECT earlier reform efforts. Never before, however, MEASURES OF STUDENT has the interest in them been so pervasive.4 LEARNING Much of the current attention to testing derives from the standards-based systemic re- Setting standards for learning form movement. Spurred by the first-ever Edu- Measuring student learning requires defin- cation Summit between the President and the ing what students need to know and be able to nation’s governors in 1989 which set results- do. The 1990s saw the first widespread efforts oriented national education goals,5 standards- to develop content standards for academic based reform has dominated education policy achievement. CED’s earlier hope6 that rigor- making at both the state and federal level for ous, substantive national standards and related roughly a decade. The strategy underlying this assessments would be the foundation for state approach to school improvement begins with and local curriculum and instructional reform defining standards for student performance has not been realized. The tradition of local that create high expectations for all students. control is still strong in education. We under- Linking assessments to these standards, it is estimated the political potency of local control argued, will then provide a means for parents, and the importance of involving many con-

5 Measuring What Matters stituencies in a consultative approach to stan- highly specific, spelling out in detail the con- dard-setting. Thus, the standards that are driv- tent knowledge students should demonstrate, ing reform are the product of state-by-state whereas others are more general—or vague, as standard-setting activities involving subject critics contend. The degree to which standards matter experts, educators, and representatives are ‘challenging’ also varies, with some states of the public. demanding much more of their students than The good news is that all states but one others.”9 (Iowa) have or are developing content stan- Standards, therefore, are not yet sufficiently dards in some or all of four core academic rigorous and substantive. The National Re- areas (English, mathematics, science, and so- search Council (NRC)10 has also pointed out cial studies or history). (See Figure 2.) An that standards differ in the degree to which early tendency in many states to create “a diz- they guide policy and practice, some being so zying array of fuzzy, nonacademic goals that general that they make it difficult to make valid are overly subjective and highly controversial”7 inferences about student performance or to has given way to more focused attention on provide clear guides for instruction. Some stan- academic achievement. dards are so extensive that teachers cannot This does not mean that standard-setting address them comprehensively, and test de- has become noncontroversial or the results signers cannot include them all on assessments. uniform. Education is inherently controver- Education researcher Paul Hill argues that sial, because it is “as much concerned with many states initially used a “logrolling” process central public values as it is with schools per se, for setting standards, resulting in requirements and central values that Ameri- 8 cans hold dear may conflict.” Figure 2 Differences in values are of- ten reflected in raucous de- Standards and Assessments at the State Level bates over standards that can roil a state politically: witness the recent decision by the Adopted standards Kansas State Board of Educa- in at least one tion to omit mention of evo- subject 45 4 lution and the “big bang” theory of the origin of the Adopted standards universe from the state’s sci- in English, math, ence standards and assess- social studies, 38 6 3 ments, leaving local schools and science free to teach these topics if they wish. (That decision be- Assessments aligned came a major issue in the with standards in at 35 6 3 least one subject 2000 election of state board members. As this statement Assessments aligned I 1997–1998 goes to press, the newly- with standards in I 1999–2000 elected board appears ready English, math, 17 4 2 I to reverse the prior board’s social studies, 2000–2001 action.) and science 0 1020304050 The results of state stan- dard-setting processes vary SOURCE: Education Week (2000:64, 2001:94-95). widely. “Some standards are

6 Chapter 2: Measuring Student Achievement that reflected the aspirations of advocates for particular academic specialties. Now many are CRITERIA FOR EVALUATING TESTS: seriously grappling with the question of “what VALIDITY, RELIABILITY, EQUITY is an externally valid standard (i.e., one that is Tests should be judged by three key criteria: closely linked to an important life outcome for • Validity: the degree to which a test measures 11 students).” what it was intended to measure. Setting standards on a state-by-state basis is • Reliability: the consistency and dependabil- an incremental and slow process. Despite the ity of assessment results. slow pace, we are optimistic that if policy mak- ers and the public stay the course, rigorous • Equity: fairness, lack of bias toward any particular group of examinees. and clear standards can be achieved. To insure that standards can effectively undergird mea- SOURCE: Klein and Hamilton (1999:4). surement systems and instructional improve- ments, we recommend that states (and districts where applicable) periodically review their stan- tiquing are multiple-choice or “fill in the dards for clarity, focus, rigor, and validity, us- bubble” tests, which they appear to equate with ing independent outside reviewers to help “standardized tests.” In fact, large-scale stan- benchmark against other states and against dardized tests are tests that are administered to models of good practice. The results of out- students from many schools under uniform side reviews should be readily available to the conditions, as opposed to “teacher-made tests” public. In this way, the process of setting stan- administered in individual classrooms. “[E]ven dards will not only improve assessment and a written examination, one that is scored by instruction but also provide a continuing frame- teachers or other human judges and not by work within which citizens can discuss and, if machine, is considered standardized if all stu- necessary, revise their goals for public school- dents respond to the same (or nearly the same) ing. questions and take the examination under simi- lar conditions.”12 Thus, any test intended to Designing assessments measure academic achievement across multiple Assessments that are intended to measure classrooms and schools will be “standardized” academic achievement should meet three key if the results are intended to be comparable. criteria: they should be valid, reliable, and fair The often unspoken assumption of stan- measures of the student learning they seek to dardized-test critics—that teacher-made tests describe. (See box: Criteria for Evaluating Tests: are immune from the weaknesses attributed to Validity, Reliability, Equity.) Tests that are not standardized and multiple-choice tests—is not valid, reliable, and fair will obviously be inaccu- borne out by research.13 Moreover, for large- rate indicators of the academic achievement of scale testing like that required by standards- students and can lead to wrong decisions being based reform, the multiple-choice format has made about students and schools. some advantages. Multiple-choice tests can be Public discussion about the quality of tests scored by machine more quickly and at lower is often hampered by a semantic confusion. cost than other forms of assessment (essays, Test critics often decry “standardized tests,” portfolios) which must be graded by hand. alleging that such tests are invalid and unfair They also represent the form of assessment —that they are incapable of capturing critical with which experts in assessment and test de- differences in student achievement, reflect rote velopment have the most experience and the learning rather than measuring higher-order greatest confidence in their ability to make thinking skills, and lend themselves to being tests that are valid, reliable, and fair. Reliable “gamed.” What these critics are actually cri- scores for individual students can be achieved

7 Measuring What Matters in shorter amounts of testing time when mul- Figure 3 tiple-choice tests rather than other test formats are used. Types of Assessment Instruments Nevertheless, standards-based reform, by Used by States, 2000–2001 requiring measures that can capture the rich array of knowledge and skills embedded in Number of States new content standards and that can be used to 50 49 hold students and educators accountable, has 45 encouraged wider experimentation with new 46 forms of standardized testing. States and dis- 40 tricts are increasingly using new test formats, 35 38 some considered more “authentic” or better aligned with ambitious new goals for educa- 30 tion than fill-in-the-blank or multiple-choice 25 questions. (See Figure 3 and box: There is More to Tests Than Multiple-Choice Ques- 20 tions.) Issues of reliability, validity, fairness, and 15 cost-efficiency still pertain, however; and 10 tradeoffs among these criteria are sometimes 5 7

0 2 THERE IS MORE TO TESTS THAN Multiple Short Extended Extended Portfolio Choice Answer Response Response MULTIPLE-CHOICE QUESTIONS (English) (Other Subjects) Though people tend to equate standardized tests with multiple-choice questions, tests in- SOURCE: Education Week (2001:94). creasingly use other formats as well: • Constructed response: students write their necessary. In some cases, tradeoffs can be ad- own answers, rather than choose among a dressed through policy. For example, if policy handful of pre-selected responses. Con- makers are willing to trade off cost efficiency in structed response, or open-ended, test favor of comparatively expensive assessment items might include essays or mathematical methods like essay exams or open-ended re- problems in which the student is required to show the steps used in reaching a solu- sponse items (to allow for richer and more tion. extensive content coverage) and to permit suf- ficient testing time to ensure reliability, test • Hands-on performance tasks: students are developers can produce tests including such given practical tasks, involving instruments and equipment, and are assessed on their items. content and procedural knowledge as well Sometimes, though, tradeoffs are unavoid- as their ability to use that knowledge in able because of the technical limitations of test reasoning and problem solving. They may design. Kentucky and Vermont were leaders in be evaluated individually or as part of exploring the use of portfolios of student work teams. in an effort to align assessment with instruc- • Portfolios: students are assessed on the tion and to evaluate students in a way that basis of a representative sample of the work reflected the breadth of their academic work. they produce during the school year. Portfolios proved useful to teachers and schools but not suited to use in accountability systems.

8 Chapter 2: Measuring Student Achievement

Teachers reported that learning to score port- should also help monitor the intended and folios had a strong positive influence on in- unintended consequences of assessment sys- struction.14 Reliability was significantly lower tems, a point we will pursue further in our than with traditional multiple-choice tests, how- discussion of accountability. ever. Kentucky reintroduced multiple-choice items to its state assessment after outside ex- Reporting scores perts found that the items were needed to Test scores are the most visible aspect of ensure content coverage, score reliability, and measurement. They can convey information stability of proficiency standards over time.15 about how students and schools measure up Vermont, too, uses multiple-choice, short-an- against standards, about whether they are mak- swer, and extended response items as well as ing progress toward meeting standards, and portfolios in its state test system. about how they compare to other students and An NRC committee reflected the view of schools. Too often, in our view, the press and many test experts when it found that “policy the public overemphasize the comparisons. A and public expectations of testing generally complex story gets reduced to rankings and a exceed the technical capacity of the tests them- preoccupation with whose scores are highest selves. One of the most common reasons for or lowest, when what matters more for school this gap is that policy makers, under constitu- improvement efforts is what knowledge and ent pressure to improve schools, often decide skills students have and whether their achieve- to use existing tests for purposes for which they ment is improving over time. Transforming edu- were neither intended nor adequately vali- cation to an outcome- and performance-based system dated.”16 We recognize that there are other demands that we look beyond the “horse race” and signs of strain in the assessment system as well, focus on the information measurement provides that such as the failure of several testing contrac- can be used to improve teaching and learning. tors to provide timely and accurate score re- Extracting this information from test scores ports to states and districts. requires understanding how assessment results These growing pains seem to us inevitable are and should be reported. and remediable aspects of transforming a huge, Traditionally, the results of large-scale test- complex enterprise into a performance-ori- ing programs have been reported as norms: ented endeavor. Like standard-setting itself, how well students perform relative to the per- building an assessment system capable of meet- formance of a national sample of students who ing the new demands being placed on it will took the same test. They result in familiar per- take time, sustained commitment, and money. centile or grade-referenced scores. A student Nevertheless, creating an assessment system may be described as performing at the 75th capable of supporting schools that focus on percentile (that is, better than 75 percent of outcomes and performance is an urgent prior- the national sample). Some percentage of stu- ity. We urge business leaders, parents, and the dents may be described as scoring at or above a public to provide vigorous support for efforts specified grade level, which reflects not what to build assessment systems that support per- students in that grade are expected to know formance-oriented education. Our recommen- but how these students performed relative to dations about standard setting apply to the national sample enrolled in the same grade. assessment as well. States and districts should Large-scale testing was born from a need to periodically review their assessments for reli- make selection decisions (to make college ability, validity, and fairness. They should call selection decisions, for example, and to iden- on independent outside reviewers to help iden- tify students for gifted programs). Norm-refer- tify and use the best available testing technolo- enced scores were designed to assist in selection gies and should keep the public informed of decisions by spreading out applicants along a the results of these evaluations. Outside review score distribution.

9 Measuring What Matters

Norm-referenced scores are less good at answering today’s most pressing question: what CHALLENGES TO MEASUREMENT do students know? An NRC report cites “a com- PRACTICES monly used analogy: norm-referenced scores The emphasis currently being given to mea- tell you who is farther up the mountain; they surement as a tool of educational reform height- 17 do not tell you how far anyone has climbed.” ens the importance of using tests appropriately, To extend the analogy, they also provide no including all students in assessment programs, information about what the climber needed to and acknowledging that new demands on test- accomplish to reach the level reported (i.e., ing cannot be met on the cheap. was the mountain high or low or particularly steep?). Today in education we want to know Appropriate test use more than how students and schools compare. Individual tests are not one-size-fits-all. Pro- We also want to know what students know about fessional guidelines about the circumstances things we believe are important. That is, we in which use of a particular test is appropriate need criterion-referenced scores, where the have existed since the mid 1950s and have criteria are educational standards. been revised and expanded several times, most The distinction between norm-referenced recently in 1999.18 Test experts argue, however, scores (which emphasize comparisons) and that the standards are insufficiently understood criterion-referenced scores (which emphasize and adhered to by policy makers and test users, performance against standards) is important especially as they build new test-based account- to keep in mind as we pursue our discussion ability systems.19 We recommend that policy about using measurement for various purposes. makers and test users respect professionally- Scores appropriate for one purpose (e.g., norm- developed testing standards when designing referenced scores for informing parents about and implementing assessment and accountabil- how their children are performing relative to ity systems. We will touch on a number of these others) may be quite inappropriate for another standards here and in the Chapter 4 discussion (e.g., for holding schools and teachers account- of test use for accountability purposes. able for improving the performance of their A key concern of test experts is that policy students relative to state standards). makers and the public frequently do not un- Reporting test scores for groups of students derstand that the validity and reliability of a and making inferences from them, especially test is determined by the use to which it will be for purposes of accountability, raise additional put. Tests that are valid and reliable for one considerations. Scores that show average purpose (to influence classroom practice, for achievement of test takers at one specific time example) may not be so for another (to hold are less useful for gauging how well particular schools accountable for performance). The schools are performing than are scores that temptation is nevertheless strong to use tests track changes over time for the same students for multiple purposes to save money and time. (or students in the same grade cohort). Com- We recommend, in accordance with profes- paring scores or score changes across schools sional standards, that tests be used only for and districts without considering differences purposes for which they have been validated— in the backgrounds of test takers can lead to that is, where the strengths and limitations of inaccurate judgments about which schools and the testing program and the test itself have districts are really successful at raising student been evaluated for its intended use. performance, since student backgrounds are To be powerful instruments of reform, most known to influence academic achievement. tests should be tied to (“aligned with”) the We address these issues in more detail in Chap- specific expectations that states and districts ter 4. have for their students. Alignment refers to

10 Chapter 2: Measuring Student Achievement

how well an assessment measures the contents The barriers to alignment are more serious and skills laid out in a standard. “Alignment when states use so-called “off-the-shelf” com- ensures that the tests match the learning goals mercial tests rather than developing their own. embodied in the standards. At the same time, Such tests are designed to be used in many aligned assessments enable the public to deter- states. Given the variety of standards across the mine student progress toward the standards.”20 states, they are unlikely to line up with the Judgments about the extent to which tests and standards in any one state. In a country that standards are aligned go deeper than “yes” or has no nationally-agreed upon benchmarks for “no” evaluations and examine both the con- learning, there is still room for norm-refer- tent that test items are measuring and the depth enced tests that gauge comparative standing of performance that a student is supposed to and for off-the-shelf and other tests that give demonstrate. general information about subject-matter Alignment is easier to achieve when states knowledge. deliberately design tests to measure their stan- Nevertheless, the purposes we identified for dards, though this does not ensure alignment. measurement in the first section of this report Tests still may not measure all of the state’s are best served by tests that are aligned to standards, particularly in states where the list standards. (See Figure 4.) Otherwise, teachers of standards is extensive. The ability of tests to get no guidance from test scores about how measure standards may be further limited by well their students have mastered the standards, constraints imposed by policy makers, such as so they do not know whether their instruc- testing time and costs. tional programs are working. Accountability

Figure 4 Number of States Using Criterion-Referenced Assessments Aligned to State Standards

2000–2001 50

45

40 I Elementary School 43 42 41 41 I Middle School 35 38 38 I High School 30

25

20 23 24 22 15 19 17 10 13

5

0 English Math History Science

SOURCE: Education Week (2001:95).

11 Measuring What Matters systems that do not make clear what teachers students are reaching high standards. And with- are expected to teach and students expected to out such an accurate gauge, schools and school learn are unfair and may well be found illegal districts cannot be held accountable for re- by the courts. sults. Accurate assessment tools, then, are the Using tests for “high stakes” purposes such glue that holds the reform effort together.”25 as denying students promotion or graduation and assigning rewards and sanctions to schools Testing all students raises the stakes for the tests themselves. When Test programs must include all students if adverse consequences for individuals may re- they are to contribute to improving education sult, tests become subject to the possibility of for all. In the past, many testing programs ex- legal challenge on the grounds that the way empted students with disabilities and students they are used is discriminatory or otherwise who are not yet fluent in English from testing inappropriate. (See Chapter 4.) Some civil or did not include the scores of such students rights groups have long opposed the use of in public reports on district and school perfor- tests, at least when tests are used by themselves mance. Testing these special needs students to make important decisions about students or and accurately interpreting their scores raises when students do not have equal access to a variety of technical challenges. Nevertheless, high-quality instruction.21 Their wariness is un- special needs students should be included in derstandable; history has given them plenty of testing programs to the maximum extent cause for concern. Early in the 20th century, possible, and district and school performance “[i]n their worst manifestations, the uses [of reports should always include information on tests] were racist and xenophobic”: prominent any exceptions to this policy. scientists used IQ test results to argue that blacks Special needs students represent a signifi- and immigrants from Southern and Eastern cant proportion of the school population. In Europe were mentally inferior.22 Southern 1996-97, almost 13 percent of elementary and schools resisting desegregation used tests to secondary school students were served by fed- resegregate black students into lower tracks. In eral programs for children with disabilities.26 1994, publication of The Bell Curve 23 sparked a Seven percent participated in bilingual educa- “highly-charged, racialist debate” over the au- tion or English as a second language pro- thors’ claim that social and economic inequal- grams.27 Excluding these students from testing ity among racial and ethnic groups can be sends signals that such students don’t matter explained by differences in intelligence as mea- or can’t be expected to meet high standards sured by tests. Despite detailed critiques of the and that schools aren’t responsible for their book’s methods and analysis, it provided a ra- academic progress. Since these students are tionale for those desiring to limit education not distributed uniformly across schools and and social welfare policies aimed at reducing districts, the validity of test score comparisons inequalities.24 is compromised when nonstandard policies are If attention is given to using tests that are valid, used to exempt students from regular assess- reliable, and fair, however, we believe that measures ment programs and when schools and districts of academic performance can be powerful instru- fail to provide data about the proportion of ments of educational opportunity because of the spot- special needs students included in their scores. light they shine on students whose needs have too Federal law now requires that all students often been neglected in the past. As the Citizens’ be included in assessment and accountability Commission on Civil Rights has noted, “with- mechanisms and that appropriate accommo- out an accurate means of measuring what stu- dations and modifications be made if neces- dents know and can do, responsible school sary.28 For example, a braille version of a test authorities have no way of gauging whether might be provided for a blind student or the

12 Chapter 2: Measuring Student Achievement test read aloud to him or her, or a student with creating assessment instruments that are a reading disability might be given extra time aligned with standards and that can measure a to complete a test. A student not yet proficient range of skills and (2) including all children in in English might be given tests (in subjects assessment programs. Measuring academic other than English itself) in his or her native achievement and using the results to improve language. performance demands other new investments States face major challenges in implement- as well: for example, enhanced test security ing the law. There is little research on the measures for accountability tests; more frequent effects of testing accommodations on the valid- testing so that improvement and not just ity and reliability of test score information for current status can be assessed; linked data sys- either students with disabilities or for English- tems to support public reporting; and (most language learners. It is not clear whether dif- critical of all) appropriate remediation for low- ferent versions of tests in different languages performing students and assistance for schools in fact measure the same thing.29 Indeed, states and teachers to improve their capacity to assist report that the assessment of special popula- all students in meeting high academic stan- tions is among the greatest challenges they dards. face in developing assessment systems.30 There is already evidence that the inability Including all students in assessments is an or unwillingness to make the necessary invest- important goal, but one which makes unprec- ments affects how educational performance is edented demands on testing expertise. It is measured and the results used. Few states have essential that users of test scores understand implemented standards-based testing systems the complexity of the inclusion issue and uti- that cover all grades and major subjects; the lize information about who is and isn’t yet cov- norm is to use standards-based tests at selected ered by testing programs in their interpretation grades in selected subjects. Few have as yet of performance data. States and districts should developed student information systems that develop clear guidelines for accommodations enable comparison with matched sets of schools that permit special needs students to partici- or student level data bases that allow students’ pate in testing. They should also describe the performance to be tracked over time so perfor- methods they use to screen students for ac- mance changes can be identified. Many report commodations and should report how often that they are unable to provide the follow-up students receive accommodations or are still assistance to schools that test results suggest is excluded altogether. Meanwhile, government needed. (See box: States Lack the Capacity to and foundation funding agencies should sup- Serve All Schools in Need of Improvement.) port research aimed at addressing gaps in our The Maryland State Board of Education re- knowledge about the validity and reliability of cently delayed the date at which its high school different accommodations and alternate assess- graduation tests will be used to deny diplomas ment practices that can accurately gauge the after the governor and state legislature failed academic achievement of those special needs to supply full funding for an academic support students for whom standard testing approaches program the board believed was needed to are unsuited. help students who are behind their peers prior to entering high school.31 Cost implications We urge supporters of a truly performance- Shifting to a performance-based education system based educational enterprise to acknowledge and developing the measurement tools to assess per- the costs involved and to advocate the neces- formance requires new investment; it cannot be con- sary funds to build and implement measure- structed on the cheap. We have already described ment systems that support and hold schools two major new demands on testing systems: (1) accountable for instructional improvement.

13 Measuring What Matters

Reallocation of dollars from lesser priority ac- STATES LACK THE CAPACITY TO tivities and more efficient uses of resources SERVE ALL SCHOOLS IN NEED OF may address part of the need but are highly IMPROVEMENT unlikely to supply all of the investment required. A 1998 Department of Education Survey examined states’ capacity to follow up on infor- mation about low school performance: LIMITATIONS OF TESTS AS • Only 9 states reported that they could pro- MEASURES OF STUDENT vide support to at least half the schools in LEARNING need of improvement in their states. Although Americans strongly support test- • 12 states reported that they served less than ing, they do have some concerns about relying half of the schools in need of improve- on statewide tests in public schools. Two major ment. concerns relate to the limits of tests in accu- • 24 states said they had more schools in rately measuring learning: that some children need of support services than they had the perform poorly on tests and that statewide tests resources to provide. cannot measure many important skills that chil- 32 SOURCE: U.S. Department of Education (1999b:64). dren should learn. (See Figure 5.) Of course, the public is right to recognize these limita- tions. Tests are not perfect measures of indi- vidual students’ knowledge, Figure 5 nor do they reflect the full range of a rich educational Public Concerns About the Limits of Tests program. The issue for policy is not whether tests are perfect but whether they help in efforts to im- prove the performance of students Some children and schools and whether they con- perform poorly 49% 6% tribute positively to decisions (like on tests even 32% 10% promotion and graduation) that though they know are going to be made whether tests the material exist or not. We strongly believe I Strongly Agree I Somewhat Disagree that they do. I Somewhat Agree I Strongly Disagree

Statewide tests cannot measure 36% 17% many important skills children 35% 10% should learn

NOTE: Percentages do not add to 100 due to rounding. SOURCE: Belden Russonello & Stewart and Research/Strategy/Management (2000:44).

14 Chapter 3 USING ACHIEVEMENT MEASURES TO IMPROVE TEACHING AND LEARNING

All aspects of a performance-based educational system, including the measurement system itself, MEASUREMENT THAT ENHANCES should ultimately be judged by their contribution to INSTRUCTION teaching and learning. Data can be powerful catalysts for instruc- The growing use of tests for accountability tional change. Creative superintendents and purposes has led to charges that new assess- principals use the information provided by ment systems hurt educational quality. Critics assessments to spur teachers to identify the argue that overemphasis on tests results in cur- strengths and weaknesses of individual students ricular “compression” (slighting subjects that and make plans for addressing problems. are not tested) and in “teaching to the test” Teachers are stimulated to work together on (for example, spending class time on new curricular models that align lesson plans worksheets and test-prep material that imitate to content standards and that encourage cur- test questions rather than on richer forms of ricula to flow from subject to subject and from instruction). Such unintended consequences grade to grade within schools.33 Good teachers do occur, and when they occur they are indeed use test results to help them learn how better worrisome. We do not think that curricular com- to teach the curriculum, not to teach to the test pression and teaching to the test are inevitable results itself. of testing, and we believe that with thoughtful atten- State assessments are being used by many tion they can be avoided. schools to motivate instructional improvement, Moreover, the evidence is mounting that but these tests are actually not necessarily the measurement systems designed to enhance in- preferred form of measurement for purposes struction, when accompanied by efforts to give of affecting the everyday interactions between teachers the capacity to use the results, can be teachers and students. Teacher-made tests, di- powerful levers for instructional improvement. agnostic tests, and other forms of local assess- ment are better suited to provide some kinds Assessment is assisting educators in identifying of information crucial for enhancing teaching. and helping students who in earlier periods Tests used for accountability and monitoring would have been left behind by schools that purposes are designed to provide a range of had no clear idea about how to measure information about students’ mastery of a given progress toward their objectives, if indeed they subject at a given grade level. They are usually had any. However, assessments should be ac- administered once a year. They provide too companied by greater efforts to develop the little information too infrequently to allow teacher capacity necessary to make full use of teachers to adjust their instruction to reflect new information about student learning and changing student needs during the course of translate it into improved instruction. the academic year.

15 Measuring What Matters

Local assessments can compensate for these performance assessments at the district level; shortcomings. Because they can be given often and standards-referenced tests at the state level. and aren’t used for accountability or to reflect All of these are compiled into reports that content coverage comprehensively, they can show important constituencies what they need go into specific topics in depth and cover sub- to know about student performance.”34 (See jects that may not be on state assessments. They box: How Two Districts Blend Local and State can more easily make use of test formats (es- Assessments to Improve Instruction.) says, open-ended questions, hands-on activi- States, districts, and schools can work to- ties, portfolios) that give students a variety of gether to address concerns over inappropriate ways to demonstrate their knowledge and that teaching to the test, cheating, and other test are better suited than multiple-choice items misuses that can distort instruction. Encourag- for classroom instruction. ing careful monitoring of test use by indepen- Districts that understand the potential ben- dent organizations (see Chapter 5) is one efits of measurement can creatively blend local approach. Another is continued improvement and state assessments to improve instruction. A of tests themselves, so that they become some- National Research Council (NRC) report cites thing worth teaching to. Meanwhile, researcher two examples of districts pursuing standards- Richard Phelps argues that the state testing based reform that have fashioned “a mosaic of directors, who are technically proficient and assessment information that includes frequent independent of test publishers, can “deploy a assessment of individual student progress at number of relatively simple solutions” to com- the classroom level; portfolios and grade con- bat test misuse: “not revealing the contents of ferences on student work at the school level; tests beforehand; not using the same test twice;

HOW TWO DISTRICTS BLEND LOCAL AND STATE ASSESSMENTS TO IMPROVE INSTRUCTION

Community District 2 in New York City began setts has developed its own test, and the district its reform effort by changing the curriculum, uses a commercial test. In addition, schools have rather than the assessments. The district admin- developed parallel assessments. One elementary isters a citywide mathematics and reading test, school, for example, begins each September by and a state test as well. Each year, the district assessing every student, from early childhood to reviews the results, school by school, with princi- grade 5, using a variety of methods: observation pals and the board, setting specific goals for for young children (through grade 2), running raising performance, especially among the low- records, writing samples. They repeat the run- est-performing students. In addition, schools ning records and writing samples every four to also administer additional assessments that they six weeks. They examine the data in January and found are aligned with the curriculum. In that again in June to determine the children’s way, the intensive staff development around progress. In that way, every teacher can tell you curriculum, which the district has made its hall- how her students are doing at any point. Teach- mark, and the professional development the ers can adjust their instructional practices ac- district provided on the assessment, produce the cordingly, and principals have a clear picture of same result: teachers with significantly enhanced how each classroom is performing. The district knowledge and skills about how to teach stu- and state tests, meanwhile, provide an estimate dents toward challenging standards. of each school’s performance for policy makers. Schools in Boston also use a multifaceted approach to assessment. The state of Massachu- SOURCE: National Research Council (1999d:49-50).

16 Chapter 3: Using Achievement Measures to Improve Teaching and Learning requiring that non-tested subjects also get opment of new curricula to implement the taught (or testing them, too); and maintaining standards, and to change the way they interact strict precautions against cheating during test with students. New findings from research in administrations.”35 Keeping test content secure the science of learning38 underscore the need and not using the same test twice make it futile for teachers to develop teaching practices that to try to teach to specific test items. However, are adapted to the diverse learning needs of these steps drive up the cost of testing, requir- their increasingly heterogeneous students: prac- ing that publishers prepare more forms of each tices that reflect the prior knowledge, as well as test, that districts purchase a new form for each the skills, attitudes, and beliefs that individual test administration, and that policies be imple- learners bring to school and that are sensitive mented to discourage cheating.36 to the cultural and language practices of stu- We are not naïve in thinking either that dents and the effects of those practices on such good test practices are followed every- classroom learning. Finally, teachers are being where or that some teachers and schools will given unprecedented amounts of data about not respond to the “horse race” mentality about the achievement of their students from district test scores found in the media and among the and state tests and are being asked to use infor- public by focusing instruction in some instances mation on student outcomes to improve their too narrowly on tests. We (business leaders, our teaching.39 colleagues, and our fellow citizens) contribute to test All of these new expectations assume that misuse when we focus our own attention too nar- teachers know how to do these things. Stan- rowly or uncritically on test scores as the sole indica- dards-based reform further assumes that teach- tor of school and student performance. We can be ers would institute effective practices if they part of the solution by being informed consumers of had the freedom and motivation to do so. An test results. We should pay attention to how tests are NRC study committee concluded, however, that linked to standards and instruction, interpret test these assumptions might be “overoptimistic” scores carefully, focus on the progress of schools and and that “knowledge about how to implement students rather than on winners and losers in the effective instructional strategies to help all stu- “horse race” game, and support building the capacity dents learn to challenging standards is also of school-level educators to use test data well for largely unknown.”40 Teachers feel unprepared instructional improvement. (1) to implement new curriculum and perfor- mance standards and teaching methods, (2) to address the special needs of students with dis- TEACHER CAPACITY abilities or limited English proficiency and those Assessments will only contribute to improved teach- from diverse cultural backgrounds, and (3) to ing and learning if educators know how to interpret integrate educational technology into their in- and use test results and how to change their instruc- struction.41 tional practices to make them more effective. There is Part of the solution is better preparation of a growing realization that along with the standards, future teachers, especially more emphasis on assessment, and accountability pieces of standards- subject-matter knowledge. Teachers already in based reform, more attention needs to be given to schools, though, will have to be helped through foster teachers’ capacity to change what they do in the professional development. We recommend that classroom.37 policy makers and educators recognize, as busi- Standards-based reform makes unprec- nesses do, that staff development and edented demands on teachers. It calls on them workforce retooling are important investments to learn about rigorous new standards that re- in the future. As we pointed out in our 1994 quire them to teach very different material, to report, in education these activities have typi- understand and often participate in the devel- cally been less well funded than in the private

17 Measuring What Matters sector and have been among the first to be cut factors that affect teachers’ ability to provide effective when resources are lean.42 instruction and students’ ability to achieve to high Investments in professional development, standards. though, must result in change in the class- room. As much as possible, professional devel- opment activities should be designed to meet HELPING TEACHERS GET AND USE the needs of the personnel in individual schools. PERFORMANCE DATA Too often now, professional development var- In Texas, Just for the Kids (JFTK) was ies widely in quality and consists of short dis- founded in 1995 to provide community sup- trict-sponsored workshops, which are not linked port for school improvement efforts. JFTK to specific teacher needs or to school improve- uses data from the state education agency to ment plans. calculate (for each elementary school in the Teachers also need more help to under- state) an “opportunity gap”: a measure of how stand and use measurement as a tool of in- the school compares to the most successful structional improvement. The NRC found that schools serving equally or more disadvantaged teacher preparation programs provide little student populations. JFTK offers regional emphasis on measurement and that teachers training sessions where principals and commu- feel inadequately prepared in assessment.43 nity leaders are trained on how to present data and lead public discussions and provides train- Thus teachers may not know how to take full ing sessions for educators on how to use the advantage of the data on student outcomes data. Since October 1998, JFTK has trained being provided by state and local assessment more than 1000 campus leadership teams systems and by their own classroom tests. As- representing over 250 school districts through- sisted by business leaders, several states have out Texas in how to use the JFTK data to in- developed programs that help teachers and crease student achievement on their campus. principals learn how to use data to improve JFTK plans to extend its data analysis to instruction. (See box: Helping Teachers Get middle and high schools. Internet site: and Use Performance Data.) www.just4kids.org. If capacity issues are ignored, test experts In Maryland, the Maryland Business warn that a predictable pattern will develop: Roundtable for Education, in collaboration test scores, which generally start off at a low with the Maryland State Department of Educa- tion, created a “decision support system” web level, will “rise quickly for a couple of years, site, which allows schools (and others) to ac- level off for a few more, and then gradually cess and analyze state test data, compare per- 44 drop over time.” In addition to helping cur- formance with other schools, and identify rent teachers improve their instructional prac- effective practices for improving student learn- tice, the necessary changes may involve other, ing. The web site links state standards, data costly reforms such as class size reductions, analysis, school improvement planning pro- after-school and summer school programs for cesses, and resources for instructional en- students having academic difficulties, univer- hancement. Internet site: www.mdk12.org. sal pre-school for those who want it, and rede- The Illinois Business Roundtable has signed teacher training programs as well as collaborated with the Illinois State Board of policies that address health and other prob- Education and the National Central Regional lems that interfere with some children’s ability Education Laboratory to launch a similar effort in July 2000. Internet site: to learn. Assessments are no magic solution to the www.ilsi.isbe.net. challenge of real educational improvement. They serve Wisconsin unveiled the Wisconsin Informa- as an essential thermometer but cannot by themselves tion Network for Successful Schools in Sep- make teaching and learning better. Policy makers tember 2000. Internet site: www.dpi.state.wi.us. and educators must attend to the whole range of

18 Chapter 4 USING ACHIEVEMENT MEASURES TO HOLD STUDENTS AND EDUCATORS ACCOUNTABLE

A key link in the chain of standards-based reform moted from grade to grade or to graduate is accountability, which provides a badly-needed in- from high school. Principals and teachers face centive for students and educators to meet the stan- growing pressures to accept contracts tying con- dards. Without meaningful consequences for tinued employment or pay to the academic performance, the instructional utility of tests is un- achievement of their students. Policy makers likely to be realized. Accountability also provides have “jumped aboard the standards-and-ac- the mechanism for freeing schools from re- countability train,” even if many have been quirements to follow rules and procedures and slower to impose real consequences on schools instead making them responsible for results in and the teachers and principals who work in the form of improved student learning. them.46 Students and educators can be held account- Accountability systems in most states are able for their performance in various ways. too new to evaluate their effect on student Publishing school-level test results and provid- achievement. Evidence from two of the pio- ing data on individual students to parents use neering states—North Carolina and Texas—is the incentive of information (and perhaps pub- very encouraging about the positive effects of lic recognition for high achievement or public accountability systems accompanied by conse- embarrassment over poor performance) to spur quences for results that are implemented as improvement. Higher stakes accompany ac- part of a coordinated set of reform strategies. countability when tests—or tests and some com- (See box: North Carolina and Texas Show Im- bination of other measures—have more direct provement Using a Coordinated Array of Re- consequences; when, for example, they are used form Strategies.) to deny low-performing students promotion to Accountability however, exacerbates con- the next grade or a high school diploma or cerns about unintended negative consequences when schools and their staffs are given finan- of measurement which are less likely to occur cial rewards for high or improving student when tests are used for purposes with more achievement or threatened with state takeovers indirect consequences for individuals (such as or “reconstitution” if student achievement is improving instruction and monitoring the edu- unacceptably low. cational system). When applied unequally (e.g., While it can reasonably be argued that to students only and not to educators, or vice “accountability for results is more talk than versa) accountability raises concerns about fair- action” at this point, especially for the adults in ness.47 Some critics argue that the negative con- the education system, there is no question but sequences of testing (grade retention, denial that accountability is “in vogue.”45 More and of diplomas) are being felt much more by stu- more students now face or soon will face new dents, while rewards rather than sanctions are requirements that they pass tests to be pro- more often applied to teachers.

19 Measuring What Matters

NORTH CAROLINA AND TEXAS SHOW IMPROVEMENT USING A COORDINATED ARRAY OF REFORM STRATEGIES

The National Education Goals Panel, a bipar- The study concluded that the most plausible tisan and intergovernmental body of federal explanation for the test score gains were to be and state officials, was founded after the 1989 found in the policy environment established in Education Summit to track state and national each state. Both states pursued a similar set of progress toward meeting the national education policies which were consistent with each other goals. and sustained over time. The main elements In 1997, the panel reported that two states, included: North Carolina and Texas, stood out from other • State-wide standards by grade for clear teach- states in the extent to which they were achieving ing objectives these objectives. The panel commissioned an outside review to confirm the findings and seek • Holding all students to the same standards to identify the factors that could and could not • State-wide assessments closely linked to aca- account for these states’ success. demic standards The analysis confirmed that the gains in aca- • Accountability systems with consequences for demic achievement in both states were signifi- results cant and sustained. North Carolina and Texas posted the largest average gains in student • Increasing local control and flexibility for scores on tests of the National Assessment of administrators and teachers Educational Progress (NAEP) administered • Computerized feedback systems and data for from 1990 to 1997. The results were mirrored in continuous improvement state assessments administered during the same • Shifting resources to schools with more disad- period, and there was evidence that the scores vantaged students of disadvantaged students improved more than those of advantaged students. SOURCE: Grissmer and Flanagan (1998).

The more direct the consequences for indi- the public to question the usefulness of assess- vidual students and schools, the more critical ments and accountability programs and weaken become questions about the technical quality political support. Virginia was forced to ad- and fairness of accountability models. How dress this issue after only 2 percent of its schools much margin for error is there in students’ test showed satisfactory performance on the first scores and how should this so-called “measure- administration of the Standards of Learning ment error” affect decisions about promotion exams, on which school accreditation will even- and graduation that are based on test scores? tually be based. Setting high standards for stu- Do test-based accountability formulas distin- dent learning is not incompatible with a strategy guish high and low performing schools accu- of setting the initial standards at a lower level rately and consistently? Would different and then gradually ratcheting them up over methods of calculating accountability ratings time, as Texas is doing. Both strategies can get result in different “winners” and “losers” in us where we need to go, if political support accountability systems that reward the top per- remains strong and thoughtful adaptations are formers and sanction the lowest? made as experience is gained. In Texas, this More critical, too, becomes the political has meant introducing a more rigorous set of reasonableness of the results. Performance ex- standards and tests over time. In Virginia, it pectations set so high that most students or has meant keeping the Standards of Learning schools initially fail to reach them may cause in place but delaying by several years the date

20 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable when sanctions for poor-performing schools instruction and assistance are available for in- (loss of accreditation) will kick in. Both states dividuals who are denied promotion or gradu- have shown large increases in the number of ation, and whether specific steps should be students passing state standards-based exams taken to motivate students to do their best on since they were first introduced. tests rather than just score well enough to get Designers of test-based accountability sys- promoted or to graduate. tems must address a series of issues that affect students and educators, respectively. Using multiple measures Even the best test is imprecise to some de- gree. Student scores are expected to vary across USING PERFORMANCE MEASURES different versions of tests, reflecting both the TO REWARD AND SANCTION specific sample of questions asked and transi- STUDENTS tory factors, such as the student’s health on the Probably the best-known use of tests is to make decisions affecting students: “ending so- STATES USING TESTS FOR cial promotion” by employing test scores to GRADUATION DECISIONS determine readiness for promotion to the next grade and developing “high school exit tests” Students must pass on which to base the awarding or withholding tests covering of high school diplomas. (See box: States 10th grade standards Using Tests for Promotion Decisions and box: STATE to graduate States Using Tests For Graduation Decisions.) Alabama Yes These are highly sensitive decisions, so it is Alaska Class of 2002 important to consider how test scores should Arizona Class of 2002 be used in making them, whether appropriate California Class of 2004 Florida Class of 2003 STATES USING TESTS FOR Georgia Yes PROMOTION DECISIONS Louisiana Class of 2003 States that have or will have promotion policies Maryland Class of 2007 based, at least in-part, on performance on state Massachusetts Class of 2003 assessments Mississippi Class of 2003 Arkansas Nevada Yes California New Jersey Class of 2003 Delaware New Mexico Yes District of Columbia Florida New York Yes Illinois North Carolina Class of 2003 Louisiana Ohio Class of 2005 North Carolina South Carolina Class of 2005 Ohio Tennessee Class of 2005 South Carolina Texas Class of 2005 Texas Virginia Virginia Class of 2004 Wisconsin Washington Class of 2008 TOTAL = 13 TOTAL 21 SOURCE: Glidden (1999). SOURCE: Education Week (2001:79).

21 Measuring What Matters day of a test. Moreover, no single test can fully Some states are considering the use of differ- reflect what a student knows and can do. For entiated diplomas, not only to indicate who these reasons, a fundamental precept of good has and hasn’t passed the exams but also to test practice is that “an educational decision recognize exceptionally strong performance, that will have a major impact on a test taker thus giving students an added incentive to do should not be made solely or automatically on well and not just “get by.”49 Differentiated di- the basis of a single test score. Other relevant plomas might also provide a way to signal that a information about the student’s knowledge and student has shown the persistence and disci- skills should also be taken into account.”48 pline (traits employers value) to finish school, Professional standards and virtually all test pub- even if he or she hasn’t been able to pass the lishers warn against relying on tests alone to tests. While these are worthy goals, we fear that make important educational decisions. the tradeoff will be a proliferation of creden- Such warnings are not always heeded, how- tials, varying by state, that will be hard for ever. Test scores offer an easy way to make colleges and employers to interpret. decisions about large numbers of students in a We acknowledge that there may need to be short period of time, while case-by-case evalua- a transition period, as new standards and re- tions incorporating multiple measures like quirements come into being, but we urge that grades and teacher evaluations are more time- policy makers move toward diplomas that tell consuming and prone to special pleading by colleges and employers that their holders have students and parents. To the extent that pro- at minimum passed any required high school motion or graduation is heavily influenced by judgmental con- Figure 6 siderations rather than a more “objective” criterion, the mean- Uncertainty Over the Meaning of a ing of the underlying educational High School Diploma standard students are being asked to meet becomes harder to con- Percent of Respondents Saying That a Diploma is No Guarantee That the Typical High School Student Has Learned the Basics vey to the public. We believe confusion over the underlying standard is especially Employers 60 problematic for decisions to re- quire satisfactory performance on high school exit tests as a condi- Professors 65 tion for receiving a diploma. One of the major complaints that col- leges and employers have is that Students 21 they no longer know what a high school diploma signifies in terms of what a graduate knows and Teachers 25 can do. (See Figure 6.) Standards- based graduation exams hold promise for remedying this situ- Parents ation. 31 The issue, though, isn’t as simple as whether or not to deny 0% 25% 50% 75% 100% a diploma to a student who fails SOURCE: Public Agenda (2000b). graduation test requirements.

22 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable exit tests. There are practices that can be imple- of holding children back must be weighed mented to avoid too heavy a reliance on a against the consequences of promoting them single test for awarding the diploma. One is to into grades or colleges or workplaces where ensure that all students have multiple opportu- they are woefully unprepared to do the work. nities to take the tests. Another is to offer tests One crucial key to making new policies benefi- in a number of subjects and require that many, cial for students who fail promotion and gradu- but not all, be passed in order to graduate. ation tests is to provide them with instruction We think that the arguments for using mul- that will help them catch up. One reason hold- tiple measures in addition to tests are stronger ing students back led to higher dropout rates for promotion and retention decisions. Here, in the past50 is that unsuccessful test-takers were the primary concern is not sending a signal to usually recycled through the same educational the outside world about the student’s level of experiences that had proven unsuccessful for accomplishment, but trying to place the stu- them. It is encouraging to see that today some dent in the most appropriate educational set- policy makers are addressing the need for ting to support his or her learning. We urge innovative and intensive instructional invest- that promotion decisions include advice from ments to help students who have failed or who teachers who know the student and the educa- are at risk of failing “high-stakes” tests. Chicago tional opportunities available in the school and offers one example of a multi-pronged district. We note that Chicago changed its pro- approach to “ending social promotion,” cou- motion rules in August 2000 to allow teachers pling tests with a variety of new instructional to determine whether students should be pro- strategies. (See box: Chicago’s Multi-Pronged moted based on considerations of class grades, Approach to “Ending Social Promotion.”) other test scores, attendance, and behavior in Such an approach will help protect test- addition to scores on the Iowa Test of Basic based promotion and graduation policies Skills, which had previously been the only fac- against court challenges or sanctions imposed tor taken into account. The burden of adding by the U.S. Department of Education’s Office additional considerations to promotion deci- for Civil Rights (OCR). To avoid challenges sions could be minimized by bringing in mul- and sanctions, assessment policies must con- tiple measures only for students whose scores form to many rules about test use that are put them within some specified distance from rooted in the U.S. Constitution, federal civil the cut-off score for promotion.* rights statutes, and judicial decisions, as well as in state law. Availability of appropriate instruction Because of the protections offered by these and assistance rules, the National Commission on Testing and We urge policy makers to recognize that the Public Policy found that “the most common use of tests for promotion or graduation deci- way to challenge important tests is through the sions faces political hurdles and risks legal chal- courts.”51 High-school graduation tests have lenge unless students are provided with been the subject of a number of lawsuits. Re- adequate academic preparation for the tests cently Louisiana’s test-based promotion policy and with intensive instruction for those who was unsuccessfully challenged by a parents’ fail. group in federal court; the OCR has received Some educators and parents fear that hold- complaints from Louisiana and four other states ing children back academically will result in over promotion policies.52 greater numbers of school dropouts. As test- Promotion and graduation tests are most based promotion and graduation policies likely to be challenged on two grounds: that spread, this is certainly a possible consequence they are discriminatory (having disproportion- to be monitored. However, the consequences ate impact on poor and minority children) or

* See memorandum by CAROL J. PARRY (page 43). 23 Measuring What Matters

CHICAGO’S MULTI-PRONGED APPROACH TO “ENDING SOCIAL PROMOTION” In the Chicago Public Schools (CPS), students Should students fail to meet minimum test in the third, sixth, and eighth grades must meet scores at the end of the school year, they are minimum test scores on the Iowa Tests of Basic required to participate in a six-week Summer Skills (ITBS) in reading and mathematics in Bridge Program. This second major component order to be promoted to the next grade level. of the CPS initiative offers smaller classes and a Chicago’s initiative to “end social promotion” centrally mandated curriculum aligned with the goes beyond testing to provide progressively format and content of the ITBS. A decision is targeted intervention, aimed at improving the made at the end of the summer about whether achievement of students with the lowest skills. to promote or retain students who again fail to In the year before promotion, the CPS policy achieve minimum test scores in one or both aims to focus teacher attention on those students subjects. who are not mastering the material. In addition, The third component of the initiative focuses students who are at risk of not meeting the mini- on those students who are retained. Schools mum scores are given extended instructional with high proportions of retained students are time during the school year through Lighthouse, given extra teachers, both to reduce class size an after-school program where students engage and to give extra support to retained students. in a centrally developed curriculum focused on Retained students are also required to partici- reading and mathematics. pate in the Lighthouse after-school program. SOURCES: Roderick et al. (1999); Roderick et al. (2000). that they violate students’ due process rights the four states against which claims had been (by failing to give them adequate notice about filed about promotion tests, permitting the new test policies or by testing knowledge and states to use the tests if they took steps to pro- skills they have not been taught). Tests appear vide things like summer school and acceler- most likely to withstand legal challenge when ated programs to struggling students.55 (The their use is judged to be “educationally neces- fifth complaint, recently filed against Louisi- sary” (following professional test standards ana, is pending as this report is being written.) helps defendants meet their burden of proof here), when students are given sufficient ad- Motivating students to do their best vance notice of high-stakes test requirements, As Education Week recently asked, “what if when tests are judged to be fair measures of the schools gave a test and nobody—or at least what has been taught, and when there are edu- not many of the students taking it—cared?”56 cational supports in place for students who Without consequences based on test results, struggle with the exams.53 older students in particular may not bother to While policy makers and educators must try very hard on new state assessments. Since recognize that tests can and will be challenged how students perform on these assessments if they are used inappropriately, it is also im- has growing consequences for principals and portant to note that the courts and OCR have teachers (see the next section), the question generally not found reason to overturn new has implications for school as well as student test-based promotion and graduation policies. accountability. Unmotivated student test tak- In a widely-watched case, a federal court in ers may produce results that do not accurately Texas in January 2000 rejected a challenge to reflect the quality of education at their schools, the state’s use of the Texas Assessment of Aca- complicating efforts to hold educators account- demic Skills as a requirement for high school able for results. graduation.54 OCR has reached agreements with Policies linking graduation to passing state

24 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable tests, of course, do provide students with some motivation to perform at least satisfactorily. If USING PERFORMANCE MEASURES all that matters, however, is whether the stu- TO REWARD AND SANCTION dent passes or not, students still have little EDUCATORS reason to try to do their best, so that test scores Forty-five states and a number of large city will not accurately reflect their own and their school districts have established school account- school’s performance. ability systems to report on or rate school per- We urge businesses and colleges to signal formance. Test scores are the most frequently that they care about how students perform on measured outcome, sometimes accompanied standards-based tests, as well as in academic by attendance and/or dropout/graduation work more generally, by asking that test scores rates and other information.57 Most account- be included on high school transcripts and by ability systems rely on making information on using the results of these tests along with stu- student outcomes widely available to generate dent grades and judgments about the rigor of pressure for school improvement. A few offer courses taken in hiring and admission deci- financial rewards to educators in schools that sions. We encourage businesses to participate are performing well; some apply sanctions to in the Making Academics Count project, those that are performing poorly. (See box: through which companies pledge to use Accountability Systems Featuring Financial Re- evidence of students’ academic achievement wards.) Sanctions range from additional over- in their entry-level hiring decisions. (See box: sight by district and state education authorities Making Academics Count.) We encourage to penalties such as “reconstitution”: removing states and colleges to consider how standards- based testing and college admissions require- the existing principal and staff and essentially ments might be linked, as is currently being reconstructing the school from scratch. Au- explored in Illinois. (See box: The Prairie State thorities have been given the power to recon- Achievement Examination.) stitute schools in a number of states and districts and have exercised it in such places as San MAKING ACADEMICS COUNT Francisco and Baltimore. Florida has taken another route to sanctioning failing schools: Since 1997, the National Alliance of Busi- providing students with vouchers to attend ness, the Business Roundtable, and the U.S. nonpublic schools if they wish. Chamber of Commerce have led a business- Accountability systems for educators can fo- sponsored nationwide effort to reinforce the cus on individual teachers and principals or on connection between school performance and workplace success. The campaign, Making Aca- the school as a whole. At present, most focus demics Count, intends to raise student achieve- on schools. ment by encouraging employers to ask for Accountability for individual teachers student records when hiring entry-level employ- ees. By doing so, employers can send the mes- New performance-based accountability sys- sage to students that “yes, school counts.” tems for educators generally avoid singling out The initial goal of the campaign was to have individual teachers. The preference for group, at least 10,000 companies of all sizes asking for or school-based, accountability reflects a his- school records and other profiles of academic tory of unsuccessful attempts to implement in- performance as part of the hiring process. The dividual incentives such as “merit pay” in campaign has surpassed that goal. Its sponsors education. While teacher unions (about four- have vowed to increase their efforts and con- fifths of American teachers are unionized) con- tinue raising that number. tinue to oppose linking job performance to SOURCE: National Alliance of Business (2000). pay,58 some innovative experiments involving

25 Measuring What Matters

THE PRAIRIE STATE ACHIEVEMENT EXAMINATION Illinois is working with ACT to “wrap” standards-based test components around the ACT college entrance exam (and components of its Work Keys battery) for multi-score reporting. The Illinois State Board of Education (ISBE) designed the Prairie State Achievement Examination (PSAE) to contain a set of existing tests that measure the Illinois Learning Standards, including tests used for college admis- sions and for measuring workplace skills so students could receive the results of these tests as a bonus. To that end, the PSAE incorporates three separate test components: (1) ISBE developed writing, sci- ence, and social science assessments; (2) the ACT Assessment, which includes English, mathematics, reading, and science reasoning tests; and (3) two Work Keys (WK) assessments, Reading for Informa- tion and Applied Mathematics. The components are combined as shown below:

Component Tests PSAE Test Scores

ACT Reading + WK Reading = PSAE Reading ACT English + ISBE Writing = PSAE Writing ACT Mathematics + WK Mathematics = PSAE Mathematics ACT Science Reasoning + ISBE Science = PSAE Science ISBE Social Science = PSAE Social Science

PSAE scale scores and performance levels are reported for each of the five subjects. In addition, the PSAE generates an ACT Assessment Composite Score, ACT Subject Scores (4 subject scores, 7 sub- scores), and 2 Work Keys Scores. Illinois has worked hard to ensure that colleges will accept ACT As- sessment scores achieved through PSAE state-required testing. The Illinois Student Assistance Commis- sion has confirmed that it will recognize ACT Assessment scores achieved through PSAE testing for use in awarding state scholarships. Colleges and universities have indicated willingness to accept and use ACT Assessment scores reported from “state” testing (as opposed to the national ACT Assessment test). SOURCES: Illinois (1999); Illinois State Board of Education (2000). local union cooperation are underway around districts) they tended to evolve from true merit the country. pay plans, where teachers are rewarded for Merit pay, in the generic sense, is a pay better work, to plans in which teachers are system that links teacher pay to some measure rewarded for taking on more tasks.59 Instead of of performance rather than to the prevailing true pay for performance plans, some districts single salary schedule. Typically, teachers are have recently adopted compensation plans that now paid according to a state or district sched- link pay not to students’ academic achieve- ule that gives higher pay to teachers with more ment but to the acquisition of specific knowl- experience and with advanced degrees. edge and skills. Such training, like that Districts have experimented with various evidenced by advanced certification by the Na- forms of merit pay for individual teachers for a tional Board for Professional Teaching Stan- long time, but seldom have stuck with them for dards, is typically aligned with the requirements long. Merit pay programs have been unpopu- of standards-based reform and is thought to be lar with teachers and have also suffered from effective in promoting student learning.60 high costs and administrative burdens. Districts Pay for knowledge and skills is an improve- and states often failed to provide stable fund- ment on the single salary schedule and may be ing for the programs. Where merit pay pro- a necessary first step in many places toward grams did survive (generally in wealthier pay for performance. It does not, however,

26 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable

ACCOUNTABILITY SYSTEMS FEATURING FINANCIAL REWARDS Some states and districts provide financial different approaches to linking individual teach- rewards to schools and teachers that are tied ers’ compensation to student performance. directly to student performance. For example: Schools can participate in the pilot if 85 percent North Carolina establishes student performance of their teachers approve. Teachers are eligible goals for each school and gives $750 rewards to for up to $1,500 in bonuses. Three different ways all the teachers at schools that meet their goals. of measuring teacher performance will be tested: Teachers at schools exceeding their goals by 10 increases in student performance on standard- percent or more receive $1,500 each. ized tests; increases in student performance on teacher-developed assessments; and increases in Dallas rewards its most effective schools on the teachers’ skills and knowledge. basis of a statistically-sophisticated School Effec- tiveness Index that adjusts test scores and other Lansdale, PA (Colonial School District) began school performance measures to take explicit implementing an achievement award program in account of differences among students and school year 1999-2000 that rewards both indi- among schools. Adjusted index scores are com- vidual teachers and groups of teachers within a pared across schools, with the top schools (the school. Individual teachers are ranked on how specific number dependent on fund availability) their students’ academic achievement compares receiving rewards. All professional staff in to those of a comparison group of teachers. selected schools receive awards of $1,000, with Group achievement awards are granted based on support staff receiving $500 and the school the success of schools in meeting agreed-upon receiving $2,000 for its activity fund. annual goals. Individual awards range from $1,389 to $2,778, while the maximum award an California enacted a law in July 1999 that pro- individual teacher can receive under the group vides a one-time performance bonus award of up to $25,000 per teacher in underachieving award program is $2,500. schools that improve their Academic Perfor- Maryland began offering monetary bonuses in mance Index (API) by at least 2 times the 1996 to schools showing significant and sus- school’s annual growth target. The California tained improvement on the state assessment. Certificated Staff Performance Incentive Act Schools that show two years of statistically signifi- provides bonuses of $25,000 to 1,000 teachers in cant progress on a School Performance Index schools with the largest absolute API gain, may be eligible for the reward. School Improve- $10,000 to 3,750 teachers in schools with the ment Teams are the recipients of the awards and second-largest gain, and $5,000 to 7,500 teachers must use them on educational programs, but not in schools with the third-largest gain. Bonus on staff salary or bonuses. The amount of each distribution is determined by negotiation be- school’s award depends on the number of tween the local governing board and the teach- schools qualifying for the awards and the enroll- ers’ union. ment of each school receiving an award. In 2000, Denver teachers agreed in late 1999 to a four- the average financial award was $49,000 per year pilot program that will investigate three school. create accountability for individual teachers, operatively in designing and evaluating these which we believe should be the ultimate objec- new arrangements. Teacher engagement is im- tive. portant, for both substantive and political rea- We encourage districts to undertake more sons. But we believe teachers must become aggressive experimentation with new pay and more receptive to these plans than has histori- bonus plans that link individual teachers’ com- cally been the case. Their concerns about pos- pensation directly with their ability to foster sible negative effects are not unfounded, but student learning and that engage teachers co- neither are they unique to merit pay arrange-

27 Measuring What Matters ments in educational settings. Moreover, pay rooms. If used effectively, school-level account- for most workers outside education is based at ability can motivate changes in teaching more least in part on merit considerations. Teachers directly than accountability diffused across cannot in our view continue to exempt them- many schools or districts. It should make it selves from accountability for outcomes and easier to see which schools are performing well expect public understanding and support.* and which are not. We acknowledge, though, that there is still Using tests to hold schools accountable, however, much to learn about how to design perfor- heightens the importance of the issues regarding the mance-based compensation plans for individual use of tests we have discussed in earlier chapters, teachers and about their effects. Merit pay plans especially issues of fairness that are politically and have tended to come and go rather quickly technically complex. How should accountability and have not been carefully evaluated.61 While systems deal with the well-established fact that some states (such as Florida and Delaware) many factors besides school influence a have mandated pay for performance, they have student’s academic performance at any given often left the details to be worked out through time: factors such as prior educational experi- processes that have not yet taken place. Only a ences, family background, and nonschool edu- few districts have adopted or are experiment- cational opportunities? Are there differences ing with performance-based compensation sys- in the stability and reliability of different mea- tems. Linking teacher performance to student sures of school performance? How should the performance requires state and district systems fact that student performance varies more that both measure student achievement and within individual schools than it does among link students to the teachers who have taught schools affect the design of accountability rat- them. As we’ve already seen, most schools can- ings and rankings? not now make such linkages for teachers in Accountability systems today typically rely many grade levels and subjects. Therefore, we on average test scores or percentage pass rates believe that careful experimentation and evalu- to measure school performance. Many analysts ation rather than a head-long rush to adopt are critical of this approach, because such “sta- new pay-for-performance programs for indi- tus” indicators reflect many factors affecting vidual teachers is the right approach at this achievement other than those controlled by stage. the school. Such indicators can therefore be misleading about what the school contributes Accountability for schools to student achievement, failing to show how The growing popularity of school-based ac- well the school actually educates its students. countability marks an important step on the The path-breaking Texas accountability system road to a performance-based education sys- partially addresses these concerns by basing its tem. Holding the educators in a school collec- school ratings not just on the average test pass tively responsible for students’ academic rate in each school, but also on the pass rates achievement and its growth emphasizes out- of selected subgroups. (See box: The Texas comes, not inputs, and puts the locus of ac- Approach to Accountability.) The state also countability at the school level for the first issues a “comparative improvement” report that time. Historically, very little information was shows each school how much year-to-year im- collected about individual schools; most of what provement its students have achieved compared we knew about education was based on district to 40 other schools serving similar populations. or state-level data. More elaborate “value-added” accountabil- Moving accountability to the school level is ity models based on improvements in student important because it directly affects those who performance are sometimes used to avoid the actually interact everyday with students in class- problems caused by using average or status

*See memorandum by CAROL J. PARRY (page 44). 28 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable

schools, such as financial resources, that are THE TEXAS APPROACH TO not under the school’s control. Unfortunately, ACCOUNTABILITY the information needed to make these adjust- The Texas Education Agency rates every ments is not typically available. States and dis- district and every school within a district as tricts instead have developed less complete either Exemplary, Recognized, Academically models that use prior test scores in various Acceptable/Acceptable, or Academically Unac- ways to determine how much of the change in ceptable/Low-Performing, based on three student performance should be attributed to indicators: (1) student scores on the Texas the school’s efforts.63 While not widely used, Assessment of Academic Skills (TAAS) tests in such value-added accountability systems have reading, writing, and mathematics, (2) student been tried in several states—North and South dropout rates, and (3) student attendance Carolina and Tennessee—and districts— rates. To assign ratings, the agency breaks down each district and school into four student Dallas and Minneapolis. subgroups – African American, Hispanic, An ideal scoring system designed to gauge white, and economically disadvantaged. For a educational improvement would measure the school or district to receive a specific rating, progress made by individual students over time. not only must the school/district as a whole Often, however, states and districts do not have meet the specified criteria, so too must each of administrative systems that can link individual the four student subgroups. For example, to students and their test scores from year to year receive a rating of Exemplary: and must use other approaches. For example, • At least 90% of all students must pass each they may look at the gains achieved by the TAAS subject area test, and at least 90% of same cohort of students over time, even though each subgroup of students must also pass the population in the cohort is not exactly the the tests. same (because of student mobility, for ex- • The dropout rate cannot exceed 1% for all ample). students, or 1% for any student subgroup. Different measures of school performance • There must be at least a 94% attendance may give very different results. A recent study rate for all students and for all student sub- compared nine different measures developed groups. from the scores of fifth graders in South Caro- lina schools, all based on their standardized To be rated at a specific level, each student 64 subgroup as well as the total student popula- test scores. These measures included several tion must meet the minimum criteria for that variants of average scores and value-added level. A school or district can only be ranked as scores, some adjusting for nonschool factors high as its lowest performing subgroup of stu- such as socioeconomic status and race and oth- dents would be ranked. ers not. There were substantial differences in SOURCE: Texas Education Agency (1999). rankings depending on the measure employed. Schools with more white and relatively affluent students tended to score highly using average scores in school accountability programs.62 They scores, but this advantage almost disappeared report scores in terms of the growth in student in most cases when value-added scores were achievement rather than the absolute level of used. achievement and attempt to identify the There are important tradeoffs to be consid- amount of improvement for which the school ered in deciding how much information to is responsible. Ideally, statistical procedures include in accountability systems. Value-added would be used to eliminate the effects of (1) measurements that track individual students nonschool factors that contribute to the growth and provide information on student progress in achievement and (2) differences among that can be linked back to individual teachers

29 Measuring What Matters and classrooms demand costly new commit- substantially changed school rankings. These ments by districts and states. They require that corrections raised the effectiveness measure tests be given more frequently than is now for many schools with low average test scores typically the case: every year or every other year and reduced it for many schools with high instead of in just three or four grades. They average scores. also require new data systems that combine Another study used the North Carolina data student test scores with information on stu- from a five-year period to determine the stabil- dent, family, and community characteristics and ity of school rankings over time.69 It found that that link students and their teachers.65 To keep schools at both the top and the bottom of the costs down, administrators may be tempted to annual relative rankings tended to be small re-use tests rather than develop new ones and schools, even though the average performance use inexpensive tests that are less effective than of small and large schools was similar. It ap- those envisioned by standards-based reform- pears that the annual rankings of schools over ers.66 time are quite unstable and that the measured Because test-based accountability is intended to annual differences may not indicate real differ- create incentives for schools to improve their perfor- ences in school performance.70 mance, it is important to get the incentives right. These studies indicate that the design of Even the most sophisticated of the value-added accountability systems is critical to their effec- accountability models currently in use cannot tiveness and credibility and underscore the definitively identify the effects on achievement need for improved understanding of the ef- that are under the control of the educators in fects of specific design features. the school. These models are nevertheless use- Continuous improvement is essential to make ful because they provide important informa- accountability work. The higher the stakes attached tion to parents, citizens, and policy makers. to accountability, the more crucial it is that measure- Poor performance draws attention to schools ment systems be reliable, valid, fair, and comprehen- where changes are required to make them more sible. We have much to learn about designing effective. accountability systems that accomplish our goals. To When test-based accountability is used as this end, we urge accountability supporters to the basis for rewards and sanctions for specific promote investment in research on and analy- schools and their personnel, however, the ac- sis of such systems. curacy and reliability of these models become far more important. A model that incorrectly labels a school as “low-performing” for example, GUIDELINES FOR GOOD could lead high-quality principals and teachers PRACTICE to shun employment there in favor of “high- Robert Linn, one of the country’s foremost performing” schools that promise better re- experts on testing and assessment, has recently wards. offered seven suggestions for “enhancing the There is evidence urging caution in this validity, credibility, and positive impact of as- regard. Recent research examined accountabil- sessment and accountability systems while mini- ity models in North Carolina, which rated mizing their negative effects.”71 They capture schools serving students with high average well the implications of our discussion of using scores as also more effective in improving an- achievement measures to hold students and nual performance than schools serving students educators accountable. We commend the fol- with low scores.68 Detailed analysis indicated lowing suggestions to business leaders and other that correcting for measurement error and (and citizens as guidelines to good practice. We to a lesser extent, for a misspecified model) should insist that accountability systems:

30 Chapter 4: Using Achievement Measures to Hold Students and Educators Accountable

1. Provide safeguards against the selective ex- clusion of students from assessments; e.g., by including all students in accountability calculations.72 2. Utilize new high-quality assessments each year that are statistically equated to those of previous years. 3. Not put all of the weight on a single test; instead, seek multiple indicators. 4. Place more emphasis on comparisons of school performance from year to year than from school to school. This allows for dif- ferences in starting points while maintain- ing an expectation of improvement for all. 5. Consider both value added and status in the system. Value added provides schools that start out far from the mark a reasonable chance to show improvement while status guards against institutionalizing low expec- tations for those same students and schools. 6. Recognize, evaluate, and report the degree of uncertainty in the reported results. 7. Put in place a system for evaluating the effects of the system: positive and negative, intended and unintended.

31 Chapter 5 MONITORING EDUCATIONAL PROGRESS AND THE MEASUREMENT SYSTEM ITSELF

Besides helping improve instruction for in- tests to each other. However, the National dividual students and providing a means for Research Council studied this issue in response holding students and educators accountable, to a congressional mandate and concluded that measurement of academic performance also creating linkages that would permit multiple provides information about the status of the commercial and state achievement tests to be education system. Shifting our focus from in- compared to one another is not feasible.73 structional improvement and accountability to Thus, the growth in state and district testing does monitoring raises three additional issues that not diminish the importance of continuing support should be part of the discussion of educational for the National Assessment of Educational Progress measurement: (1) the importance of continu- (NAEP) and for American participation in interna- ing support for national and international tional assessments. assessments to provide some common metrics For 30 years NAEP has served as “the nation’s in our otherwise fragmented assessment sys- report card,” providing a continuing measure tem; (2) the need to expand and improve ef- of student achievement in key subject areas. forts to make information about the NAEP periodically tests a sample of students in performance of the education system available three grades, one each in elementary, middle, to the public; and (3) the desirability, given the and high school. For two decades, scores were growing consequences of testing, to build insti- reported on a national basis only. For the last tutions capable of monitoring the assessment ten years, scores have also been reported on a system itself. statewide basis for those states that choose to participate. NAEP thus serves not only as a national barometer of educational progress, NATIONAL AND INTERNATIONAL but also as a check on states’ progress as mea- ASSESSMENTS sured by their own assessments. Though we applaud the progress that has State participation in NAEP is voluntary. been made over the last decade by states and Forty-eight of the fifty states signed agreements school districts in developing standards, assess- to participate in the NAEP 2000 mathematics ments, and accountability systems, we also ob- and science assessments, up from 37 participat- serve that this decentralized approach to ing states when the state assessment program standards-based reform does not provide data began in 1990. With the growth in other test- with which to compare performance across ing programs, however, schools are apparently states, nor does it provide an overall picture of finding it increasingly difficult to allocate the the health of the American educational sys- time and resources to participate in NAEP. tem. Some have hoped that it would be pos- Eight of the states that originally committed to sible to link different state and commercial NAEP 2000 had to drop out because too few

32 Chapter 5: Monitoring Educational Progress and the Measurement System Itself schools agreed to participate to meet statistical sampling requirements. Participation in state REPORTING TO THE PUBLIC NAEP requires significant commitments of Though 45 states reported in 2000 that they time, resources, and effort at both the district planned to issue report cards on schools, only and local levels. 32 percent of parents reported that such re- Over the next few months, the NAEP gov- port cards were available.75 This discrepancy erning board will be considering recommen- suggests that there is still much to do to get perfor- dations74 from an Ad Hoc Committee on NAEP mance data into the public’s hands and help parents Participation to create incentives for schools to and others learn how to use it. participate. Some would involve new costs (e.g., Earlier in this report we talked about put- using paid contractors rather than school ad- ting performance data into the hands of edu- ministrators to administer the versions of NAEP cators so that data become tools of instructional that result in state-specific scores, as is already improvement. Data for parents and the public done in the version of NAEP that results in serve different needs: monitoring the perfor- national scores), and some would require con- mance of schools and (where school choice is gressional action (e.g., providing school- permitted) providing information upon which specific scores for schools who want them). We parents can decide where to enroll their chil- urge the NAEP board to adopt policies aimed dren. at improving school participation and urge Con- In this report we cannot give adequate at- gress to approve these changes, including bud- tention to the full range of issues, including getary increases if necessary. how to select and format information to meet International assessments such as the Third public demands as well as to reflect school International Mathematics and Science Study performance accurately, that must be addressed 76 (TIMSS—conducted in the mid-1990s and re- in the design of school report cards. As cently updated) supplement NAEP by provid- potential consumers, however, we recommend ing additional benchmarks against which to that report cards include the following fea- measure the performance of American stu- tures: dents. TIMSS taught the United States impor- • Common formats across districts. While tant and sometimes sobering lessons, such as states may require districts to prepare school that even the best of our high school seniors report cards and may specify the informa- do not perform as well in math and science as tion that must be included, they do not their counterparts in other countries. The fed- necessarily require that the information be eral government has been a key supporter of reported in common formats. Different for- international assessments, funding U.S. par- mats make it hard to pull some useful infor- ticipation and providing essential financial as- mation from report cards, such as whether sistance and design advice to the international schools in different districts are equally suc- coordinating agencies. Future international cessful in the rate at which their students testing depends on this kind of support, along are meeting academic standards. School re- with the continued willingness of schools to port cards should include comparable in- participate. We urge Congress to provide suffi- formation about the number of students who have been excluded from the testing cient funding to the National Center for Edu- program or whose scores have been omit- cation Statistics to support international ted from accountability calculations. assessments of student achievement, and we encourage American schools to accept when • Contextual information. Most school report invited to participate. cards still present test score averages, which

33 Measuring What Matters

reflect what students know but not neces- error becomes more significant; so reports sarily how well schools are performing.77 As should include information on the margins we explained in Chapter 4, test scores are of error associated with the data. highly correlated with students’ socioeco- • Explanations of measures. To make aca- nomic and ethnic status and reflect out-of- demic achievement data more meaningful school as well as in-school experiences. It is to parents and others, numerical scores are therefore helpful in interpreting individual often given labels that represent levels of school results to have background informa- accomplishment: e.g., basic, novice, profi- tion about the students enrolled. Other cient, and advanced. The labels applied to kinds of contextual information can also be different tests may sound similar when in useful: e.g., student mobility rates (in some fact they might mean quite different things. urban schools, a majority of students move The solution is to include clear explana- in or out in the course of a single year), tions of these labels on report cards. school and class size, per-pupil spending, • “School success” and school environment and the number of children who are data. The point of report cards is that they English-language learners. While focus be used. In this regard, it is significant that group research has suggested that the pub- the public may have different views about lic rates demographic data low on its list of what information should be included in re- desirable information about schools,78 we port cards than other groups for whom such support the inclusion of this information reports are designed. For example, research and urge the public to make use of the in two cities (Greensboro, NC and Sacra- more nuanced picture of school perfor- mento, CA) found that school board mem- mance it provides. bers differed from parents in their informa- • Disaggregated data. Whether achievement tion priorities. Standardized test scores were data are reported as averages or as some far more important for school board mem- measure of improvement, it will be more bers than for parents, who preferred school informative if they are disaggregated for success and school environment data particular groups of students and not just (graduation and promotion rates being reported as school totals. Since the goal of examples of the former, school safety and standards-based reform is to have all chil- parental involvement indicators being dren achieving at high levels, it is important examples of the latter).79 While we believe to know whether schools are boosting the that parents should pay attention to direct performance of all their students. Report- measures of academic achievement and ing results for different subjects and differ- should have ready access to this informa- ent grade levels by student groups (e.g., tion, we also think it important to acknowl- African Americans, Hispanics, white, and edge other parental concerns and include economically-disadvantaged students as is them in school reports. done in Texas) may be too cumbersome for • Review and continuous improvement. Like summary report cards routinely distributed standards and assessments, school “report to all parents. The data should be readily cards” should periodically be reviewed and available to anyone who wants them, how- improved based on experience with their ever, and should be considered by the press usefulness to the public and on any unex- and others who report on school perfor- pected effects on parent or school behav- mance. Since subgroup reporting reduces ior. For example, Maryland State Superin- the number of students in each group, rela- tendent Nancy S. Grasmick reports that state- tive to the school as a whole, measurement wide groups met for months in 1990 to

34 Chapter 5: Monitoring Educational Progress and the Measurement System Itself

hammer out the details of a public report- tions of state standards. They use different ing mechanism that would include test re- criteria, so that their ratings vary. By mak- sults. From an initial listing of 100 possible ing their criteria transparent, however, they report card measures, 13 were initially cho- foster informed discussion about the con- sen for inclusion, along with data to provide tent of standards and whether they are mea- the context for viewing the accountability suring what the public believes to be impor- measure. A five-year review cycle was estab- tant. lished. After the 1995 review, the indicator • Achieve, Inc.—was founded by governors and for student promotion rate was pulled from business leaders after the 1996 Education the report because reviewers found it had Summit to promote standards, assessments, become a compelling incentive for social 80 and accountability as means to school im- promotion. provement. Achieve provides information on state standards and on state progress in MONITORING THE implementing standards-based reform. At MEASUREMENT SYSTEM state request, Achieve conducts indepen- dent evaluations of state standards and tests. Independent organizations that stand out- Achieve and 10 states have established the side the everyday fray of education politics and Mathematics Achievement Partnership to policy making can make important contribu- develop an internationally-benchmarked 8th tions to improving the design and use of assess- grade math assessment which will give par- ment and accountability systems. They can ticipating states a common metric for com- monitor and report on the quality of educa- paring the performance of their students. tional standards and assessments, on the in- tended and unintended consequences of • Consortium on Chicago School Research—con- accountability systems, and on the meaning of ducts research activities designed to advance test scores and test score changes over time. school improvement in Chicago’s public Such groups, which may choose to focus on schools and to assess the progress of school national, state-wide, or district issues, can serve reform. It is an independent, university- a tremendously useful function in keeping based organization sponsored jointly by the policy makers and educators honest about what city’s foundations and the public school sys- education reform efforts are accomplishing. tem central office. The central office pro- We encourage support from foundations, vides test scores and access to the schools businesses, and other funding sources for such for consortium surveys, but the researchers organizations. We recommend that policy mak- conduct their analyses according to profes- ers seek out the services of these groups and sional standards and make the ultimate de- help create them where they do not exist. We cisions about what will be published and urge the public to expect policy makers to when. subject assessment and accountability systems • Maryland Assessment Research Center for Edu- to independent evaluation. The following or- cational Success—a research arm of the Uni- ganizations currently provide the kind of ser- versity of Maryland Department of Measure- vices we envision; the state and local ones can ment, Statistics, and Evaluation, the center serve as models for replication. conducts basic and applied research to • The American Federation of Teachers, the enhance the quality of assessment practice Council for Basic Education, and the and knowledge. The state Department of Thomas B. Fordham Foundation—these Education has contracted with the center to organizations provide independent evalua- provide assessment support for the Mary-

35 Measuring What Matters

land State Performance Assessment sions aimed at building a fair system and Program. helps legitimate the resulting system (whose • Dallas Accountability Task Force—the complex inner workings are “less than trans- 81 Dallas Public School System has created its parent” to outsiders) in the eyes of par- own accountability program operating on ents, teachers, and the public. top of the Texas state accountability system. These and similar groups help spur the con- Dallas employs a statistically-sophisticated tinuous improvement of assessment and ac- value-added model for calculating school countability systems that we call for in this effectiveness and allocating performance- statement. They also serve as safeguards against based awards. The district established an the possibility that undesirable effects of these Accountability Task Force, including teach- systems will go unrecognized or unaddressed. ers, administrators, members of the school The information they put into the hands of the board, and community representatives, to public better equips all of us to engage in our oversee the design of the system and to democratic society’s constant debate over the review and revise it as needed. The task goals of American education and how best to force helps make difficult technical deci- reach them.

36 ENDNOTES

1. This felicitous phrase was used by Judge Lynn N. dards are developed jointly by the American Educational Hughes, who cited “the buffeting of letters, press confer- Research Association, the American Psychological ences, speeches, meetings, and the rest of the wonderful Association, and the National Council on Measurement cacophony of a free people disagreeing” in rejecting a in Education. They address professional and technical complaint that the Houston (TX) Independent School issues of test development and use in education, psychol- District board had denied segments of the community ogy, and employment. the right to speak before selecting a new school superin- 19. National Research Council (1999a:250). tendent. Quoted in McAdams (2000:120). 20. National Research Council (1999d:43-4). 2. Committee for Economic Development (1994). 21. National Research Council (1999a:45). 3. Committee for Economic Development (1985, 1987). 22. National Research Council (1999a:32). 4. Linn (2000). 23. Herrnstein and Murray (1994). 5. The national education goals, as adopted at the 1989 Education Summit and later modified by Congress, 24. National Research Council (1999a:32). essentially state that by the year 2000: 25. Citizens’ Commission on Civil Rights (1998:10). All children will start school ready to learn. 26. U.S. Department of Education (1999a:Table 53). The high school graduation rate will increase to at least 27. U.S. Department of Education (1999a:Table 59). 90 percent. 28. Improving America’s Schools Act of 1994, Public Law All students will become competent in challenging 103-382; Individuals with Disabilities Education Act subject matter. Amendments of 1997, Public Law 105-17. Teachers will have the knowledge and skills that they 29. National Research Council (1999d). need. 30. U.S. Department of Education (1999b:46). U.S. students will be first in the world in mathematics 31. Maryland State Department of Education (2000). and science achievement. 32. Belden Russonello & Stewart and Research/Policy/ Every adult American will be literate. Management (2000). Schools will be safe, disciplined, and free of guns, drugs, 33. Duggan and Holmes (2000:4-5). and alcohol. 34. National Research Council (1999d:49). Schools will promote parental involvement and partici- pation. 35. Phelps (1999:24). 6. Expressed in Committee for Economic Development 36. Koretz (1996:186). (1994). 37. See, for example, Goertz (2000:79) and National 7. Committee for Economic Development (1994:37). Research Council (1999d). 8. National Research Council (1999c:22). 38. National Research Council (1999b). 9. National Research Council (1999d:25). 39. Goertz (2000:68). 10. National Research Council (1999d:25). 40. National Research Council (1999d:19-20). 11. Hill (2000:3). 41. National Center for Education Statistics (1999:51). 12. National Research Council (1999a:29). 42. Committee for Economic Development (1994:14). 13. Phelps (1999:25). 43. National Research Council (1999d:81). A new survey of teachers by Education Week provides additional evi- 14. Two RAND studies, cited in National Research Coun- dence that teachers receive little training on understand- cil (1999d:81). ing and using academic standards and test results. See 15. Klein and Hamilton (1999:13). Education Week (2001:45). 16. National Research Council (1999a:30). 44. Hoff (2000). 17. National Research Council (1999d:66). 45. Finn et al. (1999:46). 18. American Psychological Association (1999). The stan- 46. Finn et al. (1999:44).

37 47. Porter (2000). 65. Meyer (2000:4). 48. National Research Council (1999a:3). 66. Linn (2000:13). 49. Olson (2000). 67. Ladd and Walsh (forthcoming). 50. Research on the connection between grade retention 68. The South Carolina findings are based on the and drop-out rates is summarized in National Research accountability system that was put in place in the mid- Council (1999a:128-132). 1980s and may not reflect the state’s current system. 51. National Commission on Testing and Public Policy, 69. Kane (n.d.:1-2). From GATEKEEPER to GATEWAY: Transforming Testing in 70. Other research does suggest that small schools are America, 1990, quoted in National Research Council more effective than large ones. The study under discus- (1999a:251). sion was not designed to investigate this point, but rather 52. Robelen (2000:14). to explain why the highest and lowest ranked schools in 53. National Research Council (1999a:Chapter 3); North Carolina were virtually all small schools. Robelen (2000:14). 71. Linn (2000:15). 54. GI Forum v. Texas Education Agency, 87 F.Supp.2d 667, 72. The one exception is that it is appropriate to exclude 142 Ed. Law Rep. 907 (D. TX 2000). from a school-based accountability system students who 55. Robelen (2000:14). The Office for Civil Rights has have not spent some minimum number of days in the just published a resource guide to help educators and school. policy makers develop and implement test-use policies 73. National Research Council (1999e). consistent with legal requirements. See Office for Civil 74. Ad Hoc Committee on NAEP Participation (2000). Rights (2000). 75. See Education Week (2001) for state responses; Public 56. Keller (2000). Agenda (2000a) for parental responses. 57. Education Week (2001:80-81). 76. Gormley and Weimer (1999). 58. For example, at their 2000 convention, National 77. Gormley and Weimer (1999). Education Association members rejected a plan to use job performance evaluations in paying wage bonuses. See 78. A-Plus Communications (1999:5-6). Archer (2000). 79. Jaeger, Richard, Barbara Gorney, Robert Johnson, 59. Research on merit pay plans is summarized in Sarah Putnam, and Gary Williamson, A Consumer Report National Research Council (1999c:177). on School Report Cards. Greensboro, NC: Center for Educa- tional Research and Evaluation, University of North 60. Odden and Kelley (1997). Carolina at Greensboro (1994), cited in Gormley and 61. Hanushek (1996:131). Weimer (1999:107). 62. Clotfelter and Ladd (1996); Meyer (1996, 2000). 80. Grasmick (2000:54). 63. Ladd and Walsh (forthcoming). 81. Mendro et al. (1999). 64. Clotfelter and Ladd (1996:37).

38 REFERENCES

Ad Hoc Committee on NAEP Participation Duggan, Terri, and Madelyn Holmes, eds. 2000 Enhancing Participation in the National Assessment 2000 Closing the Gap: Report on the Wingspread Confer- of Educational Progress. Washington, DC: ence “Beyond the Standards Horse Race: Implemen- National Assessment Governing Board, tation, Assessment, and Accountability – The Keys to September. Improving Student Achievement.” Washington, DC: Council for Basic Education. A-Plus Communications 1999 Reporting Results: What the Public Wants to Education Week Know. A companion report to Education Week’s 2000 Quality Counts 2000: Who Should Teach? Quality Counts ’99. Education Week 19(18), January 13. American Psychological Association Education Week 1999 Standards for Educational and Psychological Test- 2001 Quality Counts: A Better Balance—Standards, ing. Washington, DC: American Psychological Tests, and the Tools to Succeed. Education Week Association. 20(17), January 11. Archer, Jeff Finn, Chester E., Jr., Marci Kanstoroom, and Michael J. 2000 NEA Delegates Take Hard Line Against Pay for Petrilli Performance. Education Week 19(42), July 12. 1999 The Quest for Better Teachers: Grading the States. Belden Russonello & Stewart and Research/Strategy/ Washington, DC: The Thomas B. Fordham Management Foundation. 2000 Raising Education Standards and Assessing the Glidden, Heidi Results. Results of a Public Opinion Survey 1999 Making Standards Matter 1999. [Online]. Conducted for the Business Roundtable, Available: http://www.aft.org/edissues/ August. standards99/index.htm. [October 25, 2000]. Citizens’ Commission on Civil Rights Washington, DC: American Federation of Teachers. 1998 Executive Summary. In Title I in Midstream: The Fight to Improve School for Poor Kids. Goertz, Margaret E. Washington, DC: Citizens’ Commission on 2000 Implementing Standards-Based Reform: Civil Rights. Challenges for State Policy. In Terri Duggan Clotfelter, Charles T., and Helen F. Ladd and Madelyn Holmes, eds., Closing the Gap: Report on the Wingspread Conference “Beyond 1996 Recognizing and Rewarding Success in School. the Standards Horse Race: Implementation, In Helen F. Ladd, ed., Holding Schools Account- Assessment, and Accountability – The Keys to able: Performance-Based Reform in Education. Improving Student Achievement.” Washington, Washington, DC: Brookings Institution Press. DC: Council for Basic Education. Committee for Economic Development Gormley, William T., Jr., and David L. Weimer 1985 Investing in Our Children: Business and the Public 1999 Organizational Report Cards. Cambridge, MA: Schools. New York, NY: Committee for Harvard University Press. Economic Development. Grasmick, Nancy S. Committee for Economic Development 2000 Looking Back at a Decade of Reform: The 1987 Children in Need: Investment Strategies for the Maryland Standards Story. In Terri Duggan Educationally Disadvantaged. New York, NY: and Madelyn Holmes, eds., Closing the Gap: Committee for Economic Development. Report on the Wingspread Conference “Beyond the Committee for Economic Development Standards Horse Race: Implementation, Assessment, and Accountability – The Keys to Improving Student 1994 Putting Learning First: Governing and Managing Achievement.” Washington, DC: Council for the Schools for High Achievement. New York, NY: Basic Education. Committee for Economic Development.

39 Grissmer, David, and Ann Flanagan Ladd, Helen F., and Randall P. Walsh 1998 Exploring Rapid Achievement Gains in North Forthcoming Implementing Value-Added Measures Carolina and Texas. Washington, DC: National of School Effectiveness: Getting the Education Goals Panel, November. Incentives Right. Economics of Education Review. Hanushek, Eric A. Linn, Robert L. 1996 Comments on Chapters Two, Three, and Four. In Helen F. Ladd, ed., Holding Schools Account- 1999 Assessments and Accountability. Educational able: Performance-Based Reform in Education. Researcher 29(2): 4-16, March. Washington, DC: Brookings Institution Press. Maryland State Department of Education Herrnstein, Richard J., and Charles Murray 2000 State Board Moves Ahead With Assessments. 1994 The Bell Curve: Intelligence and Class Structure in News Release, May 24. American Life. New York: Free Press. McAdams, Donald R. Hill, Paul T. 2000 Fighting to Save Our Urban Schools…and 2000 The Changing State Role. Basic Education Winning! Lessons from Houston. New York, NY: 44(8), School Governance. Washington, DC: Teachers College Press. Council for Basic Education, June. Mendro, Bob, Ruben Olivarez, Maureen Peters, Tony Hoff, David J. Milanowski, Eileen Kellor, and Allan Odden 2000 Testing’s Ups and Downs Predictable. 1999 A Case Study of the Dallas Public School-Based Education Week 19(20), January 26. Performance Award Program. Madison, WI: University of Wisconsin, Wisconsin Center for Illinois Education Research, Consortium for Policy 1998 Illinois’ Response to the 1999 Action State- Research in Education. ment. [Online]. Available: http: Meyer, Robert H. www.achieve.org/achieve/achievestart.nsf states/Illinois [October 25, 2000]. 1996 Value-Added Indicators of School Perfor- mance. In National Research Council, Illinois State Board of Education Improving America’s Schools: The Role of Incentives. 2000 Illinois State Board of Education (ISBE) Washington, DC: National Academy Press. Prairie State Achievement Examination Meyer, Robert H. (PSAE) Questions and Answers. [Online]. Available: http://www.isbe.state.il.us/isat 2000 Value-Added Indicators: A Powerful Tool for QandA.htm. [September 2000]. Evaluating Science and Mathematics Programs and Policies. NISE Brief (National Institute for Kane, Thomas Science Education) 3(3), June. No Improving K-12 School Accountability Systems. National Alliance of Business Date Cambridge, MA: Harvard University, unpublished paper. 2000 Making Academics Count. [Online]. Available: http://www.bcer.org/macc/. [October 31, Keller, Bess 2000]. 2000 Incentives for Test-Takers Run the Gamut. National Center for Education Statistics Education Week 19(34), May 3. 1999 Teacher Quality: A Report on the Preparation and Klein, Stephen P., and Laura Hamilton Qualifications of Public School Teachers. 1999 Large-Scale Testing: Current Practices and New Washington, DC: US Department of Directions. Santa Monica, CA: RAND Education. Education, Office of Educational Research and Improvement, NCES 1999-080. Koretz, Daniel National Research Council 1996 Using Student Assessments for Educational Accountability. In National Research Council, 1999a High Stakes: Testing for Tracking, Promotion, and Improving America’s Schools: The Role of Incentives. Graduation. Committee on Appropriate Test Washington, DC: National Academy Press. Use, Jay P. Heubert and Robert M. Hauser, eds. Commission on Behavioral and Social

40 Sciences and Education. Washington, DC: Porter, Andrew National Academy Press. 2000 Doing High-Stakes Assessment Right. The National Research Council School Administrator 57(11), December. 1999b How People Learn: Brain, Mind, Experience, and Public Agenda School. Committee on Developments in the 2000a Reality Check 2000. Conducted in association Science of Learning, John D. Bransford, Ann with Education Week and funded by the Pew L. Brown, and Rod R. Cocking, eds. Commis- Charitable Trusts and the GE Fund. Education sion on Behavioral and Social Sciences and Week 19(23), February 16. Education. Washington, DC: National Academy Press. Public Agenda 2000b Understanding the Issue: Quick Takes - National Research Council Education: People’s Chief Concerns. [Online]. 1999c Making Money Matter: Financing America’s Available: http://www.publicagenda.org/issues Schools. Committee on Education Finance, angles.cfm?issue_type=education. [October 25, Helen F. Ladd and Janet S. Hansen, eds. 2000]. Commission on Behavioral and Social Sciences and Education. Washington, DC: National Robelen, Erik W. Academy Press. 2000 Parents Seek Civil Rights Probe of High-Stakes Tests in La. Education Week 20(06), October 11. National Research Council 1999d Testing, Teaching, and Learning. Committee on Roderick, Melissa, Anthony S. Bryk, Brian A. Jacob, Title I Testing and Assessment, Richard F. John Q. Easton, and Elaine Allensworth Elmore and Robert Rothman, eds. Board on 2000 Ending Social Promotion: Results From the First Testing and Assessment, Commission on Two Years. Chicago, IL: Consortium on Behavioral and Social Sciences and Education. Chicago School Research, December. Washington, DC: National Academy Press. Roderick, Melissa, Jenny Nagaoka, Jen Bacon, and John National Research Council Q. Easton 1999e Uncommon Measures: Equivalence and Linkage 2000 Update: Ending Social Promotion. Chicago, IL: Among Educational Tests. Committee on Consortium on Chicago School Research, Equivalency and Linkage of Educational Tests, Charting Reform in Chicago Series: Data Brief, Michael J. Feuer, Paul W. Holland, Bert F. September. Green, Meryl W. Bertenthal, and F. Cadell Hemphill, eds. Commission on Behavioral and Texas Education Agency Social Sciences and Education. Washington, 1999 1999 Accountability Manual: The 1999 DC: National Academy Press. Accountability Rating System for Texas Public Schools and School Districts and Blueprint for Odden, Allan and Carolyn Kelley the 2000-2003 Accountability Systems. 1997 Paying Teachers for What They Know and Do: New [Online]. Available: http:/www.tea.state.tx.us and Smarter Compensation Strategies to Improve perfreport/account/99/manual/. [April, 1999]. Schools. Thousand Oaks, CA: Corwin Press. U.S. Department of Education Office for Civil Rights 1999a Digest of Education Statistics, 1998. Washington, 2000 The Use of Tests As Part of High-Stakes DC: National Center for Education Statistics, Decisions for Students: A Resource Guide for U.S. Department of Education. Educators and Policy-Makers. [Online] Available: http://www.ed.gov/offices/ocr/testing/. U.S. Department of Education [December 2000]. 1999b Promising Results, Continuing Challenges: The Final Report of the National Assessment of Title I. Olson, Lynn Washington, DC: Office of the Under 2000 States Ponder New Forms of Diploma. Secretary, Planning and Evaluation Service, Education Week 19(41), June 21. U.S. Department of Education. Phelps, Richard P. 1999 Why Testing Experts Hate Testing. Washington, DC: Thomas B. Fordham Foundation, January.

41 MEMORANDA OF COMMENT, RESERVATION, OR DISSENT

On the report as a whole, WILLIAM E. our neglect of poor children’s schooling in BROCK, with which ALAN BELZER and inner cities, and if our K-12 schools in general PATRICK W. GROSS have asked to be do not measure up to the standard of interna- associated tional competition, we must broaden our view of solutions to encompass a redefinition of This report is an outstanding contribution public education which includes all schools to our continuing national conversation on that educate the public at public expense, pro- the subject of using assessment and account- vided they are subject to public accountability ability to improve learning. and curricular standards. I remain concerned, however, that these Market-based mechanisms work in every discussions have not adequately incorporated other sphere and yet have not been tried in the much of the modern research on learning. delivery of our most expensive publicly-financed Children come to schools today with a host service, K-12 education. We have, arguably, the of learning challenges. The factors range from best university system in the world—one based genetic, to nutritional, to environmental, but completely on free choice—where there is no they impose a huge burden on student and compulsion to attend a college or university teacher alike. based on residence, and yet we do not offer the It is long past time we used assessments as a same in K-12 where our failings are egregious diagnostic tool, allowing schools and parents to identify the cause of learning struggles and and scandalous. remedy them to the extent possible before the I believe CED should support a full multi- damage is done. This can be done and is being year test of a scholarship-type voucher system done today—but all too rarely. in several American cities, allowing parents to opt out of the present government-run schools, and use stipends to attend private, parochial, On the report as a whole, ROBERT C. or for-profit schools at their own option, pro- WAGGONER, with which JAMES Q. vided that these schools adhere to state-ap- RIORDAN has asked to be associated proved curricular standards, do not practice I have voted to disapprove this report be- discrimination, and offer open admission, with cause I am concerned that improved testing lotteries to determine entry if there is an ex- and assessment, and greater accountability in cess of applicants over vacancies. the existing public school system do not ad- I believe a full test of choice will show its dress the essential failing of public K-12 educa- capacity to deliver superior K-12 education, tion, which is systemic. probably at lower cost than at present. The The public school system, as it is currently opponents of choice are blocking even a test of structured, is devoid of effective choice for all the concept because they fear its success. except those who can afford to opt out and pay America’s children and the greater public are tuition. If our biggest public policy failing is suffering the results.

42 dren to think on their feet; teaching children Page 2, CAROL J. PARRY, with which to solve problems that exist in their communi- FLETCHER L. BYROM has asked to be ties, not just in their text books; teaching chil- associated dren to appreciate art and music. The report may give the impression that testing is the most important reform or change. Understanding that the purpose of the report Page 4, CAROL J. PARRY, with which is to discuss testing, we have to be wary of the FLETCHER L. BYROM has asked to be report being used by those that advocate test- associated ing above all other changes. I believe (as men- The report recognizes that creating the “per- tioned at several points in the report) that a fect” test will be very difficult, expensive, and number of changes are equally critical: enhanc- time consuming. If we are serious about creat- ing the profession of teaching through appro- ing excellent measurement and assessment priate compensation, on-going training and tools that are fully comparable across states, education, and changing the public percep- then we need a national project, under the tion of teachers so that they are well respected direction of the federal government, but with in our society; improving the physical school the input and cooperation of states to make facilities that children attend so that they are this happen in a quality way. Unfortunately, clean, safe and have appropriate space for stu- this may offend those who believe in local con- dents to learn; creating an environment in the trol of schools. public schools of discipline and respect so that teachers that want to teach and students that want to learn can do so without disruption Page 23, CAROL J. PARRY, with which from those that do not share these goals; in- FLETCHER L. BYROM has asked to be volving parents in both the educational pro- associated cess and in governance of the schools their In recommending that standardized tests children attend; developing programs and be used to determine if students get promoted, mechanisms so that parents, teachers and ad- we risk creating a self-fulfilling prophecy and a ministrators work as a team to improve the permanent underclass of students. When schools and enhance a child’s educational ex- schools and teachers are evaluated based on perience. I am sure there are many more. test results, they are motivated not only to im- I also question whether standardized mul- prove the quality of the schools so that chil- tiple-choice tests can really test the ability to dren will get better scores, but also to weed out think and reason. The report recognizes that those that cannot make it on these tests so that testing has its limitations. However, it also pro- overall school scores will rise. The report does poses that standardized tests (which at least recommend that individual improvement in now are primarily of the multiple-choice vari- student scores be part of the evaluation of the ety) be used to make decisions about school school and the teacher, but it also suggests that performance, teacher performance, and stu- students that cannot make it to the next level dent performance and advancement. Over the may drop out—only exacerbating the problem years, I have been involved in many efforts to we have today. The report does provide sugges- improve the public schools. The most exciting tions on how to assist these students, but these have not involved testing but creative teaching solutions take resources, specially trained teach- —teaching that goes way beyond things you ers, and special teaching environments. It is can measure in a test: teaching children to be naïve of us to think that these opportunities responsible members of society; teaching chil- for weaker students will exist everywhere. It is

43 important that the evaluation of a school or district should not only depend on the test scores but also on the school’s ability to pre- vent dropouts.

Page 28, CAROL J. PARRY, with which FLETCHER L. BYROM has asked to be associated If we tie teacher evaluations and compensa- tion as well as funding for the school to test results, teachers will feel they must teach the test. The report suggests that changing the test every year, not giving out questions in advance, etc. can prevent this. But just as there are very successful tutoring programs for SAT’s (tests that are supposedly kept top secret), schools and teachers will develop ways to teach what they presume will be on the tests, based on the curriculum and established local or state stan- dards.

44 OBJECTIVES OF THE COMMITTEE FOR ECONOMIC DEVELOPMENT

For more than 50 years, the Committee for foundations, and individuals. It is independent, Economic Development has been a respected nonprofit, nonpartisan, and nonpolitical. influence on the formation of business and Through this business-academic partner- public policy. CED is devoted to these two ship, CED endeavors to develop policy state- objectives: ments and other research materials that To develop, through objective research and commend themselves as guides to public and informed discussion, findings and recommenda- business policy; that can be used as texts in tions for private and public policy that will contrib- college economics and political science courses ute to preserving and strengthening our free society, and in management training courses; that achieving steady economic growth at high employ- will be considered and discussed by newspaper ment and reasonably stable prices, increasing pro- and magazine editors, columnists, and com- ductivity and living standards, providing greater mentators; and that are distributed abroad to and more equal opportunity for every citizen, and promote better understanding of the Ameri- improving the quality of life for all. can economic system. To bring about increasing understanding by CED believes that by enabling business present and future leaders in business, government, leaders to demonstrate constructively their con- and education, and among concerned citizens, of the cern for the general welfare, it is helping busi- importance of these objectives and the ways in which ness to earn and maintain the national and they can be achieved. community respect essential to the successful CED’s work is supported by private volun- functioning of the free enterprise capitalist tary contributions from business and industry, system.

45 CED BOARD OF TRUSTEES

Chairman JOHN BRADEMAS, President Emeritus FRANK P. DOYLE, Retired Executive Vice President New York University GE WILLIAM E. BROCK, Chairman Bridges LearningSystems, Inc. STEPHEN L. BROWN, Chairman and Chief Vice Chairmen Executive Officer ROY J. BOSTOCK, Chairman John Hancock Mutual Life Insurance Company 3 B/com Group, Inc. JOHN H. BRYAN, Chairman of the Board JOHN H. BRYAN, Chairman of the Board Sara Lee Corporation Sara Lee Corporation THOMAS J. BUCKHOLTZ, Executive Vice President DONALD R. CALDWELL, Chairman and Chief Beyond Insight Corporation Executive Officer MICHAEL BUNGEY, Chairman and Chief Cross Atlantic Capital Partners Executive Officer RAYMOND V. GILMARTIN, Chairman, President Bates Worldwide and Chief Executive Officer W. VAN BUSSMANN, Corporate Economist Merck & Co., Inc. DaimlerChrysler Corporation * FLETCHER L. BYROM, President and Chief Treasurer Executive Officer MICASU Corporation REGINA DOLAN Executive Vice President, Chief Financial Officer and DONALD R. CALDWELL, Chairman and Chief Chief Administrative Officer Executive Officer PaineWebber Group, Inc. Cross Atlantic Capital Partners FRANK C. CARLUCCI Washington, D.C. MARSHALL N. CARTER, Retired Chairman and Chief Executive Officer State Street Corporation ROGER G. ACKERMAN, Chairman and Chief ROBERT B. CATELL, Chairman and Chief Executive Officer Executive Officer Corning Incorporated KeySpan Corporation REX D. ADAMS, Dean JOHN B. CAVE, Principal The Fuqua School of Business Avenir Group, Inc. Duke University JOHN S. CHALSTY, Senior Advisor PAUL A. ALLAIRE, Chairman Credit Suisse First Boston Xerox Corporation RAYMOND G. CHAMBERS, Chairman of the Board IAN ARNOF, Retired Chairman Amelior Foundation Bank One, Louisiana, N.A. MARY ANN CHAMPLIN, Retired Senior Vice JAMES S. BEARD, President President Caterpillar Financial Services Corp. Aetna Inc. HENRY P. BECTON, JR., President and CLARENCE J. CHANDRAN, Chief Operating Officer General Manager Nortel Networks Corporation WGBH Educational Foundation ROBERT CHESS, Chairman THOMAS D. BELL, JR., Special Limited Partner Inhale Therapeutic Systems, Inc. Forstmann Little & Co. CAROLYN CHIN, Chief Executive Officer ALAN BELZER, Retired President and Chief Cebiz Operating Officer * JOHN L. CLENDENIN, Retired Chairman AlliedSignal Inc. BellSouth Corporation PETER A. BENOLIEL, Chairman, Executive Committee FERDINAND COLLOREDO-MANSFELD, Chairman Quaker Chemical Corporation and Chief Executive Officer MELVYN E. BERGSTEIN, Chairman and Chief Cabot Industrial Trust Executive Officer GEORGE H. CONRADES, Chairman and Chief Diamond Cluster International, Inc. Executive Officer DEREK BOK, President Emeritus Akamai Technologies, Inc. Harvard University KATHLEEN B. COOPER, Chief Economist and JON A. BOSCIA, President and Chief Executive Officer Manager, Economics & Energy Division Lincoln National Corporation Exxon Mobil Corporation ROY J. BOSTOCK, Chairman JAMES P. CORCORAN, Executive Vice President, 3 Government & Industry Relations B/com Group, Inc. American General Corporation

*Life Trustee GARY L. COUNTRYMAN, Chairman Emeritus FREDERICK W. GLUCK, Of Counsel Liberty Mutual Insurance Company McKinsey & Company, Inc. STEPHEN A. CRANE, Chairman, President and Chief CAROL R. GOLDBERG, President Executive Officer The Avcar Group, Ltd. Stirling Cooke Brown Holdings Limited ALFRED G. GOLDSTEIN, President and Chief RONALD R. DAVENPORT, Chairman of the Board Executive Officer Sheridan Broadcasting Corporation AG Associates JOHN T. DEE, Chairman and Chief JOSEPH T. GORMAN, Chairman and Chief Executive Officer Executive Officer Volume Services America TRW Inc. ROBERT M. DEVLIN, Chairman and Chief RICHARD A. GRASSO, Chairman and Chief Executive Officer Executive Officer American General Corporation New York Stock Exchange, Inc. JOHN DIEBOLD, Chairman EARL G. GRAVES, SR., Publisher and Chief John Diebold Incorporated Executive Officer REGINA DOLAN, Executive Vice President, Chief Black Enterprise Magazine Financial Officer and Chief Administrative Officer WILLIAM H. GRAY, III, President and Chief PaineWebber Group, Inc. Executive Officer IRWIN DORROS, President The College Fund Dorros Associates ROSEMARIE B. GRECO, Principal FRANK P. DOYLE, Retired Executive Vice President Greco Ventures GE GERALD GREENWALD, Chairman Emeritus E. LINN DRAPER, JR., Chairman, President and Chief UAL Corporation Executive Officer BARBARA B. GROGAN, President American Electric Power Company Western Industrial Contractors T. J. DERMOT DUNPHY, Retired Chairman PATRICK W. GROSS, Founder and Chairman, Sealed Air Corporation Executive Committee CHRISTOPHER D. EARL, Managing Director American Management Systems, Inc. Perseus Capital, LLC JEROME GROSSMAN, Chairman and Chief W. D. EBERLE, Chairman Executive Officer Manchester Associates, Ltd. Lion Gate Management Corporation JAMES D. ERICSON, Chairman and Chief RONALD GRZYWINSKI, Chairman and Chief Executive Officer Executive Officer The Northwestern Mutual Financial Network Shorebank Corporation WILLIAM T. ESREY, Chairman and Chief JUDITH H. HAMILTON, President and Chief Executive Officer Executive Officer Sprint Classroom Connect KATHLEEN FELDSTEIN, President WILLIAM A. HASELTINE, Chairman and Chief Economics Studies, Inc. Executive Officer Human Genome Sciences, Inc. RONALD E. FERGUSON, Chairman, President and Chief Executive Officer WILLIAM F. HECHT, Chairman, President and Chief General RE Corporation Executive Officer PPL Corporation E. JAMES FERLAND, Chairman, President and Chief Executive Officer JOSEPH D. HICKS, Retired President and Chief Public Service Enterprise Group Inc. Executive Officer Siecor Corporation * EDMUND B. FITZGERALD, Managing Director Woodmont Associates HEATHER HIGGINS, President Randolph Foundation HARRY L. FREEMAN, Chair The Mark Twain Institute STEVEN R. HILL, Senior Vice President, Human Resources MITCHELL S. FROMSTEIN, Chairman Emeritus Weyerhaeuser Company Manpower Inc. RODERICK M. HILLS, President JOSEPH GANTZ, Partner Hills Enterprises, Ltd. GG Capital, LLC HAYNE HIPP, President and Chief Executive Officer E. GORDON GEE, Chancellor The Liberty Corporation Vanderbilt University RONALD N. HOGE, Former President and Chief THOMAS P. GERRITY, Dean Emeritus Executive Officer The Wharton School MagneTek, Inc. University of Pennsylvania DEBORAH C. HOPKINS, Executive Vice President RAYMOND V. GILMARTIN, Chairman, President and and Chief Financial Officer Chief Executive Officer Lucent Technologies Inc. Merck & Co., Inc.

*Life Trustee MATINA S. HORNER, Executive Vice President MICHAEL D. LOCKHART, Chairman and Chief TIAA-CREF Executive Officer Armstrong Holdings, Inc. PHILIP K. HOWARD, Vice Chairman Covington & Burling JOSEPH T. LYNAUGH, Retired President and Chief Executive Officer ROBERT J. HURST, Vice Chairman NYLCare Health Plans, Inc. Goldman, Sachs & Co. BRUCE K. MACLAURY, President Emeritus ALICE STONE ILCHMAN The Brookings Institution Bronxville, New York COLETTE MAHONEY, RSHM, President Emeritus WILLIAM C. JENNINGS, Chief Executive Officer Marymount Manhattan College US Interactive, Inc. ELLEN R. MARRAM, Partner JEFFREY A. JOERRES, President and Chief North Castle Partners Executive Officer Manpower Inc. ALONZO L. MCDONALD, Chairman and Chief Executive Officer JAMES A. JOHNSON, Chairman and Chief Avenir Group, Inc. Executive Officer Johnson Capital Partners EUGENE R. MCGRATH, Chairman, President and Chief Executive Officer ROBERT M. JOHNSON, Chairman and Chief Consolidated Edison Company of New York, Inc. Executive Officer Bowne & Co., Inc. DAVID E. MCKINNEY, President The Metropolitan Museum of Art H.V. JONES, Office Managing Director Korn/Ferry International, Inc. DEBORAH HICKS MIDANEK, Principal Glass & Associates PRES KABACOFF, President and Co-Chairman Historic Restoration, Inc. HARVEY R. MILLER, Senior Partner Weil, Gotshal & Manges JOSEPH J. KAMINSKI, Corporate Executive Vice President NICHOLAS G. MOORE, Chairman Air Products and Chemicals, Inc. PricewaterhouseCoopers EDWARD A. KANGAS, Chairman, Retired DIANA S. NATALICIO, President Deloitte Touche Tohmatsu The University of Texas at El Paso JOSEPH E. KASPUTYS, Chairman, President and MARILYN CARLSON NELSON, Chairman, President Chief Executive Officer and Chief Executive Officer Thomson Financial Primark Carlson Companies, Inc. JAMES P. KELLY, Chairman and Chief MATTHEW NIMETZ, Partner Executive Officer General Atlantic Partners United Parcel Service Inc. THOMAS H. O’BRIEN, Chairman of the Board THOMAS J. KLUTZNICK, President PNC Financial Services Group, Inc. Thomas J. Klutznick Company LEO J. O’DONOVAN, S.J., President CHARLES F. KNIGHT, Chairman and Chief Georgetown University Executive Officer DEAN R. O’HARE, Chairman and Chief Emerson Electric Co. Executive Officer CHARLES E.M. KOLB, President Chubb Corporation Committee for Economic Development JOHN D. ONG, Chairman Emeritus ALLEN J. KROWE, Retired Vice Chairman The BFGoodrich Company Texaco Inc. ROBERT J. O'TOOLE, Chairman and Chief C. JOSEPH LABONTE, Chairman Executive Officer The Vantage Group A.O. Smith Corporation KURT M. LANDGRAF, President and Chief STEFFEN E. PALKO, Vice Chairman and President Executive Officer Cross Timbers Oil Company Educational Testing Service SANDRA PANEM, Partner J. TERRENCE LANNI, Chairman of the Board Cross Atlantic Partners, Inc. MGM Mirage CAROL J. PARRY, Retired Executive Vice President CHARLES R. LEE, Chairman and Co-Chief The Chase Manhattan Corporation Executive Officer VICTOR A. PELSON, Senior Advisor Verizon Communications Warburg Dillon Read LLC ROBERT H. LESSIN, Chairman DONALD K. PETERSON, President and Chief Wit Soundview Group, Inc. Executive Officer WILLIAM W. LEWIS, Director Avaya Inc. McKinsey Global Institute PETER G. PETERSON, Chairman McKinsey & Company, Inc. The Blackstone Group IRA A. LIPMAN, Chairman of the Board and President RAYMOND PLANK, Chairman and Chief Guardsmark, Inc. Executive Officer Apache Corporation

*Life Trustee ARNOLD B. POLLARD, President and Chief JOHN C. SICILIANO Executive Officer San Marino, California The Chief Executive Group RUTH J. SIMMONS, President S. LAWRENCE PRENDERGAST, Executive Vice Smith College President of Finance RONALD L. SKATES, Former President and Chief LaBranche & Co. Executive Officer HUGH B. PRICE, President and Chief Data General Corporation Executive Officer FREDERICK W. SMITH, Chairman, President and National Urban League Chief Executive Officer NED REGAN, President Federal Express Corporation Baruch College HUGO FREUND SONNENSCHEIN, President Emeritus WILLIAM R. RHODES, Vice Chairman The University of Chicago Citigroup Inc. ALAN G. SPOON, Managing General Partner JAMES Q. RIORDAN Polaris Ventures Quentin Partners Inc. JOHN R. STAFFORD, Chairman, President and E. B. ROBINSON, JR., Chairman Emeritus Chief Executive Officer Deposit Guaranty Corporation American Home Products Corporation ROY ROMER STEPHEN STAMAS, Chairman Former Governor of Colorado The American Assembly Superintendent, Los Angeles Unified School District JOHN L. STEFFENS, Vice Chairman DANIEL ROSE, Chairman Merrill Lynch & Co., Inc. Rose Associates, Inc. PAULA STERN, President HOWARD M. ROSENKRANTZ, Chief Executive Officer The Stern Group, Inc. Grey Flannel Auctions DONALD M. STEWART, President and Chief MICHAEL I. ROTH, Chairman and Chief Executive Officer Executive Officer Chicago Community Trust The MONY Group Inc. ROGER W. STONE, Chairman and Chief LANDON H. ROWLAND, Chairman, President and Executive Officer Chief Executive Officer Box USA Group, Inc. Stillwell Financial Inc. MATTHEW J. STOVER, President NEIL L. RUDENSTINE, President edu.com Harvard University RICHARD J. SWIFT, Chairman, President and Chief WILLIAM RUDER, President Executive Officer William Ruder Incorporated Foster Wheeler Corporation GEORGE RUPP, President RICHARD F. SYRON, President and Chief Columbia University Executive Officer GEORGE F. RUSSELL, JR., Chairman Thermo Electron Corporation Frank Russell Company HENRY TANG, Chairman EDWARD B. RUST, JR., Chairman and Chief Committee of 100 Executive Officer FREDERICK W. TELLING, Vice President Corporate State Farm Insurance Companies Strategic Planning and Policy Division ARTHUR F. RYAN, Chairman and Chief Pfizer Inc. Executive Officer RICHARD L. THOMAS, Retired Chairman The Prudential Insurance Company of America First Chicago NBD Corporation MARGUERITE W. SALLEE, Chairman and Chief JAMES A. THOMSON, President and Chief Executive Officer Executive Officer Frontline Group RAND STEPHEN W. SANGER, Chairman and Chief CHANG-LIN TIEN, NEC Distinguished Professor of Executive Officer Engineering General Mills, Inc. University of California, Berkeley HENRY B. SCHACHT, Director and Senior Advisor THOMAS J. TIERNEY, Director (former Chief Executive) E.M. Warburg, Pincus & Co. LLC Bain & Company MICHAEL M. SEARS, Senior Vice President and STOKLEY P. TOWLES, Partner Chief Financial Officer Brown Brothers Harriman & Co. The Boeing Company ALAIR A. TOWNSEND, Publisher WALTER H. SHORENSTEIN, Chairman of the Board Crain’s New York Business The Shorenstein Company STEPHEN JOEL TRACHTENBERG, President * GEORGE P. SHULTZ, Distinguished Fellow George Washington University The Hoover Institution JAMES L. VINCENT, Chairman and Chief Stanford University Executive Officer Biogen, Inc.

*Life Trustee ROBERT C. WAGGONER, Chief Executive Officer Burrelle’s Information Services LLC DONALD C. WAITE, III, Director McKinsey & Company, Inc. HERMINE WARREN, President Hermine Warren Associates, Inc. ARNOLD R. WEBER, President Emeritus Northwestern University JOHN F. WELCH, JR., Chairman and Chief Executive Officer GE JOSH S. WESTON, Honorary Chairman Automatic Data Processing, Inc. CLIFTON R. WHARTON, JR., Former Chairman and Chief Executive Officer TIAA-CREF DOLORES D. WHARTON, Chairman and Chief Executive Officer The Fund for Corporate Initiatives, Inc. MICHAEL W. WICKHAM, Chairman and Chief Executive Officer Roadway Express, Inc. HAROLD M. WILLIAMS, Of Counsel Skadden Arps Slate Meagher & Flom LLP J. KELLEY WILLIAMS, Chairman and Chief Executive Officer ChemFirst Inc. LINDA SMITH WILSON, President Emerita Radcliffe College MARGARET S. WILSON, Chairman and Chief Executive Officer Scarbroughs KURT E. YEAGER, President and Chief Executive Officer Electric Power Research Institute MARTIN B. ZIMMERMAN, Vice President, Governmental Affairs Ford Motor Company

*Life Trustee CED HONORARY TRUSTEES

RAY C. ADAM, Retired Chairman JOHN M. FOX NL Industries Sapphire, North Carolina O. KELLEY ANDERSON DON C. FRISBEE, Chairman Emeritus Boston, Massachusetts PacifiCorp ROBERT O. ANDERSON, Retired Chairman RICHARD L. GELB, Chairman Emeritus Hondo Oil & Gas Company Bristol-Myers Squibb Company ROY L. ASH W. H. KROME GEORGE, Retired Chairman Los Angeles, California ALCOA SANFORD S. ATWOOD, President Emeritus WALTER B. GERKEN, Chairman, Executive Committee Emory University Pacific Life Insurance Company ROBERT H. B. BALDWIN, Retired Chairman PAUL S. GEROT Morgan Stanley Group Inc. Delray Beach, Florida GEORGE F. BENNETT, Chairman Emeritus LINCOLN GORDON, Guest Scholar State Street Investment Trust The Brookings Institution HAROLD H. BENNETT KATHARINE GRAHAM, Chairman of the Salt Lake City, Utah Executive Committee The Washington Post Company JACK F. BENNETT, Retired Senior Vice President Exxon Corporation JOHN D. GRAY, Chairman Emeritus Hartmarx Corporation HOWARD W. BLAUVELT Keswick, Virginia JOHN R. HALL, Retired Chairman Ashland Inc. MARVIN BOWER Delray Beach, Florida RICHARD W. HANSELMAN, Chairman Health Net Inc. ALAN S. BOYD Lady Lake, Florida ROBERT A. HANSON, Retired Chairman Deere & Company ANDREW F. BRIMMER, President Brimmer & Company, Inc. ROBERT S. HATFIELD, Retired Chairman The Continental Group, Inc. HARRY G. BUBB, Chairman Emeritus Pacific Mutual Life Insurance ARTHUR HAUSPURG, Member, Board of Trustees Consolidated Edison Company of New York, Inc. THEODORE A. BURTIS, Retired Chairman of the Board PHILIP M. HAWLEY, Retired Chairman of the Board Sun Company, Inc. Carter Hawley Hale Stores, Inc. PHILIP CALDWELL, Chairman (Retired) ROBERT C. HOLLAND, Senior Fellow Ford Motor Company The Wharton School University of Pennsylvania EVERETT N. CASE Cooperstown, New York LEON C. HOLT, JR., Retired Vice Chairman Air Products and Chemicals, Inc. HUGH M. CHAPMAN, Retired Chairman NationsBank South SOL HURWITZ, Retired President Committee for Economic Development E. H. CLARK, JR., Chairman and Chief Executive Officer GEORGE F. JAMES The Friendship Group Ponte Vedra Beach, Florida A.W. CLAUSEN, Retired Chairman and Chief DAVID KEARNS, Chairman Emeritus Executive Officer New American Schools BankAmerica Corporation GEORGE M. KELLER, Chairman of the Board, Retired DOUGLAS D. DANFORTH Chevron Corporation Executive Associates FRANKLIN A. LINDSAY, Retired Chairman JOHN H. DANIELS, Retired Chairman and Itek Corporation Chief Executive Officer ROY G. LUCKS Archer-Daniels Midland Co. San Francisco, California RALPH P. DAVIDSON ROBERT W. LUNDEEN, Retired Chairman Washington, D.C. The Dow Chemical Company ALFRED C. DECRANE, JR., Retired Chairman and IAN MACGREGOR, Retired Chairman Chief Executive Officer AMAX Inc. Texaco, Inc. RICHARD B. MADDEN, Retired Chairman and ROBERT R. DOCKSON, Chairman Emeritus Chief Executive Officer CalFed, Inc. Potlatch Corporation LYLE EVERINGHAM, Retired Chairman FRANK L. MAGEE The Kroger Co. Stahlstown, Pennsylvania THOMAS J. EYERMAN, Retired Partner STANLEY MARCUS, Consultant Skidmore, Owings & Merrill Stanley Marcus Consultancy JOHN T. FEY AUGUSTINE R. MARUSI Park City, Utah Lake Wales, Florida WILLIAM F. MAY, Chairman and Chief JAMES J. RENIER Executive Officer Renier & Associates Statue of Liberty-Ellis Island Foundation, Inc. IAN M. ROLLAND, Former Chairman and Chief OSCAR G. MAYER, Retired Chairman Executive Officer Oscar Mayer & Co. Lincoln National Corporation GEORGE C. MCGHEE, Former U.S. Ambassador AXEL G. ROSIN, Retired Chairman and Under Secretary of State Book-of-the-Month Club, Inc. JOHN F. MCGILLICUDDY, Retired Chairman WILLIAM M. ROTH and Chief Executive Officer Princeton, New Jersey Chemical Banking Corporation RALPH S. SAUL, Former Chairman of the Board JAMES W. MCKEE, JR., Retired Chairman CIGNA Companies CPC International, Inc. GEORGE A. SCHAEFER, Retired Chairman of the Board CHAMPNEY A. MCNAIR, Retired Vice Chairman Caterpillar, Inc. Trust Company of Georgia ROBERT G. SCHWARTZ J. W. MCSWINEY, Retired Chairman of the Board New York, New York The Mead Corporation MARK SHEPHERD, JR., Retired Chairman ROBERT E. MERCER, Retired Chairman Texas Instruments, Inc. The Goodyear Tire & Rubber Co. ROCCO C. SICILIANO RUBEN F. METTLER, Retired Chairman and Beverly Hills, California Chief Executive Officer DAVIDSON SOMMERS TRW Inc. Washington, D.C. LEE L. MORGAN, Former Chairman of the Board ELMER B. STAATS, Former Controller Caterpillar, Inc. General of the United States ROBERT R. NATHAN, Chairman FRANK STANTON, Former President Nathan Associates, Inc. CBS, Inc. J. WILSON NEWMAN, Retired Chairman EDGAR B. STERN, JR., Chairman of the Board Dun & Bradstreet Corporation Royal Street Corporation JAMES J. O’CONNOR, Former Chairman and Chief ALEXANDER L. STOTT Executive Officer Fairfield, Connecticut Unicom Corporation WAYNE E. THOMPSON, Past Chairman LEIF H. OLSEN, President Merritt Peralta Medical Center LHO Group CHARLES C. TILLINGHAST, JR. NORMA PACE, President Providence, Rhode Island Paper Analytics Associates HOWARD S. TURNER, Retired Chairman CHARLES W. PARRY, Retired Chairman Turner Construction Company ALCOA L. S. TURNER, JR. WILLIAM R. PEARCE, Director Dallas, Texas American Express Mutual Funds THOMAS A. VANDERSLICE JOHN H. PERKINS, Former President TAV Associates Continental Illinois National Bank and Trust Company JAMES E. WEBB RUDOLPH A. PETERSON, President and Chief Washington, D.C. Executive Officer (Emeritus) BankAmerica Corporation SIDNEY J. WEINBERG, JR., Senior Partner The Goldman Sachs Group, L.P. DEAN P. PHYPERS New Canaan, Connecticut ROBERT C. WINTERS, Chairman Emeritus Prudential Insurance Company of America EDMUND T. PRATT, JR., Retired Chairman and Chief Executive Officer ARTHUR M. WOOD Pfizer Inc. Chicago, Illinois ROBERT M. PRICE, Retired Chairman and RICHARD D. WOOD, Director Chief Executive Officer Eli Lilly and Company Control Data Corporation WILLIAM S. WOODSIDE, Retired Chairman R. STEWART RAUCH, Former Chairman LSG Sky Chefs The Philadelphia Savings Fund Society CHARLES J. ZWICK Coral Gables, Florida CED RESEARCH ADVISORY BOARD

Chairman JOHN P. WHITE ALAIN C. ENTHOVEN LINDA YUEN-CHING LIM Professor Marriner S. Eccles Professor of Public Professor Harvard University and Private Management University of Michigan Business School John F. Kennedy School of Government Stanford University Graduate School of Business ROBERT LITAN Vice President, Director of Economic RONALD F. FERGUSON Studies Professor The Brookings Institution Harvard University John F. Kennedy School of Government PAUL ROMER JAGDISH BHAGWATI STANCO 25 Professor of Economics Arthur Lehman Professor of Economics BENJAMIN M. FRIEDMAN Stanford University Columbia University William Joseph Maier Professor of Graduate School of Business RALPH D. CHRISTY Political Economy Harvard University CECILIA E. ROUSE J. Thomas Clark Professor Associate Professor of Economics Department of Agricultural, Resource, ROBERT W. HAHN and Public Affairs and Managerial Economics Resident Scholar Princeton University Cornell University American Enterprise Institute Woodrow Wilson School JOHN COGAN HELEN F. LADD DAVID WESSEL Senior Fellow Professor of Public Policy Studies and Assistant Bureau Chief/Columnist The Hoover Institution Economics The Wall Street Journal Stanford University Sanford Institute of Public Policy Duke University

CED PROFESSIONAL AND ADMINISTRATIVE STAFF

CHARLES E.M. KOLB President

Research Communications/Government Relations Development VAN DOORN OOMS CLAUDIA P. FEUREY MARTHA HOULE Senior Vice President and Vice President for Communications Vice President for Development and Director of Research and Corporate Affairs Secretary of the Board of Trustees MICHAEL J. PETRO GLORIA Y. CALHOUN Vice President and Director of Development Assistant JANET HANSEN Business and Government Policy Vice President and Director and Chief of Staff SANDRA L. MARTINEZ of Education Studies Development Associate CHRIS DREIBELBIS ELLIOT SCHWARTZ Business and Government Policy RICHARD M. RODERO Vice President and Director Associate Assistant Director of Development of Economic Studies JIM KAPSIS Finance and Administration TAREK ANANDAN Public Affairs Associate Research Associate CARY DAVIS VALERIE MENDELSOHN Director of Finance and Administration MICHAEL BERG Conference Manager and Secretary of Research Associate the Research and Policy Committee KAREN CASTRO Accounting Manager SETH TUROFF KIMBERLY MOORE Public Affairs and Development Assistant PETER E. COX Research Associate Operations Manager ROBIN SAMERS Advisor on International Public Affairs Associate SHARON A. FOWKES Economic Policy Executive Assistant to the President HELENA ZYBLIKEWYCZ ISAIAH FRANK Manager of Business and Public Affairs ARLENE M. MURPHY William L. Clayton Professor Executive Assistant to the President of International Economics and Office Manager The Johns Hopkins University AMANDA TURNER Office Manager STATEMENTS ON NATIONAL POLICY ISSUED BY THE COMMITTEE FOR ECONOMIC DEVELOPMENT

SELECTED PUBLICATIONS:

Improving Global Financial Stability (2000) The Case for Permanent Normal Trade Relations with China (2000) Welfare Reform and Beyond: Making Work Work (2000) Breaking the Litigation Habit: Economic Incentives for Legal Reform (2000) New Opportunities for Older Workers (1999) Investing in the People's Business: A Business Proposal for Campaign Finance Reform (1999) The Employer’s Role in Linking School and Work (1998) Employer Roles in Linking School and Work: Lessons from Four Urban Communities (1998) America’s Basic Research: Prosperity Through Discovery (1998) Modernizing Government Regulation: The Need For Action (1998) U.S. Economic Policy Toward The Asia-Pacific Region (1997) Connecting Inner-City Youth To The World of Work (1997) Fixing Social Security (1997) Growth With Opportunity (1997) American Workers and Economic Change (1996) Connecting Students to a Changing World: A Technology Strategy for Improving Mathematics and Science Education (1995) Cut Spending First: Tax Cuts Should Be Deferred to Ensure a Balanced Budget (1995) Rebuilding Inner-City Communities: A New Approach to the Nation’s Urban Crisis (1995) Who Will Pay For Your Retirement? The Looming Crisis (1995) Putting Learning First: Governing and Managing the Schools for High Achievement (1994) Prescription for Progress: The Uruguay Round in the New Global Economy (1994) *From Promise to Progress: Towards a New Stage in U.S.-Japan Economic Relations (1994) U.S. Trade Policy Beyond The Uruguay Round (1994) In Our Best Interest: NAFTA and the New American Economy (1993) What Price Clean Air? A Market Approach to Energy and Environmental Policy (1993) Why Child Care Matters: Preparing Young Children For A More Productive America (1993) Restoring Prosperity: Budget Choices for Economic Growth (1992) The United States in the New Global Economy: A Rallier of Nations (1992) The Economy and National Defense: Adjusting to Cutbacks in the Post-Cold War Era (1991) Politics, Tax Cuts and the Peace Dividend (1991) The Unfinished Agenda: A New Vision for Child Development and Education (1991) Foreign Investment in the United States: What Does It Signal? (1990) An America That Works: The Life-Cycle Approach to a Competitive Work Force (1990) Breaking New Ground in U.S. Trade Policy (1990)

*Statements issued in association with CED counterpart organizations in foreign countries. Battling America's Budget Deficits (1989) *Strengthening U.S.-Japan Economic Relations (1989) Who Should Be Liable? A Guide to Policy for Dealing with Risk (1989) Investing in America's Future: Challenges and Opportunities for Public Sector Economic Policies (1988) Children in Need: Investment Strategies for the Educationally Disadvantaged (1987) Finance and Third World Economic Growth (1987) Reforming Health Care: A Market Prescription (1987) Work and Change: Labor Market Adjustment Policies in a Competitive World (1987) Leadership for Dynamic State Economies (1986) Investing in Our Children: Business and the Public Schools (1985) Fighting Federal Deficits: The Time for Hard Choices (1985) Strategy for U.S. Industrial Competitiveness (1984) Productivity Policy: Key to the Nation's Economic Future (1983) Energy Prices and Public Policy (1982) Public-Private Partnership: An Opportunity for Urban Communities (1982) Reforming Retirement Policies (1981) Transnational Corporations and Developing Countries: New Policies for a Changing World Economy (1981) Stimulating Technological Progress (1980) Redefining Government's Role in the Market System (1979) Jobs for the Hard-to-Employ: New Directions for a Public-Private Partnership (1978) CED COUNTERPART ORGANIZATIONS

Close relations exist between the Committee for Economic Development and independent, nonpolitical research organizations in other countries. Such counter- part groups are composed of business executives and scholars and have objec- tives similar to those of CED, which they pursue by similarly objective methods. CED cooperates with these organizations on research and study projects of common interest to the various countries concerned. This program has resulted in a number of joint policy statements involving such international matters as energy, East-West trade, assistance to developing countries, and the reduction of nontariff barriers to trade.

CE Circulo de Empresarios Madrid, Spain

CEDA Committee for Economic Development of Australia Sydney, Australia

EVA Centre for Finnish Business and Policy Studies Helsinki, Finland

FAE Forum de Administradores de Empresas Lisbon, Portugal

FDE Belgian Enterprise Foundation Brussels, Belgium

IDEP Institut de l’Entreprise Paris, France

IW Institut der Deutschen Wirtschaft Cologne, Germany

Keizai Doyukai Tokyo, Japan

SMO Stichting Maatschappij en Onderneming The Netherlands

SNS Studieförbundet Naringsliv och Samhälle Stockholm, Sweden