Policy Analysis November 13, 2018 | Number 854 Fixing the Bias in Current State K–12 Education Rankings By Stan J. Liebowitz and Matthew L. Kelly EXECUTIVE SUMMARY tate education rankings published by U.S. News states in the South and Southwest score much higher & World Report, Education Week, and others play than they do in conventional rankings. Furthermore, we a prominent role in legislative debate and public create another set of rankings on the efficiency of educa- discourse concerning education. These rankings tion spending. In these efficiency rankings, achieving are based partly on achievement tests, which successful outcomes while economizing on education Smeasure student learning, and partly on other factors not expenditures is considered better than doing so through directly related to student learning. When achievement lavish spending. These efficiency rankings cause a further tests are used as measures of learning in these convention- increase in the rankings of southern and western states al rankings, they are aggregated in a way that provides mis- and a decline in the rankings of northern states. Finally, leading results. To overcome these deficiencies, we create our regression results indicate that unionization has a a new ranking of state education systems using demo- powerful negative influence on educational outcomes, and graphically disaggregated achievement data and exclud- that, given current spending levels, additional spending ing less informative factors that are not directly related has little effect. We also find no evidence of a relationship to learning. Using our methodology changes the order of between student performance and teacher-pupil ratios or state rankings considerably. Many states in New England private school enrollment, but some evidence that charter and the Upper Midwest fall in the rankings, whereas many school enrollment has a positive effect. Stan Liebowitz is the Ashbel Smith Professor of Economics and director of the Center for the Analysis of Property Rights and Innovation at the Jindal School of Management, University of Texas at Dallas; he is also an adjunct fellow at the Cato Institute. Matthew Kelly is a graduate student and research fellow at the Colloquium for the Advancement of Free-Enterprise Education at the Jindal School of Management, University of Texas at Dallas. 2 INTRODUCTION the efficiency of educational spending reorders Most existing Which states have the best K–12 education state rankings in fundamental ways. As we show rankings of systems? What set of government policies and in this report, employing our improved ranking “ education spending levels is needed to achieve methodology overturns the apparent consen- state K–12 targeted outcomes in an efficient manner? An- sus that schools in the South and Southwest education are swers to these important questions are essen- perform less well than states in the Northeast unreliable and tial to the performance of our economy and and Upper Midwest. It also puts to rest the misleading. country. Local workforce education and quality claim that more spending necessarily improves of schools are key determinants in business and student performance.5 residential location decisions. Determining Many rankings, including those of U.S. which education policies are most cost-effective News, provide average scores on tests admin- ” is also crucial for state and local politicians as istered by the National Assessment of Edu- they allocate limited taxpayer resources. cation Progress (NAEP), sometimes referred Several organizations rank state K–12 educa- to as “the nation’s report card.”6 The NAEP tion systems, and these rankings play a promi- reports provide average scores for various sub- nent role in both legislative debate and public jects, such as math, reading, and science, for discourse concerning education. The most students at various grade levels.7 These scores popular are arguably those of U.S. News & World are supposed to measure the degree to which 1 Report (U.S. News). It is common for activists students understand these subjects. While and pundits (whether in favor of homeschool- U.S. News includes other measures of educa- ing, stronger teacher unions, core standards, tion quality, such as graduation rates and SAT etc.) to use these rankings to support their ar- and ACT college entrance exam scores, direct guments for changes in policy or spending pri- measures of the entire student population’s orities. As shown by the recent competition for understanding of academic subject matter, Amazon’s HQ2 (second headquarters), politi- such as those from the NAEP, are the most cians and business leaders will also frequently appropriate measures of success for an edu- cite education rankings to highlight their states’ cational system.8 Whereas graduation is not advantages.2 Recent teacher strikes across the necessarily an indication of actual learning, country have likewise drawn renewed atten- and only those students wishing to pursue a tion to education policy, and journalists inevi- college degree tend to take standardized tests tably mention state rankings when these topics like the SAT and ACT, NAEP scores provide arise.3 It is therefore important to ensure that standardized measures of learning covering such rankings accurately reflect performance. the entire student population. Focusing on Though well-intentioned, most existing NAEP data thus avoids selection bias while rankings of state K–12 education are unreliable more closely measuring a school system’s abil- and misleading. The most popular and influen- ity to improve actual student performance. tial state education rankings fail to provide an However, student heterogeneity is ignored 4 “apples to apples” comparison between states. by U.S. News and most other state rankings that By treating states as though they had identical use NAEP data as a component of their rank- students, they ignore the substantial variation ings. Students from different socioeconomic present in student populations across states. and ethnic backgrounds tend to perform dif- Conventional rankings also include data that ferently (regardless of the state they are in). As are inappropriate or irrelevant to the educa- this report will show, such aggregation often tional performance of schools. Finally, these renders conventional state rankings as little analyses disregard government budgetary con- more than a proxy for a jurisdiction’s demogra- straints. Not surprisingly, using disaggregated phy. This problem is all the more unfortunate measures of student learning, removing inap- because it is so easily avoided. NAEP provides propriate or irrelevant variables, and examining demographic breakdowns of student scores by 3 state. This oversight substantially skews the THE IMPACT OF HETEROGENEITY current rankings. Students arrive to class on the first day of Our main goal Perhaps just as problematic, some educa- school with different backgrounds, skills, and is to provide tion rankings conflate inputs and outputs. life experiences, often related to socioeco- “ a ranking of For instance, Education Week uses per pupil nomic status. Assuming away these differenc- expenditures as a component in its annual es, as most state rankings implicitly do, may public school rankings.9 When direct measures of student lead analysts to attribute too much of the vari- systems achievement are used, such as NAEP scores, ation in state educational outcomes to school that more it is a mistake to include inputs, such as edu- systems instead of to student characteristics. cational expenditures, as a separate factor.10 Taking student characteristics into account is accurately Doing so gives extra credit to states that one of the fundamental improvements made reflects the spend excessively to achieve the same level of by our state rankings. learning success others achieve with fewer resources, An example drawn from NAEP data il- when that wasteful extra spending should in- lustrates how failing to account for student that is taking stead be penalized in the rankings. heterogeneity can lead to grossly misleading place. Our main goal in this report is to provide a results. (For a more general demonstration of ranking of public school systems in U.S. states how heterogeneity affects results, see the Ap- that more accurately reflects the learning that pendix.) According to U.S. News, Iowa ranks ” is taking place. We attempt to move closer to 8th and Texas ranks 33rd in terms of pre-K–12 a “value added” approach as explained in the quality. U.S. News includes only NAEP eighth- following hypothetical. Consider one school grade math and reading scores as components system where every student knows how to in its ranking, and Iowa leads Texas in both. By read upon entering kindergarten. Compare further including fourth grade scores and the this to a second school system where students NAEP science tests, the comparison between don’t have this skill upon entering kindergar- Iowa and Texas remains largely unchanged. ten. It should come as no surprise if, by the end Iowa students still do better than Texas stu- of first grade, the first school’s students have dents, but now in all six tests reported for those better reading scores than the second school’s. states (math, reading, and science in fourth and But if the second school’s students improved eighth grades). To use a baseball metaphor, this more, relative to their initial situation, a value- looks like a shut-out in Iowa’s favor. added approach would conclude that the But this