TAUBMAN CENTER POLICY BRIEFS

PB-2005-4 | November 18, 2005

The “Third Way” of Education Reform? By Brian A. Jacob, Kennedy School of Government

Education reformers agree on the need abilities to meet other goals that for high-quality teaching, but they are valued by parents. Empowering Brian A. Jacob disagree sharply on how to identify principals with greater authority could Brian Jacob is Assistant Professor of Public Policy at the John F. Kennedy School of good and bad teachers. The 2002 No thus be an effective way to increase Government, a Faculty Research Fellow in Child Left Behind Act ushered in a accountability while respecting a the Program on Children and the Program on Education at the National Bureau of new era of standardized test-based wider range of educational goals and Economic Research, and a Faculty As- accountability, with state statutes now the particular circumstances of each sociate with the Center for Basic Research in the Social Sciences. He has previously rewarding and sanctioning school classroom and school. served as a policy analyst in the New York staff based on student performance City’s Mayor’s Offi ce and taught middle school in East Harlem. and ten states directly linking teacher How Do Principals Measure Up? 1 This Policy Brief integrates research from pay to test scores. Many parents and As front-line managers in schools, Brian Jacob’s papers listing in the “Related educators, however, continue to view principals must distinguish high- Publications” section at the end of the brief. high-stakes testing as overly focused quality from low-quality teaching. on narrow educational goals. Parents In research with Lars Lefgren, we A. Alfred Taubman Center for often want teachers to be role models confi dentially surveyed elementary State and Local Government and classroom managers, to develop school principals in a mid-sized school The Taubman Center and its affi liated strong relationships, foster creativity, district, questioning them about their institutes and programs are the Kennedy School of Government’s focal point for encourage self esteem, or to teach teachers on a variety of performance activities that address urban policy, state a broad curriculum. Standardized measures.2 We then compared how and local governance and intergovern- mental relations. exams, they argue, fail to capture these well these subjective assessment Taubman Center Policy Briefs are short values and may detract from their measures of teachers matched overviews of new and notable research on these issues by scholars affi liated pursuit. Both views have merit, but the subsequent test-score gains in with the Center. the battles between them often lead to individual classrooms, teacher salaries, A. Alfred Taubman Center for State gridlock. and the classrooms that parents request and Local Government Using a variety of data sources not for their children the following year. John F. Kennedy School of Government, 79 JFK St., Cambridge, MA 02138 normally available to researchers, we Conventional measures of teacher Telephone: (617) 495-2199 fi nd that principals are an overlooked capabilities are weak predictors of Email: [email protected] resource in these debates. Principals student performance on standardized www.ksg.harvard.edu/taubmancenter reliably predict which teachers will exams. The most comprehensive way perform best at improving student to assess teachers based on student test scores. Moreover, compared to test scores takes into consideration standardized exams or traditional students’ prior performance and their determinants of teacher pay, principals demographic characteristics. After do a better job at assessing teachers’ constructing such a “value-added” The “Third Way” of Education Reform? TAUBMAN CENTER POLICY BRIEFS

index of teachers ability to improve student What Do Parents Value in Education? test scores, we examine the relationship Parent satisfaction is an alternative to test between teacher salaries and these value- scores as an indicator of teacher quality. We added measures. We fi nd that teachers with gauge parent satisfaction by the number of characteristics that lead to higher compensation parent requests for placement of their child – experience and educational attainment – do with a specifi c teacher. Roughly 30 percent no better than their colleagues at improving of parents made such requests according to student test scores. First-year teachers and principals’ records from 11 of the 13 schools in untenured teachers do perform slightly worse, the district that provided this information. We but better-paid teachers overall are not notably fi nd that principals’ assessments predict which better at improving test scores. teachers will be requested by parents in their Principals’ own subjective assessments of schools better than either traditional factors teacher capabilities are better predictors of in teacher compensation – education and teacher value-added performance than the experience – or our own teachers’ value-added factors that determine pay levels. Principals assessment measures. However, the types of appear particularly good at identifying teachers teachers preferred by parents vary considerably depending on the demographic characteristics Many parents and educators of the student population in a particular school. view high-stakes testing as On average, parents strongly prefer teachers overly focused on narrow that principals describe in our survey as good educational goals. at promoting student satisfaction. Parents place less value on a teacher’s ability to raise standardized math or reading achievement at either end of the spectrum of performance – as measured either by test scores or principal – i.e., those who will produce the largest assessments. Parent requests correspond more and smallest value-added achievement closely to principals’ subjective assessments of gains in their schools. However, they have teachers’ ability to raise student achievement far less ability to distinguish differences than to the value-added measure, which among teachers in the middle.3 Upon closer indicates either that parents prefer the qualities examination, we fi nd two reasons that captured in the principal assessments more than principals may not do as well with this group. the value-added measures or that parents can In determining the ratings they give teachers, more easily observe the qualities identifi ed by principals appear to place too much emphasis the principal. on the recent performance of their teachers and do not suffi ciently account for the differences These average preferences, however, in initial student ability across classrooms. mask striking differences across family Perhaps not surprisingly, a teacher’s prior demographics. Parents in lower-income value-added score is an even better predictor schools strongly prefer teachers who principal of subsequent student performance than either assessments and value-added indices indicate the teacher’s education and experience, or the are good at improving test scores, as opposed to principal’s assessment of the teacher’s quality. those teachers that principals describe as good at increasing student satisfaction.4 Interestingly, they appear to place little value on the softer teacher qualities recognized by principals. The results are reversed for families in higher- 2 The “Third Way” of Education Reform? TAUBMAN CENTER POLICY BRIEFS

income schools. These parents strongly prefer Our fi ndings may also have implications for teachers who principals describe as good at school choice programs. On one hand, our promoting student satisfaction, and place analysis suggests that low-income families not no value on teachers described as good at only value student achievement, but also will promoting student achievement. be able to recognize which teachers will best It thus seems that what parents want promote these goals. This fi nding may alleviate from school depends largely on family concerns that disadvantaged students will not circumstances. Insofar as children from more benefi t from school-choice programs. On the advantaged households are more likely to read other hand, systematic differences in parent well by the second or third grade, for example, values may also help explain why school- their parents will not prioritize basic phonics choice programs can lead to greater segregation instruction. Parents whose children are still between racial and income groups. Other struggling with basic literacy will place greater studies have shown that parents tend to choose value on instruction that emphasizes basic schools with racially and socially homogenous skills. environments for their children. Our fi ndings suggest that parents may not desire segregation This has important implications for current per se, but instead view the socioeconomic school reform strategies, especially those composition of a school as a signal about that rely on mechanisms of parent choice whether others at the school share their interest in particular types of curriculum or pedagogy. Parents in lower-income schools Of course, because we study the choice of strongly prefer teachers who are teachers within schools, rather than the choice good at improving test scores, of schools within a district or metropolitan area, as opposed to those teachers these conclusions must be considered tentative. that principals describe as Finally, our analysis suggests caution in good at increasing student interpreting school-choice programs in general. Low-income families, for instance, may choose satisfaction. The results are to attend a school with a high proportion of reversed for families in higher- other low-income families and low average test income schools. scores because they believe that the parents in these schools have preferences that, like their in selecting schools. Our fi ndings suggest own, prioritize student achievement over other that communities will react differently to goals. Conversely, high-income parents may accountability policies, such as those embodied choose to attend schools with high test scores in the federal accountability legislation No not because those schools engage in the basic Child Left Behind (NCLB), depending on skills and test preparation that is most helpful their demographic makeup. Specifi cally, since for increasing scores, but for completely schools where fewer students are eligible for opposite reasons – namely, because the income-tested free-lunch programs prefer preferences of families in those schools signal teachers that promote student satisfaction that teachers will engage in less basic skills over testing gains, we predict that higher- instruction and offer instead a broad curriculum income communities will express greater and activities that increase student engagement. dissatisfaction with the achievement emphasis of NCLB. 3 The “Third Way” of Education Reform? TAUBMAN CENTER POLICY BRIEFS

Empowering Principals in pedagogy, classroom management, or curriculum. Principals can also evaluate Our fi ndings suggest that principals should teachers on the basis of a broader spectrum of have greater authority for school-wide staffi ng educational outputs than test scores alone. decisions, as they do at most charter schools. Currently, principals have some authority New powers for principals should also include over the promotion of untenured teachers, but safeguards. One reason that teachers formed virtually no role in determining the promotion unions in the fi rst place was to defend against or compensation of tenured teachers. In fact, abuse and patronage by earlier generations for a host of complicated social and political of superintendents and principals. Our own reasons, principals have been reluctant to results suggest, for instance, that principals exercise the authority they do have over may discriminate against male and untenured untenured teachers. It is rare, for example, for faculty.5 If teachers are to be held accountable an untenured teacher in a large school district to principals, then principals must be held to be dismissed for mere incompetence. One accountable by other mechanisms. immediate way that superintendents could Another question is whether empowering improve schools is by setting high standards principals might lead to the kinds of for teaching excellence and then creating an unanticipated negative effects as those from environment in which principals are not only high-stakes testing. Our analysis takes place in allowed, but actually expected, to dismiss a context where principals were not explicitly ineffective untenured teachers evaluated by their superiors on the basis of Principals’ subjective performance evaluations their ability to identify effective teachers. It of teachers may thus offer a way out of the is possible that moving to a system where “testing/accountability” versus “professional principals had more authority and responsibility autonomy” dilemma. Insofar as principals for monitoring teacher effectiveness would can observe inputs as well as outputs, they enhance principals’ ability to identify various may be able to ensure that teachers increase teacher characteristics. On the other hand, it is student achievement through improvements possible that principals would be less willing to

“Teaching to the Test” and Cheating Test score gains are not synonymous with improved student learning. One reason many people view the scores as an imperfect indicator of educational success is that the impres- sive gains produced by high-stakes testing can also be artifi cially induced by “teaching to the test” and even cheating. My own research on the introduction of high-stakes testing in Chicago, for instance, indicates that some portion of the achievement gains in certain grades likely resulted from teachers instructing in test-specifi c skills, de-emphasizing sub- jects without high-stakes exams like science and social studies, and placing more students into special education where their scores often do not count.6 Moreover, policies that attach stricter consequences for low test scores prompt outright cheating by teachers and adminis- trators that can be discerned by tell-tale patterns of test answers along with unusually large test score jumps. Using such methods, Steve Levitt and I found serious cheating in at least 4 to 5 percent of elementary classrooms in the Chicago Public Schools, a fi nding largely confi rmed by subsequent retesting that launched offi cial investigations.7

4 The “Third Way” of Education Reform? TAUBMAN CENTER POLICY BRIEFS

honestly assess teachers under such a system, meaningful principal evaluations would perhaps because of social or political pressures. improve both student achievement and parent Just as high-stakes exams spur cheating and satisfaction. The relative merits of principal- “teaching to the test” (see side bar), high-stakes versus-test assessment is less clear.8 But principal assessments could lead to perverse empowered principals would likely enhance outcomes. parent satisfaction, identify teachers who Nonetheless, our results suggest that are particularly good or bad at promoting switching from the current compensation student achievement, and mitigate the negative system toward one that incorporates more consequences of high-stakes exams.

Figure 1: How Three Teacher Assessment Systems Predict Student Test Gains and Parent Preferences

5 The “Third Way” of Education Reform? TAUBMAN CENTER POLICY BRIEFS

Endnotes RELATED PUBLICATIONS 1 “Teacher Merit Pay Tied to Educational Gains,” The New York Times, October 4, 2005 “What do Parents Value in Education: An Empirical Investigation of Parents Revealed 2 The district, located in the western United States, Preferences for Teachers,” gave permission for these February 2003 interviews and has requested to remain anonymous. The district is not By Brian A. Jacob and Lars Lefgren, National unrepresentative of American public schools as a whole. Bureau of Economic Research Working Paper While students in the district are predominantly white (73 11494 (2005) [http://ksghome.harvard.edu/~bjacob/ percent), Latinos comprise 21 percent of the elementary parents27.pdf] population. Nearly half of all students (48 percent) in the district receive free or reduced price lunch. Testing “Principals as Agents: Subjective Performance achievement levels in the district are almost exactly at the Measurement in Education” average for the nation (49th percentile on the Stanford By Brian A. Jacob and Lars Lefgren, National Achievement Test). The sample consists of 202 teachers Bureau of Economic Research Working Paper in grades two through six. (2005) [http://ksghome.harvard.edu/~bjacob/ 3 Principals appear somewhat better at identifying top and principals35.pdf] bottom-quality teachers for math than reading. 4 Disadvantaged parents are relatively less likely to “Accountability, Incentives and Behavior: The request any teacher. But their relative proclivity to Impact of High-Stakes Testing in Chicago Public choose teachers based on test-score results is particularly Schools,” striking given that these parents are arguably less likely By Brian A. Jacob, Journal of Public Economics, to be adept at inferring judgments based on comparative 89, 5-6 (2005) [http://ksghome.harvard.edu/ numerical data. ~bjacob/highstakes16.pdf] 5 Principals do not report worse personal relationships between themselves and male or untenured teachers, “Rotten Apples: An Investigation of the but they give systematically give lower ratings to these Prevalence and Predictors of Teacher Cheating,” teachers than the teachers’ value-added scores would By Brian A. Jacob and Steven D. Levitt, Quarterly otherwise dictate. Journal of Economics, 118, 3 (2003) [http:// 6 Once high-stakes consequences were introduced, ksghome.harvard.edu/~bjacob/cheating.pdf] students seem to have left fewer exam answers blank where there was no penalty for guessing. Fewer students “Catching Cheating Teachers: The Results of an also seem to give up in the fi nal sections of exams. Unusual Experiment in Implementing Theory” 7 We look for cases where particular teachers’ students By Brian A. Jacob and Steven D. Levitt, Brookings- experience unusually large test score gains, followed by Wharton Papers on Urban Affairs, William G. unusually small gains or even declines in the following Gale and Janet Rothenberg Pack, eds. (Brookings year. We also look for patterns of suspicious answer Institution Press, 2003) [http://ksghome.harvard. strings, such as identical blocks of answers for many edu/~bjacob/bw03cht_abs.htm] students in a classroom, or cases where students are unable to answer easy questions correctly, but do exceptionally well on the most diffi cult questions. Following our fi ndings, the Chicago Public Schools conducted extensive retesting of suspicious classrooms and controls groups. The results largely confi rmed our fi ndings and led to investigation of 29 classrooms. 8.Widespread implementation of value-added assessment of test scores might itself face obstacles because the approach implicitly designates certain groups as harder to teach at the time. It does this by statistically anticipating how students of certain races, gender, and income will have greater or lesser diffi culty achieving testing gains.

6