ROTTENAPPLES: AN INVESTIGATION OFTHE PREVALENCE AND PREDICTORS OFTEACHER CHEATING* BRIAN A. JACOB AND STEVEN D. LEVITT Wedevelopan algorithm for detecting teacher cheating that combines infor- mationon unexpectedtest score uctuationsand suspicious patterns of answers forstudents in a classroom.Using data from the Chicago public schools, we estimatethat serious cases of teacheror administratorcheating on standardized testsoccur ina minimumof 4 –5percentof elementary school classrooms annu- ally.The observed frequency of cheatingappears to respond strongly to relatively minorchanges in incentives. Our resultshighlight the fact thathigh-powered incentivesystems, especially those with bright line rules, may induce unexpected behavioraldistortions such as cheating. Statistical analysis, however, may pro- videa meansof detecting illicit acts, despite the best attempts of perpetratorsto keepthem clandestine. I. INTRODUCTION High-poweredincentive schemes are designed toalign the behaviorof agentswith theinterests of theprincipal implement- ing thesystem. A shortcomingof suchschemes, however, is that theyare likely to induce behavior distortions along other dimen- sionsas agentsseek to game the rules (see, for instance, Holm- stromand Milgrom[1991] and Baker[1992]). Thedistortions maybe particularly pronouncedin systemswith bright linerules [Glaeserand Shleifer2001]. It maybe impossible to anticipate themany ways in whicha particular incentivescheme may be gamed. Test-based accountability systemsin educationprovide an excellentexample of the costs and benets ofhigh-powered in- centiveschemes. In an effortto improve student achievement,a numberof states and districts haverecently implemented pro- gramsthat usestudent testscores to punish orreward schools. *Wewould like to thank Suzanne Cooper, Mark Duggan, SusanDynarski, ArneDuncan, Michael Greenstone, James Heckman, Lars Lefgren, two anony- mousreferees, the editor, Edward Glaeser,and seminar participants too numer- ousto mention for helpful comments and discussions. We also thank Arne Dun- can,Philip Hansen, Carol Perlman, and Jessie Qualles of the Chicago public schoolsfor their help and cooperation on the project. Financialsupport was providedby the National Science Foundation and the Sloan Foundation. All remainingerrors are our own. Addresses: Brian Jacob, Kennedy School of Gov- ernment,Harvard University, 79 JFK Street,Cambridge, MA 02138;Steven Levitt,Department of Economics, University of Chicago, 1126 E. 59thStreet, Chicago,IL 60637. © 2003by thePresident and Fellowsof Harvard Collegeand theMassachusetts Institute of Technology. The Quarterly Journal ofEconomics, August 2003 843 844 QUARTERLY JOURNALOF ECONOMICS Recentfederal legislation institutionalizes this practice,requir- ing statesto test elementary students eachyear, rate schools on thebasis ofstudent performance,and intervenein schoolsthat do notmake suf cient improvement. 1 Severalprior studies suggest that suchaccountability policiesmay be effective at raising stu- dent achievement[Richards and Sheu1992; Grissmer,Flanagan, etal. 2000; Deereand Strayer2001; Jacob2002; Carnoyand Loeb 2002; Hanushekand Raymond 2002]. At thesame time, however, researchershave documented instances of manipulation,includ- ing documentedshifts away fromnontested areas or “ teachingto thetest” [Klein etal.2002; Jacob2002], and increasingplacement in special education[Jacob 2002; Figlioand Getzler2002; Cullen and Reback2002]. In this paper weexplore a verydifferent mechanismfor inating testscores: outright cheating on thepart ofteachers and administrators. 2 As incentivesfor high testscores increase, un- scrupulousteachers may be more likely to engage in arangeof illicit activities,including changingstudent responseson answer sheets,providing correctanswers to students, or obtaining copies ofan examillegitimately prior to the test date and teaching students using knowledgeof the precise exam questions. 3 While suchallegations may seem far-fetched, documented cases of cheatinghave recently been uncovered in California [May 2000], Massachusetts [Marcus 2000], NewYork [Loughran and Comis- key1999], Texas[Kolker 1999], and GreatBritain [Hofkins1995; Tysome1994]. Therehas beenvery little previous empirical analysis of teachercheating. 4 Thefew studies that doexist involve investi- 1.The federal legislation, No Child Left Behind, was passed in 2001.Prior to thislegislation, virtually every state had linked test-score outcomes to school fundingor requiredstudents to pass an exitexamination to graduate high school. Inthe state of California,a policyproviding for merit pay bonusesof asmuchas $25,000per teacher in schools with large test score gains was recently put into place. 2.Hereinafter, we uses the phrase “ teachercheating” to encompass cheating doneby eitherteachers or administrators. 3.We have no way ofknowingwhether the patterns we observe arise because ateacherexplicitly alters students’ answer sheets, directly provides answers to studentsduring atest,or perhapsmakes test materials available to students in advanceof the exam (for instance, by teachinga readingpassage that is on the test).If we had access to the actual exams, it might be possible to distinguish betweenthese scenarios through an analysisof erasure patterns. 4.In contrast, there is a well-developedstatistics literature for identifying whetherone student has copied answers from another student [Wollack 1997; Holland1996; Frary 1993;Bellezza and Bellezza 1989; Frary, Tideman,and Watts1977; Angoff 1974]. These methods involve the identi cation of unusual ROTTEN APPLES 845 gationsof specic instancesof cheatingand generallyrely on the analysis oferasure patterns and thecontrolled retesting of stu- dents.5 Whilethis earlierresearch provides convincing evidence ofisolated cheating incidents, our paper representsthe rstsys- tematicattempt to (1) identify theoverall prevalence of teacher cheatingempirically and (2) analyze thefactors that predict cheating.To address thesequestions, we use detailed adminis- trativedata fromthe Chicago public schools(CPS) that includes thequestion-by-question answers given by everystudent in grades 3to8 whotook the Iowa Testof Basic Skills (ITBS) from 1993 to2000. 6 In addition tothe test responses, we also have accessto eachstudent’ s full academicrecord, including past test scores,the school and roomto which a student was assigned,and extensivedemographic and socioeconomiccharacteristics. Ourapproach todetecting classroom cheating combines twotypes ofindicators :unexpectedtestscore uctuationsand unusual patterns ofanswers for students within aclassroom. Teachercheating increases the likelihoo dthat students in a classroomwill experiencelarge, unexpecte dincreasesin test scoresone year, followed by verysmall testscore gains (oreven declines)the following year. Teacher cheating, especially if donein an unsophisticated manner,is alsolikely to leave tell-tale signs in theform of blocks of identical answers,un- usual patterns ofcorrelat ionsacross student answerswithin theclassroom ,orunusual responsepatterns within astudent’s exam(e.g., a student whoanswers a numberof very dif cult patternsof agreement in student responses and, for the most part, areonly effectivein identifyingthe most egregious cases of copying. 5.In the mid-eighties, Perlman [1985] investigated suspected cheating in a numberof Chicago public schools (CPS). Thestudy included23 suspect schools— identied onthebasis of a highpercentage of erasures,unusual patterns of score increases,unnecessarily large orders of blankanswer sheets for the ITBS and tips tothe CPS Ofce of Research—along with 17 comparisonschools. When a second formof thetest was administered to the 40 schools under more controlled condi- tions,the suspect schools did much worse than the comparison schools. An analysisof several dozen Los Angeles schools where the percentage of erasures andchanged answers was unusually high revealed evidence of teachercheating [Aiken1991]. One of themost highly publicized cheating scandals involved Strat- eldelementary, an award-winningschool in Connecticut.In 1996 the rm that developedand scored the exam found that the rate of erasures at Strateld was up to vetimes greater than other schools in the same district andthat 89 percentof erasuresat Strat eld were from an incorrect toa correct response.Subsequent retestingresulted in signi cantly lower scores [Lindsay 1996]. 6.We do not, however, have access to the actual test forms that students lledout so we are unable to analyze these tests for evidence of suspicious patternsof erasures. 846 QUARTERLY JOURNALOF ECONOMICS questionscorrectl ywhilemissing many simple questions ).Our identication strategy exploits the fact that thesetwo types of indicators arevery weakly correlate dinclassroomsunlikelyto havecheated, but veryhighly correlatedin situationswhere cheatinglikely occurred .Thatallows us tocredibly estimate theprevalenc eofcheatingwithout having toinvoke arbitrary cutoffs as towhat constitutescheating. Empirically, wedetect cheating in approximately4 to5 per- centof the classes in oursample. This estimate is likelyto understatethe true incidence of cheating for two reasons. First, wefocus only on the most egregious type ofcheating, where teacherssystematically alter student testforms. There are other moresubtle ways in whichteachers can cheat, such as providing extratime to students, that ouralgorithm is unlikelyto detect. Second,even when test forms are altered, our approach is only partially successfulin detectingillicit behavior.As