Rappaport Institute for Greater Boston POLICY BRIEFS Taubman Center for State and Local Government

February 2008

The Seven Big Errors of PerformanceStat By Robert D. Behn, John F. Kennedy School of Government

Maybe you’ve read about T.E.A.M.S. (for Total Effi ciency “CompStat.” Perhaps a friend told Accountability Management System), Rappaport Institute/Taubman Center Policy Briefs are short overviews of new you about “CitiStat.” But who is this and the Probation Department created and notable research on key issues by “PerformanceStat” — and, if he is S.T.A.R.S (for Statistical Tracking, scholars affi liated with the Institute and the Center. making so many errors, why hasn’t the Analysis & Reporting). Robert D. Behn manager taken him out of the lineup? Next came ’s CitiStat, Robert D. Behn, a lecturer in public the adaptation of this innovation to policy at Harvard’s Kennedy School of A Short History of PerformanceStat Government, is the faculty chair of the improve performance in an entire Kennedy School’s executive-education It all began in 1994, when program on “Driving Government jurisdiction, created in 2000 by Mayor Performance: Leadership Strategies Commissioner and Martin O’Malley. This prompted That Produce Results.” He writes the his leadership team at the New York online monthly, Bob Behn’s Public similar approaches in other cities Management Report, and is the author Police Department created CompStat, — from the large, such as Atlanta of numerous articles, books, and reports their leadership and management (ATLStat) and San Francisco (SFStat), on performance leadership. strategy designed to reduce the city’s to the small, such as Palm Bay, © 2008 by Robert D. Behn. The content crime rate. Quickly other police refl ects the views of the author (who is Florida (PalmStat), and Somerville, responsible for the facts and accuracy departments adopted this innovation. Massachusetts (SomerStat). And then, of the research herein) and do not represent the offi cial views or policies of Only fi ve years later, a survey by 3 as Ellen Perlman noted in Governing , the Rappaport Institute or the Taubman the Police Foundation found that “‘Stat’ Fever” really got hot. Center. approximately a third of the 515 After all, this management approach departments in the U.S. with 100 or Rappaport Institute for is not uniquely applicable to more sworn police offi cers reported Greater Boston John F. Kennedy School of Government 1 municipal government. In 2002, the implementing a version of CompStat . 79 JFK Street, Cambridge, MA 02138 Ohio Department of Job and Family Today, police departments around Telephone: (617) 495-5091 Services created its “Performance the world employ this strategy, often Email: [email protected] Center.” In 2005 in Washington, giving it their own name; in Australia, http://www.ksg.harvard.edu/rappaport Governor Christine Gregoire numerous police organizations conduct A. Alfred Taubman Center for developed GMAP (for Government what are often called “Operational State and Local Government Management Accountability, and John F. Kennedy School of Government, Performance Reviews.”2 Performance), and, in 2007, when 79 JFK Street, Cambridge, MA 02138 Then other agencies Martin O’Malley became governor of Telephone: (617) 495-2199 adapted the approach. For example, Email: [email protected] Maryland, he created StateStat. www.ksg.harvard.edu/taubmancenter the Parks Department created Moreover, at least one unit of the ParkStat, the Human Resources federal government, the San Diego Administration created JobStat, district of the U.S. Border Patrol, the Correction Department created has created its own version of this The Seven Big Errors of PerformanceStat Rappaport Institute | Taubman Center POLICY BRIEFS

approach, which it labeled BorderStat. And they also employ these data to compare how overseas, several cities in Scotland, including well different sub-units are doing, to set Aberdeen, Edinburgh, and Sterling, have targets for future results, and thus to motivate experimented with CitiStat.4 the individuals in those units to achieve their targets. As I have studied diff erent And yet, as I have studied different examples examples of PerformanceStat, of PerformanceStat, examined formal examined formal descriptions, descriptions, questioned key executives about questioned key executives about their approach, and observed many in action,7 their approach, and observed I’ve been struck by how many don’t quite many in action, I’ve been struck appreciate (or at least employ) some of the core principles that can make the strategy effective. by how many don’t quite Yes, they do the formal, visible things that appreciate (or at least employ) you would associate with a PerformanceStat some of the core principles that approach. Yet, something subtle but important can make the strategy eff ective. is missing. Specifi cally, I have identifi ed seven big mistakes. All of these adaptations of the original First, however, I should provide my defi nition CompStat innovation — regardless of whether of “PerformanceStat”: their names include the ***Stat suffi x — are based on the same premise: Government A jurisdiction or agency is employing a needs to improve its performance, and, to PerformanceStat performance strategy do so, it needs a demanding and strategic if it holds an ongoing series of regular, approach. To capture these various but similar frequent, periodic, integrated meetings performance strategies, I have chosen the name during which the chief executive and/ “PerformanceStat.”5 or the principal members of the chief executive’s leadership team plus the What Is PerformanceStat? individual director (and the top managers) of different sub-units use data to analyze Every one of these PerformanceStat strategies the unit’s past performance, to follow-up is different. They have to be. Neither the nature on previous decisions and commitments of the performance each seeks to improve, to improve performance, to establish nor their political and organizational context, its next performance objectives, and to is the same. The leadership team of each examine the effectiveness of its overall jurisdiction and each agency has to adapt the performance strategies. basic principles of PerformanceStat to its own objectives and circumstances. This is not very restrictive. Lots of managerial activities fi t within this defi nition. Nevertheless, effective adaptations of PerformanceStat strategy all use data — and Thus, I’m not complaining about public do so in two important ways. First, they executives who fail to implement the collect and analyze data to determine the type idiosyncratic technicalities of my own, narrow, and level of results that the organization is eccentric doctrine. Rather, I’m concerned about producing, to detect its important “performance jurisdictions or agencies that miss something defi cits,”6 and to suggest policies and practices very basic, something that should be central to that might produce improvements. Second, any effort to improve performance (whether 2 Rappaport Institute Taubman Center POLICY BRIEFS The Seven Big Errors of PerformanceStat |

or not it is a PerformanceStat approach), These responsibilities could be to reach specifi c something that can divert a real opportunity to output targets: The director of the public works produce improved results into little more than department (and the head of each public works an utterly simplistic, noticeably ineffective, and district) could be given the responsibility thus purely symbolic sham. of fi lling every pothole that citizens report within 48 hours. Such output responsibilities Error #1: No Clear Purpose are relatively easy to achieve. Organizations In the public sector, any undertaking has actually produce outputs. So if the organization to begin with a defi nition of the purpose to has (or can obtain) the necessary capabilities9 be achieved. In New York, Commissioner — people, equipment, knowledge — it can Bratton began with a brash — and very explicit directly produce the outputs for which it is — purpose: to reduce the city’s crime. In responsible. Baltimore, Mayor O’Malley had his own clear objective: to improve the delivery of traditional As Nietzsche once noted, city services. “forgetting our objectives Yet, as Nietzsche once noted, “forgetting is the most frequent act of our objectives is the most frequent act of stupidity.” Indeed, too often, 8 stupidity.” Indeed, too often PerformanceStat PerformanceStat is nothing more is nothing more than the latest government fad. Upon hearing about the approach, the manager than the latest government fad. exclaims, “Ooh, cool hammer,” and goes looking for some convenient nails to pound. These responsibilities could be to reach specifi c outcome targets: The superintendent of schools As always, however, public managers need to and the principal of every elementary school start with a clear purpose: “What results are could be given the responsibility of ensuring we trying to produce?” “What would better that all sixth-grade students can add, subtract, performance look like?” “How might we know multiply, and divide (as measured on some if we have made some improvements?” Only yardstick test). The police commissioner and after the members of the leadership team have the commander of every police precinct could agreed to some common answers to these be given the responsibility of reducing the questions can they adapt the PerformanceStat number of GPS devices stolen from cars by 10 strategy to help them achieve these — now percent this year. Such outcome responsibilities very explicit — purposes. are harder to achieve. Organizations produce Error #2: No One Has Specifi c the outputs that contribute to these outcomes Responsibilities — but they don’t directly produce the outcomes themselves. Lots of other people — parents in Who will do all this? Who is responsible for particular — make a signifi cant contribution to what? To produce results in any organization, how much students learn. And lots of factors someone must do this producing. But who? affect the number of GPS systems stolen, And what? including whether GPS owners leave their Indeed, no one can answer the Who? question devices in their cars or whether automobile until they have fi rst answered the What? owners simply purchase more of them. question. This requires converting the clear Or, these responsibilities could be to develop purposes into specifi c responsibilities. These new strategies. If a school system or a specifi c responsibilities can take on various forms. 3 The Seven Big Errors of PerformanceStat Rappaport Institute | Taubman Center POLICY BRIEFS

school is not achieving its educational outcome each sub-unit. And, they keep the management targets, or if a police department or precinct leadership team of each sub-unit focused on is not achieving its crime reduction targets, achieving its targets. it could be charged with developing a new To accomplish all this, the participants in these strategy. This might appear more squishy meetings examine performance of each sub- — less of a target and more of a wish. Yet, unit since the last meeting: Has the sub-unit if the organization charged with developing done what it promised at the last meeting? the strategy must also implement it in a way Has it hit its targets? How? Why not? What that eventually achieves an output or outcome are the successes? What are the problems, target, the responsibility is no less real. inadequacies, weaknesses, and shortfalls? Eventually the people who created the strategy Should the targets be changed for the next have to demonstrate their own brilliance and period? Do specifi c problems need to be fi xed? their organization’s competence by using the If so, by whom and by when? Should the strategy to produce the real results. sub-unit be charged with developing a new strategy? If so, by whom should it be crafted The chief executive needs and by when should it be implemented? to delegate — offi cially and When Baltimore launched CitiStat, Jay Sakai unequivocally — a key deputy was one of the fi rst members of its analytic to conduct every meeting. staff. A few years later, he moved to head Otherwise, from one meeting the city’s Bureau of Water and Wastewater. to the next, there will be no Thus, Sakai’s relationship to CitiStat changed consistency of purpose — no signifi cantly. Originally, he had been analyzing ability to focus on the completion the data of the city’s operating agencies. Now he was managing one. of specifi c responsibilities. Moreover, Sakai’s thinking about the CitiStat’s routine of bi-weekly meetings for every agency At almost all PerformanceStat meetings, the changed. As a CitiStat analyst, he thought discussion is about results. But that does not the meetings were too frequent. But when he guarantee that this discussion ever touches on began as a line manager, he realized that the bi- who is responsible for what results. Indeed, an weekly schedule made sense. Why? Because if organization can hold meeting after meeting the meetings were less frequent, he could leave without ever clarifying who specifi cally is the meeting relieved, thinking: Thank goodness supposed to accomplish what exactly. that meeting is over; I won’t have to worry about that for a while. But, if the next meeting Error #3: The Meetings are Held Irregularly, is only 14 days away, it isn’t really over: I have Infrequently, or Randomly to start to work right now on the problems that have been identifi ed and the commitments that An important component of the have been made; otherwise, in two weeks, I and PerformanceStat strategy is the ongoing my bureau will be embarrassed. series of regular, frequent, periodic, integrated meetings. They provide feedback on both There is nothing magic about Baltimore’s achievements and failures. They seek to two-week interval. These meetings could identify lessons for improving performance be biweekly or monthly (or perhaps even in the future. They keep the organization’s quarterly). After all, the frequency of the leaders up-to-date on what is happening in meetings depends upon how frequently new 4 The Seven Big Errors of PerformanceStat Rappaport Institute | Taubman Center POLICY BRIEFS

data become available. If performance data are Ideally, this would be the chief executive: the available only monthly, it makes little sense to police commissioner, the mayor, or the agency hold bi-weekly meetings. head. In many circumstances, however, this Nevertheless, if an agency or jurisdiction individual has too many other responsibilities. is serious about using this strategy to A mayor, for example, has so many demands improve performance, it ought to hold its on his or her time — demands that cannot PerformanceStat meetings on a regular and be conveniently scheduled around the frequent basis. If individual managers and their PerformanceStat calendar. units are to improve performance, they need Consequently, the chief executive needs to to know three things: (a) when the next data delegate — offi cially and unequivocally cycle ends, (b) what specifi cally they need to — a key deputy to conduct every meeting. do to demonstrate improvement by the end of Otherwise, from one meeting to the next, that cycle, and (c) when they will report back there will be no consistency of purpose — no in front of their organizational superiors, peers, ability to focus on the completion of specifi c and subordinates on how much they have responsibilities. accomplished on making these improvements. Error #5: No Dedicated Analytic Staff These meetings are both substantive and symbolic. Because these meetings examine PerformanceStat requires data — data that specifi c performance defi cits, explore possible illustrates the current level of performance. solutions, and produce specifi c commitments Who, however, looks at the data? Who analyzes for specifi c actions to be completed by specifi c the data in an attempt to fi gure out whether dates, they create real opportunities to produce performance is improving or not? Who better results. And because these meetings examines the data and tries to fi gure out what require the active engagement of most of the new approaches should be considered? agency’s or jurisdiction’s key executives, The managers of the various sub-units need they dramatize that the issues being analyzed, to do this. But the leadership team of the discussed, and debated are important. jurisdiction or agency needs a few people to do this too. Error #4: No One Person Authorized to Run the Meetings And these people can’t also have ten other, higher-priority tasks. For the PerformanceStat For these meetings to work, however, someone strategy to produce meaningful results, it needs has to conduct each meeting. Moreover, that a few analytical people working on it full-time someone ought to be the same someone. The to understand — through the use of data — PerformanceStat strategy depends upon the what kind of results are really being produced. regular, periodic discussions of performance, but these discussions will have little impact if Error #6: No Follow-Up they are conducted by a rotating collective of What is the relationship between the issues random offi cials. discussed at the previous meeting and those The person who conducted last month’s examined at the meeting today? Did today’s meeting must conduct this month’s meeting. meeting build on the problems identifi ed, Otherwise, the continuity in the analysis of solutions analyzed, and commitments made performance is lost. Moreover, to do this, at the previous meeting? Or are we, yet again, the person who runs the meeting needs clear starting all over? If the PerformanceStat authority. approach is to produce real improvements in 5 The Seven Big Errors of PerformanceStat Rappaport Institute | Taubman Center POLICY BRIEFS

results, it has to focus on the key results that analytic staff, and if it has failed to conduct need improvement. And it has to focus on them any follow-up since the previous meeting, it at meeting, after meeting, after meeting. is unable to do much more than applaud this Of course, if there is no clear purpose or delightful show. no clear responsibilities, there will be little Still, the leadership team can’t let sub-units off on which to follow-up. And if there is no the hook when they offer bland assertions of analytical staff, there will be no one charged wonderful progress without offering any data with providing the briefi ng materials to suggest as evidence. Conversely, they can’t also rely on what to follow-up. Finally, if there is no one purely on brutal censure without offering an individual authorized to run the meetings, there opportunity to improve and earn compliments. will be no one who can follow-up. To truly improve any sub-unit’s performance, And, with no follow-up, the PerformanceStat the leadership team needs to both pressure its will be little more than PerformanceSham. managers and help them to succeed.

Error #7: No Balance Between the Brutal Adapting the Principles of and the Bland PerformanceStat Both NYPD’s CompStat and Baltimore’s PerformanceStat isn’t a model. It can’t simply CitiStat are known for being tough and be copied. It isn’t a system. It can’t be airlifted uncompromising with poor performers. A from one organization into another. Obtaining report by the Police Foundation found that the benefi ts of this approach to performance CompStat had “a reputation among line offi cers — using this strategy to produce real as brutal and punitive rather than collaborative improvements in results — requires more than and creative.”10 The CitiStat meetings in the mindless mimicry of the most visible and Baltimore and the ATLStat meetings in most superfi cial elements of the approach. Atlanta have been described as “brutal, unsentimental affairs.”11 Indeed, both NYPD Obtaining the benefi ts from and Baltimore have accumulated an image of this approach to performance being aggressively demanding, sometimes even — using this strategy to produce sarcastically demeaning. real improvements in results Yet in an overreaction to NYPD’s and — requires more than the mindless Baltimore’s reputation, some jurisdictions mimicry of the most visible and and agencies have consciously tried to make most superfi cial elements of the their meetings as harmonious as possible. approach. As a result, their meetings have become mostly show-and-tell. The director of each subordinate unit (be that a precinct commander Yet this is what Eli Silverman of the or an agency head) essentially runs the John Jay College of meeting, showing a series of PowerPoint slides suggests many police departments have and presenting yet another glowing picture of done. He reports that many of the efforts the unit’s latest accomplishments. at “replication are frequently based on a superfi cial understanding” of the strategy.12 Unfortunately, if the leadership team has failed PerformanceStat is more than some fancy to specify what it is trying to accomplish, technology and a series of meetings; yet, if it has failed to designate someone to run every meeting, if it has failed to create its own 6 The Seven Big Errors of PerformanceStat Rappaport Institute | Taubman Center POLICY BRIEFS

when you visit any particular example, that is (Edinburgh: Scottish Executive Social Research, primarily what you will see. 2006). PerformanceStat is a leadership and 5 Thus, PerformanceStat is not the latest, multi- management strategy that public executives variable statistic invented by the Sabermetricians can employ to produce real results in a to fi nally determine who really is the best baseball variety of government jurisdictions and player. It has absolutely nothing to do with baseball, let alone Bill James, or Billy Beane, or public agencies. But to do so, they need not Theo Epstein. only to ascertain the key components of 6 the strategy but also to develop their own Robert D. Behn, “On Why Public Managers complex appreciation of the cause-and-effect Need to Focus on Their Performance Defi cit,” relationships among these components and the Bob Behn’s Public Management Report, vol. 4, no. 1 (September 2006). results to be produced. To make an intelligent adaptation of any leadership or management 7 I have observed: CompStat in Lowell and strategy, public executives must understand Boston, Massachusetts, and in ; how this approach can work — what the cause- Traffi cStat, JobStat, T.E.A.M.S., and S.T.A.R.S and-effect connection is between their actions in New York City; DPSSTAT at the Los Angeles County Department of Public Social Services; and their results. Then, they need to adapt these CitiStat in Baltimore, SomerStat in Somerville, cause-and-effect concepts to refl ect their own Massachusetts, ProvStat in Providence Rhode unique circumstances as well as the purposes Island, SyraStat in Syracuse, New York, ATLStat they are attempting to achieve. in Atlanta, PalmStat in Palm Bay Florida, ColumbusStat in Columbus, Ohio, SFStat in San Francisco; GMAP in Olympia, Washington and Endnotes the Performance Center in Columbus Ohio.

1 David Weisburd, Stephen D. Mastrofski, Rosann 8 Friedrich Nietzsche, The Wanderer and His Greenspan, and James J. Willis, The Growth of Shadow, (online at http://www.davemckay. Compstat in American Policing, Police Foundation co.uk/philosophy/nietzsche/nietzsche. Reports (Washington, D.C.: Police Foundation, php?name=nietzsche.1878.humanalltoohuman. April 2004), pp. 4, 6. See also: David Weisburd, zimmern.12), p. 206. Stephen Mastrofski, Ann Marie McNally and 9 Another error of PerformanceStat is the Rosann Greenspan, Compstat and Organizational failure to ensure that the organization charged Change: Findings from a National Survey, Report with a specifi c responsibility has the necessary submitted to the National Institute of Justice by operational capacity. But this is a common failure the Police Foundation, (Washington, D.C.: Police in government, and so I have not included it Foundation, 2001). on this list. See Robert D. Behn, Performance 2 Lorraine Mazerolle, Sacha Rombouts, and Leadership: 11 Better Practices That Can Ratchet James McBroom, “The Impact of COMPSTAT Up Performance (Washington, D.C.: IBM Center on Reported Crime in Queensland,” Policing: for the Business of Government, 2004), p. 16. An International Journal of Police Strategies & 10 James J. Willis, Stephen D. Mastrofski, and Management, vol. 30, no. 2 (2007), p. 238. David Weisburd, Compstat in Practice: An 3 Ellen Perlman, “‘Stat’ Fever,” Governing, January In-Depth Analysis of Three Cities (Washington, 2007, p. 48. D.C.: The Police Foundation, 2003), p. 21. 4 Cathy Sharp, Jocelyn Jones, and Alison M. Smith, 11 Shirley Franklin, “After New Orleans,” Esquire, What Do We Measure and Why? An Evaluation of December 2005. the Citistat Model of Performance Management and its Applicability to the Scottish Public Sector 7 Rappaport Institute | Taubman Center POLICY BRIEFS

12 Eli B. Silverman, “Compstat’s Innovation,” in “The Core Drivers of CitiStat: It’s Not Just David Weisburd and Anthony A. Braga (eds.), About the Meetings and the Maps,” Robert Police Innovation: Contrasting Perspectives (New D. Behn, International Public Management York: Cambridge University Press, 2006) p. 267. Journal, Vol. 8, No. 3 (2005), pp. 1-25 “Can CitiStat Work in Greater Boston?” Further Reading Phineas Baxandall and Charles Euchner, Cambridge, MA: Rappaport Institute for Greater Boston, 2003. Online at http://www. “What All Mayors Would Like to Know About ksg.harvard.edu/rappaport/downloads/citistat/ Baltimore’s CitiStat Performance Strategy,” fi nal_citistat_report.pdf. Robert D. Behn, Washington, D.C.: IBM Center for the Business of Government, 2007. “The Baltimore CitiStat Program: Online at http://www.businessofgovernment. Performance and Accountability,” Lenneal org/pdfs/BehnReportCiti.pdf. J. Henderson, Washington, D. C: IBM Center for the Business of Government, 2003. “Designing PerformanceStat: Or What are the Online at: http://www.businessofgovernment. Key Strategic Choices that a Jurisdiction org/pdfs/HendersonReport.pdf. or Agency Must Make When Adapting the CompStat/CitiStat Class of Performance “The SchoolStat Model,” Strategies?” Robert D. Behn, Presented at the Christopher Patusky, Leigh Botwinik, and Twenty-Ninth Annual Research Conference of Mary Shelley, Washington, D.C.: IBM Center the Association for Public Policy Analysis and for the Business of Government, 2007. Management, Washington, D.C., November 9, Online at http://www.businessofgovernment. 2007 org/pdfs/PatuskyReport.pdf. “The Theory Behind Baltimore’s CitiStat,” “What Do We Measure and Why? An Robert D. Behn, Presented at the Twenty- Evaluation of the Citistat Model of Eighth Annual Research Conference of the Performance Management and its Association for Public Policy Analysis and Applicability to the Scottish Public Management, Madison, Wisconsin, November Sector,” Cathy Sharp, Jocelyn Jones, and 4, 2006 Alison M. Smith, Edinburgh, Scotland: Scottish Executive Social Research, 2006. “The Varieties of CitiStat,” Robert D. Behn, Online at http://www.scotland.gov.uk/Publica Public Administration Review, Vol. 66, No. 3 tions/2006/07/21102410/0. (May-June 2006), pp. 332-340

RAPPAPORT Institute for Greater Boston Kennedy School of Government, Harvard University

The Taubman Center and its affi liated The Rappaport Institute for Greater Boston institutes and programs are the Kennedy aims to improve the governance of Greater School of Government’s focal point for Boston by fostering better connections activities that address urban policy, state between scholars, policy-makers, and civic and local governance and intergovernmental leaders. More information about the Institute is relations. available at www.ksg.harvard.edu/rappaport.

8