The Right Decision-Police.Indd

THE RIGHT DECISION Evidence-based Decision Making for Police Service Professionals

Canadian Association of Chiefs of Police Research Foundation

Paul S. Maxim, Len Garis, Darryl Plecas and Mona Davies

THE RIGHT DECISION

Evidence-based Decision Making for Police Service Professionals

Paul S. Maxim, Len Garis, Darryl Plecas and Mona Davies

Paul Maxim, Professor, Wilfrid Laurier University

Paul Maxim obtained his MA in criminology at the University of Ottawa and his PhD in sociology at the University of Pennsylvania where he specialized in criminology and research methods. He is currently a professor in the Department of Economics and the Balsillie School of International Affairs at Wilfrid Laurier University in Waterloo, Ontario. His primary areas of research interest are population and labour economics.

Len Garis, Adjunct Professor, University of the Fraser Valley

Fire Chief for the City of Surrey, British Columbia, Len Garis and is an Adjunct Professor in the School of Criminology and Criminal Justice at the University of the Fraser Valley, a member of the Afﬁ liated Research Faculty at John Jay College of Criminal Justice in New York, and a faculty member of the Institute of Canadian Urban Research Studies at Simon Fraser University. His focus is on addressing public safety challenges through evidence-based decision making and innovation.

Darryl Plecas, Professor Emeritus, University of the Fraser Valley

Darryl Plecas is Professor Emeritus in the School of Criminology and Criminal Justice at the University of the Fraser Valley. Prior to his retirement, he held the RCMP Senior University Research Chair at UFV. He received the University’s Teaching Excellence Award, and in 2003 received an Innovative Excellence in Teaching, Learning and Technology Award at the Fourteenth International Conference on College Teaching and Learning. He has published numerous articles relating to Canadian criminal justice. He also has extensive experience in applied policy and program evaluation, and effective decision making.

Mona Davies, Legal Analyst

Mona Davies has an LL.B. from the University of London and her Masters in Political Science from the University of Toronto. She also has a professional degree from the Institute of Chartered Secretaries and Administrators. She has worked both in the private and public sectors with over 10 year’s experience in international law, specializing in investor state arbitration.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Table of Contents

2 Forward 3 Introduction 17 Deﬁ ning the Problem 31 Thinking Critically 43 Collecting Evidence Canadian Association of Chiefs 55 Statistics of Police Research Foundation 67 Experimental Designs This project was commissioned by 79 Costing Analysis the Canadian Association of Chiefs of Police Research Foundation, 97 Making Decisions which seeks to create and develop the highest standards of effectiveness in law enforcement through research. Other partners include the City of Surrey, British Columbia and the University of the Fraser Valley, B.C.

For more information about this book This manual was adapted from or its authors, contact Dr. Irwin M. Th e Right Decision: Evidence-based Cohen at the Centre for Public Safety Decision Making for Fire Service and Criminal Justice Research in the Professionals, developed through the School of Criminology and Criminal support of the Canadian Safety and Justice at the University of the Fraser Security Program, led by Defence Valley, at [email protected] or 604- Research and Development Canada’s 854-4553. For additional public safety Centre for Security Science, in publications, please visit www.cjr.ufv.ca partnership with Public Safety Canada. Page 2

Forward

Canadian police services are key to Transparent decision-making model maintaining public order and security, Social & but their capital and labour costs can economic Evidence- pose signifi cant fi nancial challenges. Strategic +=costing based planning analysis decision Police service professionals make crucial making decisions regarding the level of service they can provide their communities and the Evidence-based decision making is not demands they are going to place on those new. Rather, it is a framework that brings communities. In an effort to stay current together strategic planning with social and improve their effectiveness, Canadian and economic costing analysis within a police services have begun to adopt transparent decision-making model. innovative approaches to: (1) rethinking what services are of highest priority; (2) This manual provides an overview of generating improved outcomes; and (3) some of the more crucial components of better controlling service delivery costs.1 evidence-based decision making. Some of Policing is one of the major budget items the material may appear a little daunting for most cities and municipalities. at fi rst. But we would ask that you read it in small chunks and go over it more than While understanding the invaluable once. As with any volume of this type, the role of police services, both the public material often makes more sense when and municipal leaders are asking that linked to a problem or issue in your own signifi cant decisions be based on hard organization. Overall, we hope you fi nd this evidence. Questions such as what are manual useful in improving your decision police forces’ underlying strategic value making and justifying your choices. and what are the associated costs and benefi ts are commonly raised. For their assistance, we would like to thank: Saanich Police Chief Cst. Robert Downie, This manual has been created to help Ottawa Police Cst. Ian Matyas, Surrey Fire decision makers address those concerns. Service Strategic Planning Analyst Alex Evidence-based decision making is Tyakoff, and Dr. Charles Jennings, Director one of the more effective tools you of the Regenhard Centre for Emergency can use to rationalize why a particular Response Studies at the John Jay College of approach or program option was chosen. Criminal Justice in New York.

1. Research Brief No. 31. Police Performance Metrics http://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/plc-prfrmnc-mtrcs/plc-prfrmnc-mtrcs-eng.pdf

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 3

Introduction

Effective Decision Making in a Changing World

While the primary function of police departments has remained consistent The days are gone—if, indeed, they ever over time—to ensure the safety and security of all persons and property— existed—where government and taxpayers how departments deliver their services take a request for more equipment is becoming more complex. Crime rates and more personnel at face value. have generally declined over the past two decades, but the public is increasingly asking police departments to respond to and taxpayers take a request for more a broader range of calls. Those calls often equipment and more personnel at face require more sophisticated equipment and value. Politicians, city managers and higher better or differentially trained personnel. executives are increasingly forced to make Furthermore, many jurisdictions have choices within tight resource constraints. been asking its police services to integrate their functions with other fi rst responder More than ever, leaders in police agencies to include fi re services and departments need to make decisions in emergency medical services. ways that are transparent and justifi able. Good decision making, we will argue, As a result, leaders and managers needs to be informed as much as continually face this question: How can possible by evidence, research, and sound we provide quality service in light of more information. We term this approach complex demands while being sensitive evidence-based decision making. We to resource and economic restraints? make and justify evidence-based decisions Choices and trade-offs need to be made, by referencing independently supported and consequences need to be considered. and verifi able facts. This approach helps The pressure increases on decision makers ensure that the decisions we make are when politicians, municipal staff, and sound and defensible. Used effectively, ultimately, the public scrutinize these evidence-based approaches can help you decisions. The days are gone—if, indeed, produce the results for which you are they ever existed—where government searching. Page 4

So why is evidence-based research important? Why does this approach to problem solving matter for police This manual will help you: services? Among some key reasons are the following: 1. Find and use information and research to make • Policies and programs that are not evidence-based decisions. guided by sound evidence frequently cost too much, waste resources, or 2. Put your decisions simply yield poor or unknown results; • External decision-makers who into a compelling approve departmental budgets may framework to convince not view departmental requests as others of their merit. justiﬁ ed if they lack compelling evidence; and • Policies and strategies that are evidence-based often produce Yet, even in those circumstances, better results, which can increase evidence-based decision making can your credibility and support for the help you link the values, principles, and department as a whole. ideologies that guide your department to independent evidence and supportive This manual will help you understand research. how to ﬁ nd and use the information and research needed to make evidence-based The evidence that we will learn to decisions. It will also help you to put your use comes from a variety of sources. decisions within a compelling framework Some is available as administrative data to convince others of their merit. that government and other formal organizations routinely collect. Some is Of course, not all decisions are or can be generated in the course of formal policy based on facts. Both professionally and and program evaluations, and some will in our personal lives, we refer to ethics, come from the work of government values, preferences and political choices. and academic scientists. Other sources To believe or do otherwise would be to of information will include your own deny the complexities of social life. organization and, often, your own unit or department.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 5

Learning to Navigate the World of You will learn why those frequently form Evidence part of information collection before a new policy or program is developed, or This manual will help you to navigate before strategic priorities are determined. the world of evidence without feeling You will learn about cost-benefi t analyses intimidated by it. As we will discover, and costing studies, which are critical not all evidence or data is of equal value. components of strategic planning when Even good information needs to be resources are tight. placed in a context where we can evaluate its accuracy and meaning. In other words, Using examples from police services this manual will help you fi gure out what around the world, this manual will show you need to know about data generation you how to defi ne a problem. It will help without having to be a scientist or scholar. you to think critically and creatively about it, and fi nd the evidence you need to Besides learning how to assess evidence, inform your decision. Additionally, it will we will also discuss how to use evidence provide simple explanations of various to formulate a persuasive argument. forms of research so you will know how Data alone is not suffi cient to inform and when to use them to support your and support your decisions. We need case. to frame public justifi cations for our policy or program decisions logically and Before we begin, though, it is helpful coherently. Requests not grounded in a to think more deeply about the reasons sound strategic or business plan will have for doing all of this. How and why has very little chance of success. We will learn evidence-based decision making become that many arguments or justifi cations so important? Why should you, or anyone that are put forward simply do not make else, care about the process? sense. We will examine some major logical fallacies that are to be avoided at all costs.

This manual will also explain how to conduct an environmental scan and a SWOT analysis (an assessment of an organization’s Strengths, Weaknesses, external Opportunities, and Threats). Page 6

Medicine and Health Care Professionals That Have Led the Way

We can trace the origins of evidence- As anyone who has been in their fi eld for based approaches back to the 1980s. any length of time knows, the world is full Faced with signifi cant fi nancial of scam artists selling the latest managerial challenges, the government of the United elixir or practice. Within the UK, it was Kingdom started to emphasize the need obvious to the government that investments for policies and best practices supported were needed, but those investments needed by compelling evidence and empirically- to be effective and effi cient.1 sound research. Decision makers had wasted too many resources, they believed, This approach infl uenced many other on choices that had no evidence to back fi elds, most particularly health sciences, them up. They too often decided on the where researchers could directly link poor basis of personal preference, traditional practices to increased levels of harm practices, and ideas that had little more to for patients. Evidence-based medicine support them than they were popular at evolved as a way to reduce the gap the time. between academic research and clinical practice. Ideally, this would ensure the This had profound implications for the best possible outcomes and the most practice of how police intelligence was appropriate care for patients. Researchers being delivered, thereby prompting a and health care professionals scrutinized shift away from traditional practices to policies and procedures to see how they Intelligence-Led Policing (ILP) models could run medical facilities in more which have been adopted internationally. effi cient and effective ways.2

The need to change existing ways of doing things in the world of medicine was becoming increasingly apparent. For example, one major study suggested that it took approximately 15 years to incorporate the results of research into recommended policy. As a dramatic example, let us consider that the research basis underlying a cure for a particular form of cancer might already exist.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 7

However, the lag between that discovery While research on all aspects of policing and even partially implementing it in a grew substantially in the late twentieth clinical setting takes about a decade and century, the application of research to a half. Even after that extended period, police practice intensifi ed in the early only about 40 per cent of practitioners twenty-fi rst century. This is, especially are using that information. 3 so for the three tasks that make up the “triple-T” strategy of policing: targeting, Meanwhile, people who could benefi t testing, and tracking. Evidence-based from the results of that research continue targeting requires systematic ranking to suffer or die because information and comparison of levels of harm had not infl uenced the practices of the associated with various places, times, medical profession in a timely way. Worse people, and situations that policing can still, implementing the answer might be lawfully address. Evidence-based testing intentionally delayed if other groups helps assure that police practices neither saw greater benefi t and fi nancial profi t increase crime nor waste money. Tracking in “managing” the disease rather than in whether police are doing what police actually curing it. leaders decide should be done is likely to grow rapidly in the coming years. We are An evidence-based approach tries already seeing the use of GPS records of to use the best available information where police go and the implementation generated through research, experiments, of body-worn video devices to record observation, and other factual sources to what happens during encounters with infl uence the creation of the best decisions citizens.4 and policies possible. Sometimes, this can directly confl ict with other forces, values Post World War II Developments and interests, as the previous hypothetical After World War II patrols in police cars example illustrates. were promoted on the theory that police

“omnipresence” would deter crime.5 Case Study In the 1960s, the advent of three-digit Evidence-based policing is a method of emergency phone numbers such as 911, making decisions about “what works” in turned random patrol into an airport-style policing: which practices and strategies “holding pattern” for rapid response, also accomplish police missions most cost- based on a theory of deterrence. This effectively. In contrast to basing decisions produced a new theory of organizational on theory, assumptions, tradition, or action based on the distinction between convention, an evidence-based approach reactive and proactive actions. continuously tests questions with empirical research ﬁ ndings. Page 8

What police did when they reacted to a citizen call was not subject to much By 1975, the “three Rs”— police agency direction or analysis. The main organizational requirement was random patrol, rapid response, to arrive, do something, and leave as and reactive investigation—had quickly as possible. Policing in the mid- become the standard model of 1970s was largely delivered in a one-size- urban policing. By 2012, this fi ts-all strategy, sometimes described was evolving into “triple-T”: as the “three Rs”: random patrol, rapid targeting, testing and tracking. response, and reactive investigations. This is broadly known as the traditional or standard model of policing, elements of which still remain with us today. While the standard model is far from gone, its resources are increasingly guided By 1975, the three Rs had become the by statistical evidence. With the emerging 6 standard model of urban policing across triple-T strategy, both patrol and the United States as well as in the United detective managers had moved toward far Kingdom and other predominantly greater proactive management of police Anglo-common law cultures (e.g., resources. Compared with practices from Canada, Australia, and New Zealand). the 1970s, these focused police strategies There was almost no targeting of patterns are far more elaborate and differentiated, or predictions of crime or disorder. Little choosing from a wider range of priorities testing was done of what worked best to and objectives on the basis of extensive prevent or solve crimes and problems. data analysis. Few agencies were involved in tracking and managing what police were doing, Evidence-based targeting for problem- where, when, and how, in relation to any oriented policing (POP) has gradually specifi c objectives. Most police agencies become recognized as real police work, lacked computers. The tradecraft of not “social work,” supported in some all but the most rudimentary crime and agencies with new case management intelligence analysis was largely in its early systems. During this time additional infancy. policing models started to gain traction, namely Intelligence-Led Policing, a By 2012, the three Rs were changing into business organizational model that what is described as the “triple-T” of emphasizes evidence-based decision targeting, testing, and tracking. making, accountability, and targeting prolifi c offenders.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 9

Underlying all of these models is the need The growing adoption of those three to do more with less and measurably drive principles has given shape to what down crime and disorder in communities. is increasingly called evidence-based These policing models are not mutually- policing (EBP). Interestingly, the rise of exclusive; that is, they can be blended and the triple-T strategy did not emerge from tailored to meet local conditions. any theoretical plan to use evidence in such a coherent strategy. The best test of evidence-based policing is whether it has improved public safety The marshalling of evidence around three and police legitimacy. There is certainly a key strategic tasks was driven as much by correlation over time between increases innovative police leadership as by police in the use of systematic evidence and scholarship. Its success resulted from a decline in serious crime, in both the the surprisingly rational convergence United States and the United Kingdom. of police reform with a fl ood of new research in criminology and business Targeting, Testing, and Tracking fi elds. There are many other strategies Both the demand for, and uses of, that can use EBP besides triple-T. research evidence have become focused on three strategic principles: Evidence can be useful, for example, in the recruitment and promotion of more 1. Police should conduct and apply good women police leaders, in developing research to target scarce resources on better ways to train police, or in fi nding predictable concentrations of harm lighter-weight body armor with greater from crime and disorder. protection. Evidence can also be used to 2. Once police choose their high- understand the dispersion of crime and priority targets, they should review disorder in communities, and address or conduct tests of police methods these problems before they escalate to help choose what works best to beyond control. reduce harm. 3. Once police agencies use research to target their tested practices, they should generate and use internal evidence to track the daily delivery and effects of those practices, including public perceptions of police legitimacy. Page 10

“To understand is to perceive patterns.” – Isaiah Berlin, theorist, philosopher

Mapping Data, Informing Decisions Geographic information systems were fi rst introduced to modern day policing as Applying geographic information systems a way for crime analysts and criminologists or GIS to crime mapping and analysis is a to better major innovation of police organizations understand the around the world. It enables crime geography of analysts, uniformed offi cers, investigators, crime. In fact, administrators and police executives to a rudimentary access and analyze data in maps so they form of can make better decisions. In its simplest mapping was form, tabular data can be imported into used by the a GIS and expressed spatially in the physician John form of maps. These maps can show the Snow in 1854 distribution of crime types across your to successfully counter a severe outbreak community, or how traffi c volume and of cholera that occurred in London, the built environment can impact police England. By isolating the water-borne response times. It can also be a way to disease to a public water pump on Broad redistrict police boundaries and inform Street in the Soho district, he disabled the the location of a community police well pump by removing its handle. This station. Like most things in life, you are very act had helped to stem the fl ow of only limited by your creativity. the outbreak. The application of mapping to disease control was subsequently applied to policing; the prevailing wisdom was that criminal activity is much like the behavior of infectious disease.

Using mapping technologies, police can target resources on most crimes by identifying the small fraction of localized places in any city where crime happens repeatedly.7

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 11

These hot spot concentrations are most obvious at a micro level, such as a single In the not-too-distant past, offi cers address, a cluster of addresses, or a “block- stuck push pins into wall maps to face” street segment from one corner to show locations of crime. More the next on a single street. Hot spots can be inventive types tied strings from mapped, ranked, classifi ed by offense type, pin-to-pin to indicate linkages and analyzed in many ways relevant to police between events or sequence of operations. In fact, mapping technologies crimes. This is akin to writing are central to most CompStat-led police offi cer’s notes on paper napkins, a departments across North America. practice we wouldn’t recommend.

Beyond descriptive or hot spot mapping Thankfully, with the arrival of lies the power of analytical mapping and affordable and powerful computers predictive policing. Retrospectively, analysts and GIS software, most police have been able to spot the bad guy hiding services can readily access and apply in data by mapping the location of stolen mapping technologies to examine, vehicles relative to where they were dumped understand and solve problems, and recovered. Often, a stolen vehicle is and to make informed decisions. abandoned within a short walking distance of the auto thief ’s residence, which helps to connect the dots in an investigation. Mapping has also been used along with The good news is GIS technologies are linkage analysis to carry out telephone relatively affordable and accessible to pattern analysis in major crime cases such most police departments, including small as homicide and forcible confi nement. to mid-sized operations with limited By charting the location of cell towers budgets, and there are plenty of excellent relative to whom calls were made (and industry courses available. One only has to received) during “crime time,” and where visit the geography department of a local they pinged, investigators are able to university or polytechnic institute for ideas. triangulate data points and make inferences. For those who are not very familiar with Geographic profi ling goes a bit further in the workings of GIS programs, we would theory and method, and in capable hands suggest you start by dropping by your city is able to predict - with amazing precision planning and engineering departments - when and where the serial offender will for assistance. Most of these municipal strike next. By leveraging GIS technologies departments have the necessary technical to discover spatial and temporal patterns, expertise and base maps to jump-start you are better able to deploy your offi cers in your journey into GIS. Speak with a crime a more effi cient and cost-effective manner analyst and IT technician on how to best and to keep communities safe. deploy GIS across your department. Page 12

Effective Decision Making: The Task of Good Leaders and Managers

Decision making is possibly one of the most important roles of leaders and Poorly made decisions increase managers. Their decisions infl uence the confl ict and diminish morale. direction of their units and affect the Well-made decisions that lead morale and well-being of personnel who work for them. Poorly made decisions to tangible, positive results increase confl ict and diminish morale. can increase departmental Well-made decisions that lead to tangible, success and improve morale. positive results can increase departmental success and improve morale. Using evidence-based research helps to Nevertheless, even when leaders and ground our experiences and opinions in managers see the value in an evidence- a broader context of information that based approach, several factors can get is ultimately more convincing. Besides, in the way. Some administrators feel practices evolve. The police service of pressured to make decisions quickly the early twentieth century is not that of and with incomplete information while the new millennium. others might use outdated information. Policing is a fast-paced environment; When developing a new strategy or policy there is a need for speed. However, this it is best to assess what you know, what tendency should be governed by sober others around you know, and what the refl ection and consideration of the latest wider fi eld of research tells you about it. data to inform decisions and better It is also prudent to commit to evaluating practices. Additionally, most people rely that new policy or plan after you have on personal experience, observation, started it so you can generate your own or gut instinct when having to make a evidence to show its effectiveness. That choice. As trained professionals, our helps to advance the fi eld as a whole, personal experiences and judgments are and your department’s research can then often valid, but they comprise part of the inform other departments on what works, picture only. Cognitive science indicates what does not, and why. Often we are that we tend to see what we expect to reluctant to assess a program or practice see. The mind is poorly “wired” to deal because we might fi nd that it does not effectively with both inherent uncertainty work. That is not a problem. Both as and the induced uncertainty of dealing individuals and as a society, we typically with complex, multi-faceted issues in law learn more from our failures than from enforcement. our successes.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 13

The Nature of Empirical Research

What are we really talking about when This means that other independent we use the term evidence? Unlike the scholars and researchers examined the evidence that might come out of a police research to see if it was credible and well investigation of a crime scene, evidence designed. This does not mean to say that in this context has a specifi c meaning. It the work is either perfect or infallible. refers to the results of empirical research Nevertheless, it does increase your ability coming from systematic data collection to trust in the results. Research must be grounded in formal assessments, peer-reviewed before it is published in experiments, or other research models. most academic journals. Some academic It is a systematic approach to answering journals can be highly technical and very a research question that generates intimidating to those outside the fi eld. information or facts that are replicable, Fortunately, many sources summarize observable, credible, verifi able, and signifi cant academic fi ndings or translate supportable. the results into everyday language.

When assessing the research available to Common Research Methods you, some of it will be: In the medical fi eld, the gold standard • Quantitative, generating numbers and for research has been the randomized statistics, or controlled trial. Here researchers • Qualitative, generating subjective randomly assign individuals to receive information that is helpful in various preventive, therapeutic or determining preferences, values, or diagnostic interventions, and then follow perspectives of those responding to up to see the effect of the intervention. the questions. One possible intervention might be no intervention at all. This enables Either of those approaches can generate researchers to compare the control group valid data. The key is in knowing when (which received no intervention) to the and where to use what kind of evidence, test groups, which received the various and to be able to fi nd out whether it is interventions in question. Drug testing is adequate for the purposes at hand. frequently done this way. In a later chapter we will examine different frameworks While there are many good sources of for collecting evidence and discuss supporting evidence, academic research why researchers hold the randomized has the added benefi t of being peer- controlled trial in such high esteem. reviewed. Page 14

In the social sciences, having randomized Making Better Decisions tests involving a control group is also possible. For example, we could By now, you probably can see that there are randomly assign people with security benefi ts in making decisions infl uenced by alarms in their homes as a test group sound, credible research. Quite simply, if for comparison with another random you have done your homework, it is likely group without alarms, which would be you will have a better-informed decision. the control group. This is one way of Defending your decision is also easier answering the research question, “Do since the process is more transparent and people with security alarms have fewer is based on something other than your break-in incidents than people without hunch, best guess, or personal opinion. security alarms?” Researchers will set up such experiments to ‘control’ the factors It is important to recognize, though, that that might skew (or distort) the results. evidence-based decision making is best This increases the validity of the research, suited for objective questions. As we so that you can have greater confi dence noted at the outset of this chapter, other or trust in the measurements and results. decisions are infl uenced primarily by our Researchers are also concerned about preferences, values, or beliefs, and are less the reliability of their result—meaning, likely linked to research. if we continued to replicate this study repeatedly, would we get the same results? However, the two merge when we want Would we get the same results if we ran to fi nd the most effective ways to address this test in a different community? Or, is issues that ultimately correspond with it unique to this community only and, if our values. Improving our quality of life so, why is that? Research needs to be both by providing quality police services and valid and reliable so you know the results crime reduction is a social value that are legitimate and trustworthy, and not a provides the motivation to do things fl uke or coincidence. differently. Evidence-based research helps us to know what to do and how to do it.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Introduction Page 15

Notes

1. Tranﬁ eld, D., Denyer, D. and Smart, P. (2003) “Towards a methodology for developing evidence- informed management knowledge by means of systematic review,” British Journal of Management, Vol. 14, No.3, pp. 207-222. 2. Vishwanath V. Baba, Farimah HakemZadeh, (2012) “Toward a theory of evidence based decision making,” Management Decision, Vol. 50 Issue 5, pp. 832-867. 3. Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F., Chalmers, T.C. (1992) “A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction.” Journal of the American Medical Association, Vol. 268, No. 2, pp. 240-248. 4. Sherman, L. (2013) The Rise of Evidence-Based Policing: Targeting, Testing, and Tracking, Chicago: University of Chicago Press 5. Wilson, O.W. (1950) Police Administration. New York: McGraw-Hill. 6. Skogan, W. and K. Frydl (Eds.) (2004) Fairness and Effectiveness in Policing: The Evidence. Washington, DC: National Research Council, p 223 7. Sherman, Lawrence W., Patrick R. Gartin, and Michael E. Buerger. 1989. “Hot spots of predatory crime: Routine activities and the criminology of place.” Criminology 27: 27-56 Weisburd, David, Elizabeth R. Groff and Sue-Ming Yang (2012) The Criminology of Place: Street Segments and our Understanding of the Crime Problem. New York: Oxford University Press. Page 16

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 17

Deﬁ ning the Problem

Not all Decisions are Alike

We make hundreds of decisions daily, ranging from what to have for breakfast, Evidence-based decision making can help to deciding in which room to hold a meeting, to whether or not we should us in those circumstances where we need to buy a new car. Many of those decisions make an economically, socially or politically are informal, relatively insignifi cant and signifi cant decision. have few consequences. Some decisions incur a degree of risk or uncertainty. Each of us takes risks every day. For most of us, reasonable risks don’t prevent us Evidence-based decision making can from doing our daily tasks and routines. help us in those circumstances where we Working through a formal process to need to make an economically, socially or address those issues would cause our lives politically signifi cant decision. to grind to a halt. An advantage of evidence-based decision making is that it allows us to use known On the other hand, there are signifi cant results to estimate a measurable outcome. decisions we face in our personal The good news is that anything can be and our professional lives where the measured. No matter how “fuzzy” the consequences are not small or we need measurement is, it’s still a measurement if others to be engaged in making the it tells you more than you knew before.1 decision. Examining the issues in detail and working through a formal process is One can never know the actual worth our time and effort. Generally, that consequences of a decision before the formal process involves creating a clear event. However, by drawing on experience defi nition of the problem, outlining the and the available evidence, generating a alternatives, and weighing the costs and reasonable and defensible expectation of benefi ts associated with selecting any of a specifi c outcome is possible. those alternatives. Page 18

All of us will make decisions that lead to While getting a less than ideal outcome undesired outcomes at times. That is a from a good decision is unfortunate, reality of life. The fact that we made the one other advantage of having made a wrong choice is different from making good decision is that we can draw lessons a bad decision. There is a difference from it. If the decision making process is between not making the correct decision transparent, it is possible to consider why and bad decision making. it resulted in a negative outcome. Did we make some incorrect assumptions? Were As we will outline, bad decisions are we missing some important information? avoidable. Bad outcomes from good decisions, however, are fortuitous events In this chapter, we will consider the over which we might have little control. following So what then, distinguishes a good decision from a bad decision? Simply, • What is the issue and how do we good decisions are ones that ﬂ ow from problematize it? where the problem is clearly articulated. • How can we identify the options and alternatives? They are ones where we bring as much • How can we think creatively to of the appropriate and available evidence generate new ideas? to bear as possible. A good decision is • How do we generate alternatives? one where you can look back and with a clear conscience assert that under the same circumstances, and with the same evidence, you would come to the same conclusion.

❖

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 19

What is the Issue?

Typically, even rational, systematic decision makers will start by making a list Proper planning makes of alternatives. Lists are good and they decision making defensible deﬁ nitely have their place. Nevertheless, as John D. Rockefeller once said in a different even when the results are context, “A list is not a plan.” not as expected, and in an environment of increased police Before we start to generate options, we accountability, this is crucial. need to ask: What is the purpose of the decision? What is our intended goal? Those questions are embedded in an analysis of the problem. The framework That is, we can show critics that the choice of that analysis is generally a strategic or a we made was rational and reasonable business plan. Making a decision without under the circumstances. Proper planning planning is common. As the old adage makes decision making defensible even goes, “if we fail to plan, we plan to fail.” when the results are not as expected, and Without an explicit plan, however, we in an environment of increased police generally do not know if an undesirable accountability, this is critical. outcome is a result of a bad approach or the fact that we encountered new or When we ask the question, “What is the different circumstances. An open and issue?” we are essentially asking, “How formally structured process allows us to does the decision we are facing ﬁ t into and accumulate knowledge so that we are less advance the mandate of our organization?” likely to make the same mistake in the future.

Often, unplanned decisions do not end well. Planning allows us to make decisions logically and systematically. Proper planning makes decision making simpler and it makes it transparent. Page 20

Before Doing Anything, Ask “Why?”

Too often, we fi nd ourselves backed Where can we see evidence of your into a corner when confronted with contributions? Who in your unit the seemingly simple request about have you helped or supported this whether we should choose Option year? When can we expect to see the A over Option B. This is a popular returns on your performance? strategic move by someone who wishes to force an issue. For example, Perhaps these are not the most an employee may ask for a meeting appropriate questions to ask in the to discuss performance and salary. circumstances, but you get the idea. As an opening gambit, the employee The notion is to tie the request back to might ask, “Are you going to give me the goals of the unit or organization the same raise as last year or will I and to ensure that the choices that we also get the promotion I have coming are considering are consistent with in recognition of my service to the those goals. Typically, we are trying to company?” ensure the bases for the choices are not irrelevant. Decisions to reward Clearly, the employee is attempting employees simply because they are to force a false choice. In this friendly, consistently show up for instance, we call it a false dichotomy work on time, or always dress neatly because the question assumes that are diffi cult to defend. only the two options A or B are possible. In fact, many options may When all else fails, ask yourself, “Can exist. Before considering the many I defend my decision to others in the possibilities, assessing the employee’s organization, my boss, or the public?” contributions to the organization is As a former colleague once said, “I necessary. Ideally, there should be make every major decision assuming a performance assessment policy it will appear on the front page of in place. Lacking that, however, tomorrow’s newspaper. If I can you might ask some of the fi ve accept that, then I have likely made Ws. Why should you be rewarded a reasonable choice on reasonable based on your performance? What grounds.” have you contributed to enhancing the effectiveness of your unit?

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 21

That mandate is normally part and parcel of our strategic plan. Sometimes it is embedded in our operational plan or By embedding the decision standard operating procedures (SOPs). within the framework of a pre-existing plan—such as a As an example, let us assume that a police strategic plan—the choices chief of a medium-sized police department made are defensible on has just returned from a conference on crowd strategically assessed grounds. control and policing large public events or demonstrations. The general consensus is that such recent events in Vancouver as the Stanley Cup riots and in Toronto during the G20 meetings have dictated the need The goals outlined in the department’s for law enforcement to be prepared to strategic plan say that its primary goal confront high-risk situations. The issue has is to provide for a safe and engaged been reinforced in the press by widespread community and to strive for organizational coverage of the events. Although the and service excellence. Consequently, the majority of public events the Chief ’s real needs of the department might be department oversees are planned peaceful renewing the patrol car ﬂ eet or an upgrade gatherings with minimal public disruption, in the department’s dispatch system. a small number are demonstrations that By referring to a planning framework, can lead to public disruption and arrests. we can see that creating an Emergency While the department’s personnel have Response Team (ERT), team is not a gone through basic crowd control training priority. Furthermore, in all likelihood, for such a response, the Chief now thinks the incremental investment in that area the frequency and level of training of his would be wasted and provide little, if any, personnel may be inadequate. return on investment.

The issue the Chief faces is whether he Investing in an ERT team, however, fi ts should create his own Emergency Response with the overall goal of the department Team (ERT) or rely on the services of to improve the safety of the community. a larger neighbouring department. At The real issue, however, is whether the face value, it is diffi cult to argue that proposed investment fi ts with the real one should not pursue extra preparation and immediate needs of the community. for such events. The reality is, however, The issue is not one of improving the that in developing its strategic plan, the overall safety of the community; the issue department needed to balance different is really how best to address the most requirements of the local community. likely threats the community faces. Page 22

Undoubtedly, the Chief could have listed Again, by embedding the decision within the options available to the department for the framework of a pre-existing plan an ERT and the best alternative among or operational framework, the choices those options could be selected. The point, made are defensible on strategically however, is that decision was not the only assessed grounds. In that case, a delay in one to be considered. The key was to refer implementing a new program to provide to the department’s operational focus or, for an ERT is justiﬁ able. ideally, its strategic plan.

❖

Generating Ideas

Often, choices appear obvious. Do we Often, this generates a group-think spend more on equipment or personnel? mentality where we reinforce the belief in a Is our communications equipment at the limited number of options. Furthermore, end of its working life expectancy or not? colleagues and subordinates may be more In other instances, the alternatives are not concerned about reinforcing what you always self-evident. It is not an A or not-A have said or telling you what they think choice. In later chapters, we will examine you want to hear rather than offering how to conduct environmental scans unique suggestions. Outsiders, however, and SWOT analyses. These are relatively may face similar situations but approach formal procedures that systematically the issue entirely differently. review what others have done or might do in similar circumstances. Engage in a group brainstorming session Before resorting to those approaches, however, several more modest ways exist Possible group-think tendencies aside, to generate alternatives. You might want sometimes the people around you are to consider the following options: the best source of ideas. They know the organization and understand the problems. Furthermore, they are less expensive than Talk to people outside your consultants since they are already on normal circles payroll. Ask for individual suggestions. Too often we limit our social and Sometime a group session where we ask professional circles to those we already people to come up with “crazy” alternatives know or with whom we work. is effective.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 23

The semblance of a little competition can Often these connections have something sometimes unleash new ideas. Remember, of value to offer. Understanding the today’s innovations were yesterday’s outsider’s view can pay huge dividends. impossibilities. Brainstorming can be either informal or structured, the intent Hire a reputable consultant is to generate as many ideas as possible Often, you are the local expert at your and seek solutions to our most vexing and core business or activity. That is why you persistent problems. are in your position. On the other hand, not all of your decisions relate to your Read more books and journals; core business function. Most businesses surf the web engage outside design fi rms, marketing The more you read, especially outside your agencies, web designers or management area of policing, the more novel ideas you consultants. The key is to identify the area are likely to come across. Business books of expertise that you require. Once done, are an obvious choice but sometimes great ask your associates if they can recommend ideas come from works of fi ction. Most a consulting fi rm or individual. Usually, of us like to stretch ourselves. Professional smaller fi rms are more creative and less journals are a good way of keeping up costly, but creativity is a business. with new trends. As always, the internet is anarchy and generally fi ts the adage that Of course, you need to be willing to be you get what you pay for. Still, gems are to open to new perspectives. Don’t let your be found and modern search engines are prejudices get in the way. Just because you amazingly good at ferreting them out. As have a low opinion of someone does not Stephen Covey stated, it’s always wise to mean they have bad ideas. Also, do not feel “sharpen the saw.”2 intimidated because someone can generate better ideas than you. Especially if that Focus on the people you serve — person is a subordinate, you automatically both internally and externally get credit for being smart enough to having such a creative employee on your team. Look at the world from the perspective of the people you serve both internally and Finally, be willing to accept that sometimes, externally. How they see your organization the best options are the obvious ones. A is probably very different from how you and consultant who gives you a report that your immediate colleagues see it. Besides tells you what you already know, may the people you serve, other great sources not simply be lazy or uncreative. It could of ideas are from your partners such as be that what is obvious to you is indeed professional associations, community the best option. Consider it that your partners, educational institutions and other suspicions have been confi rmed. areas of government. Page 24

Get a Plan

Whatever its size or complexity, every Four elements of a plan: organization can beneﬁ t from having a plan. Whether it is termed a strategic, organizational, or business plan, the Statement of organization values point is the same: an organization needs to know why it is doing what it is doing, where it is going, and how it intends to get there. Statement of goals and objectives

Without a plan, people make decisions arbitrarily. At best, those decisions will lack consistency and, at worse, they will be Outline of how to achieve the goals contradictory. A plan does not guarantee organizational success or effi ciency. Not having one, however, invariably dooms an organization to mediocrity or failure. Indication of how to measure success Much material outlining how to put together an organizational plan is available both in bookstores and on the internet. Topics range from project management Plans vary in complexity but there are practices and principles to the latest in advantages to keeping it simple. Complex major case management. Time spent plans are often diffi cult to remember reviewing some of that material would be and can be highly constraining. As most a good investment. battlefi eld generals know, once the action starts, little goes as expected. Often, the Essentially, a plan consists of four best one can hope for is that the troops elements: know what they are fi ghting for, that they 1. A general statement of organizational remember the overall goals and objectives, values. and that the line offi cers are suffi ciently 2. A statement of goals and objectives. trained to react to unexpected tactical 3. An outline of how the organization challenges and setbacks. Thus, there is a lot intends to carry out or achieve its to be said for keeping things simple. goals. 4. An indication of how to measure success.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 25

Keeping Th ings Simple Holmes asks, “And, what does that tell you?” The idea of keeping things simple and staying true to the obvious is illustrated Watson replies, “Astronomically, it in this humorous exchange between tells me there are millions of galaxies Sherlock Holmes and Dr. Watson: and potentially billions of planets. Astrologically, it tells me that Saturn is in Sherlock Holmes and Dr. Watson Leo. Theologically, it tells me that God is went on a camping trip. After sharing great and we are small and insignificant. a good meal and a bottle of wine, they Horologically, it tells me that it’s about retire to their tent for the night. 3am. Meteorologically, it tells me that we will have a beautiful day tomorrow. At about 3 am, Holmes nudges Watson What does it tell you, Holmes?” and asks, “Watson, look up into the sky and tell me what you see?” Holmes retorts, “Watson you idiot — someone stole our tent.” Watson said, “I see millions of stars.”

Statement of Organizational Values Well-crafted statements can be inspiring, and make for eloquent poster boards that It is currently in vogue among can be placed on ofﬁ ce walls and in annual management gurus to spend a great reports. Poorly crafted statements do deal of time identifying the fundamental little more than provide a source of levity. values underlying our organization. As always, the best practical advice is to Typically, we outline organizational values keep things simple and straightforward. in one or more of: a mission statement, a Simple, unambiguous statements are easy vision statement, and a values statement. to remember and easy to follow.

Well-crafted mission, vision or value statements can be inspiring; poorly crafted statements do little more than provide a source of levity. Page 26

Essentially, a statement of value should Statement of Goals and Objectives outline the reason for the organization’s existence. This is known as the mission An organization’s statement of goals statement. For many organizations, and objectives contains the targets it sets such as police departments, the mission for itself. Organizational goals are the may be obvious. Your raison d’être is to broader targets for which one is aiming; protect lives and property or, in a broader objectives are the midterm step one sets sense, to create a safer community. The to achieve those goals. Broad goals may mission statement is where you answer be such things as setting targets reducing the great existential question, “What is the number of property crime and violent your purpose?” crimes in the community, or increasing the unit’s capacity to handle a broader range Value statements should also provide of service demands. To achieve the goal some expectation of where the of reducing crime, it is often necessary organization plans to be in the next three to make a list of objectives that form a to ﬁ ve years. What, in other words, is series of intermediate steps. For example, the midterm vision for the organization? one objective might be to research, Perhaps you see yourself as becoming the acquire and use the best technology and regional standard for performance. infrastructure to assist ofﬁ cers in the detection and investigation of crime. Finally, a values statement suggests something about your core beliefs. Implementation Procedures These are meant to be foundational and inspirational. For Google, it was, “Do No As we noted earlier, a list is not a plan. Harm.” In your case, it may be, “Serve Simply outlining the organization’s goals the Community.” While this might seem and objectives is a necessary part of, trite, it is useful to recall the core value but not a complete planning process. A when decision making starts to focus too true plan involves a discussion of how much on what is in the best interest of we can carry out the goals. What is the the organization. In this instance, what mechanism or what are the procedures you do is not about the organization; it is that are being put in place to meet the about serving your community. desired outcomes? For example, one objective may be to reduce on-the-job injuries. We may link this to the overall goal of increasing worker safety.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 27

❖ Napoleon’s goals and objectives Sometimes it is easy to confuse the concepts of goals and objectives. Too often, the two are used interchangeably. While related, the two are distinct notions. A good example is to consider Napoleon Bonaparte’s intentions in 1799.

Goal Objective Rule all of Become head of state in France Europe Conquer Italy Conquer Spain Defeat Prussian Army Defeat the Austro-Hungarian Army Incorporate Poland into the French Empire Conquer Russia

Ironically, Bonaparte achieved all of his objectives except for the last. Despite this impressive achievement, he ultimately failed to achieve his overarching goal. He failed to consider the impact of Russia’s brutal and unforgiving winters.

The important issue under Another mechanism might be to implementation is: How do we make this enhance information-sharing and happen? Obviously, the mechanism we working relationships between the police choose depends upon the circumstances. department and external community Perhaps more resources should go into partners. training personnel to be better equipped to address mentally-ill persons in confl ict This applies to all of the goals and with the law. On the other hand, people objectives identifi ed in the plan, whether may have adequate training but they have they are “soft” objectives, such as not had suffi cient opportunity to practice increasing employee morale, or “hard” the procedures. objectives, such as reducing work-related injuries or damage to police vehicles. Page 28

Implementation procedures are the actionable items in our plan. Too often, Too often, strategic and strategic and business plans identify what the organization intends to achieve but business plans identify what not the means by which it hopes to meet the organization intends those intentions. to achieve, but not the means by which it hopes Put another way, if goals and objectives are to meet those intentions. the nouns in a sentence, implementation procedures are the action components or verbs. Typically, outcome measures will cover a Measuring Outcomes spectrum of issues, ranging from internal performance metrics, to levels of service Measuring outcomes is essentially keeping provision, to fi nancial accountability. a scorecard. Before you can do this, Many discussions on strategic plans however, it is necessary to indicate within suggest creating a table where we list your plan what specifi c performance operational objectives in one column indicators you are going to use. You should and their corresponding measures of closely link those indicators to the specifi c success in the next. These linkages are objectives you have identifi ed and, in a judgment calls, but complex objectives general sense, to the overall goals outlined usually require more varied indicators in the plan. As the eminent management than simple, one-dimensional objectives. guru, Peter Drucker, once stated: “What gets measured gets managed.” Because goals are longer term and higher level notions than objectives, it is Obviously, clear quantitative measures are often more diffi cult to identify specifi c easiest to use, such as changes in calls for measures. Furthermore, goals often service, response time, clearance rates, require a more qualitative assessment than and so on. However, qualitative measures do intermediate objectives. One thing should not be overlooked. Indicators of to keep in mind, however, is that while community satisfaction or fear of crime, there ought to be a consistency between for example, may be hard to quantify but the outcome measures of objectives are crucial performance elements for any and goals, there need not be a perfect service provider. correspondence.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Deﬁ ning the Problem Page 29

It is possible to meet most or all of environment and circumstances in which one’s objectives but not one’s goals. the organization ﬁ nds itself. Regardless, Similarly, the failure to meet one or more those components become part of the objectives does not necessarily mean that crucial list of elements to which we refer the organization has missed its overall when we need to make a critical decision. goals. Practical strategic or business plans sometimes contain other items or provide Often we pose questions or decisions more detail on certain dimensions. vaguely. A good decision maker will deﬁ ne and clarify the issue and relate it to Details might also be put in place about the organization’s plan. Having done that, what forms the organization’s ”value- one can then ask subsidiary questions added” for your community, or how it such as: Does the issue warrant action? differs from similar organizations or If so, when should we carry it out? Is the service providers. Whether these items matter urgent, important, both or neither? are relevant depends on the particular

❖

Evidence-based Decision Making

Good evidence-based decision making is general framework of our plans. The tightly linked to an organization’s plans. primary benefi t of a good plan is that This does not mean that occasionally it allows decision makers to be able to we must make important decisions that justify how and why they are assessing are beyond what we planned to do. the choices they are considering. Raising Environments change and new issues the criticism that certain options have arise, all the while we have to be able to been considered is easy. In fact, for many maintain a focus on our core business decisions there may be an almost infi nite functions. The world is not static. This is list of possible options. We can reduce particularly true in the world of policing. that list substantially if we point out that the suggestions may have merit, but are Effective managers, however, need to outside the realm of the strategic plan. be suffi ciently fl exible to deal with those situations. Regardless, going through A good plan, then, lets us know what a planning exercise often provides a questions or issues are relevant, what broad enough perspective or suffi cient options are worthy of consideration, and guideposts that “out of the blue” consequently, what evidence we need to challenges can be placed within the consider in weighing those options. Page 30

Notes

1. Douglas W. Hubbard (2014) How to Measure Anything: Finding the Intangibles in Business. New Jersey, NJ: John Wiley & Sons. 2. Stephen R. Covey (1989). 7 Habits of Highly Eff ective People, New Jersey: Simon & Schuster.

A good plan lets us know what issues are relevant, what options are worthy of consideration, and what evidence we need to consider in weighing those options.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 31

Thinking Critically

Clarity of Thought

Evidence and data alone are not suffi cient for making good and useful decisions. Humans are stubborn beasts with a How we formulate an argument or explanation is just as important as the tendency to defend any coveted untruth quality of the information we might bring against the best of reason and evidence. to bear. When we consider evidence- based decision making, we need to keep two aspects in mind. First, as in Then, there is the internet. Its rise has been making any type of case, the underlying the greatest venue for half-baked ideas, arguments need to be based on sound conspiracy theories and their supporters logic. An argument that can lead to more since the invention of walls and graffi ti. than one conclusion is generally not very Fundamentally, evidence and sound logic useful. Second, how most people think rarely sway people. When was the last evidence or proof shores up an argument time, for example, someone listened to is typically not the most powerful way of you make a case and said, “Thank you making a case. for pointing out my logical fallacies. I see that I was wrong on this issue and I If there are two things that seem to will from now on change my perspective characterize humanity, they are that on the matter.” A positive outcome is people like to argue and, even when typically one where they change the topic; someone shows that their position is false a negative outcome is where they turn or illogical, they generally won’t change away muttering something about you and their world view. Humans are stubborn your kind having always been idiots. beasts with a tendency to defend any coveted untruth against the best of The fact is, there are some discussions to reason and evidence. which no solution exists, either logical or empirical. Arguments over the existence Evidence seems to abound that of God; who is the best looking actor or argumentation is one of humanity’s most actress; or, whether Aunt Helen made the favoured social activities. Go to any sports world’s best muffi ns will never be resolved. bar on a Saturday night and you will see what seems to be inexhaustible evidence. Page 32

Generally speaking, matters of values Although, it is admitted that charlatans are issues that are based on emotional abound and thrive in those domains as in preferences. all others.

On the other hand, there are situations The focus of this chapter is on those where evidence and rationality sway instances where, either individually or us (or, at least, some of us). Economic in groups, we are willing to consider issues, for example, typically command rational and evidence-based input into our more rational sentiments. Matters of our decision-making processes. Since health, and life and death—immunizing those instances appear rarely in the affairs your children against the measles, of humans, it is obligatory for us not to for instance—tend to elicit a rational miss the opportunity for making a sound response. decision by using faulty logic.

❖

Logical Fallacies

Logical statements are generally of As youngsters we are taught that what our the form, if A leads to B and B leads parents, teachers and other “experts” say to C, then the occurrence of A will is generally true. It is an accumulation of lead to C. Logical fallacies are ones knowledge passed from one generation where inherent gaps, contradictions or to the next that distinguishes humans simple irrelevancies in arguments go from other beings. This has allowed us to unacknowledged or unchallenged. Some develop antibiotics, to build skyscrapers logicians and philosophers have made and to distribute spam to those little boxes careers listing almost inﬁ nite varieties of we call cell phones. Without accepting fallacies (again, see the internet). For the knowledge passed on from authorities, most part, however, logical fallacies fall civilization could not exist. into a small group. Learn to identify these and you will be less likely to be led astray, However, while we may be willing to whether intentionally or not. accept the received wisdom from our resident Yodas, we should not be blind to Appeals to Authority the fact that Yoda may be wrong. There is nothing wrong with asking for further None of us has the capacity to generate evidence to back up some authority’s all human knowledge from scratch. claim.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Thinking Critically Page 33

Personal or Ad Hominem Arguments

Be suspicious of opening Ad hominem is Latin for against the person. lines such as: “But, it has Essentially, ad hominem arguments are always been done that way,” ones where someone attacks the person or “The experts agree that...” making the statement personally. Usually, the person’s sanity, morals or parentage is called into question. An ad hominem While we do not have the time to question argument is an attempt to “blow-off ” all authority, certain appeals should raise the proponent by undermining their your suspicion. credibility. Among some more polite ad hominem attacks are such statements as: Typical openings that should cause you to be suspicious are lines such as: • “What do you expect from a couple of fascists (socialist, liberals, academics, • “But, it has always been done that whatever)?” way.” • “That’s a typical statement from • “Everyone knows that’s the way it is.” someone who is clearly out of touch • “What do you (we) know? So-and-so with today’s realities.” is an expert in these matters.” • “That’s a typical male (feminist) • “Science tells us that . . .” response.” • “The experts agree that . . .” • “Gee, you would think s/he is an expert in the matter the way s/he is In such instances, there is nothing wrong going on.” with saying that, “If that is the case, • “So, how many years have you been then there should clearly be some hard in the ﬁ eld?” evidence to back it up. Perhaps we should check it out in more detail.” Or, “Gee, The key here is to separate the argument that’s interesting because some (scientists, or assertion from the speaker. Just experts, etc.) say just the opposite. How because one has a low opinion of the are we to resolve this?” other person, doesn’t necessarily mean that what they have to say is wrong or Usually, appeals to authority are code for irrelevant. It may be difﬁ cult at times, but either, “I am too lazy to check this out,” trying to respect the idea is essential if or, “I am blowing smoke.” not the person presenting it. Page 34

The “Red Herring” Pink Herrings True red herrings are items that are Red herrings are irrelevant issues that clearly unrelated to the issue at hand. someone brings up in a discussion. For Sometimes, however, someone may raise example, it is asserted in a council meeting, an issue that is suffi cient to address the and may be the case, that too much problem but is not necessarily a solution. money is being spent on travel, toys for We might refer those to as pink herrings. administrators or overtime. Someone then Perhaps the biggest pink herring is for asserts that this would not have happened administrators to argue the problems if we had invested in the appropriate exist in their organization because of a technology a couple of years ago. lack of fi nancial resources. The problem here is that inappropriate Certainly, money can purchase resources. spending that has gone unchecked is due All too often, however, more money just to a lack of fi nancial oversight. Effective leads to more of the same. Money, itself, fi nancial oversight has existed before the doesn’t necessarily solve the problem. time of the Romans and long before Proper oversight, a more effective use of computers were available. Investing in existing physical and human resources, or the appropriate technology may help in a more creative approach to the issue may the oversight process but does not ensure be more effective than simply throwing oversight in itself. Examples are bountiful more money at the problem. What of solutions that have merely added to the is necessary is that existing or future problem rather than solving it. resources are directed toward developing or enhancing mechanisms related to the The key to addressing red herrings is problem. to ask how the herring is related to the problem being considered. How will the As with the red herring, we need to technology be used to enhance oversight? address the open-ended call for money Is the appropriate software available? questions by asking how the money will Are the auditors properly trained in the be used. The answer will likely be to equipment to be able to enhance their purchase more equipment or hire more performance? Computers, after all, only offi cers. The subsidiary question then do what we tell them to do. becomes: In what way will that equipment or those offi cers enhance a process that is currently broken or ineffective?

The Right Decision: Evidence-based Decision Making for Police Service Professionals Thinking Critically Page 35

Circular Arguments Similarly, an often heard comment in city councils Circular arguments are those of the form is that a particular group that A causes B because B is the result of will not support tax A. Circular arguments abound, particularly increases because they in political debates. A favorite of teachers have made it part of is students who come after an exam and their platform. When assert that they can’t get a C because they asked why that is part of are A students. (So, explain how you earned the platform, the answer the C if you are an A student?) is that tax increases are not in the interest of the people. Another good example is sometimes found in salary negotiations. Bargaining To break the circularity, we need to units will sometimes insist that they know why a body passed the law in the need to get a larger increase than their ﬁ rst instance: what was its supposed colleagues because they have historically purpose? Likewise, we need to know in been the highest paid police unit in the what way not increasing taxes beneﬁ ts the group of comparable organizations. If electorate. What is the exact economic you don’t give the raise, how can they be mechanism supposed to be at play? the highest paid? Usually, most ratcheting effects that we see in labour negotiations Other Fallacies are based on circular reasoning. Group A has it in their contract that they are to have People call upon many other logical fallacies a 10 per cent premium on the rest of the when rationality and evidence fail them. jurisdiction because of the high cost of They range from the teenager’s perennial living in their area. Group B argues that appeal to popularity: “But everyone at to remain competitive, they need to be school has one,” to appeals to nature: within 10 per cent of Group A regardless “That is just not natural.” Parallels, of of productivity or other factors. A change course, abound in the professional sphere. in the compensation of any one group Every municipality or department in the automatically ratchets the pay of the other. region has a Nouveau Widget so, obviously, we need one too. A current bureaucratic Sometimes we use the term begging the favorite is the rationale for why we keep a question to describe a circular argument. current practice or why things don’t change. The form of the argument is essentially The cliché du jour is: “It is what it is,” which the same: “You know the reason that has replaced the formerly abused, “Well, action is illegal is because it is against the that is the nature of organizations.” All of law.” these are logically non-starters. Page 36

Just because two things appear associated, doesn’t necessarily mean that one causes the other–or that, in fact, they are causally connected in any way.

Causal Linkages One clinker of a fallacy we did not discuss Of course, this fallacy can also be previously goes by the formal name of post applied to fi re engines and fi res as well as hoc, ergo propter hoc, which means “after ambulances and injuries. this, therefore because of this.” Those who might have studied statistics will recognize Obviously, association or correlation this as a variant of the “correlation does is somehow related to causation. The not prove causation” fallacy. question is how can we identify or recognize a causal relationship when we see Just because two things appear associated, one? The issue is important because causal doesn’t necessarily mean one causes the thinking and causal imagery have become other—or that, in fact, they are causally entrenched in our everyday view of the connected in any way. The possible absurdity world. Whenever we see something we do of assuming that because two things are not quite understand, our fi rst inclination correlated they are connected is presented is to ask, how did that come about? In in the police car fallacy. The story here is other words, what was the cause? that a Martian comes to Earth and notices that wherever a crime has occurred, there From an historical perspective, formal is invariably a police car at the scene. The causal thinking is a relatively recent idea. Martian, therefore, erroneously concludes Most scholars use David Hume’s writings that police cars cause crime. as the starting point for explaining what is a cause and how we might identify one.

David Hume (26 April 1711 – 25 August 1776) was a Scottish philosopher, historian, economist, and essayist known especially for his philosophical empiricism and skepticism. He was one of the most important ﬁ gures in the Scottish Enlightenment, and in the history of Western philosophy. He is the philosopher “widely regarded as the greatest who has ever written in the English language.” Hume is often grouped with John Locke, George Berkeley, and a handful of others as a British Empiricist.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Thinking Critically Page 37

Hume was a Scottish philosopher who lived in the early to mid-1700s. Without Hume’s conditions for a belabouring the issue, Hume identiﬁ ed causal relationship three necessary conditions for a causal relationship. The ﬁ rst condition is 1. The cause and effect that the cause and the effect must be must be coincidental. coincidental or “conjoined,” as he said. This is the correlation part where two 2. The cause must come things generally appear together. before the effect.

The second condition is that the cause 3. There is no underlying must come before the effect. Therefore, third factor resulting in if the Martian had been around a little longer, he would have noticed that the the cause and effect to crime occurred fi rst and that the police be appearing together. car generally turned up later. Thus, it was crime that caused the police to respond; crime was not a consequence of the Spuriousness means that a relationship existence of police cars. between two or more factors is coincidental. The real cause is an The third element of causation is the most underlying third factor. The problem diffi cult issue and that is what we call the here is that even if we take away the condition of non-spuriousness. Non- apparent cause, the effect will remain. spuriousness means the cause is not just Thus, with crimes and police cars, if a enough or suffi cient to cause the effect, prank caller instigates a call that makes but that it necessarily produces the effect police cars appear, then they will appear or outcome. This is sometimes easier to whether a crime occurs or not. From an understand in the negative. What non- evaluator’s or a scientist’s perspective, spuriousness means is that no third factor non-spuriousness is generally the most is resulting in the apparent cause and effect diffi cult factor to control. Observing to be appearing together. An example here that two events generally coincide is not might be the strong correlation between diffi cult, nor is it diffi cult to see that one crime, the number of police offi cers and event generally precedes the other. the number of crimes across jurisdictions. Neither may be a cause of the other; both, however, are driven by an underlying increases or decreases in population density. Page 38

The diffi cult issue is assessing whether Unfortunately, we conduct much research some other underlying mechanism is that does not consider all three of those driving both of those events. Essentially, issues. That is why, for example, we often we have devised two ways to deal with the hear of some medical survey where spuriousness issue. The fi rst is to try to some factor (say, pomegranates) are develop explanatory theories to explain supposed to reduce the risk of cancer. how or why something should cause Typically, the study is correlational such something else. In formal terms, we need that someone conducts a survey and it is to fi nd what we call a causal mechanism. found that people who eat pomegranates Logically, why should X produce Y? As have a lower incidence of cancer. We can we say in the trade, “What’s the story?” probably determine that the consumption of pomegranates preceded the onset or The second way of dealing with the non- non-onset of cancer. spuriousness issue is through the physical manipulation of conditions. That is, What those studies generally do not do can we physically reproduce the effect is to control for spurious or confounding ourselves? We call this manipulation an factors. For example, pomegranate experiment. eaters may be also less likely to smoke, get more exercise, eat a healthier diet Over time, we have developed a series and generally have a healthier lifestyle of experimental designs or ways of than non-pomegranate eaters. Those manipulating situations so that we can factors are likely the real causal agents. isolate what we believe are the cause Including pomegranates in the diet or not and effect factors from other possible is irrelevant. or spurious infl uences. We will highlight those techniques in a later chapter. Of course, once we start to believe that pomegranates are related to cancer, we can In summary, then, it is suffi cient at this generate any number of possible causal point to consider that all three conditions explanations after the fact. For example, must exist for us to be reasonably we might argue that high levels of vitamin confi dent that something is truly the cause C or antioxidants in pomegranates fi ght of something else. Those are the elements the onset of cancer. of coincidence or correlation; temporal sequencing where the cause precedes or comes before the effect; and, the condition of non-spuriousness where no other underlying mechanism is generating both the apparent cause and the effect.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Thinking Critically Page 39

Linking Evidence to Explanations

A common mistake people make is that by collecting sufﬁ cient evidence, one can To prove a theory: “prove” that a hypothesis or theory is correct. In fact, the relationship between 1. We must ﬁ nd an explanation that an explanation and what forms evidence is consistent with at least most is complex. of the evidence we have to date.

To prove a relationship we generally need 2. We must then conduct to use data or evidence in two ways. First, secondary tests to see whether when we consider an explanation, we must those explanations hold up. fi nd one that is consistent with at least most of the evidence or facts that we have to date. If an explanation does not explain more proof that one causes the other. most of what we know, it is unlikely to be On the other hand, a few instances where a good candidate for what we need. crime occurred with no police cars about soon disproves the hypothesis. Once we have narrowed our plausible explanations to ones that make sense That is perhaps the single most important logically, and ones that generally fi t the point that Hume made in his discussion existing evidence, we need to conduct of causation. It is very diffi cult to prove secondary tests to see whether those something is true; it is much easier to show explanations hold up under critical that it is not true. circumstances. Obviously, we have selected an explanation that fi ts the One example Hume used was that just known facts, so simply collecting more because the sun has risen in the east and data under the same circumstances likely set in the west since time immemorial, it won’t give us more hard evidence. does not “prove” that this will necessarily happen tomorrow. On the other hand, For example, the fact that crime rates all we need is one instance where the sun in inner-city neighbourhoods with doesn’t rise in the east to disprove the graffi ti tend to be higher than other pattern. As contrived as that example might neighbourhoods does not provide proof be, it does make the point about the relative that graffi ti causes crime. Going back imbalance between evidence that appears to our Martian example, seeing ever to show a relationship and evidence that more instances of crime and police cars appears to dispel a relationship. appearing together does not provide Page 40

Working and Null Hyphotheses If we fail to reject or falsify the null hypothesis (so, in fact, there really is no Hypothesis testing is an approach difference in attitudes) then we must routinely applied in science to help logically reject the working hypothesis establish knowledge. It requires a true that probation offi cers who have prior (or false) statement to be made that criminal justice experience are less offers a plausible explanation about the supportive of rehabilitation than those problem. Testing the hypothesis results who have no prior criminal justice in our coming to some conclusions.1 A experience. working hypothesis might be something like: probation offi cers who have prior It is that strategy that scientists use to criminal justice experience (as a police test hypotheses and theories. We cannot offi cer or corrections offi cer, for example) prove the working hypothesis directly. are less supportive of rehabilitation than Instead, we create a null hypothesis that is those who have no prior criminal justice the opposite of the working hypothesis. experience. This is perhaps due to the If we fi nd support for the null hypothesis fact that the probation offi cers have been (that is, we fi nd that prior criminal justice on the front-lines and feel that they have experience has no infl uence whatsoever a good understanding and pre-disposed on the outcome) we toss out the working prejudices as to why some people who hypothesis. Or, at least, we need to commit crimes are not remediable. seriously reconsider what it says. If we do There really is no absolute proof to not fi nd support for the null hypothesis this. Instances could exist where such (we do not fi nd an alternate support for probation offi cers are in fact supportive. the opinion) we have very strong reasons to believe that our working hypothesis To provide evidence of whether this is is valid. As we fi nd that fewer and fewer really so, we would test the hypothesis by alternatives pan out, the greater credibility looking at instances where the opposite we have in the working hypothesis. could be the case. This leads us to what we call the null hypothesis: there is no statistically signifi cant difference in the attitude toward rehabilitation between probation offi cers with prior criminal justice experience and those without this experience.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Thinking Critically Page 41

An Example of a Working Hypothesis and a Null Hypothesis2 Between November 2010 and Safety Partnership sought to analyze February 2011, the district of Oldham the correlation of the two facts. in Greater Manchester, experienced At the beginning of the study, the an 18% increase in burglaries in analysts needed to state the working residential properties compared with hypothesis and the null hypothesis. the previous 4-month period. This In this case, we might state them as was an increase of 91 burglaries. follows: One of the hypotheses put forward was that the increase in burglary is Working hypothesis: attributable to an increase in burglary The increase in burglary is attributable offenders in the area, mainly as a to an increase in burglary offenders, direct result of an increase in prison mainly as a direct result of an increase releases. in prison releases.

Some crime prevention offi cers Null hypothesis: claim however that there is no direct correlation between the two. It is unlikely that an increase in Ex-offenders receive intensive individuals released from prison was supervision immediately after their the main explanation for the increase release from prison, with the aim to in burglaries in Oldham. minimize re-offending and maximize their rehabilitation. Perhaps it is a The researchers test the null failure in the system, or the breakdown hypothesis. If the evidence is of social support services that causes consistent with it, they conditionally recidivism and not the fact that they assume that it is true and essentially have been released from prison. reject the working hypothesis. If they fi nd the evidence is not consistent As a result of this, Oldham’s with the null hypothesis, they reject Neighbourhood Policing Teams, the null hypothesis and have strong a police Superintendent, Crime reason to assume the working Prevention Offi cers and analysts hypothesis is true. from the Oldham Community Page 42

Notes

1. Chainey, S. “Improving the Explanatory Content of Analysis Products using Hypothesis Testing”, Policing Advance Access. March 14, 2012. http://policing.oxfordjournals.org/content/early/2012/03/14/ police.pas007.full.pdf 2. Chainey, S. (2012)

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 43

Collecting Evidence

Environmental Scans

Evidence-based decision making is infl uenced by the plans we create to An environmental scan makes us aware help us set priorities, and by concerns that affect our organization’s ability to of internal and external realities, important fulfi ll its mandate. When issues arise issues, and trends that affect our organization. and decisions have to be made, we need evidence to help us decide the likely impact or effectiveness of our decisions. Why should we spend the time and Police departments can use the approach energy to conduct one? Among the for example, to improve performance most prominent are the following. and seek and establish an advantage in Environmental scans can provide: relation to criminals and criminal activity.1 • A fresh, objective look at issues within A common strategy for gathering this the organization’s goals and mandate, information is through an environmental with an eye toward how to rank them scan. Simply put, an environmental scan most effectively; gives us an informed, comprehensive • An opportunity to access information, picture of the current circumstances in research, statistics, and other data that which our organization exists. It makes someone else took the time to collect; us aware of internal and external realities, • An opportunity to involve community important issues, and trends affecting the stakeholders, organizations, individuals, organization. Information of this kind helps and groups in decisions that affect confi rm or refute our perceptions. It can them, by giving them an opportunity to guide us with future programming, strategic provide input, perspective, and advice; priorities, and budgeting. An environmental • An opportunity to discover the scan can also be useful in determining future strengths and assets in the larger strategies and in developing appropriate, community to address the issue; well-informed responses. • A framework or point of comparison to understand the assets and strengths What benefi ts do organizations receive of own organization; and from conducting an environmental scan? Page 44

• An opportunity to learn how your Generally, people who are more connected organization’s programs and practices with their colleagues, who read the trade are affecting other organizations, literature, and who regularly attend agencies, individuals, or groups, and conventions and workshops tend to fi nd to what degree your programs and this process easier. practices are effective in fulfi lling your organization’s mandate. A second part might involve a more formal review. Depending on the issue, you might Conducting an environmental scan is a seek out journal or news articles that have sequential process that involves gathering been written on the topic. A good place to information from secondary sources, start is to check the internet. Search engines including existing research reports, such as Google, Bing and Webcrawler statistics, or other information. This is can retrieve a tremendous amount of supplemented by fi rst-hand or primary information very quickly. One of the big sources of information, from individuals challenges in using general search engines is or groups that you will contact yourself. that it is sometimes diffi cult to identify the Analysis of this information leads to exact search terms you need. Consequently, establishing where your organization fi ts the search generates more chaff than wheat. within the broader social ecology. Using Internet Search Engines Unlike many other management procedures, there are few formal guidelines There are some tricks to using search for conducting environmental scans. What engines. If you are fortunate enough to we will do, however, is to provide you with have access to a municipal librarian or a local an overview of the procedure and some college or university library, there are usually suggested tools for moving forward. experienced people who can provide some assistance. Some tips for narrowing Google Types of Environmental Scans searches are provided on the next page.

There are essentially two types of Either online or by visiting a library, it is environmental scans. The ﬁ rst approach also possible to search the professional is a less formal type of scanning that you literature. Trade magazines and journals conduct yourself, based on your own often provide coverage of general issues. knowledge and what you or an assistant can gather sitting at your desk. The ﬁ rst step Speaking with a librarian well-versed in is to write out what you know about how criminology or police sciences at a local others are dealing with similar situations. university or college, or knowledgeable In other words, you are looking to see how staff at a justice library, is a good place others in your social environment do things. to start.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Collecting Evidence Page 45

For more detailed sources of information, the United Nations Ofﬁ ce on Drugs and it might be necessary to enter the formal Crime (UNODC) can offer a wealth of research or academic literature. This latter information. The box on the next page step can be a little daunting at times since gives a brief listing of some of the major there is a lot of variation in how technical policing and criminal justice journals. A few articles are written. Some are very key web links are also provided. Many local accessible while others require extensive universities have academic partnerships prior knowledge of the topic. The key is with police services and are a great source not to become discouraged. of information and potential joint projects.

Sometimes it is worthwhile looking further If you require information on aﬁ eld. In this case, ofﬁ cial websites such characteristics of your community or other as those of the US National Criminal statistics, a great deal of information is Justice Reference Service (NCJRS) or available on the Statistics Canada website.

Eff ective Searches on Google 1. Be specific. 4. Limit your search by Find pages within sites using excluding unwanted terms. site:[website URL] and your Put a minus sign in front of terms search phrase, fi nd authors you wish to exclude. For example, using author:[name], and type alarms -burglar will exclude the term intitle:[word] to fi nd a page with “burglar” from your search. To limit that word in the title. a search numerically, use the range (two dot) indicator. For example 2. Format. “used armoured vans 2010 .. 2014” Use filetype:[pdf or other extension] will limit results to those years. to fi nd images and all sorts of fi les (such as docs and jpgs). 5. Use specific search engines. Google scholar, for example, 3. Broaden your search. is an excellent way to fi nd both Use an asterisk (*) as a wildcard academic and other articles on search operator to fi ll in the selected topics. Webcrawler looks blanks. For example, “polic*” across a series of search engines. will return information on Also check out the website for police, policing and so on. Amazines (www.amazines.com) for a database of free articles. Page 46

❖ Sources of information

A Sample of Professional Journals Aggression and Violent Behavior International Journal of Police Science Alcoholism Treatment Quarterly and Management American Criminal Law Review Journal of Alcohol and Drug Education American Journal of Criminal Justice Journal of Child and Adolescent Substance British Journal of Criminology Abuse Campus Law Enforcement Journal Journal of Child Sexual Abuse Canadian Journal of Criminology and Journal of Contemporary Criminal Justice Criminal Justice Journal of Crime and Justice Canadian Society of Forensic Science Journal of Criminal Law and Child Abuse and Neglect Criminology Child Abuse Review Journal of Drug Issues Child and Youth Services Journal of Elder Abuse and Neglect Correctional Compendium Journal of Ethnicity in Substance Abuse Corrections Today Journal of Experimental Criminology Crime and Delinquency Journal of Family Violence Crime Prevention and Community Safety Journal of Forensic Identification Criminal Behavior and Mental Health Journal of Forensic Psychology Practice Criminal Justice and Behavior Journal of Forensic Sciences Criminal Justice Policy Review Journal of Gang Research Criminal Justice Review Journal of Interpersonal Violence Criminology and Public Policy Journal of Offender Rehabilitation Drug and Alcohol Review Journal of Police Crisis and Negotiations European Journal on Criminal Policy Journal of Quantitative Criminology and Research Journal of Research in Crime and Federal Probation Delinquency Global Crime Justice Quarterly International Criminal Justice Review Juvenile and Family Court Journal International Journal of Comparative Law and Order and Applied Criminal Justice Law and Policy International Journal of Offender Therapy Law Enforcement Technology and Comparative Criminology Legal and Criminological Psychology

Continued on next page

The Right Decision: Evidence-based Decision Making for Police Service Professionals Collecting Evidence Page 47

❖ Sources of information

A Sample of Professional Journals (cont.) Police Practice and Research: An Substance Use and Misuse International Journal Trauma, Violence, and Abuse Residential Treatment for Children and Violence Against Women Youth Western Criminology Security Journal Women and Criminal Justice Studies in Conflict and Terrorism

Online Sources National Institute of Justice. Statistics Canada: www.statcan.gc.ca http://www.crimesolutions.gov/ US Department of Justice Community RAND Center on Quality Policing. Oriented Policing Services: http://www.rand.org/jie/centers/ http://www.cops.usdoj.gov/ quality-policing.html

A signiﬁ cant amount of information on In this instance, you might consider policing and the criminal justice system bringing in an outside facilitator and can is available through Statistics Canada.2 conducting a formal scan. The process of doing a formal scan is outlined in the To do a scan most effectively, make sure second part of the chapter on SWOT you have collected information in more analyses. The primary difference between than one way. By doing this you can check an environmental scan and a SWOT and cross-reference to see if the same analysis is that the focus or range of issues issues and concerns are surfacing through considered by an environmental scan is your various sources of information. generally much broader. SWOT analyses are typically limited to issues relating to Occasionally, it is worthwhile conducting challenges and opportunities confronting a formal process where others in the the organization. organization are involved. Page 48

Framing Your Environmental Scan Try to determine how these people have ranked the concerns that also showed The information that you will be gathering up in your secondary research. Do they is inﬂ uenced by the question you are trying see it the same way? Or have they raised to answer. To frame the environmental different thoughts, ideas, or concerns scan, we can start by asking some focused that have not shown up in the secondary questions, such as the following: research?

• What is the key issue? Once you or your team have agreed on • What do we need to know about the the ranking of the issues, beginning with issue? the most serious and urgent, then you can • What are the trends and drivers begin to consider the strategies, program affecting these factors? activities, and practice that will help you address them. You will also need to Once the question has been carefully consider the budget implications involved framed, and the research has been in meeting these strategic priorities. gathered from primary and secondary sources, then the analysis begins. As we noted, environmental scans are First, we need to consider what themes, often accompanied by a SWOT analysis, concepts, issues, or concerns surfaced in which determines the internal and external the secondary research. In other words, strengths, weaknesses, opportunities and how have other groups, organizations, threats that are affecting the organization’s communities or police departments ability to fulﬁ ll its organizational mandate. elsewhere been affected by this issue? The SWOT analysis is explained more How have they ranked those concerns? fully in the second part of this chapter.

Compare the results of your surveys with the qualitative data that is emerging from your focus groups. Consider what people have been saying in the one-to-one interviews. What common themes are emerging? How are the results showing consistency and repetition?

The Right Decision: Evidence-based Decision Making for Police Service Professionals Collecting Evidence Page 49

Example: Police Sector Council Environmental Scan3

In 2009, the Police Sector Council The environmental scan was the result (PSC) commissioned a report to of a literature review and survey of understand better the changing environmental scans among police Canadian landscape. The report sought forces in Canada, conducted in 2008. to offer a snapshot of Canadian society Over 300 organizations, from small and global issues that are relevant regional police forces to colleges to the policing community. It was with policing courses, and all levels motivated by a perceived increase in between provided their feedback. the complexity of current social trends, along with increased interrelations Assume you are the police chief among various demographic, social of a mid-size police department and economic factors. A key element in a community with its own set of was the recognition that Canadian policing challenges. Someone in police forces had to work together your organization has suggested that with one another against this backdrop it would be interesting to examine of social change. The report sought to what challenges departments in provide feedback about implications other communities are facing. You for policing in an effort to pull together could do an environmental scan a more connected and cohesive view of your own and scan the various of the policing world in Canada. community, regional police websites, but you are having a hard time ﬁ nding Police sector stakeholders were asked information as most departments use what they would like to see in an data for internal planning purposes. environmental scan produced by the Or you could use the PSC report as it PSC. Based on their suggestions, the already contains the information from document highlighted demographics over 300 organizations. This not only and public safety & security, and saves you a lot of time, but it also will introduced social and human provide you with a comprehensive resource management sections to the view of what other organizations are list of topics. facing.

Continued on next page Page 50

Summary of Key Trends Explored in PSC Report

Th e world population, like Canada’s, is Th e proportion of Canadians with a growing. Canada relies more and more post-secondary education is increasing, on international migration to increase along with the level of computer its population and renew its workforce. and technology literacy. Technology Canadians are aging, and with one in represents a double-edged sword for three Canadians considered a baby police. On the one hand it can be used boomer, the country faces the reality to solve and prevent crimes along with of mass retirement. There will be new gathering and storing information, but challenges for the Canadian policing on the other, it is increasingly used to community, as workers retire while support a range of white collar crimes today’s youth show only moderate including fraud, identity theft and interest in policing as a career. related predatory actions.

While, on the world scale, Canada Malls, gated communities and other is economically advantaged, poverty new forms of property blur the remains an issue for several sub- distinction between public and private populations. The current economic places, requiring police to collaborate uncertainty is also causing greater levels further with private security forces. of insecurity for some. Homeless Blurred boundaries and jurisdictional and poor populations, as well as issues over crimes in cyberspace are Aboriginal populations, continue to be other challenging dimensions that overrepresented in the justice system will require further international as both victims and perpetrators. The coordination by police. policing world will have to adapt to new realities in the near future, in particular Diversity in culture, lifestyle, and moral if, as some argue, socio-economic codes is posing greater challenges disadvantage is linked to greater crime to politicians and the policing world rates. alike, as Canadian society struggles to ﬁ nd a middle ground encompassing the acceptance of diversity and a fair interpretation of the law.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Collecting Evidence Page 51

Based on this and other information, Another option would be to identify the you might decide to refocus the service four or ﬁ ve key areas in which all other components of your own department. departments engage and focus on those Clearly, several options are available. as your core functions. Depending on your department’s location (a major metropolitan area as opposed Again, what you get out of an to a small, rural community), you might environmental scan is determined by the wish to broaden your range of activities. initial question you are trying to resolve.

❖

SWOT Analyses

A SWOT analysis is an assessment of When that information is combined with an organization’s Strengths, Weaknesses, the results of the SWOT, you will be Opportunities, and Threats. Keep in mind better equipped to identify your strategic that, typically, the strengths and weaknesses priorities and future directions. are internal to the organization, while the opportunities and threats are characteristics The SWOT adds to the results of the of the external environment. environmental scan by engaging various members of your organization in a SWOT is easy to use. It can be a useful discussion of the strengths and weaknesses complement to the environmental scan. A that exist within your department. Looking SWOT can generate crucial information outside the department allows you to with relatively little effort, and it brings that consider opportunities that could be seized information together in a framework that to advance the interests of the organization. provides a good base for further analysis. The SWOT also explores threats: those It is an excellent decision support tool, and external factors, realities, or trends that aids us in making an important decision – can make the ongoing functioning of the especially the right decision. department more challenging.

As we discussed earlier in the chapter, the A SWOT analysis is sometimes conducted environmental scan will provide you with as a group session with a facilitator. This primary and secondary information to might be preceded by a survey that each determine pressing issues and concerns member of the group completes in related to your research questions. advance, so they have a chance to consider their own assessment before group discussion begins. Page 52

Even simpler, one can give each group As with any activity, consultation has member a blank SWOT template that a price. While employees are being they can use to jot down their thoughts consulted they are not doing their normal in advance, and then have them bring it activities. Furthermore, group dynamics to the meeting. can generate unexpected results. Group politics come into play and red herrings Conducting a SWOT Analysis can occupy a signifi cant amount of time. For those reasons, it is often benefi cial While it is possible to conduct a SWOT to have an outside facilitator lead the analysis by yourself, the real benefi t exercise. The advantages of consultation, of the exercise is usually seen when however, are numerous. several members of the organization are involved. First of all, groups tend to generate crucial ideas that a single manager or even One of the paradoxes managers face is a management group might overlook. that on the one hand, employees and Second, people from different segments others expect leaders to lead but, at the of the organization interact with different same time, they expect to be part of the audiences, suppliers, community groups, decision-making process. clients or customers, regulators and, other service providers.

Example: Completed SWOT Analysis Strengths Weaknesses • Community and “problem solving” • Aging staff and pending philosophies retirements • Services to community • Recruitment challenges • Fiscal responsibility • Lack of diversity • Innovation

Opportunities Th reats • Collaboration with neighbouring • Increased reliance on technology community concerning disaster • Population aging / decline management planning. • Increasing budget costs • Access to senior government funds to • Need for additional civilian expand disaster response strategies. support staff

The Right Decision: Evidence-based Decision Making for Police Service Professionals Collecting Evidence Page 53

This gives them different perspectives analysis meeting. Ensure you include the on the organization, particularly with guiding research question that is behind regard to outside infl uences. Third, the environmental scan and SWOT even participants who do not see their process, as that will create the framework input refl ected in the fi nal product for the discussion. Create helpful ground generally feel they have had some say in rules for the discussion. the process. This typically has a positive effect on morale and often creates more SWOT Discussion Ground Rules “buy in” when choices have to be made • Focus on one quadrant at a time. and different options are implemented. • Listen to understand, and

acknowledge what you are hearing In a group situation, one of the fi rst others say. Avoid interrupting or questions when conducting an analysis is: criticizing the contributions of others. Who will participate? It is helpful to have • Establish reasonable time limits to a diverse cross-section of individuals keep the discussion moving forward. to ensure the most comprehensive Respect each other—it’s acceptable assessment. While no guarantee, this to have differing points of view and helps to increase the likelihood that no perspectives crucial aspect is overlooked. As a general • Agree on how distractions such as rule, the SWOT analysis should be done cell phones and interruptions from by no less than mid-level management, support staff will be managed. It is and preferably even a higher level of suggested that cell phones be turned leadership. In addition, the analysis should off and administrative staff interrupt include representative employees from only for emergencies. throughout the organization. Front-line • Confi dentiality: What can be shared supervisors should be included. Again, outside the room? Where will the while not always the case, leaders in the information go in the end? How will organization often have greater insight anonymity be protected? into those external and internal issues • All team members should participate. that need to be considered. This comes from their experience as well as their As the group considers the issues and relationships with a wide variety of people concerns that have resulted from the inside and outside the organization. environmental scan, ask them to consider each quadrant in turn as a means of Before starting the analysis, and fi lling in assessing how those issues and concerns the matrix, it is often worthwhile providing could be more fully addressed or the team with the environmental scan understood. results to read in advance of the SWOT Page 54

As you proceed through your SWOT In summary, the SWOT analysis analysis, keep these factors in mind: combines with the environmental scan • SWOT analysis is a subjective to create strategic plans that are realistic, process, not a science. However, the researched, and supported by internal quantitative and qualitative data that personnel and external stakeholders. emerged from the environmental Evidence-based decision-making beneﬁ ts scan will help the participants trust from using tools such as these, leading that the results are well-founded. to plans and decisions that will be solidly • Keep it simple by focusing on a few grounded in facts and research, and issues only. If other matters emerge, guided by a wide array of perspectives they can be addressed later through and input. a subsequent process. Without these limitations, the process may bog down with too much data and information to be dealt with at one time. • Be realistic about the strengths and weaknesses of the organization. Create safety and transparency so participants will be honest.

❖ Notes

1. “Strategic Planning in Policing, Part II.” http://mennozacharias.com/tag/environmental-scanning/ 2. http://www5.statcan.gc.ca/subject-sujet/theme-theme.action?pid=2693&lang=eng&more=0&MM 3. “Environmental Scanning Labour Market Information Project 2009.” Environmental Scan prepared for the Police Sector Council. http://www.policecouncil.ca/wp-content/uploads/2013/03/PSC- Environmental-Scan-2009.pdf

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 55

Statistics

A Tool for Decision Making

Statistics is probably one of the most misunderstood of disciplines. Most The key to understanding statistics is to university students dread having to study it, and most professors who teach it often see it as a way of organizing and making do so with great reluctance. Furthermore, sense of a world dominated by uncertainty. the topic is often reviled as a tool of charlatans. As Mark Twain once claimed, “There are lies, damned lies and statistics.” It should not be used as a last resort to Yet, used appropriately, statistics can be rationalize or support a hastily made prior one of the most useful and powerful decision based on intuition. tools in the decision maker’s toolbox. What is key for most decision makers is not Our suspicion is that statistics’ bad to get tangled in the details of statistical name stems from two sources. First, analysis, but, instead, to understand the many people see it as an outcropping fundamental principles or logic behind of math—with which most of us had the activity. Those fundamental principles a less than excellent experience in high are few and, generally, quite simple. Once school. Second, most people who teach understood, however, the principles of statistics are not themselves statisticians statistics can be used to great advantage, and, while they may come to master the even if one doesn’t have a detailed technical details, they rarely grasp the knowledge of the underlying math or underlying logic. Statistics does entail technical aspects. some math, but most of that math is no more complicated than being able Statistics consists of two basic activities. to balance one’s chequebook. The key The fi rst is the collection of data in an to understanding statistics is to see it as attempt to describe something. The a way of organizing and making sense second is the use of data to help make of a world dominated by uncertainty. In decisions or inferences. The fi rst activity fact, one defi nition of statistics is that it we call descriptive statistics; the second, is the science of decision making under we call inferential statistics. conditions of uncertainty. Page 56

A Discussion of Measurement

We often refer to the process of observing and recording data as a measurement. What distinguishes the way statisticians view Inherent error relates to what measurement from most other people is we are measuring—e.g. a that statisticians assume all measurement breathalyzer test, which may contains an element of error. In other be affected by whether there words, in the world of statistics, having is alcohol in your mouth. something measured with one hundred per cent accuracy is more good luck than good management. From a statistical Operational error relates perspective, error in measurement has to how we are conducting two basic sources: inherent error or the measurement—e.g., a instability, and operational error. problem with the measuring device or how we read it. When we speak of inherent error or instability, we are referring to the property of the thing we are measuring. For example, if you were to ask someone to While relatively stable in a range, most tell you on a 100-point scale how satisfi ed people’s actual level of job satisfaction is they were with their job (assuming 0 is inherently unstable. total dissatisfaction and 100 represents total satisfaction), they might respond The same applies to breathalyzer tests. 71. If you asked the person the same Breath analysis is by far the most question on several different occasions, commonly used method of testing for they would likely give you a range of blood alcohol (BAC) in impaired driving answers somewhere close to 71. cases. Assume a police offi cer takes two separate readings from a driver he has just The reality is, most people have a general pulled over. He will likely get different idea of their level of job satisfaction but BAC levels between the fi rst and the have a hard time giving a precise number. second reading depending on whether Furthermore, while they may be mostly the driver had just burped or vomited; satisfi ed with their job, their exact level if there was electrical interference from of satisfaction would vary according to a cell phone and police radio; or if there numerous factors ranging from the time of was tobacco smoke, dirt, or moisture in day, to whether they just had an altercation the environment. with their superior, to the weather.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Statistics Page 57

Consequently, from a statistical It is far better to admit that error in perspective the BAC level is inherently measurement is everywhere, so why not variable. admit it and try to get an estimate of the size of that error? How can we do that? To the notion of inherent variability, we The answer is that we need to either take can also add operational error. Perhaps several measurements of the same item, the police ofﬁ cer forgot to perform a or to measure several items assumed to manual calibration check on the device. be the same. The battery was not fully charged. The device was improperly used. The breathalyzer forms were not completed correctly. There was an error in copying down the results, 0.8 instead of 0.08., or between testing the BAC and recording it the ofﬁ cer forgot the actual number.

The point is that, try as we might, it is generally difﬁ cult, if not impossible, to have totally accurate measurement. Believing we can do so is simply fooling ourselves. Furthermore, for most From the statistician’s perspective, people who situations, “close” is good enough. What believe that total accuracy of measurement is possible does it matter if the BAC is 0.08 or 0.085? are like ostriches with their heads in the sand. One thing that makes statistics powerful is that statistics assumes some error will appear in our measurement.

What is also great about statistics is that, when used appropriately, we can estimate how much error exists in the measurement process.

From the statistician’s perspective, people who believe that total accuracy in measurement is possible are like ostriches with their heads in the sand. Page 58

Descriptive Statistics

Remembering the characteristics of a single item is relatively easy, whether that Descriptive statistics summarize item is a person, an event like the eclipse the characteristics of a group of the moon, or the colour of one’s motor vehicle. Similarly, most of us can so we can make sense easily recall the characteristics of several of a mass of information. items. The larger the number of items becomes, however, the more difﬁ cult it is We may measure typicality for us to remember the individual items by determining the average that make up the group. For example, or median age in the group. we may recall the ages of all of our colleagues in a police station. Recalling We may measure variability the age of all police personnel in a region by determining the youngest is virtually impossible. If we want to be and oldest ages in the group, able to say something about the ages of police ofﬁ cers in a region, we need to the spread of ages within the somehow aggregate or summarize the group, or how much the results data. This is where descriptive statistics deviate from the average. come into play.

What descriptive statistics do is summarize We might fi nd, for example, that the the characteristics of a group so that we average police offi cer in our region is 38 can make sense of a mass of information. years of age. Other measures of typicality Even if we could remember them, listing include the median and the mode. The the ages of 600 police men and women median is that point in the age distribution is not a very useful exercise. Descriptive below and above which half of the ages statistics allows us to identify certain fall. The median age might be 35. In other useful characteristics of the list. Often, words, half the police offi cers in our region the fi rst two things we want to know are above age 35 and half are younger. about a list or bunch of observations are The mode is another term for the most what is typical and how much variability common age. The mean, the median and is there? the mode are the most commonly used measures of typicality. We can also think The most common measure of typicality of those measures as a central anchor is the arithmetic average or mean. point for the list or distribution of ages.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Statistics Page 59

Measures of variability give us an idea of To deal with distributions that have the odd how widely a bunch of measures range extreme case, we sometimes use a statistic or vary. It is one thing to know that the known as the interquartile range. To get average age of a police offi cer in our the interquartile range, we need to fi gure region is 35; it is something else to know out the age of the person who is at the that most are between the ages of 30 and 25th percentile point of the distribution, 40 as opposed to 25 and 50. The most and the age of the person who is at the common measures of variability are what 75th percentile. The interquartile range is we term range statistics and variance simply the difference between those two statistics. numbers. Again, like the min-max range, the interquartile range gives us an idea of Range statistics are simple measures of the spread of the ages. the distance between two points. For example, among our police offi cers, the Besides ranges, we often use statistics youngest may be 24 and the oldest 58. The known as variability statistics to give us range would simply be 58-24, or 34 years. some notion of how the data are spread This range measurement is based on the or disbursed about the measure of central difference between the minimum value in tendency. The two most commonly used the distribution and the maximum value. variability statistics are the variance and Min-max ranges are interesting but can something called the standard deviation. sometimes be misleading. For example, At fi rst sight, these statistics may appear the oldest person in a region might be a little daunting but conceptually, they are 65 while most of the other “elderly” quite simple. The key in understanding employees are less than 55. Here, we them is not to focus on the math but to sometimes call the 65-year-old an outlier. consider the underlying ideas.

See the following pages for examples of typicality and variability. Page 60

An Example of Typicality

Even simple descriptive statistics can be useful in decision making. Let’s examine the number of robberies in two neighbouring town over a week (7 days). The reported offences for each day are shown in the accompanying table.

Day Town A Town B Mon 10 9 Tues 11 16 Wed 8 7 Median Point Thurs 9 7 Fri 9 8 Sat 9 7 Sun 5 7 Sum 61 61 Mean 99 Median 97 Mode 97

For measures of typicality, we can calculate the average or arithmetic mean, the median and the mode. The average or arithmetic mean is simply the sum of the robberies divided by the number of days. The median is that point below and above which 50 per cent of the numbers fall. The mode is the most commonly recorded response time. The data in the boxes represent the actual numbers of robberies. Even from this limited amount of information, there are several points of interest. First, both stations have a total of 61 robberies in a week. This resulted in an average robbery rate of 9 robberies a day. Examining the numbers, however, it appears that Town B had one day when there were 16 robberies. In statistical language, we call exceptional values such as this outliers. The arithmetic mean is very sensitive to outliers. This is easy to visualize if we replace the 16 with a value of 30. All the other values stay the same but the mean would shoot up to 10.7 robberies. A measure that is much less sensitive to outliers is the median (or midpoint, as it is sometime called). As we have noted, the median is the value that breaks the distribution into the upper and lower 50th percentile. In the table, the median or midpoint is 9 which coincidentally fall on Thursday, the middle day of the week. For Town A, the median or midpoint of the distribution is 9 and for Town B, the median is 7. That Town B has a lower median than mean is a consequence of the fact that, except for the outlier value of 16 robberies, Town B generally has lower numbers of robberies than Town A. Because we are only dealing with a few values, this is easy to see. It would be less obvious with a large data set. Regardless, the principles hold.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Statistics Page 61

An Example of Variability

In this example, we will use the robbery data from the previous box. We have seen that the typical or average robbery occurrences are about the same for both stations. However, looking at the raw data suggests that there might be more variability in the occurrences in Town B times as opposed to Town A. The fact that the mean and the median were slightly different provides numerical support for this view.

Town A Robberies Deviation from Mean Deviation Squared Mon 10 1 1 Tues 11 2 2 Wed 8-11 Th urs 900 Fri 900 Sat 900 Sun 5-416 Mean 9 0 2.7 Town B Mon 900 Tues 16 7 49 Wed 7-24 Th urs 7-24 Fri 8-11 Sat 7-24 Sun 7-24 Mean 9 0 9.4

One measure of variability is the range. Town A’s robbery rates go from a minimum of 5 to 11, providing a range of 6. Town B’s robbery rates go from a minimum of 7 to a maximum of 16, providing a range of 9. Another two commonly used measures of variation are the variance and the standard deviation. While seemingly complex, these measures are conceptually simple. In the second column of numbers, we have subtracted the mean from each individual response time. For example, in Town A, the ﬁ rst deviation is 10-9=1. We do that for each individual robbery. In column three, we simply square the deviations from the means (that is, multiply the value by itself). When we do this for all of the observations, we discover two things. First, the average of the deviations from the mean is zero. This will always be the case because the mean is in the “middle” of the distribution and the positive deviations will cancel out the negative ones. That is why we calculated the third column: the squared deviations. The mean or average of the squared deviations is known as the variance. The variance for Town A is 2.7 and for Town B it is 9.4. This suggests that there is much more variation in the robbery rates of Town B than for Town A. The variance is a statistic that is used a great deal. In slightly more advanced statistics, our goal is to try to explain why there is more variance or variation in one set of numbers than another. Perhaps, across the two towns, the level of economic prosperity is substantially different. The difference might also be due to variations in the effectiveness of policing. Those are notions or hypotheses we might want to test. Since squared values generate big numbers, we often compare the square root of the variances. This brings the values back to the size of the original measurement (raw numbers as opposed to squared ones). The square root of the variance is known as the standard deviation. The standard deviation for Town A is 1.6 and for Town B it is 3.1. This suggests that the variation in the robbery rates in Town B is nearly twice that of Town A. Page 62

An Aside Subsequent investigation may show that the ﬁ rst person has to perform the We can also use variance statistics as an action under a variety of conditions while estimate of how much error in measurement the second faces fewer environmental exists. For example, two people may take 25 challenges. It may also be that the ﬁ rst minutes on average to complete an activity. person lets things “slide” for a while and The variance for one person might be eight then turns on the juice to get the numbers minutes and for the second person three back up to an acceptable average. minutes. Based on the average both people appear equal in performance, but the Regardless, knowing differences in variance measures suggest that the second variances can sometime tell us more than person is much more consistent and, in simply knowing differences in averages or that sense, better. From a management central tendency. perspective, the interesting question is why one person is more consistent in their performance than the other.

❖

Inferential Statistics

The second leg on which the discipline of For readers who have no background in statistics stands is what we term inferential the area, it might be worthwhile investing statistics. Inferential statistics help us to is a one-semester course in a local college. draw conclusions and make decisions. Otherwise, understanding the concepts is Unlike for most descriptive statistics, sufﬁ cient; just leave the details to an expert. the math behind inferential statistics can get complicated. Consequently, we will Inferential statistics are used for many restrict our focus to the logic underlying purposes. However, the two primary inferential statistics and examine how they ones are to be able to estimate or infer can be used to help us make decisions. the characteristics of a population Learning inferential statistics by oneself from a sample, and to estimate whether from a book is typically not easy. signiﬁ cant differences exist between two or more populations or samples.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Statistics Page 63

Population Estimates Random selection This implies the households in the sample Let’s start with the issue of making are chosen using a chance mechanism inferences from samples of populations. – things like coin tosses and computer If we wanted to know the proportion random number generators. In other words, of the population of a city that uses someone cannot choose the households burglar alarms, we could contact each based on availability or door colour. Random household and pose the question. selection implies that a listing of households Collecting information from everyone in (say a city directory) exists where the a jurisdiction is known as conducting a households are listed or numbered from 1 to census. In a city of 300,000 households, 300,000. For a sample of 1,200 households, that could be an expensive and time- we would use a random number generator consuming proposition. That is why to give a listing of 1,200 numbers between 1 censuses are done only rarely and under and 300,000. Once we have those numbers, limited circumstances. Fortunately, early in we would then identify the households that the twentieth century, statisticians ﬁ gured hold those positions or numbers on the list. out how to estimate the characteristics of the whole (a population) from a sub Equal chance group or sample. This implies that each household has the same chance or likelihood of selection. The key to being able to do this, however, Lists with duplicate addresses or lists is in the way in which the sample is that omit certain a type of household drawn or collected from the population. (say, all apartments or all households in Essentially, “any old sample” doesn’t a particular neighbourhood) mean some cut it. The sample has to be taken from households either have a greater likelihood the population in a particular way. There of selection, or no chance of selection. are some variations on the theme, but let us keep this simple and consider the Independence basic case. What we want is something This implies that the selection of one statisticians call a simple random sample. household does not determine or affect A simple random sample is one where the selection of another. For example, the each household in the population has an person selecting the sample might notice equal chance of being selected, and that two houses on the same block or two chance of being selected is independent houses next to each other appear on the list. of the other selections. Let us break Thinking they might be too much alike, the that down into the constituent parts: researcher drops one household in favour random selection, equal chance, and of another selection. That is not acceptable. independence. The selections that appear must be included despite anything else. Page 64

If we follow these rules, then estimating Many different ways of generating the characteristics of the entire population estimates are available, but you have from the sample is possible. Some other no way of knowing if they are close things need to be considered, such as the to the actual value in the population or size of the sample, but those are details somewhere out near the planet Mars. that are best discussed with a professional. If we follow the basic rules outlined Signifi cant Differences above, we can estimate what proportion of the population of households that Another primary use of inferential have burglar alarms within a given likely statistics is to be able to estimate whether range. two samples are similar or different. For example, over a year, a Police Chief In other words, the sample estimate will might wish to know whether differences be close to what actually exists in the in response times exist across police population but will probably not be the stations. Typically, data such as response exact fi gure. What differentiates statistical times are collected though an automated sampling from other procedures, however, dispatch system. At the end of a period, is that it is possible to estimate the range calculating the mean or average response within which the population fi gure will time is possible. As discussed earlier, the likely fall. Thus, we could conclude that mean value will be an estimate based the likely proportion of homes with on error-prone data and there will be burglar alarms we would see is X percent a distribution of values around that within plus or minus Y percentage points estimate. Thus, the question is, if the in, say, 19 surveys out of 20. response time of one department is eight minutes and another one is nine, does The uninitiated often disparage statistical that one minute difference refl ect a real estimates for not being able to provide difference or is it simply within the realm exact values. But, as we discussed earlier, of possible measurement error? the fundamental assumption in the world of statistics is that all measurement entails Some differences are big and substantively error, so the best we can do is come up meaningful and do not require statistics with a point estimate and a reasonable to help us make a decision. For example, notion of its level of accuracy. This is if the difference in response time were something no other procedure can do. 10 minutes, then we know a real and With a statistical estimate, you get an idea important difference exists. However, of whether an estimate is precise enough when we get to one minute, it is not clear to be useful or too variable for practical that the difference is real or just within purposes. the realm of normal variability.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Statistics Page 65

What statistics can do is let us know consequently, not substantively signifi cant whether that difference is within or outside either. that range of normal variability. If it is outside, then we say that the difference is Inferential statistics are of even more use statistically signifi cantly diff erent. when we have multiple comparisons to make. Typically, a large city may have 10 or We should note, however, that just because more police stations. Are the differences something is statistically signifi cantly across all 10 signifi cantly different? More different, it does not necessarily mean advanced techniques can help us to fi gure that it is substantively different. On out what factors might be related to those the other hand, if something is not differences. That brings us to our fi nal statistically signifi cantly different, then topic in this chapter and that is the role of we should assess the difference as being statistical modelling. within the normal range of variation and,

❖

Statistical Modelling

For most decision makers, the real power neighbourhoods can be used to model of statistics lies in the ability to model which neighbourhoods or what types of social, natural and mechanical processes. dwellings are most likely to experience Statistical models allow us to examine burglaries or break-ins. Currently, one complex issues where multiple factors of the more active and dynamic areas might affect a particular outcome. For of modelling is in predictive policing. example, statistical models have been Predictive policing is the application used to model response times to incidents of analytical techniques, particularly allowing for trafﬁ c and weather patterns. quantitative techniques, to identify Based on the result of those models, an promising targets for police intervention optimal placing of police vehicles and and prevent or solving crime. Predictive patrol routes can be determined. Models policing can offer several advantages to can also be used to ﬁ nd whether changes law enforcement agencies.1 It does not in the number of personnel dispatched or replace traditional policing. Instead, it whether certain patrol patterns are more enhances existing approaches such as effective than others. problem-oriented policing, community policing, intelligence-led policing and hot Similarly, the physical and socio- spot policing.2 demographic characteristics of Page 66

In statistical modelling, most of our • What is the relative impact or rank focus is on trying to explain variation. ordering of various components of Thus, we go back to one of our basic the model on the outcome factor? statistical concepts—that of the variance. • Are those impacts large enough to So, for example, we might ask: What are be meaningful from a substantive or the factors that likely affect the different policy perspective? variations in response times for the same • How do the various sub components call types between Detachment No. 1 and in the model interact with one another Detachment No. 2? Based on the outcomes as to their impact on the outcome? of those and other modelling exercises, it is possible to identify what form of As we indicated, statistics is not the intervention works and what doesn’t work. magic bullet for all decision making. Improving the allocation and effi ciencies of Used appropriately, however, statistical their resources is also possible for decision techniques can provide a great deal makers. Whatever the complexity of the of insight into the questions we are model or underlying process, statistical examining. analyses help us to fi gure out many useful results with an estimable level of accuracy. Decision making is a complex process, and the best processes are those where Among the key questions we can address we use the many tools at our disposal with statistical modelling are the following: to help come up with an answer. Often, trade-offs have to be made. Something • Does the overall model accurately may be statistically signifi cant but not refl ect the process we are trying to substantively signifi cant. Similarly, just describe or emulate? In other words, because one choice is more effective is it statistically signifi cant? than another does not mean that it can • How much of the variation in the be justifi ed socially or economically. outcome factor is explained by the Regardless, knowing whether something model? has a “real” impact or not is a good • Which elements in the model are starting point. statistically signifi cant and which are not?

❖ Notes

1. “Predictive Policing: Forecasting Crime and Law Enforcement.” Research Brief RAND Corporation. http://www.rand.org/content/dam/rand/pubs/research_briefs/RB9700/RB9735/RAND_RB9735. pdf 2. “Predictive Policing, National Justice Institute” http://www.nij.gov/topics/law-enforcement/strategies/ predictive-policing/Pages/welcome.aspx

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 67

Experimental Designs

How Do We Know What it Means?

A basic notion underlying this book is that making decisions based on evidence has To understand the meaning of a fact, we advantages over other forms of decision making. By evidence, we are referring to need an appropriate point of comparison. observable and measurable “facts” or data. While we argue that it is generally a good thing to have facts, a single fact Similarly, your department might have or bit of data or piece of information is an overall crime clearance rate of 40%, fairly meaningless in itself. The reason for with a rate of 70% for violent offences. this is that nothing has meaning except in At a city council meeting, the question is comparison with something else. raised as to whether these are acceptable performance rates. The average citizen For example, assume you are on a trip to might have expectations that at least 90% India and you see a pair of shoes on sale of all crimes result in charges being laid for 2,859 rupees. If you are not familiar or being otherwise cleared. By referring with prices in India, you might ask to national data reported to Statistics yourself whether this is a good value or Canada, it can be shown that the overall not. The “fact” that the shoes are 2,859 clearance rate in Canada is about 60% for rupees is irrelevant to you unless you violent crimes and about 40% for crime have something with which to compare it. overall. By making this comparison, it is That comparison might be with another clear that your department is performing product or with the average hourly on par with the rest of the country for wage in India or with the equivalence overall crime, and somewhat better when in another currency. Currently, 2900 addressing violent crime. rupees is approximately equivalent to $50 Canadian. It is only by making a The point being made is that to comparison that the relative value of the understand the meaning of a fact, we shoes takes on meaning. need an appropriate point of comparison. Page 68

Within the framework of evidence-based The remainder of this chapter will focus decision making, a key question we have on some basic experimental designs that to ask ourselves is: What is the most we use to assess the value of information appropriate point of comparison? A or data related to a question about which complementary question might also be: we need to make a decision. In applied What is the best way in which to make that research, designs can become very comparison? The answer is to use a standard complex. No matter the complexities framework that program evaluators of the design, however, there are a few and applied scientists call experimental fundamental principles that underlie the designs. Experimental designs are simply value or the merits of the design. different approaches to helping us make an appropriate comparison.

❖

The “Counterfactual”

When we do or observe something, the Einstein referred to this mulling of question is: What would have happened counterfactuals as thought experiments. if the event had not occurred? What if Thought experiments consist of the Axis powers had won World War conducting an analysis in our heads to II? What if the party in power had not think through the potential impacts won the last election? What would and consequences of a particular have happened if insurance companies event or outcome. What differentiated provided police services instead of Einstein’s thought experiment from municipalities? The comparison is simple fantasizing or theorizing is that he with some theoretical model. It cannot also focused on how we might test the give us proof of something, but as a thought experiment using real situations mental exercise, it forces us to identify and observable data. the important elements of a policy or program. What are the relevant or As an example of a thought experiment, active components that are making the we might consider the issue that police difference or that we expect to have an performance is affected by the level impact? These ideas, which are counter to of stress caused by the nature the job the existing outcomes or facts, are called content; for example, the perceived “counterfactuals.” risk, long hours, shift work and level of responsibility and accountability.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Experimental Designs Page 69

We recognize that these elements can be stressors, but do they in fact affect one’s What makes sense logically level of performance? In our thought does not always work out in experiment we might consider other factors such as organizational stressors. the observable world. What we need is hard evidence based Experience tells us that other factors can on repeatable observations. affect job stress levels. Perhaps it may not be the nature of police work that generates the greatest amount of stress for police ofﬁ cers. Instead, it is the characteristics Even if it is fundamentally true, we of the organization and behaviors of perceive some evidence as more valid, the people in them that may produce more reliable and more relevant than stress. Maybe it is the lack of rewards others. So it is in science. Good evidence or recognition for a job well done that is stands up to the rigours of a good cross affecting the job performance. We should examination. Still, what makes up good also consider other job-context stressors evidence? that are likely to create stress in the police milieu—organizational structure One characteristic of good evidence is and various aspects of organizational how rigorously people have tested it. life (such as coworker relations, training, Within the framework of science, the resources, leadership and supervision, basic mechanism for testing an idea is and internal affairs).1 Through this the experimental design. Experimental thought experiment we conclude that job designs are physical applications of logic, content is not the sole causal link to job so let us examine the logic underlying stress levels, but that other stressors such experimental designs. as job context are strong contributors. Assume for a moment that we wish What Makes Up Good Evidence? to assess the impact of burglar alarms on home break-in rates. One approach When we engage in evidence-based would be to take a community and install decision making, the fundamental question burglar alarms in all residential homes. is: What makes up appropriate evidence? We could then see if a difference existed If we think of science as a mechanism between the break-in rates before and for ﬁ nding the “real” explanation of after the introduction of the alarms. something, then thinking of it within the Unfortunately, any difference might be context of a court case makes sense. In the result of other factors (recall our the courts, as in science, there are varying previous discussion of spuriousness). amounts of evidence provided. Page 70

For example, by coincidence, home break-in rates might have dropped due The inability to observe the to a decrease in the number of young same entity in two different people in a neighbourhood, or a more situations simultaneously is positive job market, changes in police patrolling, or perhaps due to a more active known as “the counterfactual neighbourhood watch program. We know problem.” That is, how do all of those factors are related highly to we measure what would home break-in incidences. have happened if the other situation had existed? Ideally, we would like to be able to observe the same community with and without burglar alarms simultaneously. In How close that comparison group is to other words, we would assess the effect the treatment or experimental group of a burglar alarm program based on determines how much credibility we can the difference in outcomes for the same have in our results. community with and without participation in the program. Nevertheless, we know There are many ways of fi nding or creating that this is impossible. Something cannot comparison groups, some of which are be in two states at the same time. At better than others. The adequacy of a any moment the community either comparison group is something that participated in the program or did not evaluators spend much time and energy participate. The inability to observe the considering. same entity in two different situations simultaneously is known in science as “the For example, we might fi nd a “sister” counterfactual problem.” That is, how do community not far from the target we measure what would have happened if community and use that as a comparison. the other situation had existed? On the other hand, we might decide to hand out burglar alarms to every second If we cannot assess what would have residence, or to residences on the south happened if the opposite or counterfactual side of the community but not on the situation occurred, then how can we north side. We might even consider decide if burglar alarms have an impact comparing our target community with all and not something else? The approach of the other communities in the province scientists and program evaluators take is or region. All of those approaches can to fi nd a comparison group that is as close provide a point of comparison against to the treatment group as possible. which we can judge the potential impact of burglar alarms in the target community.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Experimental Designs Page 71

The problem, however, is that all of The remainder of this chapter will focus those options have possible limitations. on the different ways we might identify Some conditions or circumstances make valid comparison groups to accurately the target and the comparison group reproduce or mimic the counterfactual. inherently different. Sometimes we can see Identifying such comparison groups is the those differences. For example, in selecting crux of any impact evaluation, no matter a “sister” community, it may be that the what type of program we are evaluating. residences in that town are older and tend Simply put, without a valid estimate of to have a poorer overall security design. the counterfactual, we cannot establish That might be an obvious difference, even the impact of a program with any degree to a casual observer. Often, however, the of certainty. differences are not obvious.

❖

Comparisons With Targets (The One-shot Test)

One of the simplest designs we have is to an action, the question becomes whether compare our population of interest with a we have met the target or goal. If we particular goal or standard. Often, policy achieve the target, we have reason to guidelines are based on legislated standards believe that the action (which is generally or targets set from studies of best practices. a policy or program implementation) has Targets can vary according to the context. been successful. Of course, we will use a For example, a community might target a statistical procedure to help us determine 20 per cent reduction in trafﬁ c accident whether we are close enough to the target incidents over a ﬁ ve-year period. A parts to be equal to the target. manufacturer may implement a six-sigma regime, where one expects that fewer The methodological literature sometimes than 3.4 defective parts per million will calls this approach the one-shot test. That be manufactured. Human resource policy is, an action, policy or program is carried may also dictate that organizations should out, compared with a standard and, if it strive to hire a certain percentage of meets the standard, we generally assume individuals belonging to minority groups. the action was successful. The evidence might seem reasonably convincing. The key, then, is to compare our population Unfortunately, one-shot tests have of interest with a target that is theoretically their limitations. We can see one major doable or achievable. Once we implement limitation in the following example. Page 72

The one-shot test does not account for alternate explanations for a result.

Example: One-shot Test reinforce locks and security mechanisms on their doors, windows and other access Suppose a community has an overall points. reported crime rate of 3,000 per 100,000 population and wishes to reduce the rate In other words, by heightening awareness by 20 per cent over a three year period. of crime in the community, members The Police chief decides to initiate a took actions that would have reduced the Neighbourhood Watch program as it is seen likelihood of crime regardless of whether as a cost-effective strategy that centers on they had participated in the program or citizen involvement in ﬁ ghting crime. Three not. The point here is not to argue that years later the crime rate has dropped to the Neighbourhood Watch program does about 2,400 per 100,000 which is equivalent not work in reducing crime. The point is to the desired 20 per cent decrease. Can we that there may be alternate or coincidental infer that the watch program is behind the explanations as to why the target was reduction in crime? It would seem so and met. How much credibility those alternate the evidence seems compelling. explanations have depends on different factors. First, does it make sense logically In fact, an alternate explanation might that the alternate explanations might hold? exist. The campaign generated substantial If previous publicity campaigns resulted in publicity in the local press. Police and no noticeable impact then we might wish volunteers went door-to-door distributing to stick with the Neighbourhood Watch pamphlets. Online campaigns were launched program as an effective mechanism. On the advising home and business owners how other hand, if publicity campaigns in other to safeguard their residences and buildings communities had resulted in substantial with simple, cost-effective measures. The drops in crime rates, we might be more rise in the use of cell-phone technology supportive of the alternative explanation. and social-media made the citizens feel A further explanation might be that crime more secure as they are able to contact the rates were declining overall for a variety of police more quickly when they see a crime reasons, such as a change in demographics, occurring or when they witness something or an increase in community support suspicious. Because of the publicity, people centres, overall heightened awareness, in the community became more aware of decreases in drug addiction rates, and so on. the need to be more vigilant and made Consequently, the crime rate would have changes in their residences and dwellings. declined regardless. Some installed closed-circuit televisions,

The Right Decision: Evidence-based Decision Making for Police Service Professionals Experimental Designs Page 73

Before-and-after Designs

A variation on the one-shot or target The difference is simply due to the design is the before-and-after design. increased vigilance of the police, who are Again, we have a group or community targeting younger drivers in an attempt of interest where we are looking to make to enforce the new legislation. It is likely an impact. We measure the situation similar police vigilance without the change beforehand, apply some intervention in legislation would have produced similar and then look at the outcome later. The results. That is, the important factor is assumption here is that any difference not the legislation, but simply enhanced between the after and before results is surveillance by the police that serves to due to the impact of the intervention. act as a general deterrent to young drivers. Unlike the one-shot design where the comparison is a policy goal or target, the Looking Past the Limitations implicit comparison in this design is the after results with the before baseline. The limitations of these designs do not mean the evidence collected is irrelevant. The before-and-after design shares most We would have good reason to believe the of the strengths and weaknesses of the results if we impose these interventions one-shot design. Speciﬁ cally, we can never in many communities and under different be sure if it is the intervention that had circumstances with similar outcomes. an impact or simply some coincidental Also, carrying out an intervention and effect. For example, a jurisdiction might then revoking it can tell us a lot. If want to reduce the automobile accident the intervention results in the desired rate among young drivers. The way they outcome and the revocation results in a decide to do this is by dropping the legal return to the original baseline, then we Blood Alcohol Concentration limit from have a more powerful argument that the .08 to .05 for drivers under the age of 25. intervention is the causal factor. What Examining the data from the three years we need to remember is that evidence is before the introduction of the legislation rarely absolute. It has varying degrees of with the data from three years after, an reliability or credibility associated with evaluator notices that accident rates have it. Just as in the courts, some forms of indeed dropped for younger drivers. evidence are more credible than others.

Again, we might consider the change in Given the inherent weaknesses of these legislation to be the precipitating factor. designs, we might ask what approaches On the other hand, it is possible that rates we can take to address the problem. So of drinking and BAC levels among young far, the gold standard among evaluators drivers have not changed. and scientists is what we term the classical experimental design. Page 74

The Classical Design

A rule of thumb in science is that nothing The key to the strength of this design is to is perfect and certainty is an elusive goal. ensure the comparison group is equivalent On the other hand, a lack of certainty in to the experimental or treatment group one’s death is rarely a reason for playing from the outset. This harkens back to our Russian roulette. Similarly, a one per cent earlier discussion of the counterfactual risk that one will lose all of one’s assets where, ideally, we would like to see the in the stock market generally results in a same elements exposed to the treatment different form of investment behaviour and not exposed simultaneously. This than if the risk is above 80 per cent. situation is physically impossible. So, if we do not have perfection, what However, we can ensure that both the is the current ideal or gold standard for treatment and comparison groups are experimental designs? initially as alike as possible. How do we do this? To date, evaluators and scientists have relied on the two-group, before-and-after One way is to take pairs of identical people design to provide the most valid and the (or communities or what have you), and most reliable evidence. We start with the divide them into two groups. However, before-and-after design mentioned above. unless the pairs are exact clones, we can We then complement it with a comparison never be certain that they are identical on or control group that serves as the all relevant characteristics. Fortunately, counterfactual. In other words, we have while we can rarely work with clones or one group exposed to a treatment and one identical matches, we can divide subjects group that is not. If the group exposed to into two statistically equivalent groups. the treatment exhibits a signiﬁ cant change As we have noted previously, statistically and the comparison group does not, then equivalent does not mean truly identical, we have very strong reasons for believing but it does mean that, on average, no the intervention had an impact. statistically signiﬁ cant difference exists between the two groups. In other words, for all practical purposes, they are close enough to being identical. The key to the strength of classical design experiments is to ensure The method for ensuring statistical the comparison (control) group is equivalence is to take an initial group and randomly assign the members to the equivalent to the experimental group. treatment and the comparison groups.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Experimental Designs Page 75

By random assignment, we mean using If crime rates are lower in residences something like a coin fl ip (with a fair where the residents have installed burglar coin) or a random number generator to alarms, it may not be that most or all of the make the assignment. With a large enough difference in the lower crime rates is due initial group, the resulting two sub groups to the burglar alarms. It is quite possible will be statistically equivalent. That is to that people who install burglar alarms say, any signifi cant differences among are more conscientious then people who individuals across the groups will cancel chose not to do so. In other words, those themselves out. To a point, the larger the who installed alarms are also the same initial group, the more equivalent the two people who have taken care to install high sub groups will appear. quality locks or window bars, and are active volunteers in the Neighbourhood Any systematic factors that might affect Watch. the outcome (beyond the intervention) will be distributed across the two groups. Usually, any situation where people or Thus the two sub groups will be the same subjects volunteer or select into the on all relevant characteristics, except that treatment group should be considered one is exposed to the intervention or suspect. Subjects often volunteer for a treatment and the other is not. program because they are more motivated or see the treatment as potentially more Avoiding Sample Selection Bias benefi cial. Sample selection bias can only be addressed if the evaluator or The key to having a strong classical researcher has done a random assignment design is for the researcher to conduct the to the conditions. Having said this, it is random assignment to the experimental imperative that the researcher engages and comparison or control groups. in true random assignment. It is not Situations where we have not randomly unknown for some researchers to assigned subjects to treatment and select those they think will be the most comparison groups have the potential cooperative or most likely to succeed to for what we call sample selection bias. be in the treatment as opposed to the What this means is that the treatment comparison group. and comparison groups might differ on a relevant factor. For example, we might conduct a study of residences that have burglar alarms with those that do not. Page 76

Less Than Ideal Variations

Sometimes we cannot randomly assign The assumption here, of course, is that members of a group to policy or these jurisdictions have different response program intervention and others to the standards but have similar geographical control. A situation where this often and socio-demographic characteristics to arises is when governments decide to British Columbia. legislate policy. By their nature, social policies are implemented throughout a We call this approach the matched jurisdiction and not randomly assigned comparisons procedure. We attempt to within particular areas. What happens, fi nd matching jurisdictions that are as for example, if the province of British similar as possible to the experimental Columbia wishes to introduce a new set one(s) to provide a relevant control group. of response standards? Obviously, we can Again, the issue of sample selection bias apply the before-and-after model, but we might arise, since there is likely something know that has limitations. Are there ways different about jurisdictions that decide of using the framework of the classical to implement a policy over those that do design to overcome those limitations? not. Just as with the simple before-and- ¬after approach, we need to regard these Matched Comparison Designs results with greater suspicion than those obtained from the gold standard of the The answer is, some approaches are less classical design. ideal than the classical model but perhaps more convincing than simply using the Regardless, matched comparison designs before-after approach. Since we have no have produced convincing evidence that ability to randomly assign jurisdictions certain practices are effective. Perhaps to different response standards, one one of the best examples is the early approach is to fi nd potential clones. That research into the use of daytime running is, jurisdictions with different standards lights on automobiles for reducing traffi c that we know (or, more likely assume) to accidents. On the fl ip side, matched be similar in all or most relevant aspects. comparison studies have also suggested For British Columbia, we might consider that some policies do not have the choosing Washington and Oregon intended impact. A good example here is States, and the Province of Alberta as the research into the relationship between comparators. capital punishment and homicide rates.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Experimental Designs Page 77

The preponderance of the cross- jurisdictional evidence suggests that Too often, we rituatlistically engage while capital punishment may assuage our feelings for revenge, it does little to in the same behaviour even when reduce actual amount of homicide. the evidence shows it doesn’t generate the outcome we wish. We need to make a decision and the stronger the evidence, the more likely the decision is the correct one. We could be wrong, but even wrong decisions help Too often, we ritualistically engage in the us know what does not work. Doing same behaviour even when the evidence the same thing over and over makes no shows it doesn’t generate the outcome we sense if the results do not change. When wish. For centuries, physicians engaged it becomes obvious that our current in bloodletting because, despite the practices do not have the desired impact, evidence, it seemed to make “common logic suggests we should try something sense” at the time. The fact that many different. Eventually, we are likely to ﬁ nd patients were unnecessarily weakened by something that does work. An important the practice and subsequently died, was factor is that we must be willing to change not a consideration. our view when faced with contrary evidence.

❖

The Essentials

The important point behind this In other words, everything needs a discussion is that how evidence is comparator for us to be able to make collected—the framework or design used sense of it. An intervention or an action to generate the data—is an important only makes sense in comparison with element in helping us determine how another action or a non-action (doing credible the evidence might be. Among nothing). That comparator is known as the key factors is our prior notion that the counterfactual. nothing has any meaning unless it is in comparison with something else. Page 78

Since something cannot be in two Evidence generated by these approaches different situations at once, we must look should always be considered suspect but, for the most appropriate comparison. As if the approach appears sound and there we have seen, clones are hard to come are few logical alternative explanations for by, so the best approach we have devised the effect, then we are generally willing to to date is the randomized experiment give the evidence reasonable weight until where subjects or objects of interest are we ﬁ nd something superior. randomly assigned to a treatment group and an appropriate comparison or control Even with the best designed experiments, group. The randomization process helps however, the results are not always equally ensure that there will be no systematic credible. The design is one element we sample selection bias. consider; the magnitude of the impact or size of the effect being produced is In some cases, random allocation to another factor. Obviously, interventions treatment and comparison group is not that produce large effects provide better possible, so we try to create situations that reasons for using the evidence for a come as close to that ideal as possible. decision than small or marginal effects. But that leads us to other considerations such as cost-beneﬁ t or cost-effectiveness analyses—topics of our next chapter.

❖ Notes

1. Shane, J.M. “Organizational stressors and police performance” Journal of Criminal Justice (2010) p. 2. doi:10.1016/j.jcrimjus.2010.05.008

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 79

Costing Analysis

Basic Concepts

Costing analysis comes in one of two variations. The fi rst instance deals with Costing studies allow us to identify the total the costs associated with doing something. For example, the decision to purchase a cost of a decision and the associated benefi ts. vehicle involves not only the capital cost of that vehicle, but also maintenance such as the cost of repairs, consumables In fact, it is a straightforward cost situation. such as gasoline, and support costs On the other hand, if the training relates such as insurance. Depending on the to offi cers’ health and safety matters, the circumstances, additional support costs returns may appear in lower accident and may arise, such as those associated with injury rates, fewer sick days, lower insurance having to build a new garage or rent a rates, more effi cient or productive parking space. If we are looking at the true employees and higher employee morale. cost of ownership, we should also factor We can weigh the relative value of those the depreciation of the vehicles (hopefully, benefi ts against the cost associated with we will recuperate some capital cost when the training sessions to estimate the relative we sell it in a few years) plus the interest return on investment. of the funds used to purchase the vehicle. A fundamental idea of economics is the The other form of costing analysis is what notion of opportunity cost. Assuming you we term a cost-benefi t or cost-eff ectiveness have a limited budget, deciding to do analysis. In this instance, we weigh the costs one thing necessarily precludes another. associated with the decision with the value For example, given a department’s capital of the expected benefi ts. For example, budget, the decision is made to purchase a department might choose to invest in a pick-up truck. By making that choice, further training. The question then arises: the alternatives—an SUV, a sedan, a What is the return on that investment? If motorcycle, and so on—are foregone. the training relates to how to fi ght online That is to say, the opportunity to select crime in a community where the internet an alternative is no longer available. Not does not exist, the return on investment only is the physical choice of the next best might be considered zero. alternative not available, we give up the benefi ts associated with that choice. Page 80

Costing studies help us to identify the Steps to Consider total cost of a decision and what the 1. Identifying the component in the returns or benefi ts associated with that department’s operating or strategic decision might be. Furthermore, we plan to which the question or analysis can also examine what we consider the relates. expected cost and returns associated with 2. Setting out the objectives that we the second or third best choices, and intend the decision to achieve. compare those to our preferred decision. 3. Identifying the options or choices Sometimes this exercise results in our that are available. seeing a “lesser” alternative as superior to 4. Conducting a fi nancial (cost-benefi t our initial preference. or cost-effectiveness) analysis of the option selected or the options under Monetary costs are not, nor should they consideration. be, the only factors that we consider when 5. Preparing an accounting statement we make a choice. Political and other summarizing the results. social considerations infl uence how we make choices. However, monetary costs These steps may appear to be a restatement are important and are typically easy to of what we have mentioned previously. quantify. Most products and services have This is the case. However, we need to see a monetary or market cost associated effective evidence-based decision making with them. Also, social and political costs as part of a broad framework that starts are often closely linked to economic with a consideration of what we are doing decisions. As with formally assessing and why, what are the alternatives, and monetary costs, using the general costing what evidence can we bring to bear to framework to assess the impacts of non help us make a decision. Unless we know monetary decisions is also possible. The what we are doing and why, it is almost only difference is that in those situations, impossible to identify the appropriate the costs and returns are often more information. Without knowing that, we diffi cult to quantify. Regardless, decision may collect much data but we likely won’t makers can and do use qualitative data be collecting much evidence. to weigh the impact of those types of decisions.

No matter whether we do a straight costing analysis, cost-effectiveness or cost-beneﬁ t analysis, there are ﬁ ve overall steps to consider.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 81

Cost Analysis

Straight costing studies involve estimating the total life cycle cost of a particular piece of equipment or service. By life The key to conducting accurate cycle, we are referring to the period during cost analyses is to ensure that we which we use the product or service. For include all of the appropriate costs. example, a motor vehicle might have an actual average life expectancy of about 12 years before it is ready for the scrapyard. A person or an organization might decide to buy a vehicle, keep it for ﬁ ve years That additional vehicle may require extra and then sell it. In that instance, for the personnel, the cost of whom we need to owner, the vehicle’s life cycle is ﬁ ve years. factor into the analysis.

The key to conducting accurate cost Some of you may wonder why we have analyses is to ensure that we include all just included depreciation in our list of of the appropriate costs. Generally, for items instead of the initial capital cost. equipment or capital goods, these fall into Here the assumption is that the piece of the following categories: equipment will be sold at the end of the life cycle. Consequently, the capital cost • depreciation, component here is the difference between • interest on capital, the purchase price and the selling price. • maintenance fees (consumables and This is what we call depreciation. repairs), • licensing or regulatory costs, and Different pieces of equipment depreciate • operator costs. at different rates, but it is common for that to be about 20-30 per cent per While analysts will often exclude operator year. We calculate depreciation on the costs from the analysis, those need to be outstanding value, so a $10,000 piece of considered, even if the fi nal decision is to equipment that depreciates at a rate of 20 exclude them. If the equipment is meant per cent would be worth $8,000 after the as a replacement component, then the fi rst year. The second year’s depreciation operator costs would carry over from the would be $8,000 x .2, or $1,600. Thus, the previous piece of equipment. However, total depreciation after two years would suppose a police department has decided be $2,000 + $1,600, or $3,600, and the to purchase new fl eet of police cars or to residual value of the equipment would be include a tactical vehicle in its inventory. $10,000-$3,600, or $6,400. Page 82

One item often forgotten in costing Often, suppliers will give cost studies is the interest on the purchase. comparisons with competitors’ products. Interest rates are sometimes called Beware, however, that those analyses discount rates in the literature. The need often selectively include or exclude to consider interest is generally obvious “inconvenient” line items. Make sure that when one borrows the money to make you are comparing the proverbial apples the purchase, since the bank or fi nancing with apples. Where you fi nd missing company will include that charge. items, make sure to ask for supplemental However, even where the equipment is information. purchased outright, we should include the “rental” value of the capital. The Many independent agencies also conduct reason for this is that if we had not made costing analyses of various items. Look the purchase, we could have invested the especially to professional or trade money for a given return or used it for associations. Similarly, governments and some other purpose. This, in effect, is public organizations will often make their another form of opportunity cost. budgets and costing studies available. Much of that can be found online or in Obtaining Reliable Cost Estimates a local library. Sometimes a simple phone call can result in a gold mine of data. Whether it is the total cost of hiring someone or purchasing a piece of An example of a straight costing study equipment, the key to good costing is presented in the box on the next studies is to ensure we include all items, page. Here, we are looking at the cost of and obtain the most accurate and reliable owning and operating a typical, full-size cost estimates of those items. Because pick-up truck over a fi ve-year period. The all organizations work in different cost of the operator is not included in environments, typically we gain the best this example. information from experience. Looking back over your organization’s fi nancial Straight costing studies are done to records can be revealing. Because they estimate life cycle costs to decide the refl ect actual experiences, it is easy to see affordability of a purchase. They are also where unexpected costs (and savings) useful in comparing different products. arose. Do not write those off as unique For example, one brand of pick-up or one-time occurrences; put those in as might have a higher capital cost but lower line items in your analysis. maintenance costs than another. The question then becomes: Which is the Where drawing on institutional experience better choice? is not possible, one can often obtain information from other sources.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 83

Similar analyses can be used to decide The Cost of Purchasing a New whether it is less costly overall to purchase a used vehicle as opposed to new, or to Pickup Truck for Personal Use lease as opposed to purchasing outright. Obviously, for these different scenarios, Three-year cost of purchasing we must make different assumptions and operating a pickup truck: regarding expected life cycle, operating costs and depreciation. It might also be Item Cost ($) worth repeating that the values used in Purchase price 23,500 costing studies are generally estimates. As Selling price 9,500 we discuss in the chapter on statistics, all Depreciation 18,577 values are estimates. The key, with a little Financing 3,387 research and experience, is to minimize Fuel 10,079 the error. However, many expected items, Insurance 3,471 such as the selling price of the vehicle and Taxes and 3,650 the actual cost of operation, are based on licensing fees assumptions that are out of one’s control. Maintenance 2,069 We have considered the cost of capital Repairs 821 goods but we can conduct similar analyses Total cost 42,054 for personnel. The same general principles apply. Typically, we focus on a person’s Cost per 0.47 salary when deciding to hire someone, kilometre but ancillary costs can be substantial. When pensions, taxes, insurance, beneﬁ ts and other compensation-related issues Assumptions: are considered, it is common for those to • 20,000 km driven per year add an additional 15-30 per cent to the • 2.7 per cent APR ﬁ nancing total salary cost. This is above the cost cost with $2,750.30 down of training and maintaining the person. payment Maintenance costs include the person’s • gas $1.25/l. working space and any equipment and • mileage at 10.46 l/100 km. supplies they may need to do their job. In the previous example, we noted that equipment typically needs an operator. So, too, do people often need equipment to do their jobs. Page 84

A Note on Cost-effectiveness

In the previous analysis, our attention was on the total cost of owning and operating A key element in cost- a vehicle over its life cycle. Knowing the effectiveness analyses is being total cost of something is an important consideration in decision making. Often, able to identify the appropriate however, knowing the total cost does not output measures and being able tell us the whole story. Most equipment to measure them appropriately. or other items generate some form of output or product. For a car, the output is transportation. In that instance, knowing The same applies to personnel costs. the cost per kilometre is often a more Higher salaries to people who are more valuable piece of information than the productive, who are less likely to miss total cost. work and who provide a better quality of service can outweigh “savings” accrued In the example provided in accompanying by outsourcing to lower-cost jurisdictions. box, the expected cost of the car per What is important is how many items are kilometer is about $0.47. We term the produced, how many people are served, price or cost of something per unit of and the quality of that output or service. output as its cost-effectiveness. While cost-effectiveness is clearly related A key element in cost-effectiveness to total cost, we should treat it as an analyses, however, is being able to identify independent issue for decision making. the appropriate output measures and Often, differences in total costs might being able to measure them appropriately. be irrelevant. It is the per-unit cost that Again, this is where examining the is important. One reason unit costs differ organization’s operating or strategic from total costs is the fact that total costs plans becomes important. It is in those consist of two components: fi xed or documents that the organization’s sunk costs, and variable costs. Fixed costs objectives and operational purpose are associated with the one-time cost of should be outlined. Either directly or purchase. Variable costs generally relate indirectly, an organization’s effectiveness to operating and maintenance costs. A is related to the product or service it is piece of equipment may have a higher meant to deliver. fi xed cost but, if it is more effi cient than a lower priced piece, it will generally have lower unit costs.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 85

Cost-beneﬁ t Analysis

Cost-benefi t analyses are generally Moreover, for ideological reasons, many extensions of simple cost-effectiveness people refuse to assume a monetary value studies. A primary difference is that cost- on public goods. A common refrain, for benefi t analyses look at a broader range example, is that “You can’t put a price on of returns on the investment. Most cost- the environment” or “You can’t put a price benefi t analyses include effects (benefi ts) on a human life.” The fact is, we do both. that are not easily quantifi able or outcomes The problem is that no independent or that have a broader social impact. indifferent market exists to set those prices. Regardless, this is an essential weakness of Cost-benefi t analysis is grounded in welfare cost-benefi t as opposed to straight costing economics. It differs from most branches analyses. of economics since the focus in not just on decisions of consumers and fi rms, but on Revealed and Stated Preferences public decisions that affect the economic interests of a broader community. While the philosophical issue of whether Consequently, cost-benefi t analyses often you can truly value a human life may focus on issues such as quality of life or not be answerable, welfare economists quality of the environment. A fundamental have two broad tools at their disposal. challenge for those doing cost-benefi t They term one approach the revealed analyses is how to measure the benefi ts so preference method. Revealed preferences they are comparable across issues. Among relate to how people actually behave when commodities, apples are not electrical confronted by a qualitative phenomenon. transformers. However, a market for both For example, comparing a particular piece exists and it is possible to place a monetary of real estate with similar ones could value on both. Currency is a common reveal the “eyesore value” of having a exchange unit that allows the producers fi re hydrant on a front lawn. How much of apples to purchase transformers even parents value education for their children when the producers of transformers have might be suggested by what proportion no interest in exchanging their product for of their income they are willing to spend apples. on a child’s tuition.

The difﬁ culty with many public goods The second tool in the economist’s and services is that there is no open repertoire is what we call stated marketplace in which the monetary value preferences. Stated preferences are just of those items is established. that: what someone is willing to tell you they would pay for something. Page 86

We may judge people’s value of environmental elements, for example, We can use cost-benefi t by how much of a tax increase they are analysis to: willing to support for clean air or nature conservatory initiatives. Typically, stated • decide whether a proposed preferences are determined through project should be undertaken surveys and similar procedures. • decide whether an existing While both stated and revealed preferences project should be continued have their merits, both have their • choose between alternative limitations. Using how much life insurance projects a person has to assess how much they value their lives might appear like an excellent revealed preference. However, how much they can buy is limited by how much The purpose is for “departments and insurance they can afford. Furthermore, agencies [to] assess regulatory and non- a person may value their life highly but regulatory options to maximize net not be willing to see relatives “benefi t” benefi ts to society as a whole. Hence, all from their death since life insurance goes regulatory departments and agencies are to the benefi ciary and not the insured. expected to show that the recommended Stated preferences on various aspects have option maximizes the net economic, been studied extensively by sociologists environmental, and social benefi ts to for the past century. Their overwhelming Canadians, business, and government conclusion is that what people say and over time more than any other type of what they do varies considerably. regulatory or non regulatory action.”1

Still, cost-benefi t analysis is one of the In summary, we can use cost-benefi t few techniques we have to assess the analysis in various ways. For example, to: broader impact of various policies and programs. It helps us to clarify the issues, • decide whether a proposed project or identify the constituent components, and program should be undertaken; bring some evidence to bear on the issue. • decide whether an existing project or It has gained general acceptance in the program should be continued; or, public sector and is mandatory in many • choose between alternative projects government shops. For example, the or programs. Treasury Board of Canada has mandated that any regulatory framework put in place by the federal government must be based on a cost-benefi t analysis.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 87

Components of a Cost-beneﬁ t Analysis

In setting up and executing a cost-benefi t 4. List the benefi ts. analysis, several steps need to be followed. For the alternatives outlined, what These include: is the return on investment? Is there a monetary return or an increase 1. Defi ne the problem. in productivity or effectiveness? Again, this is a statement of the issue Perhaps, the matter is not one of with a link back to your operational generating further revenues, but one or strategic plan. of reducing or avoiding costs. Are there health, safety or environmental 2. Identify any constraints or benefi ts to be gained? The issue limiting factors. might be related to overall quality of life. Are there savings to be had in This is a discussion of what equipment, time or personnel? administrative requirements and other challenges you might face. 5. How are the costs and benefi ts These include a listing of fi nancial limitations, managerial or personnel to be quantifi ed? challenges, environmental and other Clearly, market or monetary values regulations, and any other factors or of goods and services are the easiest “hurdles” you might need to address. with which to work. We have already outlined the challenge of providing 3. List the alternatives. market values. Still, fi nding a shadow or proxy price for a given cost or benefi t Every initiative has alternatives, may be possible. Social scientists have including doing nothing or staying developed ways to estimate the value the course. For example, if the issue of a human life. The cost associated is whether to close a particular police with noise levels or high traffi c station or not, it may be informative to volume in a community, for example, looking at amalgamating with another can be estimated by differences in unit, sharing space with other services housing values between noisy and such as fi re services or ambulance quiet communities or between those services, or expanding the operation with high and low traffi c volumes. to incorporate other stations. Page 88

Often, we can fi nd ways of assessing the This is the principle behind reverse value of tough-to-monetize issues by mortgages. A bank or fi nancial institution searching the appropriate literature. We will give you a fraction of your home’s have already discussed techniques for value today if you allow them to sell it conducting more focused online searches. at market value and keep the proceeds Using the expertise of economists and several years hence. This is the opposite of other social scientists in local colleges and the previous problem. In these instances, universities might also be possible. we call the interest rate the discount rate. At a three per cent discount rate, Once we have conducted these steps, we that future $2,000 endowment would be can put a report together summarizing worth $2,000 x 1/1.035 = $2000 x .863 = these elements and presenting the relative $1,725 today. costs and benefi ts. We term this current value on a future Net Present Value amount its net present value or NPV. The NPV is the opposite of the future value. As the saying goes, “A bird in the hand Since programs and capital goods have is worth two in the bush.” So it is with an expected life cycle, it is common to money. One reason we charge interest standardize costs to today’s value, that is, on borrowed money is that by giving the NPV. Another way of thinking about capital to a borrower, the lender faces an NPVs is to consider them as equivalent opportunity cost. That money cannot be to constant as opposed to real dollars used for anything else. To compensate the when we are trying to control prices for lender for the opportunity cost, borrowers infl ation. must pay interest. For example, when you buy a locked-in savings certifi cate with a In these examples, we have discussed fi ve-year redemption, you get back more what economists call the private time than you invested. A $1,000 certifi cate preference rate, since the focus is on an invested at 3 per cent would be worth individual. Within the public sphere, the $1,000 x 1.03 x 1.03 x 1.03 x 1.03 x 1.03 choice to invest public funds in a particular = $1,000 x 1.035 = $1,159. program often precludes investments in other programs of benefi t to the public. We can also consider the opposite. What Within the public or welfare sphere, would an endowment of $2,000 that you economists generally call the deferred are to receive in fi ve years be worth to you value the social opportunity cost. While today? In other words, what would you the terminology differs, the underlying be willing to pay for the benefi t of having principles are similar. the cash right now?

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 89

Benefi t-Cost Ratios of topics on which many people hold strong views. Firm opinions are held over For programs extended over time, we whether helicopters cost too much, make need to amortize both cost and benefi ts. too much noise, invade privacy or are Occasionally, the duration of the costs simply unnecessary. Is the money better may be different from the duration or life spent on more police offi cers, more expectancy of the benefi ts. Consequently, police cars or more police on bicycles? to make things comparable, analysts will Is there some rate of crime that warrants calculate the NPV of both costs and a “copper chopper” or is there some benefi ts. geographic size, size of population, or density of population that warrants, or We term the ratio of the benefi ts to not, the use of police helicopters? Is the costs as the benefi t-cost ratio or BCR. prime purpose of a police helicopter to Assuming the NPV of the benefi ts of deter crime? Is its purpose to increase a program is $13.5 million and the net operational effectiveness? Should police present value of the costs is $10 million, helicopters be on regular patrol or should the BCR would be: they be on stand-by, to be used only in (NPV Benefi ts) 13.5 certain situations or emergencies? Does BCR = = = 1.35 (NPV Costs) 10.0 a police helicopter save injuries and lives with respect to pursuits? Some believe Ideally, the BCR should be greater than that it does; others say that it does not one. Anything less assumes that the costs matter because police helicopters should outweigh the benefi ts and, all other things not exist and there should be no pursuits. being equal, the option should not be chosen. If we chose to evaluate several In 1999 the London Police commissioned alternatives, the one with the highest Th e London Police Service Helicopter Research BCR would normally be our choice. If a Study to evaluate whether: 1) helicopter program with a lesser benefi t-cost ratio is patrols have a suppression effect on the selected, then it is likely that we should incidence of various types of crime and have included the reason for that selection occurrences (residential break and enter, on the benefi t side of the ledger. commercial break and enter, auto theft, theft from auto, robbery, property damage, Example: Evaluation of the trespass by night, suspicious person and Effectiveness of a Police Helicopter2 suspicious vehicle); and 2) to evaluate whether a helicopter would increase the Whether communities should have police operational eff ectiveness and/or effi ciency helicopters continues to be a controversial of the police service. The London Police issue in a number of North American acquired and evaluated the use of a cities. The debate ranges over a wide array helicopter over a one-year period. Page 90

London Police Helicopter Cost Analysis Considerations3

Breakdown of Costs

Item Cost Helicopter Lease: $256,328.55 • Helicopter • Insurance • Pilot • Scheduled Maintenance • Unscheduled Maintenance • Component Reserve • Administration Spotlight Lease: $6,026.56 Additional Equipment: $11,155.67 • Avionics • Radio interface equipment • Siren/hailer • LCD mount • Mounts for technical equipment Cost of Installation of the Equipment: $16,775.00 Extra Maintenance Costs Incurred: $7,922.99 • Shop supplies • Expendables Cost of Fuel: $32,820.81 • Total Number of Litres Used = 43,168.92 $331,009.58 Cost of Operation or $331/hour

Notes: 1. The purchase price of the same helicopter was $268,000 (U.S.) or approximately $400,000 (Cdn.) in 1999. 2. Because the cost of fuel can vary so much across time and space the amount of fuel is provided to make it easier to make future comparisons. 3. Based on 1,000 hours of ﬂ ight time, but excludes the salaries of the Flight Ofﬁ cers (who are police constables) and the value of in kind and other contributions

Assumptions: Costs are based on the actual leasing and operational expenditures for the use of the helicopter for 1,000 hours over a one-year period. The benefi ts are measured in terms of effi ciency and effectiveness. Effi ciency is the monetised value of policing time that is saved by other police offi cers being cancelled from having to attend and by the amount of police downtime saved when the helicopter is involved. Effectiveness is measured by monetising the value of higher rates of apprehension when the helicopter is involved by taking into consideration the value of greater detective time to effect the balance of these apprehensions.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 91

Assuming that the average cost of a The table below shows that the constable is $50.84/hour and the average apparent level of efﬁ ciency over the 672 cost of a detective is $55.17/hour, the occurrences amounts to a savings of table below outlines the cost of efﬁ ciency 15,321 minutes or 255 hours. and effectiveness based on the total number of hours.

Table: Apparent Time Saved When Helicopter is Involved in Various Types of Occurrences

Helicopter patrols Service as a while Total Average Offi cer Total Average Offi cer Number of Apparent Type of Number of Downtime number of Downtime Offi cers Involved Helicopter Occurrence Occurrences (min.) occurrences (min.) (helicopter) Effi ciency (min.)* Assault 14 76.00 958 89.2 32 422 Weapons 37 69.11 367 104.5 198 7,007 Domestic IP 16 50.11 2,862 69.00 74 1,398 Missing Person 25 67.96 1,723 79.00 88 980 IP Noise IP 21 24.76 4,686 32.00 66 478 Disturbance 77 47.33 2,194 46.00 359 -477 Trouble with 85 44.82 8,5889 55.26 299 254 Person Drunk IP 12 41.37 1,800 41.00 35 -13 Check Welfare 16 72.08 4,422 61.00 58 -643 Assist Other 19 54.12 1,337 80.00 37 958 Agency Assist Fire 18 51.48 747 77.78 74 1,946 Department Break and Enter 35 52.20 540 59.22 207 1,453 (Residential) Theft 16 64.24 704 78.72 58 840 Property Damage 16 51.08 665 67.85 52 872 Prowler 26 32.17 1,028 42.01 75 738 Traffi c Motor Vehicle Hit and 22 64.03 2,208 64.65 67 42 Run Impaired Drivers 9 74.43 655 88.31 31 461 Suspicious 124 43.60 3,249 42.00 426 -682 Person Suspicious 29 50.68 789 42.00 105 -911 Vehicle Alarm IP 55 24.88 2,805 26.00 177 198 15,321 Total 672 2,518 =255 hrs * Average offi cer downtown (total police service) minus average offi cer downtime (helicopter) times number of police offi cers at the occurrences at which the helicopter attended. These are slight underestimates of effi ciency because the helicopter occurrences are also counted in the total police service occurrences. Page 92

Table: Operational Value: Effi ciency Value Plus Eff ectiveness Value Expressed in Hours and Dollars

Operational Effi ciency Eff ectiveness Value Type of Occurrence (“just occurred” or “in progress”) Hours C$ Hours C$ C$ Assault 7.03 357 80 4,414 4,771 Weapons 116.78 5,932 480 26,482 32,414 Domestic IP 23.30 1,184 1,184 Missing Person IP 16.33 830 280 15,448 16,278 Noise IP 8.50 432 432 Disturbance 1.25 64 64 Trouble with Person 14.38 731 731 Drunk IP 1.83 93 93 Check Welfare -10.72 -545 -545 Assist Other Agency 17.20 874 874 Assist Fire Department 36.38 1,848 1,848 Break and Enter 24.22 1,230 320 17,654 18,884 (Residential) Theft 16.93 860 160 8,827 9,687 Property Damage 22.30 1,133 1,133 Prowler 24.97 1,268 1,268 Traffi c 4.73 250 240 Impaired Drivers 7.68 390 390 Suspicious Persons -2.97 -151 -151 Suspicious Vehicles -11.68 -593 -593 Alarm IP 9.80 498 498 Other* 10.12 514 514 Total 338.36 17,189 1,320 72,823 90,014 * Occurrences for which offi cers were cancelled that are in addition to the 20 identifi ed in this table.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 93

The 338.36 hours of police constable Example: Evaluation of Assigned effi ciency have a value of $17,189 and Vehicles Versus Pool Vehicles3 the 1,320 hours of effectiveness have a detective value of $72,825. These A request was made regarding the cost combine to produce an approximate benefi t of having assigned vehicles versus value worth an estimated $90,014. The utilizing pool vehicles. The data provided average police offi cer downtime for is from the Manatee County Sheriff ’s each type of occurrence amounts to 543 Offi ce. The study provides a reasonable hours over the 672 occurrences on which approximation of the cost of an assigned the calculations are based. The value of vehicle program versus a pooled vehicle the remaining 457 hours (1,000 - 543) is concept. $23,334, which must be added to the cost of operation. The total cost, therefore, The table on the following page outlines the for the year of operation is $354,344 cost comparison between the two options. ($331,010 + $23,334). The benefi t as Besides the vehicle purchase cost, fuel, measured above is $90,014 or 25.4 per and maintenance, there are other inherent cent of the cost. costs, the most notable is that of a deputy’s lost time at the beginning and end of each Therefore, the benefi ts of the tangible shift for equipment change-out, vehicle features reported here amount to 25 per inspections, and equipment checks. The cent of these costs. It has further been average amount of time loss per shift is speculated that greater levels of effi ciency approximately 40 minutes per deputy. This and effectiveness are possible, to an equates to approximately 11 days per year in extent that the monetised benefi ts could lost patrol time. The analysis includes both, approximate $138,463, or 42 per cent of with and without this cost. the cost. These monetised benefi ts need to be considered in conjunction with the special and unique features of having a helicopter as part of a policing team. These include its speed, aerial perspective, ease and safety in conducting certain types of searches (e.g., roofs, railroad tracks and river banks) and the ability to ‘turn night into day’ by illuminating an area and contributing to visibility as well as citizen and offi cer safety. Page 94

Assigned Vehicle Program Pooled Car Costs Vehicle Life 5 Years Vehicle Life 1.8 Years Life Cycle Mileage 90,000 Life Cycle Mileage 90,000

Life Cycle Costs Life Cycle Costs Vehicle Cost $22,488 Vehicle Cost $22,488 Salvage $3,750 Salvage $6,063 Maintenance (Including Maintenance (Including $12,285 $21,297 Loaner Unit) Loaner Unit) Fuel Costs $19,731 Fuel Costs $19,688 Annual Lost Deputy $7,899 Time* Total Costs $50,754 Total Costs $65,309 Cost Per Mile $0.56 Cost Per Mile $0.73 Without Deputy Lost $0.64 Time Fuel and Maintenance Fuel and Maintenance $0.36 $0.46 Cost Per Mile Cost Per Mile * Based on average shift change time to load vehicle (25 min.) and unload vehicle (15 min.) and check/sign off on vehicle.

The costs tables show that a take-home other approved activity. Citizens see vehicle is the most cost effective solution a more visible police presence. So in regard to the fl eet. This is substantiated will potential offenders, offering a by the $0.56 per mile cost compared to a deterrent effect. pool vehicle of $0.64 per mile, a 14 per • They create a rapid response to cent savings. Including the real cost of emergency callouts. The offi cer loss productivity and the savings grow to doesn’t have to go to the district to 30 per cent. The initial implementation get a vehicle prior to responding. cost of additional vehicles is the biggest When there is a major event, rather obstacle for most agencies that have not than offi cers driving their personal implemented a take home or assigned cars to districts, where they would vehicle policy. gather equipment, and be briefed, they leave directly from home Additional advantages to take-home police to handle an incident. Criminal units were also identifi ed. Specifi cally: Investigators would have to come to the department to pick up a unit • Take-home programs result in more and crime scene equipment before police cars being seen on the streets responding to major crime scenes, as offi cers go to and from work, or losing potentially valuable time.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Costing Analysis Page 95

• Take-home cars allow for patrol • Take-home vehicles increase the offi cers, criminal investigators, back-up potential for offi cers on duty. and command staff to respond In critical situations, while the nearest where needed, when needed. In on-duty unit may be in another zone, situations such as a major disaster, an off-duty offi cer may be just around the department could activate all the corner from an emergency call for sworn offi cers in a relatively short service, or another offi cer needing time, and send offi cers directly to the assistance. assignment upon notifi cation. • Offi cers take ownership and treat • Take-home vehicles increase the level vehicles as their own, keeping them of enforcement. Offi cers driving clean. take-home cars off duty that observe • Offi cer morale is considerably higher serious violations and criminal activity with assigned vehicles. are obligated to take enforcement action.

❖

Summary

While costing studies are but one way or convention-based costs omit many of generating data for evidence-based ancillary costs associated with our decision making, they are often one of the activities. For example, it is common for more commonly used tools. Essentially, costing studies to omit interest payments costing studies do three things for us. or costs associated with the need for First, when done properly, they link the extra personnel. By focusing on a detailed outcomes we wish to measure with the analysis, we are more likely to ensure that goals and objectives of our operational we include those items. Furthermore, and strategic plans. They essentially help exhibiting the results of a costing analysis us focus on the question about whether to colleagues and others provides the the activity is within the organization’s opportunity for independent observers mandate. to identify potentially missed items.

Second, costing studies help us to focus Third, costing studies provide a on the many line items that make up actual transparent and fairly mechanical way of costs. Often, “back of the envelope” helping us decide on options. Page 96

The assessments are relatively objective Even if someone can put forward and focused. The assumptions underlying alternate evidence, a net beneﬁ t still the costs can be scrutinized, as can the exists since that evidence will contribute values associated with individual items. to a more accurate assessment of the The transparency of the process provides situation. In the end, a better basis for a for a more defensible decision: one that decision is put forward. is replicable by an independent observer. Furthermore, unlike purely value-based decisions, decisions based on evidence force critics to generate alternate values or analyses to validly criticize the analysis presented.

❖

Notes

1. Treasury Board of Canada (2007) Canadian Cost-Benefi t Analysis Guide: Regulatory Proposals. Ottawa: Government of Canada. Catalogue No. BT58-5/2007. http://www.tbs-sct.gc.ca/rtrap-parfa/analys/ analys-eng.pdf 2. Taken from: Whitehead, Paul C., “Operational Value of Police Helicopters: A Cost-Beneﬁ t Analysis,” International Journal of Police Science and Management. (2002) Vol 4, p. 233 3. Taken from: Manatee County Sheriff ’s Ofﬁ ce Memorandum, “re: Assigned Vehicle Costs”, November 30, 2007

The Right Decision: Evidence-based Decision Making for Police Service Professionals Page 97

Making Decisions

Using Evidence

We make decisions all the time in our private and in professional lives. Mostly, Evidence-based decision making makes the those decisions are based on what we process transparent—it is no longer a closed, learned in our training, on conventional wisdom, or on traditional practices. Often, magical process, but one where observers questioning common practice only leads can follow the logic and follow the evidence. to rediscovering the wheel. Yet, there are many circumstances where traditional practice and common knowledge does Historically, we can forgive decision not work. We may not achieve the results makers for pursuing timeworn rituals. we want, or our practices lead to less- After all, as rainmakers knew, if you than¬-effi cient outcomes. For some danced often enough, it would eventually reason, however, humans are reluctant to rain. Modern weather forecasting has change. We are a conservative species. We become suffi ciently accurate; however, become comfortable doing the same thing that rainmaking is no longer a viable repeatedly, even when we are not happy profession. with the outcome. As the Alcoholics Anonymous Handbook states, however, The reason for that is meteorology “Insanity is doing the same thing, over and has accumulated suffi cient systematic over again, but expecting different results.” knowledge that it is possible to predict local temperatures, precipitation and other phenomena with a high degree of certainty. Meteorologists have accomplished this by turning to scientifi c research and other forms of systematic study.

The reliance on systematic study and data collection, which is what underlies science, has made inconsistent inroads in most other disciplines. Page 98

This is unfortunate since, today, there We are not suggesting that you can always is a large amount of empirical evidence ﬁ nd an optimal solution to your problem. to help us make better decisions. However, evidence-based decision making Furthermore, where existing analyses helps us to identify options and practices do not exist, conducting a local analysis that do not work. In those instances, you to improve our own decision making is are likely no worse off trying something often not that difﬁ cult. This doesn’t mean new. Most often, however, a review of that one needs to become a scientist— the existing evidence or the collection of far from it. All we need to do is to use your own data will help provide a more empirical results to be able to build a fruitful direction. reliable body of evidence. Everyone draws inferences from evidence. Decision making based on evidence Inferential reasoning is a basic human skill. will generally allow you to make better Thinking analytically is a skill like drawing decisions. Evidence-based decision and painting or operating a vehicle. It making has the advantage of making can be taught, it can be learned, and it the process transparent. Outsiders can can improve with practice. However, like become privy to the foundations of the many other skills such as karate, it needs decision. It is no longer a closed, magical to be hands-on and applied. This manual, process but one where observers can companion workbook and related case follow the logic and follow the evidence. studies will afford you that opportunity.

Evidence-based decision making is In summary, how can we put the lessons using the best available research and of this book together to formulate a good information on the outcomes of police evidence-based strategy for decision work to carry out guidelines and evaluate making? Essentially, there are four main agencies, units, and personnel. steps.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Making Decisions Page 99

Without the right question, no amount of data will help provide an answer.

Identify and Frame the Question

The fi rst three chapters of this book • Are there more cost-effective or cost- are focused on identifying appropriate effi cient alternatives? questions. Without the right question, • Does this action have long-term or no amount of data will help provide short-term consequences? an answer. We have stressed repeatedly • What other resources am I likely to that good questions need to be put need if we pursue this action? into an appropriate framework. Ideally, you should draw these from your If what you are proposing to do is new organizational plan or your strategic plan. or outside the traditional scope of your This helps to focus the issue on the key organization’s mandate, consider putting purpose and objective of your unit. One together a focused business plan to main reason many organizations fail is support or justify the activity. that they lose sight of their mandate. They try to be all things to all people. Once you have identifi ed and justifi ed This is simply not achievable. the appropriate question, outline the options. Commonly, two or three viable If you lack an organizational or strategic alternatives are available. In other plan, the next best thing is to drill into the situations, the range of options and their issue. Ask several fundamental questions: relative merits is not necessarily obvious. In those situations, consider performing • Why are we proposing to do this? an environmental scan or SWOT analysis. • What are the likely outcomes? If the issue is crucial, consulting an • How does this action relate to the outside facilitator may be worthwhile. organization’s mission? • What benefi ts will this action bring to my unit or the people we serve? Page 100

Gather the Evidence

Often the best source of evidence is your own organization. You keep records Remember, a librarian of calls for service and your ﬁ nancial can be your best friend. accounts. Those and other resources can give you valuable insights. Usually, internal data will provide a good base line or a measure of the status quo. Librarians can also help you navigate a wealth of statistical databases. Most Outside your organization, other sources of provinces and provincial agencies collect information are available. Professional and and make available regional data. While trade organizations are a good place to start. most data are available to the public, Suppliers will also give you information some is limited to authorized agencies. If on comparative options and estimates of you work for a public service agency, it is lifetime service costs. Do an online search. likely that yours is one of those authorized Despite all of the trash on the internet, there agencies. The Statistics Canada website is are also nuggets to be had. Learn how to use also a valuable source of information. your favourite search engine to eliminate as much of the irrelevant material as possible. Some colleges and universities have Do not be afraid to check organizations in laboratories and research groups or outside jurisdictions. In the UK, the Ofﬁ ce institutes that focus on crime-related for National Statistics is responsible for matters. Again, these can often be found keeping information on crime incidents. through an internet search or by asking a It also produces many annual reports and local librarian for help. studies. Similarly, the US Federal Bureau of Investigation is a gold mine of information, Do keep in mind, however, that not all as is the Bureau of Justice Statistics. evidence is of equal value. Do not be afraid to be critical, or contarian, especially if Other excellent sources of information claims are at odds with your department’s are libraries and your local college or or your colleagues’ experience. While not university. Libraries have access to online always the case, if something is too good databases that can search academic articles to be true, it generally is. Ask yourself if and other specialized material. Some of the source is trustworthy. Is the agency this can be intimidating to us if we are not presenting the data operating impartially or used to using the facilities. Remember, a at arms-length, or does it have a self-serving librarian can be your best friend. Contact agenda? Has the research or the publication your municipal librarian or visit a local gone through an external review process? college to seek expert advice.

The Right Decision: Evidence-based Decision Making for Police Service Professionals Making Decisions Page 101

Organize the Evidence

Once you gather it, put your evidence together in an organized manner. A key element in presenting Costing studies are easily presented in data is putting it into context. a spreadsheet. Other material can be presented in a table. Be sure to record the source of your information and keep track of where you found it. That Is a million dollars an appropriate price way, if someone questions its veracity, for a police tactical vehicle? Is it a cheap you can refer them to the source. insurance policy or a colossal waste of funds if rarely, or ever, used? Is our level A key element in presenting data is of training adequate? Will training putting it in context. Remember, nothing requirements change in the near future. If means anything unless it is relationship to so, how? something else. Ask yourself, “compared to what?” These questions can only be answered by making reference to a comparable Is a ten-minute average response time benchmark. What is the price range for adequate for an urban police department? goods and services in the marketplace? Can we drill down to priority calls to What are industry norms or standards extract more precision? You can be assured for performance? Are there best practices that Mayor and Council, the Police Board against which you can compare your unit and others in the local community will ask. or department? Page 102

The more you engage in evidence-based decision making, the easier it will become. Knowledge is cumulative.

Review the Decision-making Process

Once you have done your analysis, it Taking a request to city council with is good practice to review the entire strong external evidence is more likely to decision-making process. What have result in a positive decision. Presentations you learned? How could the process that show prior examples of success or be streamlined or made more efﬁ cient? that have reliable estimates of returns The more you engage in evidence- on investment are powerful. Finally, if based decision making, the easier it will someone challenges you, it is fair play to become. Knowledge is cumulative. You say that you have provided evidence to will soon determine the best sources of support your request. If they disagree, information. You will discover how to then ask them to show you their numbers. make the process more efﬁ cient and how to minimize the likelihood of getting sidetracked.

While evidence-based decision making generally takes longer than other approaches, it has its beneﬁ ts. Decisions based on hard evidence are more resilient in the face of scrutiny. We owe it to ourselves and the communities we serve to be more evidence based in our thinking and application.

The Right Decision: Evidence-based Decision Making for Police Service Professionals What Others are Saying about The Right Decision

From Chief Constable Bob Downie, MA, OOM, Saanich From Chief Constable Bob Rich, LLB, OOM, Abbotsford Police Department, British Columbia Police Department, British Columbia Th e Right Decision has been endorsed, and funded, by the What does it take to be a police leader? One important component Canadian Association of Chiefs of Police Research Foundation. is the ability to make a good decision. I realized at some point as We are proud to have worked with Len Garis and his team to promotions came along, that the more you moved up, the less make this manual, its companion workbook, and related case defi ned the job gets. On the street you stop an offender and you studies available to police organizations throughout Canada. We fi gure out if you have grounds to arrest him. When and how you acknowledge the fi ne work that was done to produce the original can do it that is all laid out for you. By the time you are helping version of this manual for Fire Professionals, and we are grateful to run the organization, even what kind of decisions to make can for the offer to reproduce the manual in a police context. be pretty unclear.

The need for research has never been as important to policing This is a manual that sets out the steps to make a rigorous and as it is today. With increased demands on fi nite resources, and defendable decision. Read this manual and you will know or be increased accountability for our decisions, our programs, and the reminded how to go through a step-by-step decision making services we provide, we must be able to explain why we are doing process that increases your chance of success both by being what we are doing the way we are doing it, and also to understand “more right” and increase buy-in from those around you. Seems that what we are doing is the right decision. To do this we need like taking some time to read Th e Right Decision is the best decision to be able to fi nd or produce reliable evidence on which to make you could make next! our decisions. The days of relying on intuition alone to make decisions have passed. From Theron L. Bowman, Ph.D., Deputy City Manager and former Police Chief, City of Arlington, Texas Th e Right Decision takes the mystery out of evidence-based Th e Right Decision takes the reader on an organized, methodical yet decision making and shows us both how important it is that we simple and measured approach to understanding decision-making fi nd and rely on evidence to make our decisions, and that there processes. Professor Paul Maxim, Fire Chief and Professor Len is nothing to fear in opening our organizations’ doors to research Garis, Professor Emeritus Darryl Plecas and legal analyst Mona efforts. There was a time when police were sometimes distrustful Davies guide the reader through simple yet essential discourses in of research approaches, as we considered our environment too what questions to ask and where and how to obtain data. The book unique to be quantifi ed and understood by others. Those days touches on creating both basic and strategic plans. It moves to a are also behind us. With Th e Right Decision you will be able to higher-level discussion of research methods, statistical analysis and understand the questions you should ask when seeking evidence, experimental design. The realistic police-related examples that are the processes that should be engaged in, and the resources provided within each topic discussion simplify the learning for even available to assist you in getting the evidence you can trust and the nonacademic consumer. Important terms are clearly defi ned. rely on. The readers who use this information as a foundation for This manual can be used by anyone in your organization. It sets understanding and applying evidence-based decision-making a framework for helping people to build a business case you can should quickly evolve a strong skill set that will help them perfect rely on to make informed decisions. It ties in the importance performing a myriad of police processes and achieve objective of linking initiatives with your strategic or business plans and outcomes. These processes include conducting initial and associated goals and objectives. This approach can be used at any specialized criminal and administrative investigations, eyewitness level of your organization and can help prepare staff for taking a identifi cation, interviews and interrogations and crime scene broader approach to their thinking. search. This book is versatile enough to appeal to the novice student of policing yet provide value to seasoned offi cers, The CACP Research Foundation has done a great deal of work investigators and analysts looking to meet the high evidence- identifying research priorities in Canadian Policing. There is an based standards of today’s polity. The mild dosage of economics opportunity arising from the signifi cant amount of work being including cost-benefi t analysis and opportunity costs motivates done across this country by police organizations, academics, and the city manager in me to make sure my police chief and other government at all levels in providing evidence-based research to department heads have their own personal copies of this book. assist us in making the right decisions. This manual will help us to exploit this opportunity to its maximum advantage, and in turn Th e Right Decision alone is a great entrée into the evidence- allow us to provide the highest quality of policing to the people based decision-making arena. When coupled with its companion we serve. workbook, it should be universally adopted as at least a minimum knowledge standard for all purveyors of justice. Making the Right Decision As a professional in the police service or other public service, you make crucial decisions every day that balance need with available resources. How should you approach these decisions, and how can you justify the decisions you make?

In this manual, Professor Paul Maxim, Fire Chief and Professor Len Garis, Professor Emeritus Darryl Plecas and legal analyst Mona Davies explore the what, why and how of evidence-based decision making.

What Others Are Saying About The Right Decision Please see the inside back cover for full versions.

I recommend The Right Decision as a good reference tool for anyone who is in the business of making decisions. The material is put together in an easy-to-follow format and can serve as an aide memoire for a number of different aspects of the decision-making process. – Deputy Commissioner Craig J. Callens, OOM, Commanding Offi cer, RCMP “E” Division

The Right Decision takes the mystery out of evidence-based decision making and shows us both how important it is that we ﬁ nd and rely on evidence to make our decisions, and that there is nothing to fear in opening our organizations’ doors to research efforts. – Chief Constable Bob Downie, MA, OOM, Saanich Police Department, British Columbia

Read this manual and you will know or be reminded how to go through a step-by-step decision- making process that increases your chance of success both by being “more right” and increase buy-in from those around you. Seems like taking some time to read The Right Decision is the best decision you could make next! – Chief Constable Bob Rich, LLB, OOM, Abbotsford Police Department, British Columbia

The readers who use this information as a foundation for understanding and applying evidence- based decision-making should quickly evolve a strong skill set that will help them perfect performing a myriad of police processes and achieve objective outcomes... This book is versatile enough to appeal to the novice student of policing yet provide value to seasoned ofﬁ cers, investigators and analysts looking to meet the high evidence-based standards of today’s polity. – Th eron L. Bowman, Ph.D., Deputy City Manager and former Police Chief, City of Arlington, Texas