Meeting the Challenges of Evidence-Based Policy: the Campbell Collaboration

Meeting the Challenges of Evidence-Based Policy: The Campbell Collaboration By ANTHONY PETROSINO, ROBERT F. BORUCH, HALUK SOYDAN, LORNA DUGGAN, and JULIO SANCHEZ-MECA ABSTRACT: Evidence-based policy has much to recommend it, but it also faces significant challenges. These challenges reside not only in the dilemmas faced by policy makers but also in the quality of the evaluation evidence. Some of these problems are most effectively ad- dressed by rigorous syntheses of the literature known as systematic reviews. Other problems remain, including the range of quality in systematic reviews and their general failure to be updated in light of new evidence or disseminated beyond the research community. Based on the precedent established in health care by the international Cochrane Collaboration, the newly formed Campbell Collaboration will prepare, maintain, and make accessible systematic reviews of research on the effects of social and educational interventions. Through mechanisms such as rigorous quality control, electronic publication, and worldwide coverage of the literature, the Campbell Collaboration seeks to meet challenges posed by evidence-based policy. Anthony Petrosino is a research fellow at the Center of Evaluation, American Acad- emy of Arts and Sciences and coordinator for the Campbell Crime and Justice Group. Robert F. Boruch is University Trustee Professor at the University of Pennsylvania Graduate School of Education and cochair of the Campbell Collaboration Steering Group. Haluk Soydan is the director of the Center for Evaluation at the Center for Evalua- tion, Swedish National Board of Health and Welfare and a cochair of the Campbell Col- laboration Steering Group. Lorna Duggan is a forensic psychiatrist in the United Kingdom and reviewer for the Cochrane Collaboration. Julio Sanchez-Meca is a psychology professor at the University of Murcia in Spain and the director of the Unit for Meta-Analysis. NOTE: We appreciate the assistance of Maureen Matkovich and colleagues at the National Criminal Justice Reference Service who supplied the data in Figure 1. The first author’s work was supported by grants from the Mellon Foundation to the Center for Evaluation and from the Home Office to Cambridge University. Thanks to Brandon C. Welsh for his comments on earlier drafts of the article. 14 Downloaded from http://ann.sagepub.com at Vrije Universiteit 34820 on March 8, 2010 15 Campbell (1969) was an they trust were not the product of DONALDinfluential psychologist who advocacy? wrote persuasively about the need Donald Campbell unfortunately for governments to take evaluation did not live long enough to bear wit- evidence into account in decisions ness to the creation of the interna- about social programs. He also recog- tional collaboration named in his nized, however, the limitations of the honor that ambitiously attempts to evidence-based approach and the address some of the challenges posed fact that government officials would by evidence-based policy. The Camp- be faced with a number of political di- bell Collaboration was created to pre- lemmas that confined their use of re- pare, update, and disseminate sys- search. The limits of evidence-based tematic reviews of evidence on what policy and practice, however, reside works relevant to social and educational intervention not only in the political pressures (see http:// faced by decision makers when im- campbell.gse.upenn.edu). The target plementing laws and administrative audience will include decision makers at all levels of directives or determining budgets; government, they also reside in problems with the practitioners, citizens, media, and research evidence. researchers. This article begins with a discus- Questions such as, What works to sion of the rationale for the reduce crime in communities? are Campbell Collaboration. We then describe not easily answered. The studies that the precedent established by the bear on these questions are often Cochrane Collaboration in health care. scattered across different fields and This is followed by an overview of the written in different are languages, advent and of the sometimes disseminated in obscure early progress Campbell Collaboration. We con- or inaccessible and can be of outlets, clude with the of the such that inter- promise Camp- questionable quality bell Collaboration in meeting the is at best. How can pretation risky evidence-based and challenges posed by policy practice be informed, if not policy. persuaded, by such a fragmented knowledge base comprising evalu- RATIONALE ative studies that range in quality? Which study, or set of studies, if any at all, ought to be used to influence Surge of interest in policy? What methods ought to be evidence-based policy used to appraise and analyze a set of There are many influences on separate studies bearing on the same decisions or beliefs about what ought question? And how can the findings to be done to address problems like be disseminated in such a way that crime, illiteracy, and unemployment. the very people Donald Campbell Influential factors include ideology, cared about-the decision makers in politics, costs, ethics, social back- government and elsewhere-receive ground, clinical experience, expert findings from these analyses that opinion, and anecdote (for example, Downloaded from http://ann.sagepub.com at Vrije Universiteit 34820 on March 8, 2010 16 Lipton 1992). The evidence-based Gibbon (1999), MacDonald (1999), approach stresses moving beyond and Sheldon and Chilvers (2000), these factors to also consider the among others, articulated views results of scientific studies. Although about evidence-based education and few writers have articulated the social welfare. deterministic view that the term A more persuasive indicator that &dquo;evidence-based&dquo; suggests, it is clear evidence-based policy is having some that the vast majority of writers impact is initiatives undertaken by argue that decision makers need to- governments since the late 1990s. at the very least-be aware of the Whether due to growing pragmatism research evidence that bears on poli- or pressures for accountability on cies under consideration (for exam- how public funds are spent, the ple, Davies, Nutley, and Smith 2000). evidence-based approach is begin- Certainly the implicit or explicit goal ning to take root. For example, the of research-funding agencies has United Kingdom is promoting always been to influence policy evidence-based policy in medicine through science (Weiss and Petrosino and the social sectors vigorously (for 1999), and there have always been example, Davies, Nutley, and Smith individuals who have articulated the 2000; Wiles 2001). In 1997, its need for an evidence-based approach Labour government was elected (for example, Fischer 1978). But using the slogan, &dquo;What counts is there has been a surge of interest, what works&dquo; (Davies, Nutley, and particularly in the 1990s, in argu- Smith 2000). The 1998 U.K. Crime ments for research-, science-, or evi- Reduction Programme was greatly dence-based policy (for example, influenced by both the University of Amann 2000; Boruch, Petrosino, and Maryland report to Congress on Chalmers 1999; Nutley and Davies crime prevention (Sherman et al. 1999; Wiles 2001). 1997) and the Home OfBce’s own syn- One indirect gauge of this surge is theses (Nuttall, Goldblatt, and Lewis the amount of academic writing on 1998). In Sweden, the National the topic. For example, in Sherman’s Board of Health and Welfare was (1999) argument for evidence-based commissioned by the government to policing, decisions about where to draft a program for advancing target police strategies would be knowledge in the social services to based on epidemiological data about ensure they are evidence based the nature and scope of problems. (National Board of Health and Wel- The kinds of interventions employed, fare 2001). and how long they were kept in place, In the United States, the Govern- would be guided by careful eval- ment Performance and Review Act of uative studies, preferably random- 1993 was implemented to hold fed- ized field trials. Cullen and eral agencies responsible for identi- Gendreau (2000) and MacKenzie fying measurable objectives and (2000) are among those who made reaching them. This has led to the similar arguments about correc- development of performance indica- tional treatment. Davies (1999), Fitz- tors to assess whether there is value Downloaded from http://ann.sagepub.com at Vrije Universiteit 34820 on March 8, 2010 17 added by agencies. The 1998 Coinciding with fragmentation, reauthorization of the Safe and Drug evaluation studies are not regularly Free Schools and Communities Act published in academic journals or in required that programs funded outlets that are readily accessible. under the law be research based. The Instead a large percentage of news media now commonly ask why evaluative research resides in what police are not using research-based Sechrest, White, and Brown (1979) eyewitness identification techniques called the fugitive literature. These (Gawande 2001) or why schools use are government reports, disserta- ineffective drug prevention pro- tions and master’s theses, conference grams (for example, Cohn 2001). papers, technical documents, and All of these signs seem to indicate other literature that is difficult to more than a passing interest in evi- obtain. Lipsey (1992), in his review of dence-based policy. As Boruch (1997) delinquency prevention and treat- noted, different policy questions ment studies, found 4 in 10 were require different types of scientific reported in this literature. Although evidence. To implement the most some may argue that unpublished effective interventions to ameliorate studies are of lesser quality because problems, careful evaluations are they have not been subjected to blind needed. An evidence-based approach peer review as journal articles are, to what works therefore requires this is an empirical question worthy be that these evaluations gathered, of investigation. Such an assertion, and and that appraised, analyzed the at the very least, ignores the high- results be made accessible to influ- quality evaluations done by private ence relevant decisions whenever research firms.

Load more