Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review*

British Journal of Management, Vol. 14. 207-222 (2003)

David Tranfield, David Denyer and Palminder Smart Advanced Management Research Centre (AMRC), Cranfield School of Management, Cranfield University, Cranfield, MK43 OAl, UK Corresponding author email: D. [email protected]

Undertaking a review of the literature is an important part of any research project. The researcher both maps and assesses the relevant intellectual territory in order to specify a research question which will further develop the knowledge hase. However, traditional 'narrative' reviews frequently lack thoroughness, and in many cases are not undertaken as genuine pieces of investigatory science. Consequently they can lack a means for making sense of what the collection of studies is saying. These reviews can he hiased by the researcher and often lack rigour. Furthermore, the use of reviews of the available evidence to provide insights and guidance for intervention into operational needs of practitioners and policymakers has largely been of secondary importance. For practitioners, making sense of a mass of often-contradictory evidence has hecome progressively harder. The quality of evidence underpinning decision-making and action has heen questioned, for inadequate or incomplete evidence seriously impedes policy formulation and implementation. In exploring ways in which evidence-informed management reviews might be achieved, the authors evaluate the process of systematic review used in the medical sciences. Over the last fifteen years, medical science has attempted to improve the review process hy synthesizing research in a systematic, transparent, and reproducihie manner with the twin aims of enhancing the knowledge hase and informing policymaking and practice. This paper evaluates the extent to which the process of systematic review can be applied to the management field in order to produce a reliable knowledge stock and enhanced practice by developing context-sensitive research. The paper highlights the challenges in developing an appropriate methodology.

Introduction: the need for an evidence- practice in any discipline, is a key research informed approach objective for the respective academic and practitioner communities. Undertaking a review of the literature to provide The post-World-War-II era witnessed a sharp the best evidence for informing policy and focus of attention by academics and practitioners on the discipline and profession of management (Blake and Mouton, 1976; Tisdall, 1982). The pace of knowledge production in this field has *This paper results from research undertaken in Cran- been accelerating ever since and has resulted in a field IMRC (EPSRC) grant no IMRC19, 'Developing a methodology for evidence-informed management body of knowledge that is increasingly fragmen- knowledge using systematic review', Professor David ted and transdisciplinary as well as being inter- Tranfield and Dr David Denyer. dependent from advancements in the social

© 2003 British Academy of Management 208 D. Tranfield, D. Denyer and P. Smart sciences (Friedman, Durkin, Phillips and Volt- through the effective use of systematic reviews. singer, 2000). The following sections will compare and contrast In management research, the literature review the nature of reviews in medical science and process is a key tool, used to manage the diversity management research and evaluate the extent to of knowledge for a specific academic inquiry. The which the systematic review process can be aim of conducting a literature review is often to applied to the management field. Finally this enable the researcher both to map and to assess paper will present the challenges in designing an the existing intellectual territory, and to specify a appropriate methodology for management re- research question to develop the existing body of search. knowledge further. Management reviews are usually narrative and have been widely criticized for being singular descriptive accounts of the The origins of the evidence-based contributions made by writers in the field, often approach selected for inclusion on the implicit biases of the researcher (Fink, 1998; Hart, 1998). Not surpris- Since the 1980s, the British central government ingly they have also been condemned for lacking has placed increasing emphasis on ensuring that critical assessment. The management-research policy and practice are informed through a more community perpetuates this type of practice by rigorous and challenging evidence base. The not actively commissioning infrastructural ar- 'three E' initiatives (economy, efficiency and rangements to ensure previous investments in effectiveness) have focused attention on the literature reviews are not lost. This tolerance to delivery of public services and have led to the loss of knowledge forms a high-risk strategy that development of detailed guidance and best- will inevitably become unsustainable as organiza- practice manuals in many disciplines. Effective- tions endeavour further into the networked and ness in this context is concerned both with knowledge-based economy. appropriateness and the validity of the methods Reviews of the available evidence in manage- used by professionals in their day-to-day work to ment to assimilate 'best evidence' to provide achieve their basic aims and also with the overall insights and guidance for intervention into the ability of agencies to deliver the services they are operational needs of practitioners and policy- required to provide (Davies, Nutley and Smith, makers have largely become a secondary con- 2000). The concern for effective service delivery sideration. has attracted considerable attention, and has Sufficient momentum from academics, practi- focused interest on basing policy and practice on tioners, and government has stirred an urgent the best evidence available. Consequently, an need to re-evaluate the process by which manage- evidence-based movement has developed under ment researchers conduct literature reviews. Over New Labour, and in May 1997 Tony Blair the last fifteen years, medical science has at- announced that 'what counts is what works', the tempted to improve the quality of the review intention being to signal a new 'post-ideological' process. This paper proposes the view that approach to public policy where evidence would applying specific principles of the systematic take centre stage in the decision-making process review methodology used in the medical sciences (Davies, Nutley and Smith, 2000). to management research will help in counteracting bias by making explicit the values and assumptions underpinning a review. By enhancing the The evidence-based approach in medical legitimacy and authority of the resultant evidence, science and healthcare systematic reviews could provide practitioners and policy-makers with a reliable basis to formulate The evidence-based movement has had a major decisions and take action. This is particularly impact in certain disciplines. Pre-eminent have sobering if one considers the growing pressures been applications in medical science, where the upon practitioners in today's global trading pace of knowledge production has meant that environments to do this in shorter cycle times. making sense of an often-contradictory mass of This paper will begin by discussing the evidence has become increasingly difficult (Ohls- evidence-based approach in medical sciences son, 1994). Specifically in the late 1980s, attention Developing Evidence-Informed Management Knowledge 209 was drawn to the comparative lack of rigour in Systematic reviews differ from traditional secondary research (Mulrow, 1987). Critics ar- narrative reviews by adopting a replicable, gued that the preparation of reviews of secondary scientific and transparent process, in other words sources were dependent on implicit, idiosyncratic a detailed technology, that aims to minimize bias methods of data collection and interpretation through exhaustive literature searches of pub- (Cook, Mulrow and Haynes, 1997; Greenhalgh, lished and unpublished studies and by providing 1997). In addition, practice based on poor-quality an audit trail of the reviewers decisions, proce- evaluations of the literature sometimes had led to dures and conclusions (Cook, Mulrow and inappropriate recommendations (Cook, Green- Haynes, 1997). The process of systematic review gold, Ellrodt and Weingarten, 1997). In 1991, and its associated procedure, meta-analysis, has Smith questioned the overall wisdom of much of been developed over the last decade and now medical science, arguing that only 15-20% of plays a major role in evidence-based practices. medical interventions were supported by solid Whereas systematic review identifies key scien- medical evidence (Smith, 1991). The result, it was tific contributions to a field or question, meta- argued, was that patients were being regularly analysis offers a statistical procedure for synthe- subjected to ineffective treatments and interven- sizing findings in order to obtain overall relia- tions, and for many practices there was little or bility unavailable from any single study alone. no understanding of whether or not the benefits Indeed, undertaking systematic review is now outweighed the potential harm (Davies, Nutley regarded as a 'fundamental scientific activity' and Smith, 1999). (Mulrow, 1994, p. 597). The 1990s saw several The National Health Service (NHS) Research organizations formed with the aim of establishing and Development Strategy identified that too agreed and formalized procedures for systematic little research was being carried out in the review and to undertake systematic reviews to important clinical areas and that much of the synthesize and disseminate evidence across all existing research was ad hoc, piecemeal and areas of healthcare. These organizations included poorly conducted (Peckham, 1991). The report the Cochrane Collaboration (2001), the National also argued that researchers rather than practi- Health Science Centre for Reviews and Dissemi- tioners, managers or policymakers drove the nation (2001) and the National Institute for research agenda. Clinical Excellence (2001). Furthermore, there was Uttle dissemination, let alone diffusion, of research findings. The Strategy not only argued for an increase in the level of Evidence-based approacbes in otber research conducted but also for systematic re- disciplines views of existing research on important clinical or operational questions, assessing the best evidence The movement to base practice on the best available, collating the findings and presenting available evidence has migrated from medicine them in a way that was accessible and relevant to to other disciplines. In the UK, the Department decision-makers (Peckham, 1991). for Education and Skills (DfES) has established a Centre for Evidence Informed Policy and Prac- tice in Education. Furthermore, a 'What Works? Systematic review - a key tool in Programme' was introduced in the probation developing the evidence base service following the Crime Reduction Strategy published by the Home Office in July 1998. The Over the last decade medical science has made aim of the programme was to develop successful significant strides in attempting to improve the intervention programmes based on hard evidence quality of the review process by synthesizing so that they could be used as models for day-to- research in a systematic, transparent and repro- day probation practice (HM Inspectorate of ducible manner to inform policy and decision- Probation, 1998; Home Office, 1998). An Effec- making about the organization and delivery of tive Practice Initiative also has sought to address health and social care (Cook, Greengold, Ellrodt the difficult problem of ensuring that offender and Weingarten, 1997; Cook, Mulrow and supervision changes in line with research evidence Haynes, 1997; Wolf, Shea and Albanese, 2001). on what works (Furniss and Nutley, 2000). The 210 D. Tranfield, D. Denyer and P. Smart

Department for the Environment, Transport, and research to be both well founded and socially the Regions (DETR) commissioned a review of robust. This emphasis on producing a science the evidence base as it relates to regeneration base, which is both rigorous in formulation and pohcy and practice (Dabinett, Lawless, Rhodes relevant to practice, is a key characteristic of an and Tyler, 2001), Other disciplines such as evidence-based approach. nursing (Evans and Pearson, 2001), housing The quality of information accepted as evi- policy (Davies and Nutley, 1999; Maclennan dence in a discipline is dependent on a number of and More, 1999), social care (Macdonald, 1999) criteria. These include the broad intellectual and criminal justice (Laycock, 2000) have also approach, the value system adopted by research- adjusted the approach with varying degrees of ers and commissioning bodies and the usual success. In 2001, the Economic and Social research methods employed (Davies and Nutley, Research Council (ESRC) funded the establish- 1999), Medical science has traditionally adopted ment of a network (the Evidence Network) of a 'normal science' approach within which double- multi-disciplinary centres dedicated to the im- bhnded randomized controlled trials have been provement of the evidence base for policy and widely accepted as the most rigorous method for practice in social sciences. The Evidence Network testing interventions before use. So far, systema- aims to use systematic review to inform and tic reviews have tended to be applied in, and to improve decision-making in government, busi- emanate from, fields and disciplines privileging a ness and the voluntary sector. positivist tradition, attempting to do for research Internationally, in February 2000 the Camp- synthesis what randomized controlled trials bell Collaboration was launched in Philadelphia aspire to do for single studies (Macdonald, by about 150 pioneering social scientists. This 1999), Systematic reviews entail a series of techni- equivalent of the Cochrane collaboration aims: ques for minimizing bias and error, and as such systematic review and meta-analysis are widely 'to help people make well-informed decisions about regarded as providing 'high-quality' evidence. the effects of interventiotis in the social, behavioural and educational arenas' (Campbell Collaboration, Figure 1 highlights the hierarchy of evidence in 2001), the medical sciences (Davies and Nutley, 1999), In other disciplines such as education, social Within the approach taken by the Campbell services and criminal justice there is often both Collaboration, delegates considered questions less consensus regarding the appropriate metho- such as how practitioners might engage the dology to be used for evaluating the evidence review process, what makes research useful and base, and little agreement as to how use research useable and what standards and quality criteria evidence to inform policy and practice (Davies and distinguished reliable from unreliable research? Nutley, 1999; Laycock, 2000; Macdonald, 1999; In this sense, discussions addressed the need for Maclennan and More, 1999), Furthermore, policy

I-I Systematic review and meta-analysis of two or more double blind randomized controlled trials,

1-2 One or more large double-blind randomized controlled trials,

n-1 One or more well-conducted cohort studies,

II-2 One or more well-conducted case-control studies,

II-3 A dramatic uncontrolled experiment,

ni Expert comtnittee sitting in review; peer leader opinion,

IV Personal experience.

Figure 1. Hierarchies of evidence. Source: reproduced by kind permission of the publisher from Davies, H. T. O. and S. M. Nutley (1999). 'The Rise and Rise of Evidence in Health Care'. Public Money & Management, 19 (l),pp. 9-16. © 1999 Blackwell Publishing. Developing Evidence-Informed Management Knowledge 211 questions are rarely addressed by the use of Pettigrew (1997, p. 291), in much the same vein randomized controlled trials. For example, in as Whitley, emphasized the significance of the social care the nature of evidence is often hotly social production of knowledge in viewing disputed and there exists strong resistance to management research, emphasizing stakeholder privileging one research method over another. perspectives. His influential view was that man- Indeed, postmodern perspectives generally mis- agement research faces a series of challenges: trust any notion of objective evidence. Divergences such as these are deeply rooted in 'best captured in a series of concurrent double hurdles, which together raise a wide spectrum of the ontological and epistemological assumptions cognitive, social and political demands on [the] of specific fields. Despite these difficulties, Davies, skills and knowledge of [management] researchers.' Nutley and Smith argue optimistically: He argued for a thematic approach: 'The different ontological and epistemological start- ing points in different professional traditions un- 'to meet the double hurdle of embeddedness in the doubtedly colour the methods and enthusiasm with social sciences and the worlds of policy and which professionals engage with evidence. However, practice' (Pettigrew, 1997, p. 292). what is clear is that there remains in all of the areas Berry (1995) ofl'ered a Gallic perspective, examined great potential for research evidence to be vastly more influential than hitherto'. (2000, p. 4) arguing strongly the case for the importance of qualitative work. Several writers (Aram and Salipante, 2000; Pfefler and Sutton, 1999; Van de Ven, 1998; Wind and Nueno, 1998) have The nature of management research argued convincingly for the applied nature of The nature of the field of management research management research. Likewise, Hambrick has been subject, over the years, to considerable (1994) and Huflf (2000) both used their addresses analysis and discussion. Much of this discussion as President of the Academy of Management to and debate has focused upon the ontological address the ontological status of the field. More status of the field, particularly its fragmented and recently, Wilcoxson and Fitzgerald (2001) have divergent nature. For example, Whitley (1984a, focused on the nature of management as a 1984b), in two influential articles, investigated the discipline and the consequences of this for scientific status of management research as a researchers and practitioners in an Australasian 'practically oriented social science'. He identified context and Van Aken (2001) has developed a its fragmented state and argued that the conse- view of management research based as a design quence of this is a: science, rather than as a formal or explanatory science. By conceptualizing management research 'low degree of reputational control over significance in this way, he identifies the need for a field of standards ... (which) means that the significance of study to deliver output not only of high academic problems and preferred ways of formulating them quality but also which is practitioner and are unstable, subject to disputes, and arc assessed context-sensitive. He argues that the mission of by diffused and diverse standards.' (Whitley, 1984a, design sciences is to develop valid and reliable p. 343) knowledge in the form of 'field tested and Whitley (2000) further refined this position, grounded technological rules' to be used in suggesting that the continued fragmentation of designing, configuring and implementing solu- the management field may displace academics as tions to specific problems. key stakeholders in the research process. In The 1990s saw an extensive debate concerning comparing management research with industrial, the nature of management research within the work and organizational psychology, Hodgkin- British Academy of Management, which focused son, Herriot and Anderson (2001, s45) also on the ontological status of the field, and conclude that there is a considerable and widen- particularly the extent to which academic-practi- ing divide between academics and other stake- tioner relations were to be privileged. The work holder groups and that 'this divergence is likely to of Gibbons et al. (1994) on modes of knowledge further proliferate irrelevant theory and untheor- production has become increasingly influential in ized and invalid practice'. such debates. In particular, their notion of mode 212 D. Tranfield, D. Denyer and P. Smart

2 knowledge production, where there is 'a constant advantage of defining disciplinary boundaries, flow back and forth between the theoretical and making them easy to defend. Thus, the extent to the practical' and where 'knowledge is produced in which disciplines are opening up research ques- the context of application' has been argued to be tions, or addressing a previously defined and central to debates about the future of management agreed agenda, dictates positioning on this research (Tranfield and Starkey, 1998). Creating a dimension. management research which is both theoretically Management research is a relatively young sound and methodologically rigorous as well as field, far less well developed in terms of agenda relevant to the practitioner community has been a and question formulation than much of medical theme explored by both the British Academy of science. As a result there tends to be low Management and the Foundation for Manage- consensus concerning key research questions in ment Education (Starkey and Madan, 2001). management research. Studies in the field rarely This discussion was developed further in a special address identical problems and share a research issue of the British Journal of Management agenda or, more importantly, ask the same (Hodgkinson, 2001). questions. Therefore, it is unlikely that aggregative approaches to research synthesis, such as meta-analysis will be appropriate in management Comparing the management and research as the heterogeneity of studies prevents medical fields the pooling of results and the measurement of the net effectiveness of interventions. Tranfield and Starkey (1998), in an article which Table 1 outlines the similarities and differences both refiected and drove the debate in the British between medical science as an applied field of Academy of Management, used Becher's (1989) study stemming from the biological sciences, and dimensions drawn from the sociology of knowl- management research as an applied field with edge to characterize management research as strong connections to the social sciences. 'soft' rather than 'hard' 'applied' rather than The main question here is to what extent 'pure', rural' rather than 'urban', and 'divergent' review processes developed in fields that are rather than 'convergent'. The creation of such a striving to become evidence based, such as the profile, with the use of dimensions drawn from more convergent field of medicine, can inform the the sociology of knowledge, enabled contrasts to review process in the management field to help be made with other disciplines, particularly create rigorous and relevant reviews. As manage- medical science, where systematic review has ment research questions need to be clearly been applied to considerable effect. Comparison specified, either as replication of an existing can be made in both epistemological and study, as further development of an existing ontological realms. study, or as a new study to meet a defined 'gap' in Whereas medical research enjoys considerable the literature, a more systematic literature review and extensive epistemological consensus, this is process can help to justify/qualify the near/final untrue of management research, in general. The research question which is posed. Furthermore, consequential difiiculties of establishing agreed the process described/proposed in this paper thresholds for high-quality work result from this values and takes steps to encourage participation, lack of consensus. by both academics and by managers/policy- Key ontological differences between manage- makers, and is pragmatic in intent. ment research and medical science concern the Systematic reviews have traditionally been dimension 'convergent-divergent'. The extent to applied in fields and disciplines privileging a which a discipline resides at one end of this positivist and quantitative tradition: dimension or another is purported to depend upon similarities in research ideologies, values and quality judgements which create a shared 'Positivists seek cause-and-effect laws that are sense of nationhood amongst researchers within sufficiently generalizable to ensure that a knowledge of prior events enables a reasonable predication of the field (Becher, 1989). Agreements concerning subsequent event ... Because positivists see knowl- key research questions to be addressed lead to a edge as accumulating, they have been more inter- relatively low tolerance of deviance, but have the ested in developing approaches to research Developing Evidence-Informed Management Knowledge 213

Table I. Differences between medical research and management research

Medicine Management

Nature of the discipline Convergent. Divergent. Research culture Subjected to rigorous scientific evaluation. Split between positivist and phenomenological perspectives. Research questions High consensus over research questions. Low consensus over research questions. Interventions Can be measured through experiments. Experimentation may or may not be feasible. Research designs Based upon a hierarchy of evidence. Triangulation is recommended. Theory Concerned with what works-did the intervention Concerned with why something works or does offer overall benefits. not work and the context in which this occurs. Aims of policy Generally reducing illness and death, and Multiple and competing and the balance between improving health. them may change over time. Weight of inputs into policy Scientific evidence. Many extraneous factors. Methods Predominantly quantitative. Quantitative and qualitative. Literature reviews Systematic review and meta-analysis. Largely narrative reviews. The need for a review Reviews of effectiveness are used by clinical To develop a research question and inform practitioners. empirical research practice. Preparation of the review A review panel (including practitioners) guides Usually an informal/ad hoc process involving the the process. researcher, peers and supervisor. A brief scoping study is conducted to delimit the subject area. Review protocol A plan prior to the review states the criterion for Level of formality and standardisation in including and excluding studies, the search designing/adopting protocols is usually low. strategy, description of the methods to be used, Unacceptable to 'tightly' plan literature review, coding strategies and the statistical procedures to as this may inhibit the researchers capacity to the employed. explore, discover and develop ideas. Protocols are made available by international bodies to enhance networking the exchange of knowledge. Identifying research A comprehensive, structured search is conducted Identifying a field/sub- fields of study generally using predetermined keywords and search strings. occurs through informal consultation. Implicit idiosyncratic methods of data collection are used. Selection of studies Inclusion and exclusion criteria are expressed in Based on studies that appear relevant or the protocol to ensure a review of the best interesting. Researchers bias disables critical available evidence. appraisal. Decisions regarding choice are not Draw upon 'raw data' from 'whole studies' for recorded precluding any audit trails 'Raw data' is analysis to create a study in its own right. often not available in academic articles, which usually represent 'partial studies'. Precise inclusion/exclusion criteria are often not formally agreed, applied recorded or monitored. Study quality assessment Studies are assessed against predetermined Poor evaluating of the fit between research criteria. The internal validity of the study is methodology and research questions. judged. Assessing and including qualitative Researchers tend to rely on the quality rating of a studies is problematic. particular journal, rather than applying quality assessment criteria to individual articles. Data extraction is not formally guided by Data extraction Data extraction forms are used which act as a explicitly stated inclusion and exclusion criteria. historical record for the decisions made during Data extracted is not comprehensively recorded the process and provides the basis on which to and monitored. conduct data synthesis. Data synthesis A qualitative synthesis provides a tabulation of Generally narrative and qualitative. Higher levels key characteristics and results. Meta-analysis of subjectivity associated with what is taken from pools the data across studies to increase the an article for analysis and synthesis. Lack explicit power of statistical analysis. Aims to generate descriptive and thematic analysis. Specific tools 'best' evidence. and techniques from the field of qualitative data analysis are increasingly applied. Reporting and Standardized reporting structures used Non- Non-standardized reporting structures. Dissemination explanatory style adopted. Short scripts recorded Interpretive long scripts. The explanatory power and made widely avail able through improved through the use of analogy, metaphor internationally recognized institutions. and homology. Process of knowledge production, Comprehensible by practitioners. omitted. Sometimes incomprehensible by practitioners lack links between different literature. Evidence into practice Collaborative process and practice-oriented. Implementation of evidence is often an afterthought. 214 D. Tranfield, D. Denyer and P. Smart

synthesis than have interpretivists.' (Noblit and and Crombie, 1998). The Cochrane Collabora- Hare, 1988, p. 12) tion's Cochrane Reviewers' Handbook (Clarke and Oxman, 2001) and the National Health Service Indeed researchers from an interpretivist or Dissemination (2001) provide a list of stages in phenomenological position may suggest that conducting systematic review (see Figure 2). systematic reviews, with their positivist leanings, should not be adopted in the social sciences. Even within medical research, not everybody accepts Stage I: planning the review that systematic reviews are necessary or desirable (Petticrew, 2001). Petticrew (2001, p. 98) argues Prior to beginning the review a review panel is that the concern over systematic review has been formed encompassing a range of experts in the fuelled by the fact that they are often presented as areas of both methodology and theory. Efforts synonymous with a numerical aggregation of the should be made to include practitioners working results of individual studies through a process of in the field on the panel. The review panel should meta-analysis and 'that they are incapable of help direct the process through regular meetings dealing with other forms of evidence, such as and resolve any disputes over the inclusion and from non-randomized studies or qualitative re- exclusion of studies. The initial stages of search'. However, meta-analysis 'is simply one of systematic reviews may be an iterative process the tools, albeit a particularly important one, that of definition, clarification, and refinement is used in preparing systematic reviews' (Mulrow, (Clarke and Oxman, 2001). Within management Cook and DavidofT, 1997, p. 290). In most it will be necessary to conduct scoping studies to systematic reviews the heterogeneity of study data assess the relevance and size of the literature and prevents the use of meta-analysis. In these cases, to delimit the subject area or topic. Such studies synthesis is achieved through summarizing the need to consider cross-disciplinary perspectives findings of a group of studies. Alternative and alternative ways in which a research topic methods of research synthesis such as realist has previously been tackled. The scoping study synthesis, meta-synthesis and meta-ethnography may also include a brief overview of the have also been developed to draw comparisons theoretical, practical and methodological history and conclusions from a collection of studies debates surrounding the field and sub-fields of through interpretative and inductive methods. Whilst there are fundamental differences between meta-analysis and qualitative research synthesis Stage I-Planning the review (Campbell, Pound, Pope, Bitten, Pill, Mogan, and Phase 0 - Identification for the need for a review Donovan, 2003), both are concerned with 'putting Phase 1 - Preparation of a proposal for a review together' (Noblit and Hare, 1988, p. 7) findings Phase 2 - Development of a review protocol from a number of empirical studies in some coherent way (Dingwall, Murphy, Watson, Stage n—Conducting a review Greatbatch and Parker, 1998). Phase 3 - Identification of research The following section of the paper reports the Phase 4 - Selection of studies systematic review methodology used in medical Phase 5 - Study quality assessment science, seeks to tease out the key characteristics Phase 6 - Data extraction and monitoring progress of the approach, highlights the key challenges in Phase 7 - Data synthesis transferring the model to the management field Stage in-Reporting and dissemination and presents a number of recommendations on how these may be addressed. Phase 8 - The report and recommendations Phase 9 - Getting evidence into practice

Eigure2. Stages of a systematic review (Source: adapted by kind Conducting a systematic review permission of the publisher from NHS Centre for Reviews and Dissemination (2001). Undertaking Systematic Reviews of Despite the relative infancy of systematic review, Research on Effectiveness. CRD's Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2"'' a reasonable consensus has emerged as to its Edition) © 2001 NHS Centre for Reviews and Dissemination, desirable methodological characteristics (Davies University of York. Developing Evidence-Informed Management Knowledge 215 study. Where fields comprise of semi-independent and almost always requiring perseverance and and autonomous sub-fields, then this process attention to detail, systematic review has been may prove difficult and the researcher is likely to argued to provide the most efficient and high- struggle with the volume of information and the quality method for identifying and evaluating creation of transdisciplinary understanding. extensive literatures (Mulrow, 1994), A systematic search begins with the identification of Within medical science the researcher will also keywords and search terms, which are built from arrive at a definitive review question. The review the scoping study, the literature and discussions question is critical to systematic review as other within the review team. The reviewer should then aspects of the process flow from it. In systematic decide on the search strings that are most review the outcome of these decisions is captured appropriate for the study. The search strategy through a formal document called a review should be reported in detail sufficient to ensure protocol. The protocol is a plan that helps to that the search could be replicated. Searches protect objectivity by providing explicit descrip- should not only be conducted in published tions of the steps to be taken. The protocol journals and Hsted in bibliographic databases, contains information on the specific questions but also comprise unpublished studies, con- addressed by the study, the population (or ference proceedings, industry trials, the Internet sample) that is the focus of the study, the search and even personal requests to known investiga- strategy for identification of relevant studies, and tors. The output of the information search should the criteria for inclusion and exclusion of studies be a full listing of articles and papers (core in the review (Davies and Crombie, 1998), Once contributions) on which the review will be based. protocols are complete they are registered with Only studies that meet all the inclusion criteria the appropriate review-group editors, such as the specified in the review protocol and which Cochrane Collaboration. If satisfactory, the manifest none of the exclusion criteria need be review is published to encourage interested incorporated into the review. The strict criteria parties to contact the reviewers and to avoid used in systematic review are linked to the desire duplication of studies. to base reviews on the best-quality evidence. As Any management review protocol may contain decisions regarding inclusion and exclusion re- a conceptual discussion of the research problem main relatively subjective, this stage of the and a statement of the problem's significance systematic review might be conducted by more rather than a defined research question. Further- than one reviewer. Disagreements can be resolved more management reviews are often regarded as a within the review panel. The process of selecting process of exploration, discovery and develop- studies in systematic review involves several ment. Therefore, it is generally considered unac- stages. The reviewer will initially conduct a ceptable to plan the literature-review activities review of all potentially relevant citations identi- closely, A more flexible approach may make fied in the search. Relevant sources will be explicit what the researcher intends to do a priori retrieved for a more detailed evaluation of the but can be modified through the course of the full text and from these some will be chosen for study. The researcher needs to state explicitly what the systematic review. The number of sources changes have been made and the rationale for included and excluded at each stage of the review doing so. The aim is to produce a protocol that is documented with the reasons for exclusions. does not compromise the researcher's ability to be Within the medical domain there is a tension creative in the literature review process, whilst also between the statistical benefits of including a ensuring reviews be less open to researcher bias large number of primary studies and conducting than are the more traditional narrative reviews. high-quality reviews of fewer studies with the use of more selective methodological criteria of inclusion and exclusion (Davies, 2000), Quality Stage II: conducting the review assessment refers to the appraisal of a study's A comprehensive, unbiased search is one of the internal validity and the degree to which its fundamental diflerences between a traditional design, conduct and analysis have minimized narrative review and a systematic review. biases or errors. Individual studies in systematic Although sometimes taking considerable time. review are judged against a set of predetermined 216 D. Tranfield, D. Denyer and P. Smart

criteria and checklists to assist the process (Ox- Sandelowski, Docherty and Emden (1997) man, 1994). The relevance of a study to the claim that checklists, when applied to qualitative review depends on the relevance of its research studies, should be used with caution if they are questions and the quahty of its methodology. The used as a basis on which to exclude studies from a reviewer should avoid including: review. They go on to argue that any decisions 'all studies that meet broad standards in terms of regarding exclusion must be supported by a independent and dependent variables, avoiding any detailed explanation of the reviewer's conception judgement of quality.' (Slavin, 1986, p. 6) of 'good' and 'bad' studies and the reasons for exclusion. Systematic reviews, due to their positivistic Whereas systematic reviews draw upon 'raw origins, sit comfortably with studies that use data', in management research these data are quantitative methods such as randomized con- often not made available in articles by authors. In trolled trials, quasi-experimental designs, and many cases the articles only represent the results cost-benefit and cost-eflectiveness studies, there- of part studies that satisfy the orientation of the fore, establishing criteria for ascertaining what is editors of a particular journal. Therefore, the 'relevant' or 'good quality' in qualitative research decisions regarding the selection of studies provides a further challenge (Engel and Kuzel, actually become decisions about the selection of 1992). With qualitative studies there is no possi- 'articles' based on the more subjective findings bility of testing statistically the significance of the and conclusions of the author rather than on the results. Qualitative research, by its very nature: 'raw' data:

'is non-standard, unconfined, and dependent on the 'It is highly unlikely that such a synthesis will subjective experience of both the researcher and the involve a re-analysis of primary data which may be researched ... it is debatable, therefore, whether an in the form of transcripts from interviews, for field- all-encompassing critical appraisal checklist along notes from studies involving participant observa- the lines of the User's Guides to the Medical tion. Rather, the data to be analysed are most likely Literature could ever be developed' (Greenhaigh to be the findings of the studies involved. These and Taylor, 1997, p. 741). might take the form of substantive themes arising, for example, from in-depth interviews. Within Several authors have presented a range of qualitative research (and arguably all research) criteria that might be used to appraise and theory plays a pivotal role in informing the evaluate qualitative studies (Blaxter, 1996; interpretation of data. Whilst few authors appear Greenhaigh and Taylor, 1997; Mays and Pope, to have considered the role for theory-led synthesis 2000; Popay, Rogers and Williams, 1998). Popay, of findings across studies an argument can be made Rogers and Williams (1998) suggest that a quality for exploring the potential for this approach.' (Clarke and Oxman, 2001, section 4, p. 20) assessment would include the following:

• a primary marker: is the research aiming to explore Systematic reviews expose studies to rigorous the subjective meanings that people give to particular methodological scrutiny. Within the management experiences and interventions?; field it may be possible to conduct a quality • context sensitive: has the research been designed in assessment of the research articles by evaluating such a way as to enable it to be sensitive/flexible to the fit between research methodology and re- changes occurring during the study?; search questions. However, management re- • sampling strategy: has the study sample been selected searchers usually rely on the implicit quality in a purposeful way shaped by theory and/or rating of a particular journal, rather than attention given to the diverse contexts and meanings formally applying any quality assessment criteria that the study is aiming to explore?; to the articles they include in their reviews (i.e. • data quality: are different sources of knowledge/under- refereed journals are 'better' than practitioner standing about the issues being explored or compared?; journals). The difficulty in specifying and con- • theoretical adequacy: do researchers make explicit the process by which they move from data to interpreta- ducting quality assessments of studies is a major tion?; challenge in developing a systematic review • generalizability: if claims are made to generalizability methodology for management research. do these follow logically and/or theoretically from the To reduce human error and bias, systematic data? reviews employ data-extraction forms. These Developing Evidence-Informed Management Knowledge 111 often contain general information (title, author, question and measure the phenomenon in the publication details), study features and specific same way. Furthermore, researchers are less information (details and methods) and notes on concerned with the effectiveness of certain classes emerging themes coupled with details of synth- of intervention, and rather more concerned with esis. The Cochrane Collaboration states that understanding organizations and management data-extraction forms serve at least three im- processes. Therefore, it is unlikely that meta- portant functions. First, the form is directly analysis will be appropriate in management linked to the formulated review question and research. the planned assessment of the incorporated A number of authors have offered interpretive studies, providing a visual representation of and inductive approaches to research synthesis, these. Second, the extraction form acts as a which are more likely to provide a means of historical record of the decisions made during the drawing insight from studies and for addressing process. Third, the data-extraction form is the issues pertinent to management research. Some data-repository from which the analysis will authors contend that there are a number of emerge (Clarke and Oxman, 2001). philosophical and practical problems associated The data-extraction process requires a docu- with 'summing up' qualitative studies, whilst mentation of all steps taken. In many cases others argue that attempts to 'synthesize existing double extraction processes are employed, where studies are seen as essential to reaching higher two independent assessors analyse a study and analytic goals and also enhancing the general- their findings are compared and reconciled if izability of qualitative research' (Sandelowski, required. Data-extraction can be paper based or Docherty and Emden, 1997, p.367). Two inter- computer based. The development of the data- pretive and inductive methods, realist synthesis extraction sheets is fiexiblean d may depend upon and meta-synthesis, have been developed to fill the nature of the study. When devising the form, the gap between narrative reviews and meta- reviewers should consider the information that analysis. will be needed to construct summary tables and For Pawson (2001), realist synthesis offers one to perform data synthesis. Data-extraction forms technique for producing a synthesis of a range of should include details of the information source study types. He argues that in medical research, (title, authors, journal, publication details) and programmes (such as medical treatments) carry any other features of the study such as popula- the potential for change. The aim of a systematic tion characteristics, context of the study and an review is to classify these programmes and to evaluation of the study's methodological quality. conduct a meta-analysis to provide a reliable Links to other concepts, identification of emer- measure of net effect. The practitioner is invited gent themes, and key results and additional notes to replicate the treatment that has worked to also need to be included on the data-extraction maximum effect. In contrast, narrative reviews form. tend to explain the combination of attributes in a Research synthesis is the collective term for a programme and generally identify exemplars of family of methods for summarizing, integrating, best practice. The practitioner is invited to and, where possible, cumulating the findings of imitate the programmes that are successful. different studies on a topic or research question According to Pawson, it is not programmes that (Mulrow, 1994). The simplest and best-known work; rather it is the underlying reasons or form of research synthesis is a narrative review resources that they offer subjects that generate that attempts to identify what has been written change. Whether change occurs is also dependent on a subject or topic. Such reviews make no on the nature of the actors and the circumstances attempt to seek generalization or cumulative of the programme. Realist synthesis captures a knowledge from what is reviewed (Greenhalgh, list of vital ingredients or mechanisms (positive or 1997). Meta-analysis is an alternative approach negative) that underpin each individual pro- to synthesis, which enables the pooling of data gramme. The researcher then builds theory by from individual studies to allow for an increase in accumulating understanding across a range of statistical power and a more precise estimate of programmes. Whilst some scholars would ques- effect size (Glass, 1976). Within management tion whether contingency statements could ever research, few studies address the same research be developed, Pawson (2001) argues that a realist 218 D. Tranfield, D. Denyer and P. Smart synthesis can provide a transferable programme desire to improve upon traditional narrative theory in the form of 'what works for whom in reviews by adopting explicit and rigorous pro- what circumstances'. cesses and by: Meta-synthesis also offers an interpretative approach to research synthesis which can be used 'the bringing together of findings on a chosen to identify the: theme, the results of which should be to achieve a greater level of understanding and attain a level of 'theories, grand narratives, generalizations, or conceptual or theoretical development beyond that interpretative translations produced from the in- achieved in any individual empirical study. (Camp- tegration or comparison of findings from qualita- bell et al., 2002, p. 2) tive studies.' (Sandelowski, Docherty and Emden, 1997, p. 366) As in systematic reviews, the aim of realist syntheses and meta-syntheses is to 'have impact' Unlike meta-analysis, meta-synthesis is not lim- by being 'presented in an accessible and usable ited to synthesizing strictly comparable studies by form in the real world of practice and policy constructing 'interpretations, not analyses, and making' (Sandelowski, Docherty and Emden, by revealing the analogies between accounts'(No- 1971, p. 365). blit and Hare, 1988, p. 8). Meta-synthesis provides a means of taking into account: Stage III: reporting and dissemination 'all important similarities and differences in lan- guage, concepts, images, and other ideas around a A good systematic review should make it easier target experience.' (Sandelowski, Docherty and for the practitioner to understand the research by Emden, 1997, p.669) synthesizing extensive primary research papers from which it was derived. Within management Meta-ethnography is a method of meta-synthesis research a two-stage report might be produced. that offers three alternative techniques for The first would provide full (rough-cut and synthesising studies. 'Refutational synthesis' can detailed) 'descriptive analysis' of the field. This be used when reports give conflicting representa- is achieved using a very simple set of categories tions of the same phenomenon, 'reciprocal with the use of the extraction forms. For translations' can be used where reports address example, who are the authors, how many of the similar issues and 'lines of argument synthesis' core contributions are from the USA, how many can be used if different reports examine different are European? What is the age profile of the aspects of the same phenomenon. A meta- articles? Can the fields be divided into epochs in ethnography is analogous with a grounded terms of volume of orientation of study? Do theory approach for open coding and identifying simple categories divide up the field? For categories emerging from the data and by making example, can the field be divided sectorally? By constant comparisons between individual ac- gender? Or simple categories 'borrowed' from counts (Beck, 2001). The categories are then associated cognate disciplines such as psychology linked interpretively to provide a holistic account or sociology (interpretivist versus positivistic or of the whole phenomenon (Suri, 1999). behavioural versus cognitive studies, for exam- Many of the techniques of meta-synthesis ple). The researcher should be able to provide a remain 'either relatively untried and undeve- broad ranging descriptive account of the field loped, and/or difficult to codify and understand' with specific exemplars and an audit trail, (Sandelowski, Docherty and Emden, 1997, p. justifying his/her conclusions. 369). However, both realist synthesis and meta- Researchers also need to report the findings of synthesis challenge the positivistic orthodoxy that a 'thematic analysis', whether or not the results surrounds contemporary approaches to research were derived through an aggregative or inter- reviews, demonstrating that a synthesis can be an pretative approach, outlining that which is interpretive, inductive, hermeneutic and eclectic known and established already from data- process (Jensen and Alien, 1996). Whilst meta- extraction forms of the core contributions. They synthesis and realist synthesis approaches are may wish to focus on the extent to which fundamentally different to systematic reviews and consensus is shared across various themes. They in particular meta-analysis, they both share a may also want to identify key emerging themes Developing Evidence-Informed Management Knowledge 219 and research questions. Whatever overarching Conclusions categories are chosen for the tabulation, researchers should again provide a detailed audit trail This paper began by arguing that reviews of back to the core contributions to justify and existing research evidence in the management ground their conclusions. Linking themes across field lack both rigour and relevance. Anderson, the various core contributions wherever possible Herriot and Hodgkinson (2001) offer a four-fold and highlighting such links is an important part characterization of applied social science. They of the reporting process. term research that is low on rigour but high on Systematic review provides a means for practi- relevance 'Popularist Science'. In contrast, 'Ped- tioners to use the evidence provided by research to antic Science' is high on rigour but low on inform their decisions. However, turning the relevance, whereas 'puerile Science' is neither conclusions from systematic reviews into guidelines rigorous nor relevant. Only 'Pragmatic Science' for practice has been a challenge in medicine balances both rigour and relevance (see Figure 3). (Macdonald, 1999), as 'clinicians reason about They acknowledge that the pursuit of 'prag- individual patients on the basis of analogy, matic' research: experience, heuristics, and theory, as well as evidence' (Cook, Mulrow and Haynes, 1997, p. 'that genuinely bears the hallmarks of scientific 380). Decision-makers are likely, and should be rigour (irrespective of whether it be quantitative encouraged, to use personal experience and pro- and/or qualitative in nature), but which also blem-solving skills rather than relying solely on the engages a wider body of stakeholders in the results of systematic reviews (Bero and Rennie, knowledge production process, presents a set of 1995; Rosenberg and Donald, 1995). Within formidable challenges for the management research community at this juncture.' (Hodgkinson, Herriot management there is a need to recognize that and Anderson, 2001, p. S46) evidence alone is often insufficient and incomplete, only informing decision-making by bounding This paper has outlined the opportunities and available options. Therefore, the terms 'evidence challenges in applying ideas and methods devel- informed' or even 'evidence aware', rather than 'evidence based' (Nutley, Davies and Walter, 2002; Theoretical and Nutley and Davies, 2002), may be more appro- methodological priate in the management field, and the former has rigour infiuenced our choice of title for this paper. Low High Improving the translation of research evidence High Quadrant 1: Quadrant 2: into practice is not unproblematic as the 'rela- Practical 'Popularist 'Pragmatic tionships between research, knowledge, policy relevance Science' Science' and practice are always likely to remain loose, shifting and contingent' (Nutley and Davies, 2002, p. 11). For evidence-informed practice to Quadrant 3: Quadrant 4: be achieved, strategies need to be developed 'Puerile 'Pedantic which encourage the uptake and utilization of Science' Science' evidence that move beyond the simple construc- tion and dissemination of the research base (Nutley and Davies, 2000). Encouraging practi- Low tioners to set specific questions for reviews and to engage in the process may help in developing a Figure 3. A four-fold typology of research in industrial, work and 'context sensitive' science (Nowotny, Scott and organizational psychology. Source: adapted by G. P. Hodgkin- son, P. Herriot and N. Anderson (2001), British Journal of Gibbons, 2001) which may help to blur the Management, 12 (Special Issue), page S42,from N. Anderson, boundaries between science, policy and practice. P. Herriot and G. P. Hodgkinson, 'The practitioner-researcher Increasing the precision of a reliable evidence divide in industrial, work and organizational (IWO) psychology: base in order that policymakers and practitioners where are we now, and where do we go from here?'. Journal of can make more sensitive judgements is the Occupational and Organizational Psychology, 74, pp. 391—411. © 2001 The British Psychological Society and the British ultimate aim of the application of systematic Academy of Management. Reproduced by kind permission of review procedures to management research. both publishers. 220 D. Tranfield, D. Denyer and P. Smart

oped in medical science to the field of manage- Cochrane Collaboration (2001), The Cochrane Brochure, http:// ment, with the aim of further developing and www,cochrane,org/cochrane/cc-broch,htm#BDL enhancing the quality of management reviews Cook, D, J,, N, L. Greengold, A, G. Ellrodt, and S, R. and ensuring that they are practitioner and Weingarten (1997). 'The Relation Between Systematic Re- views and Practice Guidelines', Annals of Internal Medicine, context sensitive. The aim of systematic review m (3) August, pp, 210-216. is to provide collective insights through theore- Cook, D. J., C. D. Mulrow, and R. B. Haynes (1997). tical synthesis into fields and sub-fields. For 'Systematic Reviews: Synthesis of Best Evidence for Clinical academics, the reviewing process increases meth- Decisions', Annals of Internal Medicine, 126 (5) March, pp. odological rigour. For practitioners/managers, 376-380. Dabinett, G., P, Lawless, J. Rhodes, and P. Tyler (2001), A systematic review helps develop a reliable knowl- Review of the Evidence Base for Regeneration Policy and edge base by accumulating knowledge from a Practice, Department of the Environment Transport and the range of studies. In so doing the researcher may Regions. be able to develop a set of 'field tested and Davies, H. T. O, and L K. Crombie (1998), 'Getting to Grips grounded technological rules' (Van Aken, 2001, with Systematic Reviews and Meta-Analyses', Hospital p, 1), In this sense, systematic review can be Medicine, 59 (12), pp, 955-958. Davies, H, T. O. and S. M, Nutley (1999). 'The Rise and Rise of argued to he at the heart of a 'pragmatic' Evidence in Health Care', Public Money & Management, 19 management research, which aims to serve both (1), pp. 9-16. academic and practitioner communities. Davies, H. T. O,, S, M. Nutley and P. C Smith (2000). 'Editorial: What Works? The Role of Evidence in Public Sector Policy and Practice', Public Money & Management, 19 (1), pp, 3-5, References Davies, H. T. O., S. M. Nutley and N, Tilley (1999). 'Editorial: Getting Research into Practice, Public Money & Manage- Anderson, N,, P, Herriot and G, P, Hodgkinson (2001), 'The ment, 20(4), pp. 17-22. Practitioner-Researcher Divide in Industrial, Work and Davies, P. (2000), The Relevance of Systematic Reviews to Organizational (IWO) Psychology: Where Are We Now, Educational Policy and Practice, http://www.jiscmail.ac.uk/ and Where Do We Go From Here?', Journal of Occupational flles/BEME/oxreview.htm, and Organizational Psychology, 74 (4), pp, 391-411, Dingwall R,, E, A. Murphy, P, Watson, D, Greatbatch and S, Aram, J, D, and J, P. F, Salipante (2000), 'Applied Research in Parker (1998). 'Catching Goldfish: Quality in Qualitative Management: Criteria for Management Educators and for Research', Journal of Health Services Research and Policy, 3 Practitioner-Scholars', US Academy of Management Confer- (3), pp. 167-172, ence—Multiple Perspectives on Learning in Management Economic and Social Research Council (1999), A History of the Education, Toronto. EPSRC UK Centre for Evidence Based Policy and Practice, Becher, A, (1989), Academic Tribes and Territories: Intellectual http://vvww,evidencenetwork,org/home,asp, Enquiry and the Cultures of Disciplines, The Society for Engel, J. D. and A. J. Kuzel (1992). 'On the idea of what Research into Higher Education and the Open University constitutes good qualitative inquiry'. Qualitative Health Press, Milton Keynes. Research, 2, pp. 504-510. Beck, C, T, (2001), 'Caring with nursing education: A meta- Estabrooks, C. A., P. A. Field, and J, M, Morse (1994), synthesis'. Journal of Nursing Education, 40 (3), pp, 101-110, 'Aggregating qualitative findings: An approach to theory Bero, L, and D, Rennie (1995), 'The Cochrane Collaboration: development'. Qualitative Health Research, 4, pp, 503-511. Preparing, Maintaining and Disseminating Systematic Re- Evans, D. and A, Pearson (2001), 'Systematic Reviews: views of the Effects of Health Care', Journal of the American Gatekeepers of Nursing Knowledge', Journal of Clinical Medical Association, 114 (1), pp. 1935-1938, Nursing, 10 (5), pp. 593-599. Berry, M. (1995), 'Research and the Practice of Management: A Fink, A. (1998). Conducting Research Literature Reviews: From French View', Organizational Science, 6 (2), pp, 104-116. Paper to the Internet, Sage Publications, London. Blake, R. R. and J, S, Mouton (1976), Consultation. Addison- Friedman A., C. Durkin, M. Phillips, and E. Voltsinger (2000), Wesley Publishing, Reading, MA, The Future of UK Professional Associations, Proceedings for Blaxter, M, (1996). 'Criteria for the Evaluation of Qualitative 5th Annual Professional Association Research Network Research Papers', Medical Sociology News, 22 (1), pp. 68-71, Conference. Campbell, R., P, Pound, C, Pope, N. Britten, R. Pill, M, Morgan, Furniss, J. S. and M. Nutley (2000), 'Implementing and J, Donovan (2003), 'Evaluating meta-ethnography: What Works with Offenders—The Effective Practice a synthesis of qualitative research on lay experiences of Initiative', Public Money <£ Management, 20 (4), pp. diabetes and diabetes care'. Social Science and Medicine, 23-28, 56 (4), pp, 671-684. Gibbons, M,, C, Limoges, H, Nowotny, S, Schwartzman, P, Campbell Collaboration (2001), http://campbell,gse,upenn,edu/ Scott and M, Trow (1994), The New Production of Knowl- about.htm. edge: The Dynamics of Science and Research in Contemporary Clarke, M, and A. D, Oxman (Eds) (2001). Cochrane Reviewers' Societies, Sage Publications, London. Handbook 4.1.4 [updated October 2001], The Cochrane Glass, G, V, (1976), 'Primary, Secondary, and Meta-analysis of Library, Oxford, Research', Educational Researcher, 5 (2), pp, 3-8, Developing Evidence-Informed Management Knowledge 221

Greenhaigh, T. (1997), 'Papers that Summarise Other Papers Nowotny, H., P. Scott and M. Gibbons (2001), Rethinking (Systematic Reviews and Meta-analyses)', British Medical Science: Knowledge and the Public in an Age of Uncertainty. Journal, 315 (7109), pp. 672-675. Blackwell Publishers, Maiden, MA. Greenhaigh, T. and R. Taylor (1997). 'Papers that go Beyond Nutley, S. M. and H. T. O. Davies (2000). 'Making a Reality Numbers (Qualitative Research)', British Medical Journal. of Evidence Based Practice: Some Lessons from the Diffusion 315 (7110), pp. 740-743. of Innovations', Public Money & Management, 20 (4), pp. Halladay, M. and L. Bero (2000). 'Implementing Evidence- 35-42. based Practice in Health Care', Public Money & Management, Nutley S. M., H. T. O. Davies and I. Walter (2002). From 20 (4), pp. 43-50. Knowing to Doing: A Framework for Understanding the Hambrick, D. C. (1994). 'What if the Academy Actually Evidence-into-practice Agenda, Discussion Paper 1, Research Mattered?', Academy of Management Review, 19 (1), pp. 11. Unit for Research Utilisation, http://www.st-and.ac.uk/ Hart, C. (1998). Doing a Literature Review: Releasing the Social ~ cppm/KnowDo%20paper.pdf. Science Research Imagination, Sage Publications, London. Nutley, S. M. and H. T. O. Davies (2002). Evidence-based Policy HM Inspectorate of Probation (1998). Strategies for Effective & Practice: Moving from Rhetoric to Reality, Discussion Paper Offended Supervision. Report of the HMIP What Works 2, Research Unit for Research Utilisation, http://www.st- Project, Home Office, London. and.ac.uk/~cppm/Rhetoric%20to%20reality%20NF.pdf. Hodgkinson, G. P. (Ed.) (2001). Facing the future: the nature Ohlsson, A. (1994). 'Systematic Reviews—Theory and Prac- and purpose of management research re-assessed, British tice', Scandinavian Journal of Clinical & Laboratory Investi- Journal of Management, 12 (Special Issue), pp. S1-S80. gation. 54 (219), pp. 25-32. Hodgkinson, G. P., P. Herriot and N. Anderson (2001). 'Re- Oxman, A. D, (1994). 'Systematic Reviews—Checklists for aligning the Stakeholders in Management Research: Lessons Review Articles', British Medical Journal, 309 (6955), pp. from Industrial, Work and Organizational Psychology', 648-651. British Journal of Management, 12 (Special Issue), pp. Pawson, R. (2001). The Promise of a Realist Synthesis, Working S4I-S48. Paper No.4, ESRC Evidence Network, Centre for Evidence Home Office (1998), Reducing Offending: An Assessment of Based Policy and Practice, http://www.evidencenetwork.org/ Research Evidence on Ways of Dealing with Offending Documents/wp4.pdf Behaviour, Home Office Research and Statistics Directorate, Peckham, M. (1991). 'Research and Development for the London. National Health Service', Lancet, 338, pp. 367-371. Huff, A. S. (2000). 'Changes in Organizational Knowledge Petticrew, M. (2001). 'Systematic reviews from astronomy to Production', Academy of Management Review, 25 (2), pp. zoology: myths and misconceptions', British Medical Journal 288-293. 322 (13) January, pp. 98-101. Jensen, L. A., and M. N. Alien (1996). 'Meta-synthesis of Pettigrew, A. M. (1997). The Double Hurdles for Management qualitative findings'. Qualitative Health Research, 6 (4), pp. Research, Advancement in Organizational Behaviour, Ash- 553-60. gate, Aldershot. Laycock, G. (2000), 'From Central Research to Local Practice: Pfeffer, J. and R. I. Sutton (1999). 'Knowing "What" to Do is Identifying and Addressing Repeat Victimization', Public Not Enough: Turning Knowledge into Action', California Money & Management, 19 (1) pp. 17-22. Management Review, 42 (1), pp. 83-108. Macdonald, G. (1999), 'Evidence-based Social Care: Wheels off Popay, J., A. Rogers and G. Williams (1998). 'Rationale and the Runway?', Public Money & Management, 19 (1), pp. Standards for the Systematic Review of Qualitative Litera- 25-32. ture in Health Services Research', Qualitative Health Maclennan, D. and A. More (1999). 'Evidence, What Evidence? Research, 8 (3), pp. 341-351. The Foundations for Housing Policy', Public Money & Rosenberg, W. and W. Donald (1995). 'Evidence Based Management, 19 (I), pp. 17-24. Medicine: An Approach to Clinical Problem-Solving', British Mays, N. and C. Pope (2000). 'Assessing O^^lity in Quali- Medical Journal, 310 (6987), pp. 1122-1126. tative Research', British Medical Journal. 320 (January), pp. Sandelowski, M., S. Docherty, and C. Emden (1997). 'Quali- 50-52. tative Metasynthesis: Issues and Techniques'. Research in Mulrow, C. D. (1987). 'The Medical Review Article: State of Nursing and Health. 20 (4), pp. 365-371. the Science', Annual International Medicine, 106, pp. 485-488. Smith, R. (1991), 'Where is the Wisdom.? The poverty of Mulrow, C. D. (1994). 'Systematic Reviews—Rationale for medical evidence' [editorial], British Medical Journal, 303, pp. Systematic Reviews', British Medical Journal, 309 (6954), pp. 789-799. 597-599. Slavin, R. E. (1986). 'Best-evidence Synthesis: An Alternative to Mulrow, C. D., D. J. Cook, and F. Davidoff (1997). 'Systematic Meta-analytic and Traditional Reviews', Educational Re- reviews: Critical links in the great chain of evidence'. Annals searcher, 15 (9), pp. 5-11. of Internal Medicine, 126, pp. 389-391. Starkey, K. and P. Madan (2001). 'Bridging the Relevance Gap: National Institute for Clinical Excellence (2001), http:// Aligning Stakeholders in the Future of Management www.nice.org.uk/. Research', British Journal of Management, 12 (SI), pp. 3-26. NHS Centre for Reviews and Dissemination (2001), Under- Suri, H. (1999). 'The process of synthesising qualitative taking Systematic Reviews of Research on Effectiveness. research: A case study'. Annual Conference of the Association CRD's Guidance for those Carrying Out or Commissioning for Qualitative Research, Melbourne, http://www.latrobe.e- Reviews. CRD Report Number 4 (2"'' Edition). York. du.au/aqr/offer/papers/HSuri.htm Noblit, G. W. and R. D. Hare (1988). Meta-ethnography: Tisdall, P. (1982). Agents of Change: The Development and Synthesizing Qualitative Studies, Sage Publications, London. Practice of Management Consultancy, Heinemann, London 222 D. Tranfield, D. Denyer and P. Smart

Tranfield, D. and K. Starkey (1998). The Nature, Social Whitely, R. (2000), The Intellectual and Social Organization of Organization and Promotion of Management Research: the Sciences, Second Edition, Oxford University Press, Towards Policy' British Journal of Management, 9 (4), pp. Oxford. 341-353. Wilcoxson, L. and E. P. Fitzgerald (2001). The Nature and Role Van de Ven, A. H. (1998). Professional Science for a of Management Research in Australia and New Zealand, Professional School. Breaking the Code of Change Con- Australian and New Zealand Academy Of Management ference, Harvard Business School, Boston, MA. Conference. Van Aken, J. (2001). Management Research Based on the Wind, J. and P. Nueno (1998). The Impact Imperative: Closing Paradigm of the Design Sciences: The Quest for Eield Tested the Relevance Gap of Academic Management Research, and Grounded Technological Rules. Working Paper 01.1, International Academy of Management North America Eindhoven Centre for Innovation Studies, Eindhoven Uni- Meeting, New York. versity of Technology, Eindhoven. Wolf, F. M., J. A. Shea and M. A. Albanese (2001), 'Toward Whitley, R. (1984a), 'The Fragmented State of Management Setting a Research Agenda for Systematic Reviews of Studies: Reasons and Consequences', Journal of Management Evidence of the Effects of Medical Education', Teaching Studies. 21 (3), pp. 331-348. and Learning in Medicine. 13 (1), pp. 54-60. Whitley, R. (1984b), 'The Scientific Status of Management Research as a Practically-oriented Social Science', Journal of Management Studies. 2t (4), pp. 369-390.