sign in sign up
<<
Home , Outcomes research, Secondary research, Systematic review

What is...? series Second edition Evidence-based

Supported by sanofi-aventis What is a ?

G Systematic reviews have increasingly replaced traditional Pippa Hemingway reviews and expert commentaries as a way of summarising PhD BSc (Hons) RGN evidence. RSCN Research Fellow G Systematic reviews attempt to bring the same level of rigour to in Systematic reviewing research evidence as should be used in producing that Reviewing, School of research evidence in the first place. Health and Related G Systematic reviews should be based on a peer-reviewed so that Research (ScHARR), they can be replicated if necessary. University of Sheffield Nic Brereton PhD BSc G High quality systematic reviews seek to: (Hons) Health G Identify all relevant published and unpublished evidence Economist, NB G Select studies or reports for inclusion Consulting Services, G Assess the quality of each study or report Sheffield G Synthesise the findings from individual studies or reports in an unbiased way G Interpret the findings and present a balanced and impartial summary of the findings with due consideration of any flaws in the evidence. G Many high quality peer-reviewed systematic reviews are available in journals as well as from databases and other electronic sources. G Systematic reviews may examine quantitative or qualitative evidence; put simply, when the two or more types of evidence are examined within one review it is called a mixed-method systematic review. G Systematic reviewing techniques are in a period of rapid development. Many systematic reviews still look at clinical effectiveness, but methods now exist to enable reviewers to examine issues of appropriateness, feasibility and meaningfulness. G Not all published systematic reviews have been produced with meticulous care; therefore, the findings may sometimes mislead. Interrogating For further titles in the series, visit: published reports by asking a series of questions can uncover www.whatisseries.co.uk deficiencies.

Date of preparation: April 2009 1 NPR09/1111 What is a systematic review? What is a systematic review?

Why systematic reviews If the need for information is to be are needed fulfilled, there must be an evidence The explosion in medical, nursing and allied translation stage. This is ‘the act of healthcare professional publishing within the transferring knowledge to individual health latter half of the 20th century (perhaps professionals, health facilities and health 20,000 journals and upwards of two million systems (and consumers) by means of articles per year), which continues well into publications, electronic media, education, the new millennium, makes keeping up with training and decision support systems. primary research evidence an impossible feat. Evidence transfer is seen to involve careful There has also been an explosion in internet development of strategies that identify target access to articles, creating sometimes an awe- audiences – such as clinicians, managers, inspiring number of hits to explore. In policy makers and consumers – and designing addition, there is the challenge to build and methods to package and transfer information maintain the skills to use the wide variety of that is understood and used in decision- electronic media that allow access to large making’.1 amounts of information. Moreover, clinicians, nurses, therapists, healthcare managers, policy makers and Failings in traditional consumers have wide-ranging information reviews needs; that is, they need good quality Reviews have always been a part of the information on the effectiveness, healthcare literature. Experts in their field meaningfulness, feasibility and have sought to collate existing knowledge and appropriateness of a large number of publish summaries on specific topics. healthcare interventions; not just one or two. Traditional reviews may, for instance, be For many, this need conflicts with their busy called literature reviews, narrative reviews, clinical or professional workload. For critical reviews or commentaries within the consumers, the amount of information can literature. Although often very useful be overwhelming, and a lack of expert background reading, they differ from a knowledge can potentially lead to false belief systematic review in that they are not led via a in unreliable information, which in turn may peer-reviewed protocol and so it is not often raise health professional workload and patient possible to replicate the findings. In addition, safety issues. such attempts at synthesis have not always Even in a single area, it is not unusual for been as rigorous as might have been hoped. the number of published studies to run into In the worst case, reviewers may not have hundreds or even thousands (before they are begun with an open mind as to the likely sifted for inclusion in a review). Some of these recommendations, and they may then build a studies, once read in full text, may give case in support of their personal beliefs, unclear, confusing or contradictory results; selectively citing appropriate studies along the sometimes they may not be published in our way. Indeed, those involved in developing a own language or there may be lack of clarity review may well have started a review (or have whether the findings can be generalised to been commissioned to write one) precisely our own country. Looked at individually, each because of their accumulated experience and article may offer little insight into the professional opinions. Even if the reviewer problem at hand; the hope is that, when does begin with an open mind, traditional taken together within a systematic review, a reviews are rarely explicit about how studies clearer (and more consistent) picture will are selected, assessed and integrated. Thus, emerge. the reader is generally unable to assess the

Date of preparation: April 2009 2 NPR09/1111 What is a systematic review?

likelihood of prior beliefs or of selection or effectiveness of an intervention or publication biases clouding the review drug. Increasingly, however, they process. Despite all this, such narrative are required to establish if an intervention reviews were and are widespread and or activity is feasible, if it is appropriate influential. (ethically or culturally) or if it relates The lack of rigour in the creation of to evidence of experiences, values, traditional reviews went largely unremarked thoughts or beliefs of clients and their until the late 1980s when several relatives.1 commentators exposed the inadequacies of the process and the consequent bias in Systematic reviews are also: recommendations.2,3 Not least of the G Needed to propose a future problems was that small but important effects research agenda7 when the way were being missed, different reviewers were forward may be unclear or existing reaching different conclusions from the same agendas have failed to address a research base and, often, the findings clinical problem reported had more to do with the specialty of G Increasingly required by authors who wish the reviewer than with the underlying to secure substantial grant funding for evidence.4 primary healthcare research The inadequacy of traditional reviews and G Increasingly part of student dissertations or the need for a rigorous systematic approach postgraduate theses were emphasised in 1992 with the publication G Central to the National Institute for Health of two landmark papers.5,6 In these papers, and Clinical Excellence health technology Elliot Antman, Joseph Lau and colleagues assessment process for multiple reported two devastating findings. technology appraisals and single G First, if original studies of the effects of technology appraisals. clot busters after heart attacks had However, systematic reviews are most been systematically reviewed, the needed whenever there is a substantive benefits of therapy would have been question, several primary studies – perhaps apparent as early as the mid-1970s. with disparate findings – and substantial G Second, narrative reviews were uncertainty. One famous case is described woefully inadequate in summarising by The Library:8 a single the current state of knowledge. These research paper, published in 1998 and based reviews either omitted mention of effective on 12 children, cast doubt on the safety of therapies or suggested that the treatments the mumps, and rubella (MMR) should be used only as part of an ongoing vaccine by implying that the MMR investigation – when in fact the evidence vaccine might cause the development (if it had been collated) was near of problems such as Crohn’s and incontrovertible. . The paper by Wakefield et al9 These papers showed that there was much has since been retracted by most of the knowledge to be gained from collating original authors because of potential bias, existing research but that traditional but before that it had triggered a worldwide approaches had largely failed to extract this scare, which in turn resulted in reduced knowledge. What was needed was the same uptake of the vaccine.10 A definitive rigour in (research where systematic review by Demicheli et al on the objects of study are other research studies) MMR vaccines in children concluded that as is expected from primary research exposure to MMR was unlikely to be (original study). associated with Crohn’s disease, autism or other conditions.11 Here, then, is an area where a systematic When systematic reviews review helped clarify a vital issue to the public are needed and to healthcare professionals; preparing Conventionally, systematic reviews are such a review, however, is not a trivial needed to establish clinical and cost- exercise.

Date of preparation: April 2009 3 NPR09/1111 What is a systematic review?

The process of systematic within this search are ,12 review and language bias.13 The need for rigour in the production of systematic reviews has led to the development 3. Assessing the studies. Once all possible of a formal scientific process for their studies have been identified, they should be conduct. Understanding the approach taken assessed in the following ways. and the attempts to minimise bias can help in G Each study needs to be assessed for the appraisal of published systematic reviews, eligibility against inclusion criteria and which should help to assess if their findings full text papers are retrieved for those that should be applied to practice. The overall meet the inclusion criteria. process should, ideally, be directed by a peer- G Following a full-text selection stage, the reviewed protocol. remaining studies are assessed for methodological quality using a critical Briefly, developing a systematic review appraisal framework. Poor quality requires the following steps. studies are excluded but are usually discussed in the review report. 1. Defining an appropriate healthcare G Of the remaining studies, reported question. This requires a clear statement of findings are extracted onto a data the objectives of the review, intervention or extraction form. Some studies will be phenomena of interest, relevant patient excluded even at this late stage. A list of groups and subpopulations (and sometimes included studies is then created. the settings where the intervention is G Assessment should ideally be conducted by administered), the types of evidence or two independent reviewers. studies that will help answer the question, as well as appropriate outcomes. These details 4. Combining the results. The findings are rigorously used to select studies for from the individual studies must then be inclusion in the review. aggregated to produce a ‘bottom line’ on the clinical effectiveness, feasibility, 2. Searching the literature. The published appropriateness and meaningfulness of the and unpublished literature is carefully intervention or activity. This aggregation of searched for the required studies relating to findings is called evidence synthesis. The type an intervention or activity (on the right of evidence synthesis is chosen to fit the patients, reporting the right outcomes and so types(s) of data within the review. For on). For an unbiased assessment, this search example, if a systematic review inspects must seek to cover all the literature (not just qualitative data, then a meta-synthesis is MEDLINE where, for example, typically less conducted.14 Alternatively, a technique known than half of all trials will be found), including as meta-analysis (see What is meta-analysis?15 non-English sources. In reality, a designated in this series) is used if homogenous number of databases are searched using a quantitative evidence is assessed for clinical standardised or customised search filter. effectiveness. Narrative summaries are used if Furthermore, the (material that quantitative data are not homogenous. is not formally published, such as institutional or technical reports, working 5. Placing the findings in context. The papers, conference proceedings, or other findings from this aggregation of an unbiased documents not normally subject to editorial selection of studies then need to be discussed control or ) is searched using to put them into context. This will address specialised search engines, databases or issues such as the quality and heterogeneity of websites. Expert opinion on where the included studies, the likely impact of bias, appropriate data may be located is sought and as well as the chance and the applicability of key authors are contacted for clarification. the findings. Thus, judgement and balance Selected journals are hand-searched when are not obviated by the rigour of systematic necessary and the references of full-text reviews – they are just reduced in impact and papers are also searched. Potential biases made more explicit.

Date of preparation: April 2009 4 NPR09/1111 What is a systematic review?

A word of caution, however. Performing a of when an REA can be undertaken according rigorous systematic review is far from easy. It to the REA toolkit include: requires careful scientific consideration at G ‘When there is uncertainty about the inception, meticulous and laborious effectiveness of a policy or service and searching, as well as considerable attention to there has been some previous research methodological detail and analysis before it G ‘When a decision is required within truly deserves the badge ‘systematic’. The months and policy makers/researchers quality of a systematic review can be assessed want to make decisions based on the best by using a standard checklist. Example available evidence within that time checklists are available from the NHS Public G ‘When a map of evidence in a topic area is Health Resource Unit via the Critical Appraisal required to determine whether there is any Skills Programme (CASP)16 or from the Centre existing evidence and to direct future for Evidence-Based Medicine at the University research needs.’21 of Oxford.17 It is useful to have experience of An example of an REA to allow primary and secondary research, or to examination of the methods is a report by collaborate with those that do, prior to Underwood et al (2007), who evaluated the undertaking a systematic review and to ensure effectiveness of interventions for people with that an academic and practice partnership common mental health problems on directs the review. employment outcomes.22 The above has been an overview of the systematic review process. Clear guidance on User involvement the process of developing systematic reviews User involvement is well established as a is available electronically,18,19 from key texts prerequisite within primary research and is such as the one by Khan et al20 or via courses now increasingly expected within a systematic run at centres of excellence such as the NHS review. The Users Centre for Reviews and Dissemination at the Group proposes ‘a spectrum of user University of York or the Centre for involvement in the systematic review process, Evidence-Based Medicine at the University ranging from determining the scope of the of Oxford. review and the outcomes of relevance, to determining the need for a review and involvement throughout all stages of Some trends in systematic production and dissemination.’23 The reviewing definition of user involvement within the Rapid evidence assessment reviews systematic review protocol is recommended; Increasingly, health policy makers, clinicians thus, what is expected from a user or user and clients cannot wait the year or so required group and at which stages of the review for a full systematic review to deliver its should be clearly defined. For guidance on findings. Rapid evidence assessments (REAs) public involvement in research, access can provide quick summaries of what is INVOLVE at www.invo.org.uk already known about a topic or intervention. REAs use systematic review methods to search Mixed methods and evaluate the literature, but the Increasingly, qualitative methods are used comprehensiveness of the search and other together with a randomised controlled trial to review stages may be limited. The obtain a fuller picture of an intervention and Government Unit has the way it works.24 It is also possible to mix produced an REA toolkit which is methods within a systematic review as the recommended as a minimum standard for methods to systematically review qualitative rapid evidence reviews.21 The toolkit states evidence, such as from , that an REA takes two to six months to phenomenology and other qualitative complete and ‘is a quick overview of existing research designs, are now developed. This is research on a constrained topic and a particularly useful when different types of synthesis of the evidence provided by these data such as qualitative data and quantitative studies to answer the REA question’. Examples data are available to inform a review topic. For

Date of preparation: April 2009 5 NPR09/1111 What is a systematic review?

example, the issues of a mixed-method findings) when assessing clinical effectiveness synthesis have been described by Harden and (Box 2).26 This reflects the fact that, when well Thomas (2005) on the basis of their review of conducted, they should give us the best the barriers to, and facilitators of, fruit and possible estimate of any true effect. As noted vegetable intake among children aged four to previously, such confidence can sometimes ten years.25 The following issues arose from be unwarranted, however, and caution must the merger of two simultaneous meta- be exercised before accepting the veracity of syntheses of trial data (quantitative) and any systematic review. A number of problems studies of experiences (qualitative). may arise within reviews of clinical effectiveness. Strengths of mixed methods G Like any piece of research, a systematic G They preserve the integrity of the findings review may be done badly. Attention to of different types of studies by using the the questions listed in the section appropriate type of analysis that is specific ‘Appraising a systematic review’ can help to each type of finding. separate a rigorous review from one of G The use of categorical codes as a ‘halfway’ poor quality. house to mediate between two forms of G Inappropriate aggregation of studies data was unproblematic.25 that differ in terms of intervention used, patients included or types of data can lead Limitation of mixed methods to the drowning of important effects. G There is potential researcher bias when For example, the effects seen in some categorical subgroups are not created a priori subgroups may be concealed by a lack of and are created later on in the review.25 effect (or even reverse effects) in other subgroups. The findings from systematic reviews Finding existing reviews are not always in harmony with the High quality systematic reviews are published findings from large-scale high quality in many of the leading journals and electronic single trials.27,28 Thus, findings from databases. In addition, electronic publication systematic reviews need to be weighed against by the Cochrane Collaboration, the NHS perhaps conflicting evidence from other Centre for Reviews and Dissemination and sources. Ideally, an updated review would deal other organisations offers speedy access to with such anomalies. regularly updated summaries (Box 1). Hierarchies of evidence for feasibility or appropriateness reviews are available29 when Drawbacks of systematic reviews most of the above applies. Systematic reviews appear at the top of the ‘’ that informs evidence- Appraising a systematic review based practice (practice supported by research Not all systematic reviews are rigorous and

Box 1. Useful websites for systematic reviews

G The www.cochrane.org G The Joanna Briggs Institute www.joannabriggs.edu.au/pubs/systematic_reviews.php G The Campbell Collaboration www.campbellcollaboration.org G The Centre for Evidence-Based Medicine www.cebm.net G The NHS Centre for Reviews and Dissemination www.york.ac.uk/inst/crd G Bandolier www.medicine.ox.ac.uk/bandolier G PubMed Clinical Queries: Find Systematic Reviews www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml

Date of preparation: April 2009 6 NPR09/1111 What is a systematic review?

Box 2. Hierarchies of evidence for questions of therapy, prevention, aetiology or harm26

Level 1a Systematic review (with homogeneity) of randomised controlled trials (RCTs) Level 1b Individual RCT (with narrow confidence interval) Level 1c All-or-none studies Level 2a Systematic review (with homogeneity) of cohort studies Level 2b Individual (including low quality RCT; eg <80% follow-up) Level 2c ‘Outcomes’ research; ecological studies Level 3a Systematic reviews (with homogeneity) of case-control studies Level 3b Individual case-control study Level 4 (and poor quality cohort and case-control studies) Level 5 Expert opinion without explicit critical appraisal, or based on physiology, bench research or ‘first principles’

unbiased. The reader will want to interrogate the impact of missing information assessed any review that purports to be systematic to for its possible impact on the findings? assess its limitations and to help decide if the G Do the included studies seem to recommendations should be applied to indicate similar effects? If not, in the practice. Further guidance on appraising the case of clinical effectiveness, was the quality of a systematic review can be found in heterogeneity of effect investigated, several useful publications.16,30,31 Guidance assessed and discussed? focuses on the critical appraisal for reviews of G Were the overall findings assessed for clinical effectiveness. To reflect this, the their robustness in terms of the following questions provide a framework. selective inclusion or exclusion of G Is the topic well defined in terms of the doubtful studies and the possibility of intervention under scrutiny, the patients publication bias? receiving the intervention (plus the G Was the play of chance assessed? In settings in which it was received) and the particular, was the range of likely effect outcomes that were assessed? sizes presented and were null findings G Was the search for papers thorough? interpreted carefully? For example, a Was the search strategy described? Was review that finds no evidence of effect may manual searching used as well as electronic simply be an expression of our lack of databases? Were non-English sources knowledge rather than an assertion that searched? Was the ‘grey literature’ covered the intervention is worthless. – for example, non-refereed journals, G Are the recommendations based conference proceedings or unpublished firmly on the quality of the evidence company reports? What conclusions were presented? In their enthusiasm, reviewers drawn about the possible impact of can sometimes go beyond the evidence in publication bias? drawing conclusions and making their G Were the criteria for inclusion of recommendations. studies clearly described and fairly All studies have flaws. It is not the mere applied? For example, were blinded or presence of flaws that vitiates the findings. independent reviewers used? Even flawed studies may carry important G Was study quality assessed by blinded information. The reader must exercise or independent reviewers? Were the judgement in assessing whether individual findings related to study quality? flaws undermine the findings to such an G Was missing information sought from extent that the conclusions are no longer the original study investigators? Was adequately supported.

Date of preparation: April 2009 7 NPR09/1111 What is...? series What is a References 18. Higgins J, Green S (eds). Cochrane Handbook for 1. Pearson A, Wiechula R, Court A, Lockwood C. The JBI Systematic Reviews of Interventions, Version 5.0.1 [updated model of evidence-based healthcare. Int J Evid Based September 2008]. www.cochrane-handbook.org/ (last systematic review? Healthc 2005; 3: 207–215. accessed 19 November 2008) 2. Mulrow CD. The medical : state of the 19. NHS Centre for Reviews and Dissemination. science. Ann Intern Med 1987; 106: 485–488. Undertaking systematic reviews of research on effectiveness: 3. Teagarden JR. Meta-analysis: whither narrative review? CRD’s guidance for those carrying out or commissioning Pharmacotherapy 1989; 9: 274–281. reviews. CRD Report 4. York: University of York, 2001. 4. Spector TD, Thompson SG. The potential and limita- 20. Khan KS, Kunz R, Kleijnen J (eds). Systematic reviews tions of meta-analysis. J Epidemiol Community Health to support evidence-based medicine: how to review and apply 1991; 45: 89–92. findings of healthcare research. London: Royal Society of 5. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers Medicine Press, 2003. TC. A comparison of results of meta-analyses of random- 21. Government Social Research. Rapid Evidence ized control trials and recommendations of clinical Assessment Toolkit. www.gsr.gov.uk/professional_guid- experts. Treatments for myocardial infarction. JAMA ance/rea_toolkit/index.asp (last accessed 23 January 1992; 268: 240–248. 2009) 6. Lau J, Antman EM, Jimenez-Silva J et al. Cumulative 22. Underwood L, Thomas J, Williams T, Thieba A. The meta-analysis of therapeutic trials for myocardial infarc- effectiveness of interventions for people with common mental tion. N Engl J Med 1992; 327: 248–254. health problems on employment outcomes: a systematic 7. Torgerson C. Systematic Reviews. London: Continuum, rapid evidence assessment. 2003. http://eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=Wgr 8. The Cochrane Library. The Cochrane Library publishes %2bPGyPMD0%3d&tabid=2315&mid=4279&language=e the most thorough of MMR vaccination data which n-US (last accessed 23 January 2009) strongly supports its use. 23. Campbell Collaboration Users Group. User www.cochrane.org/press/MMR_final.pdf (last accessed 19 Involvement in the systematic review process. Campbell November 2008) Collaboration Policy Brief. http://camp.ostfold.net/art- 9. Wakefield AJ, Murch SH, Anthony A et al. Ileal-lym- man2/uploads/1/Minutes_March_2008_Oslo.pdf (last phoid-nodular hyperplasia, non-specific colitis, and per- accessed 19 November 2008) vasive developmental disorder in children. Lancet 1998; 24. Evans D, Pearson A. Systematic reviews: gatekeepers 351: 637–641. of nursing knowledge. J Clin Nurs 2001; 10: 593–599. 10. Murch SH, Anthony A, Casson DH et al. Retraction of 25. Harden A, Thomas J. Methodological Issues in an interpretation. Lancet 2004; 363: 750. Combining Diverse Study Types in Systematic Reviews. 11. Demicheli V, Jefferson T, Rivetti A, Price D. Vaccines International Journal of Social Research 2005; for measles, mumps and rubella in children. Cochrane 8: 257–271. Database Syst Rev 2005: CD004407. 26. Phillips B, Ball C, Sackett D et al; Centre for Evidence- 12. Dubben HH, Beck-Bornholdt HP. Systematic review Based Medicine. Levels of Evidence. of publication bias in studies on publication bias. BMJ www.cebm.net/index.aspx?o=1025 (last accessed 19 2005; 331: 433–434. November 2008) 13. Egger M, Zellweger-Zahner T, Schneider M et al. 27. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Language bias in randomised controlled trials published Derderian F. Discrepancies between meta-analyses and in English and German. Lancet 1997; 350: 326–329. subsequent large randomized, controlled trials. N Engl J 14. Sandelowski M, Barroso J. Toward a metasynthesis of Med 1997; 337: 536–542. First edition published 2001 qualitative findings on motherhood in HIV-positive 28. Egger M, Smith GD. Misleading meta-analysis. BMJ Authors: Huw TO Davies and women. Res Nurs Health 2003; 26: 153–170. 1995; 310: 752–754. 15. Crombie IK, Davies HTO. What is meta-analysis? 29. Evans D. Hierarchy of evidence: a framework for Iain K Crombie London: Hayward Medical Communications, 2009. ranking evidence evaluating healthcare interventions. J 16. Resource Unit. Critical Appraisal Skills Clin Nurs 2003; 12: 77–84. Programme (CASP). Making sense of evidence: 10 questions 30. Crombie IK. Pocket Guide to Critical Appraisal. This publication, along with to help you make sense of reviews. London: BMJ Publishing Group, 1996. www.phru.nhs.uk/Doc_Links/S.Reviews%20Appraisal%2 31. Moher D, Cook DJ, Eastwood S et al. Improving the the others in the series, is 0Tool.pdf (last accessed 19 November 2008) quality of reports of meta-analyses of randomised con- 17. Centre for Evidence-Based Medicine. Critical trolled trials: the QUOROM statement. Quality of available on the internet at Appraisal. www.cebm.net/index.aspx?o=1157 (last Reporting of Meta-analyses. Lancet 1999; 354: www.whatisseries.co.uk accessed 23 January 2009) 1896–1900. The data, opinions and statements appearing in the article(s) herein are those of the contributor(s) concerned. Accordingly, the sponsor and publisher, and their respective employees, officers and agents, accept no liability for the consequences of any such inaccurate or misleading data, opinion or statement.

Published by Hayward Medical Communications, a division of Hayward Group Ltd. Copyright © 2009 Hayward Group Ltd. Supported by sanofi-aventis All rights reserved.

Date of preparation: April 2009 8 NPR09/1111