What is...? series Second edition Evidence-based medicine Supported by sanofi-aventis What is a systematic review? G Systematic reviews have increasingly replaced traditional Pippa Hemingway narrative reviews and expert commentaries as a way of summarising PhD BSc (Hons) RGN research evidence. RSCN Research Fellow G Systematic reviews attempt to bring the same level of rigour to in Systematic reviewing research evidence as should be used in producing that Reviewing, School of research evidence in the first place. Health and Related G Systematic reviews should be based on a peer-reviewed protocol so that Research (ScHARR), they can be replicated if necessary. University of Sheffield Nic Brereton PhD BSc G High quality systematic reviews seek to: (Hons) Health G Identify all relevant published and unpublished evidence Economist, NB G Select studies or reports for inclusion Consulting Services, G Assess the quality of each study or report Sheffield G Synthesise the findings from individual studies or reports in an unbiased way G Interpret the findings and present a balanced and impartial summary of the findings with due consideration of any flaws in the evidence. G Many high quality peer-reviewed systematic reviews are available in journals as well as from databases and other electronic sources. G Systematic reviews may examine quantitative or qualitative evidence; put simply, when the two or more types of evidence are examined within one review it is called a mixed-method systematic review. G Systematic reviewing techniques are in a period of rapid development. Many systematic reviews still look at clinical effectiveness, but methods now exist to enable reviewers to examine issues of appropriateness, feasibility and meaningfulness. G Not all published systematic reviews have been produced with meticulous care; therefore, the findings may sometimes mislead. Interrogating For further titles in the series, visit: published reports by asking a series of questions can uncover www.whatisseries.co.uk deficiencies. Date of preparation: April 2009 1 NPR09/1111 What is a systematic review? What is a systematic review? Why systematic reviews If the need for information is to be are needed fulfilled, there must be an evidence The explosion in medical, nursing and allied translation stage. This is ‘the act of healthcare professional publishing within the transferring knowledge to individual health latter half of the 20th century (perhaps professionals, health facilities and health 20,000 journals and upwards of two million systems (and consumers) by means of articles per year), which continues well into publications, electronic media, education, the new millennium, makes keeping up with training and decision support systems. primary research evidence an impossible feat. Evidence transfer is seen to involve careful There has also been an explosion in internet development of strategies that identify target access to articles, creating sometimes an awe- audiences – such as clinicians, managers, inspiring number of hits to explore. In policy makers and consumers – and designing addition, there is the challenge to build and methods to package and transfer information maintain the skills to use the wide variety of that is understood and used in decision- electronic media that allow access to large making’.1 amounts of information. Moreover, clinicians, nurses, therapists, healthcare managers, policy makers and Failings in traditional consumers have wide-ranging information reviews needs; that is, they need good quality Reviews have always been a part of the information on the effectiveness, healthcare literature. Experts in their field meaningfulness, feasibility and have sought to collate existing knowledge and appropriateness of a large number of publish summaries on specific topics. healthcare interventions; not just one or two. Traditional reviews may, for instance, be For many, this need conflicts with their busy called literature reviews, narrative reviews, clinical or professional workload. For critical reviews or commentaries within the consumers, the amount of information can literature. Although often very useful be overwhelming, and a lack of expert background reading, they differ from a knowledge can potentially lead to false belief systematic review in that they are not led via a in unreliable information, which in turn may peer-reviewed protocol and so it is not often raise health professional workload and patient possible to replicate the findings. In addition, safety issues. such attempts at synthesis have not always Even in a single area, it is not unusual for been as rigorous as might have been hoped. the number of published studies to run into In the worst case, reviewers may not have hundreds or even thousands (before they are begun with an open mind as to the likely sifted for inclusion in a review). Some of these recommendations, and they may then build a studies, once read in full text, may give case in support of their personal beliefs, unclear, confusing or contradictory results; selectively citing appropriate studies along the sometimes they may not be published in our way. Indeed, those involved in developing a own language or there may be lack of clarity review may well have started a review (or have whether the findings can be generalised to been commissioned to write one) precisely our own country. Looked at individually, each because of their accumulated experience and article may offer little insight into the professional opinions. Even if the reviewer problem at hand; the hope is that, when does begin with an open mind, traditional taken together within a systematic review, a reviews are rarely explicit about how studies clearer (and more consistent) picture will are selected, assessed and integrated. Thus, emerge. the reader is generally unable to assess the Date of preparation: April 2009 2 NPR09/1111 What is a systematic review? likelihood of prior beliefs or of selection or effectiveness of an intervention or publication biases clouding the review drug. Increasingly, however, they process. Despite all this, such narrative are required to establish if an intervention reviews were and are widespread and or activity is feasible, if it is appropriate influential. (ethically or culturally) or if it relates The lack of rigour in the creation of to evidence of experiences, values, traditional reviews went largely unremarked thoughts or beliefs of clients and their until the late 1980s when several relatives.1 commentators exposed the inadequacies of the process and the consequent bias in Systematic reviews are also: recommendations.2,3 Not least of the G Needed to propose a future problems was that small but important effects research agenda7 when the way were being missed, different reviewers were forward may be unclear or existing reaching different conclusions from the same agendas have failed to address a research base and, often, the findings clinical problem reported had more to do with the specialty of G Increasingly required by authors who wish the reviewer than with the underlying to secure substantial grant funding for evidence.4 primary healthcare research The inadequacy of traditional reviews and G Increasingly part of student dissertations or the need for a rigorous systematic approach postgraduate theses were emphasised in 1992 with the publication G Central to the National Institute for Health of two landmark papers.5,6 In these papers, and Clinical Excellence health technology Elliot Antman, Joseph Lau and colleagues assessment process for multiple reported two devastating findings. technology appraisals and single G First, if original studies of the effects of technology appraisals. clot busters after heart attacks had However, systematic reviews are most been systematically reviewed, the needed whenever there is a substantive benefits of therapy would have been question, several primary studies – perhaps apparent as early as the mid-1970s. with disparate findings – and substantial G Second, narrative reviews were uncertainty. One famous case is described woefully inadequate in summarising by The Cochrane Library:8 a single the current state of knowledge. These research paper, published in 1998 and based reviews either omitted mention of effective on 12 children, cast doubt on the safety of therapies or suggested that the treatments the mumps, measles and rubella (MMR) should be used only as part of an ongoing vaccine by implying that the MMR investigation – when in fact the evidence vaccine might cause the development (if it had been collated) was near of problems such as Crohn’s disease and incontrovertible. autism. The paper by Wakefield et al9 These papers showed that there was much has since been retracted by most of the knowledge to be gained from collating original authors because of potential bias, existing research but that traditional but before that it had triggered a worldwide approaches had largely failed to extract this scare, which in turn resulted in reduced knowledge. What was needed was the same uptake of the vaccine.10 A definitive rigour in secondary research (research where systematic review by Demicheli et al on the objects of study are other research studies) MMR vaccines in children concluded that as is expected from primary research exposure to MMR was unlikely to be (original study). associated with Crohn’s disease, autism or other conditions.11 Here, then, is an area where a systematic When systematic reviews review helped clarify a vital issue to the public are needed