journal of Epidemiology and Community Health 1991; 45: 89-92 J Epidemiol Community Health: first published as 10.1136/jech.45.2.89 on 1 June 1991. Downloaded from REVIEW ARTICLE:Research methods in epidemiology, V The potential and limitations of meta-analysis Tim D Spector, Simon G Thompson We are currently wimessing an "epidemic" of numbers is particularly relevant when dealing meta-analyses and overviews in the scientific with subgroup analysis, for which very often the literature. This is a relatively new phenomenon randomised controlled trial was not designed. and this article addresses some of the important Combining the results of comparable trials or issues raised by their increasing use. In particular studies can reduce random sampling errors that the differing applications and limitations ofmeta- may predominate in any individual study. The analysis are discussed, with a review of the larger the sample size available, the more precise analytic methods used and the problems and the estimate of the effect, and the hypothesis of biases encountered. subgroup effects can be more reliably investigated. What is meta-analysis? It has been suggested by some authors that only Meta-analysis has come to refer to the combining randomised controlled trials should be subjected ofresults from a number ofexperiments or studies to meta-analysis.8 However this restriction is not examining the same question. Such a process is desirable; aetiological meta-analyses (ie, of case- not new, and some meta-analytic studies were control or prospective studies) have recently been reported as early as 1955.1 However, only since carried out, usually to clarify inconsistent the term meta-analysis was first used in 19762 has findings or to estimate the true effect of a risk the technique become recognised as an analytical factor. However the interpretation of a meta- method. Meta-analysis is a discipline that reviews analysis of randomised controlled trials is usually critically and combines statistically the results of simpler. If all relevant clinical trials are included previous research in an attempt to summarise the and these are free from bias (ie, trials are totality ofevidence relating to a particular medical randomised, all randomised individuals are issue. The term meta-analysis is now often used included in "intention to treat" analyses, and synonymously with overview. outcome assessments are objective or blinded), a meta-analysis will give an unbiased assessment of Why use meta-analysis? a treatment's efficacy.9 In observational http://jech.bmj.com/ Traditionally, when seeking advice in epidemiology, potential bias in individual studies controversial or novel areas, clinicians and (through confounding, misclassification, or other scientists have relied heavily on "informed" causes) will always remain a problem, especially editorials or narrative reviews. There is now good when effect sizes are small. If such biases are to an evidence to suggest that these traditional methods extent consistent over different studies, a meta- are subject to bias and inaccuracy.3 Reviewers analysis will reflect both the true effect and the are a using traditional methods less likely to detect biases. However the increasing use of meta- on September 27, 2021 by guest. Protected copyright. small but significant effect or difference analysis in observational studies should compared with reviewers using formal statistical encourage the more formal reporting of techniques.4 In controversial topics, such as aetiological studies, to facilitate the combining of reviews of the uses of new procedures, the such results. Indeed the direct comparison of Department of enthusiasm for the procedure may be associated results from meta-analyses of randomised Environmental and more with the specialty of the reviewer than with contfolled trials and of the related observational Preventive Medicine, the most current St Bartholomew's results of the trials.5 As medical studies is a novel and informative advance.'0 Hospital Medical reviews do not use scientific methods to assess and College, Charterhouse present data, different reviewers often reach Examples of meta-analysis Square, London different conclusions based on the same data.6 ECIM 6BQ For There are now many examples ofmeta-analysis in T D Spector these reasons some formal statistical process of a great variety of medical specialities that Medical Statistics review should replace the informal approach. demonstrate their potential usefulness. One ofthe Unit, London School Meta-analysis can be used to resolve uncertainty f of Hygiene and early important studies concerned the use of Tropical Medicine, when reports, editorials or reviews disagree. blockers in myocardial infarction'2 13 which Keppel St, London Although the randomised controlled clinical showed the efficacy of post-discharge treatment wC1 trial is now accepted as the gold standard method by combining the results of over 60 small studies. S G Thompson of assessing therapeutic regimes, individual trials It also produced a useful framework for future Correspondence to: may produce false positive or negative studies. Another meta-analysis has concluded Dr Spector conclusions. Small numbers and the consequent that steriods are of benefit in meningitis in Accepted for publication lack ofpower ofany individual study is usually the children,14 another that H2 antagonists are ofonly October 1990 main problem area.7 The problem of small minor benefit in the treatment of gastrointestinal 90 Tim D Spector, Simon G Thompson haemorrhage, and only in gastric ulcers.'5 independently assessed "quality", derived from a J Epidemiol Community Health: first published as 10.1136/jech.45.2.89 on 1 June 1991. Downloaded from Although the vast majority of meta-analyses large number of predetermined "quality" concern the assessment of therapies in criteria.26 The pooled estimate can then be randomised controlled trials, a few studies have adjusted accordingly, or else the quality* score addressed contentious aetiological issues such as used to exclude studies. A simpler method for the quantification ofthe effect ofpassive smoking trials has been proposed which concentrates on on the risk of lung cancer,'6 alcohol in breast three areas of potential bias, namely treatment cancer,"7 the oral contraceptive pill in rheumatoid allocation by randomisation, inclusion of all arthritis,'8 and leukaemia in refinery workers.'9 randomised individuals in analysis, and the blindness of the outcome assessments.27 Quality Study design in meta-analysis assessments have also been used in With the proliferation of meta-analyses, it has epidemiological studies.'7 28 The major problem become apparent that their design, methods and with quality weighting is that it must remain publication should be conducted in a rigorous arbitrary and to an extent subjective. A single scientific manner, akin to that currently expected choice of weights is difficult to justify; for of randomised controlled trials. This is to allow example, is it worse to have poor blinding or poor critical appraisal of each individual meta-analysis randomisation? Moreover the procedure goes in terms of its methodology and therefore the against the general purpose of meta-analysis, that validity ofits conclusions. A meta-analysis should is to obtain an objective summary of the available be a research study in its own right. Specific a evidence. Because of the time and resources priori aims should be set out and a working needed to undertake full quality assessment, its protocol established. routine use cannot be recommended unless its Having defined the aims of the study, a true worth becomes established. thorough search of relevant publications needs to be performed. Computer searches have aided the Publication bias inclusion of large numbers of trials in published Publication bias is a potential problem in all meta-analyses. However, several studies have meta-analyses.29 30 It arises from the fact that shown that less than two thirds of relevant trials unpublished papers may contradict the findings are uncovered by computer searches.20 Therefore of the overview due to the overrepresentation of computer searches should be supplemented by published "positive" (ie, statistically significant) the bibliographies of textbooks, reviews, and the studies. There is now good evidence that negative studies themselves, and information from studies in medicine are less likely to be published specialists in the field. Where possible databases than positive ones.3' 32The likelihood ofthis bias of ongoing clinical trials should be consulted. altering the conclusions will depend on the In order to reduce bias, the inclusion of studies chances of the existence of important numbers of should be based on predetermined criteria. For unpublished papers. This is less likely to occur example in clinical trials, evidence of when the result is of considerable importance (eg, randomisation is usually regarded as crucial2"; in vitamin supplementation and neural tube some situations a minimum study size might be defects)33 or when the questions can only be desirable. Ideally all studies should be assessed in answered by large costly studies which are likely a blinded fashion by independent observers, to reach publication (eg, trials ofthrombolytics on although this is often difficult and impractical to cardiovascular mortality). http://jech.bmj.com/ perform. The decision to include studies should The question of publication bias needs to be consider whether treatments,