A Proposal for Reporting Meta-Analysis of Observational

Home , Epidemiology, Observational study

Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting

Donna F. Stroup; Jesse A. Berlin; Sally C. Morton; et al. Online article and related content current as of July 29, 2008. JAMA. 2000;283(15):2008-2012 (doi:10.1001/jama.283.15.2008)

http://jama.ama-assn.org/cgi/content/full/283/15/2008

Correction Contact me if this article is corrected. Citations This article has been cited 686 times. Contact me when this article is cited. Topic collections Statistics and Research Methods; Randomized Controlled Trial Contact me when new articles are published in these topic areas. Related Articles published in April 19, 2000 the same issue JAMA. 2000;283(15):2041. Related Letters Intensivist Consultation and Outcomes in Critically Ill Patients Peter J. Pronovost et al. JAMA. 2003;289(8):986.

Subscribe Email Alerts http://jama.com/subscribe http://jamaarchives.com/alerts

Permissions Reprints/E-prints [email protected] [email protected] http://pubs.ama-assn.org/misc/permissions.dtl

Downloaded from www.jama.com at Capes Consortia on July 29, 2008 CONSENSUS STATEMENT

Meta-analysis of Observational Studies in Epidemiology A Proposal for Reporting

Donna F. Stroup, PhD, MSc Objective Because of the pressure for timely, informed decisions in public health and Jesse A. Berlin, ScD clinical practice and the explosion of information in the scientific literature, research Sally C. Morton, PhD results must be synthesized. Meta-analyses are increasingly used to address this problem, and they often evaluate observational studies. A workshop was held in Atlanta, Ingram Olkin, PhD Ga, in April 1997, to examine the reporting of meta-analyses of observational studies G. David Williamson, PhD and to make recommendations to aid authors, reviewers, editors, and readers. Drummond Rennie, MD Participants Twenty-seven participants were selected by a steering committee, based on expertise in clinical practice, trials, statistics, epidemiology, social sciences, and biomedi- David Moher, MSc cal editing. Deliberations of the workshop were open to other interested scientists. Fund- Betsy J. Becker, PhD ing for this activity was provided by the Centers for Disease Control and Prevention. Theresa Ann Sipe, PhD Evidence We conducted a systematic review of the published literature on the con- duct and reporting of meta-analyses in observational studies using MEDLINE, Educa- Stephen B. Thacker, MD, MSc tional Research Information Center (ERIC), PsycLIT, and the Current Index to Statistics. for the Meta-analysis Of We also examined reference lists of the 32 studies retrieved and contacted experts in Observational Studies in the field. Participants were assigned to small-group discussions on the subjects of bias, Epidemiology (MOOSE) Group searching and abstracting, heterogeneity, study categorization, and statistical methods. Consensus Process From the material presented at the workshop, the authors developed a checklist summarizing recommendations for reporting meta-analyses of ob- ECAUSE OF PRESSURE FOR TIMELY servational studies. The checklist and supporting evidence were circulated to all confer- and informed decisions in pub- ence attendees and additional experts. All suggestions for revisions were addressed. lic health and medicine and the Conclusions The proposed checklist contains specifications for reporting of meta- explosion of information in the analyses of observational studies in epidemiology, including background, search strat- Bscientific literature, research results must egy, methods, results, discussion, and conclusion. Use of the checklist should improve be synthesized to answer urgent ques- the usefulness of meta-analyses for authors, reviewers, editors, readers, and decision tions.1-4 Principles of evidence-based makers. An evaluation plan is suggested and research areas are explored. methods to assess the effectiveness of JAMA. 2000;283:2008-2012 www.jama.com health care interventions and set policy are cited increasingly.5 Meta-analysis, a studies are available.8 Here, we define an Author Affiliations: Centers for Disease Control and systematic approach to identifying, ap- observational study as an etiologic or ef- Prevention, Atlanta, Ga (Drs Stroup, Williamson, and Thacker); University of Pennsylvania School of Medi- praising, synthesizing, and (if appropri- fectiveness study using data from an ex- cine, Philadelphia (Dr Berlin); RAND Corporation, Santa ate) combining the results of relevant isting database, a cross-sectional study, Monica (Dr Morton), University of California, San Fran- cisco (Dr Rennie), Stanford University, Stanford (Dr studies to arrive at conclusions about a a case series, a case-control design, a de- Olkin), Calif; JAMA, Chicago, Ill (Dr Rennie); Thomas body of research, has been applied with sign with historical controls, or a co- C. Chalmers Centre for Systematic Reviews, Chil- 9 dren’s Hospital of Eastern Ontario Research Insti- increasing frequency to randomized con- hort design. Observational designs may tute, Ottawa (Mr Moher); Michigan State Univer- trolled trials (RCTs), which are consid- lack the experimental element of a ran- sity, East Lansing (Dr Becker); and Georgia State ered to provide the strongest evidence dom allocation to an intervention and University, Atlanta (Dr Sipe). 6,7 A complete list of members of the MOOSE Group ap- regarding an intervention. rely on studies of association between pears at the end of this article. However, in many situations random- changes or differences in 1 characteris- Corresponding Author and Reprints: Donna F. Stroup, PhD, MSc, Centers for Disease Control and Preven- ized controlled designs are not fea- tic (eg, an exposure or intervention) and tion, 1600 Clifton Rd NE, Mail Stop C08, Atlanta, GA sible, and only data from observational changes or differences in an outcome of 30333 (e-mail: [email protected]).

Downloaded from www.jama.com at Capes Consortia on July 29, 2008 REPORTING META-ANALYSES OF OBSERVATIONAL STUDIES interest. These designs have long been checklist of items for reporting that methods, results, discussion, and con- used in the evaluation of educational builds on similar activities for RCTs22 clusions (TABLE). programs10 and exposures that might and is intended for use by authors, re- cause disease or injury.11 Studies of risk viewers, editors, and readers of meta- Background factors generally cannot be random- analyses of observational studies. Reporting of the background should ized because they relate to inherent hu- include the definition of the problem man characteristics or practices, and ex- METHODS under study, statement of hypothesis, posing subjects to harmful risk factors We conducted a systematic review of description of the study outcome(s) is unethical.12 At times, clinical data may the published literature on the con- considered, type of exposure or inter- be summarized in order to design a ran- duct and reporting of meta-analyses vention used, type of study design used, domized comparison.13 Observational in observational studies. Databases and complete description of the study data may also be needed to assess the searched included MEDLINE, Educa- population. When combining observa- effectiveness of an intervention in a tional Resources Information Center, tional studies, heterogeneity of popu- community as opposed to the special PsycLIT (http://www.wesleyan.edu lations (eg, US vs international stud- setting of a controlled trial.14 Thus, a /libr), and the Current Index to Statis- ies), design (eg, case-control vs cohort clear understanding of the advantages tics. In addition, we examined refer- studies), and outcome (eg, different and limitations of statistical syntheses ence lists and contacted experts in the studies yielding different relative risks of observational data is needed.15 field. We used the 32 articles retrieved that cannot be accounted for by sam- Although meta-analysis restricted to to generate the conference agenda and pling variation) is expected.8 RCTs is usually preferred to meta- set topics of bias, searching and ab- analyses of observational studies,16-18 the stracting, heterogeneity, study catego- Search number of published meta-analyses rization, and statistical methods. We in- Reporting of the search strategy should concerning observational studies in vited experts in meta-analysis from the include qualifications of the search- health has increased substantially dur- fields of clinical practice, trials, statis- ers, specification of databases used, ing the past 4 decades (678 in 1955- tics, epidemiology, social sciences, and search strategy and index terms, use of 1992, 525 in 1992-1995, and more than biomedical editing. any special features (eg, “explosion”), 400 in 1996 alone).19 The workshop included an overview search software used, use of hand While guidelines for meta-analyses of the quality of reporting of meta- searching and contact with authors, use have been proposed, many are written analyses in education and the social of materials in languages other than En- from the meta-analyst’s (author’s) rather sciences. Plenary talks were given on the glish, use of unpublished material, and than from the reviewer’s, editor’s, or topics set by the conference agenda. For exclusion criteria used. Published re- reader’s perspective20 and restrict at- each of 2 sessions, workshop partici- search shows that use of electronic da- tention to reporting of meta-analyses of pants were assigned to 1 of 5 small dis- tabases may find only half of all rel- RCTs.21,22 Meta-analyses of observa- cussion groups, organized around the evant studies, and contacting authors tional studies present particular chal- topic areas. For each group, 1 of the may be useful,27 although this result lenges because of inherent biases and authors served as facilitator, and a may not be true for all topic areas.28 differences in study designs23; yet, they recorder summarized points of discus- For example, a meta-analysis of de- may provide a tool for helping to un- sion for issues to be presented to all pression in elderly medical inpatients29 derstand and quantify sources of vari- participants. Time was provided for the used 2 databases for the search. In ability in results across studies.24 2 recorders and 2 facilitators for each addition, bibliographies of retrieved We describe here the results of a topic to meet and prepare plenary pre- papers were searched. However, the au- workshop held in Atlanta, Ga, in April sentations given to the entire group. thors did not report their search strat- 1997, to examine concerns regarding the We proposed a checklist for meta- egy in enough detail to allow replica- reporting of Meta-analysis Of Observa- analyses of observational studies based tion. An example of a thorough “reject tional Studies in Epidemiology on the deliberation of the independent log” can be found in the report of a meta- (MOOSE). This article summarizes de- groups. Finally, we circulated the check- analysis of electrical and magnetic field liberations of 27 participants (the list for comment to all conference attend- exposure and leukemia.30 Examples of MOOSE group) of evidence leading to ees and representatives of several con- a table characterizing studies included recommendations regarding the report- stituencies who would use the checklist. can be found in Franceschi et al31 and ing of meta-analyses. Meta-analysis of in- Saag et al.32 Complete specification of dividual-level data from different stud- RESULTS search strategy is not uniform; a review ies, sometimes called “pooled analysis” The checklist resulting from work- of 103 published meta-analyses in edu- or “meta-analysis of individual patient group deliberations is organized cation showed that search procedures data,”25,26 has unique challenges that we around recommendations for report- were described inadequately in the ma- will not address here. We propose a ing background, search strategy, jority of the articles.10

Downloaded from www.jama.com at Capes Consortia on July 29, 2008 REPORTING META-ANALYSES OF OBSERVATIONAL STUDIES

Methods tional studies is controversial, as it is for also recommend subgroup or sensitiv- Items in this checklist section are con- RCTs,16,33 because scores constructed in ity analysis rather than using quality cerned with the appropriateness of any an ad hoc fashion may lack demon- scores as weights in the analysis.37,38 quantitative summary of the data; de- strated validity, and results may not be While some control over heteroge- gree to which coding of data from the ar- associated with quality.34 Nevertheless, neity of design may be accomplished ticles was specified and objective; assess- some particular aspects of study qual- through the use of exclusion rules, we ment of confounding, study quality, and ity have been shown to be associated recommend using broad inclusion cri- heterogeneity; use of statistical meth- with effect: eg, adequate concealment of teria for studies, and then performing ods; and display of results. Empirical evi- allocation in randomized trials.35 Thus, analyses relating design features to out- dence shows that reporting of proce- key components of design, rather than come.8 In cases when heterogeneity of dures for classification and coding and aggregate scores themselves, may be im- outcomes is particularly problematic, quality assessment is often incomplete: portant. For example, in a study of blind- a single summary measure may well be fewer than half of the meta-analyses re- ing (masking) of readers participating inappropriate.39 Analyses that stratify ported details of classifying and coding in meta-analyses, masking essentially by study feature or regression analysis the primary study data, and only 22% as- made no difference in the summary with design features as predictors can sessed quality of the primary studies.10 odds ratios across the 5 meta-analy- be useful in assessing whether study We recognize that the use of quality ses.36 We recommend the reporting of outcomes indeed vary systematically scoring in meta-analyses of observa- quality scoring if it has been done and with these features.40 Investigating heterogeneity was a key Table. A Proposed Reporting Checklist for Authors, Editors, and Reviewers of Meta-analyses feature of a meta-analysis of observa- of Observational Studies tional studies of asbestos exposure and Reporting of background should include risk of gastrointestinal cancer.41 The au- Problem definition Hypothesis statement thors of the meta-analysis hypoth- Description of study outcome(s) esized that studies allowing for a latent Type of exposure or intervention used period between the initiation of expo- Type of study designs used Study population sure and any increases in risk should Reporting of search strategy should include show, on average, appropriately higher Qualifications of searchers (eg, librarians and investigators) Search strategy, including time period included in the synthesis and keywords standardized mortality ratios than stud- Effort to include all available studies, including contact with authors ies that ignored latency. In other words, Databases and registries searched Search software used, name and version, including special features used (eg, explosion) the apparent effect of exposure would Use of hand searching (eg, reference lists of obtained articles) be attenuated by including the latent List of citations located and those excluded, including justification period in the calculation of time at risk Method of addressing articles published in languages other than English Method of handling abstracts and unpublished studies (the “denominator”), since exposure- Description of any contact with authors related deaths (the “numerator”) would, Reporting of methods should include Description of relevance or appropriateness of studies assembled for assessing the hypothesis by definition, not occur during that la- to be tested tent period (FIGURE). Rationale for the selection and coding of data (eg, sound clinical principles or convenience) Documentation of how data were classified and coded (eg, multiple raters, blinding, and In fact, the data suggested that stud- interrater reliability) ies allowing for latent periods found Assessment of confounding (eg, comparability of cases and controls in studies where appropriate) on average somewhat higher standard- Assessment of study quality, including blinding of quality assessors; stratification or regression ized mortality ratios than studies on possible predictors of study results ignoring latency. This example shows Assessment of heterogeneity Description of statistical methods (eg, complete description of fixed or random effects models, that sources of bias and heterogeneity justification of whether the chosen models account for predictors of study results, can be hypothesized prior to analysis dose-response models, or cumulative meta-analysis) in sufficient detail to be replicated Provision of appropriate tables and graphics and subsequently confirmed by the Reporting of results should include analysis. Graphic summarizing individual study estimates and overall estimate Table giving descriptive information for each study included Results of sensitivity testing (eg, subgroup analysis) Results Indication of statistical uncertainty of findings Recommendations for reporting of re- Reporting of discussion should include Quantitative assessment of bias (eg, publication bias) sults include graphical summaries of Justification for exclusion (eg, exclusion of non–English-language citations) study estimates and any combined es- Assessment of quality of included studies Reporting of conclusions should include timate, a table listing descriptive infor- Consideration of alternative explanations for observed results mation for each study, results of sen- Generalization of the conclusions (ie, appropriate for the data presented and within the domain of the literature review) sitivity testing and any subgroup Guidelines for future research analysis, and an indication of statisti- Disclosure of funding source cal uncertainty of findings.

Downloaded from www.jama.com at Capes Consortia on July 29, 2008 REPORTING META-ANALYSES OF OBSERVATIONAL STUDIES

Discussion synthesis, while indicating when more Figure. Effect of Latent Period on The discussion should include issues re- research is necessary. Heterogeneity lated to bias, including publication bias, The application of formal meta- confounding, and quality. Bias can oc- analytic methods to observational stud- Latent Period Time-at-Risk ies has been controversial.42 One reason cur in the original studies (resulting from Exposure-Related Deaths Exposure-Related flaws in the study design that tend to dis- for this has been that potential biases in Do Not Occur Deaths Occur tort the magnitude or direction of asso- the original studies, relative to the biases Start of At-Risk Period Begins in RCTs, make the calculation of a single Exposure ciations in the data) or from the way in Time which studies are selected for inclu- summary estimate of effect of exposure sion.42 Publication bias, the selective potentially misleading. Similarly, the publication of studies based on the mag- extreme diversity of study designs and list, to ensure its usefulness to journal nitude (usually larger) and direction of populations in epidemiology makes the reviewers and editors. The US Food and their findings, represents a particular interpretation of simple summaries prob- Drug Administration (FDA) receives threat to the validity of meta-analysis of lematic, at best. In addition, methodo- and reviews petitions and applica- observational studies.43-45 Thorough logic issues related specifically to meta- tions for approval of regulated prod- specifications of quality assessment can analysis, such as publication bias, could ucts and/or their labeling. The FDA’s contribute to understanding some of the have particular impact when combin- Center for Food Safety and Applied Nu- variations in the observational studies ing results of observational studies.44,47 trition is now receiving applications that themselves. Methods should be used to Despite these challenges, meta- use results of meta-analyses in sup- aid in the detection of publication bias, analyses of observational studies con- port of the requested action. The re- eg, fail-safe procedures or funnel plots.46 tinue to be one of the few methods for vised checklist should be tested dur- Schlesselman47 comments on such bi- assessing efficacy and effectiveness and ing the review of an application. One ases in assessing the possible associa- are being published in increasing num- might randomly assign FDA review- tion between endometrial cancer and bers. Our goal is to improve the report- ers who encounter systematic reviews oral contraceptives. This meta-anal- ing of these meta-analyses so that read- of observational studies to use the ysis combined both cohort and case- ers can understand what was done in a checklist or not. Since the require- control studies and used a sensitivity given analysis, who did it, and why it was ments for reporting for regulatory pur- analysis to illustrate the influence of done. If bias is a problem, we suggest that poses might not completely coincide specific studies, such as those pub- an informative approach is to use broad with those in the checklist and since lished in English. inclusion criteria for studies and then to sample size (the number of formal sys- perform analyses (when the data per- tematic reviews received by the FDA) Conclusion mit) relating suspected sources of bias might be small, this evaluation should Due to these biases in observational and variability to study findings. document any potential incompatibil- studies, the conclusion of the report Methodologic and interpretational ity between requirements for regula- should contain consideration of alter- concerns make the clear and thorough tory reporting and the checklist. native explanations for observed re- reporting of meta-analyses of observa- Second, we will work with the Coch- sults and appropriate generalizations of tional studies absolutely essential. Our rane Collaboration to promote the use the conclusion. A carefully conducted workshop was convened to address the of these recommendations by Coch- meta-analysis can reveal areas warrant- problem of increasing diversity and vari- rane collaborative review groups.49 ing further research. Finally, since fund- ability that exist in reporting meta- Members of the Cochrane Collabora- ing source has been shown to be an im- analyses of observational studies. In tion are involved routinely in perform- portant source of heterogeneity,48 the constructing the checklist, we have ing systematic reviews. Some are now sponsoring organization should be dis- attempted, where possible, to provide incorporating nonrandomized studies closed and any effect on analysis should references to literature justifying the out of necessity. A trial of use of the be examined. inclusion of particular items. checklist could be compared with the Assessment of the usefulness of rec- FDA experience. COMMENT ommendations for reporting is depen- Third, an evaluation of the check- Taking stock of what is known in any dent on a well-designed and effec- list by authors, reviewers, readers, and field involves reviewing the existing lit- tively conducted evaluation. The editors could compare objective mea- erature, summarizing it in appropri- workshop participants proposed a sures of the quality of articles written ate ways, and exploring the implica- 3-pronged approach to determine use- with and without the formal use of the tions of heterogeneity of population and fulness and implementation of these guidelines. A challenge to the use of study for heterogeneity of study re- recommendations. quality measures would be arriving at sults. Meta-analysis provides a system- First, further comments should be in- a valid measure of quality. A more im- atic way of performing this research corporated into revisions of the check- portant end point for trials in journals

Downloaded from www.jama.com at Capes Consortia on July 29, 2008 REPORTING META-ANALYSES OF OBSERVATIONAL STUDIES is process measures. Questions of in- Permanente Medical Care Program, Pasadena; Dun- analysis using individual patient data for ovarian can- can Saunders, MBBCh, PhD, University of Alberta, Ed- cer studies. Am J Epidemiol. 1997;145:1917-1925. terest include whether the use of the monton; and Stephen D. Walter, PhD, McMaster Uni- 27. McManus RJ, Wilson S, Delaney BC, et al. Re- checklist makes preparation and evalu- versity, Hamilton, Ontario, for their advice and view of the usefulness of contacting other experts when substantive comments on an earlier draft of this article. conducting a literature search for systematic reviews. ation of manuscripts easier or is oth- In addition, we thank Barbara McDonnell, CDC, for BMJ. 1998;317:1562-1563. erwise helpful. Again, defining the managing complex logistics of this project. 28. Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of un- constructs of interest present crucial published controlled trials. Pediatrics. 1989;84:374-380. challenges to this research. REFERENCES 29. Cole MG, Bellavance F. Depression in elderly medi- Less formal evaluations, based on cal inpatients. CMAJ. 1997;157:1055-1060. 1. Greenland S. Quantitative methods in the review of 30. Kheifets LI, Afifi AA, Buffler PA, et al. Occupa- comments from users in any of the epidemiologic literature. Epidemiol Rev. 1987;9:1-30. tional electric and magnetic field exposure and leu- above groups, would certainly be help- 2. Chalmers TC, Lau J. Meta-analytic stimulus for kemia. J Occup Environ Med. 1997;39:1074-1091. changes in clinical trials. Stat Methods Med Res. 1993; 31. Franceschi S, La Vecchia C, Talamini R. Oral con- ful, as well. One would need to be con- 2:161-172. traceptives and cervical neoplasia. Tumori. 1986;72: cerned about contamination of the 3. Badgett RG, O’Keefe M, Henderson MC. Using sys- 21-30. tematic reviews in clinical education. Ann Intern Med. 32. Saag KG, Criswell LA, Sems KM, et al. Low-dose control groups when evaluating the 1997;126:886-891. corticosteroids in rheumatoid arthritis. Arthritis Rheum. checklist, as journals, for example, 4. Ohlsson A. Systematic reviews. Scand J Clin Lab 1996;39:1818-1825. Invest Suppl. 1994;219:25-32. 33. Emerson JD, Burdick E, Hoaglin DC, et al. An em- might adopt the checklist even in the 5. Bero LA, Jadad AR. How consumers and policy- pirical study of the possible relation of treatment dif- absence of evidence of its efficacy from makers can use systematic reviews for decision mak- ferences to quality scores in controlled randomized clini- ing. Ann Intern Med. 1997;127:37-42. cal trials. Control Clin Trials. 1990;11:339-352. randomized trials. 6. Thacker SB. Meta-analysis. JAMA. 1988;259:1685- 34. Ju¨ ni P, Witschi A, Bloch R, Egger M. The hazards 1689. of scoring the quality of clinical trials for meta- In conclusion, the conference partici- 7. Meta-Analysis, Decision Analysis, and Petitti D. analysis. JAMA. 1999;282:1054-1060. Cost Effectiveness Analysis. New York, NY: Oxford pants noted that meta-analyses are them- 35. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Em- University Press; 1994. selves observational studies, even when pirical evidence of bias. JAMA. 1995;273:408-412. 8. Berlin JA. Invited commentary. Am J Epidemiol. 50 36. Berlin JA, for the University of Pennsylvania Meta- applied to RCTs. If a role for meta- 1995;142:383-387. analysis Blinding Study Group. Does blinding of read- 9. Peipert JF, Phipps MG. Observational studies. Clin analyses of observational studies in set- ers affect the results of meta-analyses? Lancet. 1997; Obstet Gynecol. 1998;41:235-244. 51 350:185-186. ting policy is to be achieved, stan- 10. Sipe TA, Curlette WL. A meta-synthesis of fac- 37. Hasselblad V, Eddy DM, Kotchmar DJ. Synthesis tors related to educational achievement. Int J Educ Res. dards of reporting must be maintained of environmental evidence. J Air Waste Manag As- 1997;25:583-598. soc to allow proper evaluation of the qual- 11. Ioannidis JP, Lau J. Pooling research results. Jt . 1992;42:662-671. ity and completeness of meta-analyses. Comm J Qual Improv. 1999;25:462-469. 38. Friedenreich CM, Brant RF, Riboli E. Influence of 12. Lipsett M, Campleman S. Occupational expo- methodologic factors in a pooled analysis of 13 case- The Meta-analysis Of Observational Studies in Epi- sure to diesel exhaust and lung cancer: a meta- control studies of colorectal cancer and dietary fiber. demiology (MOOSE) Group: Centers for Disease Con- analysis. Am J Public Health. 1999;89:1009-1017. Epidemiology. 1994;5:66-67. trol and Prevention, Atlanta, Ga: Gary Jeng, PhD, Rob 13. Vickers A, Cassileth B, Ernst E, et al. How should 39. Berlin JA, Rennie D. Measuring the quality of tri- Lyerla, PhD, Thomas Peterman, MD, Donna F. Stroup, we research unconventional therapies? Int J Technol als. JAMA. 1999;282:1083-1085. PhD, MSc, Stephen B. Thacker, MD, MSc, and G. David Assess Health Care. 1997;13:111-121. 40. Colditz GA, Burdick E, Mosteller F. Heterogene- Williamson, PhD; University of Pennsylvania School 14. Mann CC. Can meta-analysis make policy? Sci- ity in meta-analysis of data from epidemiologic studies. Am J Epidemiol. 1995;142:371-382. of Medicine, Philadelphia: Jesse A. Berlin, ScD; RAND ence. 1999;266:960-962. 41. Frumkin H, Berlin J. Asbestos exposure and gas- Corporation, Santa Monica, Calif: Sally C. Morton, PhD; 15. Blettner M, Sauerbrei W, Schlehofer B, et al. Tra- trointestinal malignancy review and meta-analysis Stanford University, Stanford, Calif: Ingram Olkin, PhD; ditional reviews, meta-analyses and pooled analyses [published correction appears in Am J Ind Med. 1988; University of California, San Francisco, and JAMA, Chi- in epidemiology. Int J Epidemiol. 1999;28:1-9. 14:493]. Am J Ind Med. 1988;14:79-95. cago, Ill: Drummond Rennie, MD; Thomas C. Chal- 16. Greenland S. Invited commentary. Am J Epide- 42. Blettner M, Sauerbrei W, Schlehofer B, et al. Tra- mers Centre for Systematic Reviews, Children’s Hos- miol. 1994;140:290-296. ditional reviews, meta-analyses and pooled analyses pital of Eastern Ontario Research Institute, Ottawa: 17. Lau J, Ioannidis JP, Schmid CH. Summing up evi- in epidemiology. Int J Epidemiol. 1999;28:1-9. David Moher, MSc; Michigan State University, East dence. Lancet. 1998;351:123-127. 43. Rosenthal R. The file drawer problem and toler- Lansing: Betsy J. Becker, PhD; Georgia State Univer- 18. Shapiro S. Meta-analysis/shmeta-analysis. Am J Epidemiol. 1994;140:771-778. ance for null results. Psychol Bull. 1979;86:638-641. sity, Atlanta: Theresa Ann Sipe, PhD; Centre for Sta- 44. Easterbrook PJ, Berlin JA, Gopalan R, Matthews tistics in Medicine, Oxford, England: Douglas Alt- 19. Stroup DF, Thacker SB, Olson CM, Glass RM. Characteristics of meta-analyses submitted to a medi- DR. Publication bias in clinical research. Lancet. 1991; man, PhD; Memorial Sloan-Kettering Cancer Center, 337:867-872. New York, NY: Colin Begg, PhD; Hamilton Went- cal journal. From: International Congress on Biomedi- cal Peer Review and Global Communications; Sep- 45. Dickersin K, Min YI. NIH clinical trials and publi- worth Regional Public Health Department, Hamil- tember 17-21, 1997; Prague, Czech Republic. cation bias. Online J Curr Clin Trials [serial online]. ton, Ontario: Larry Chambers, PhD; Harvard Medi- 20. Lang TA, Secic M. How to Report Statistics in 1993 Apr 28: Doc No 50. cal School and Brigham and Women’s Hospital, Boston, Medicine. Philadelphia, Pa: American College of Phy- 46. Hedges LV, Olkin I. Statistical Methods for Meta- Mass: Graham Colditz, PhD; University of Maryland, sicians; 1997. analysis. Boston, Mass: Academic Press; 1985. Baltimore: Kay Dickersin, PhD; AT&T Labs, Murray Hill, 21. Cook DJ, Sackett DL, Spitzer WO. Methodo- 47. Schlesselman JJ. Risk of endometrial cancer in re- NJ: William DuMouchel, PhD; University of Colo- logic guidelines for systematic reviews of random- lation to use of combined oral contraceptives. Hum rado, Denver: Karen Kafadar, PhD; Cleveland, Ohio: ized control trials in health care from the Potsdam con- Reprod. 1997;12:1851-1863. Tom Lang, MA; Food and Drug Administration, Wash- sultation on meta-analysis. J Clin Epidemiol. 1995; 48. Jadad A, Sullivan C, Luo D, et al. Patients’ pref- ington, DC: Lynn Larsen, PhD; University of Minne- 48:167-171. erences for Turbuhaler or pressurized metered dose sota, Minneapolis: Thomas A. Louis, PhD; University 22. Moher D, Cook DJ, Eastwood S, et al. Improving inhalers (pMDIs) in the treatment. From Annual Meet- of British Columbia, Vancouver: Parminder Raina, PhD; the quality of reports of meta-analyses of randomised ing of the American Academy of Allergy, Asthma, and University of Pittsburgh, Pittsburgh, Pa: Allan Samp- controlled trials. Lancet. 1999;354:1896-1900. Immunology; March 3-8, 2000; San Diego, Calif. son, PhD; Family Health International, Research Tri- 23. Huston P. Health services research. CMAJ. 1996; 49. Huston P. Cochrane Collaboration helping un- angle Park, NC: Ken Schulz, PhD, MSc; and Institute 155:1697-1702. ravel tangled web woven by international research. of Medicine, Washington, DC: Mike Stoto, PhD. 24. Egger M, Scheider M, Davey-Smith G. Meta- CMAJ. 1996;154:1389-1392. Acknowledgment: We thank Susan Eastwood, MA, analysis. BMJ. 1998;316:140-144. 50. Moher D, Pham B, Jones A, et al. Does the qual- University of California, San Francisco; Christine 25. Stewart LA, Parmar MK. Meta-analysis of the lit- ity of reports of randomised trials affect estimates of Friedenreich, PhD, Alberta Cancer Board and Univer- erature or of individual patient data? Lancet. 1993; intervention efficacy reported in meta-analyses? Lan- sity of Calgary, Calgary; Sander Greenland, DrPH, Cstat, 341:418-422. cet. 1998;352:609-613. MA, MS, University of California, Los Angeles; Rich- 26. Steinberg K, Smith SF, Lee N, et al. Comparison 51. Berlin JA, Colditz GA. The role of meta-analysis ard Horton, MBBCh, MD, Lancet, London, England; of effect estimates from a meta-analysis of summary in the regulatory process for foods, drugs, and de- Diana Petitti, MD, MPH, Southern California Kaiser data from published studies and from a meta- vices. JAMA. 1999;281:830-834.

Downloaded from www.jama.com at Capes Consortia on July 29, 2008