The Existence of Publication Bias and Risk Factors for Its Occurrence

Kay Dickersin, PhD

Publication bias is the tendency on the parts of investigators, reviewers, and lication, the bias in terms ofthe informa¬ editors to submit or accept manuscripts for publication based on the direction or tion available to the scientific communi¬ strength of the study findings. Much of what has been learned about publication ty may be considerable. The bias that is when of re¬ bias comes from the social sciences, less from the field of medicine. In medicine, created publication study sults is on the direction or three studies have direct evidence for this bias. Prevention of based signifi¬ provided publica- cance of the is called tion bias is both from the scientific dissemina- findings publica¬ important perspective (complete tion bias. This term seems to have been tion of knowledge) and from the perspective of those who combine results from a used first in the published scientific lit¬ number of similar studies (meta-analysis). If treatment decisions are based on erature by Smith1 in 1980. the published literature, then the literature must include all available data that is Even if publication bias exists, is it of acceptable quality. Currently, obtaining information regarding all studies worth worrying about? In a scholarly undertaken in a given field is difficult, even impossible. Registration of clinical sense, it is certainly worth worrying trials, and perhaps other types of studies, is the direction in which the scientific about. If one believes that judgments should move. about medical treatment should be community all (JAMA. 1990;263:1385-1389) made using good, available evidence, then one should insist that all evidence be made available. In reality, however, medical to have The human intellect. . .is more moved stitution is a case in Science de¬ decisions, date, mainly point. been the individual clinician's and excited than on accurate, and guided by by affirmatives by pends clear, precise and Re¬ negatives. wording in the descriptions ofwork per¬ training personal experience. Francis Bacon, 1621 formed and results obtained. It is im¬ cently, there has been a change in the have been The rise perative that there be only one possible way decisions made. EVER SINCE about 1450, the stan¬ interpretation of what is written. of consensus conferences, decision anal¬ dard method of imparting information Moreover, to advance, science de¬ ysis, expert systems, using clinical tri¬ and of has been on both in als as a basis for policy, and meta-analy- acquiring knowledge pends complete reporting, sis has decisions use of the written word. Even before terms of what experiments or studies propelled regarding medical toward a more scien¬ Gutenberg's printing press, the bene¬ were conducted and in terms of how an treatment fits of writing things down was recog¬ experiment or study was conducted. tific approach. nized on or even (Moses did not rely oral tradi¬ Practically, it is not possible PUBLICATION BIAS tion to and on the Ten desirable that or ev¬ interpret pass every experiment Historical Commandments correctly). The value ery element of an experiment be report¬ Aspects of the written tradition is manifold: it ed. Yet, there seem to be no established There seem to be no formal guidelines preserves, as in the recording of histori¬ standards by which an investi¬ in science as to when study results cal information about families or com¬ gator decides what is worth reporting: should or should not be published. The munities; it provides a basis for common the decision to report one's findings and decision as to what to include in a publi¬ understanding, as in lawmaking; and it the manner in which they are reported cation and whether to publish is largely provides the vehicle by which we share are a matter ofjudgment. personal, although dictated by the fash¬ information deemed to be important, as The question of how and when study ion of the times to a certain extent. in reporting the news of the day or the results are reported is of interest be¬ When Robert Boyle, the chemist, pub¬ latest scientific findings. cause of potential selection bias: given a lished his experiments on air in 1680, he In artistic endeavors, reinterpreta- set of characteristics about a study— was credited with being the first to re¬ tion of the written word, such as in design, operation, and outcome—could port the details of his experiments and translations, is acceptable, even wel¬ one predict the likelihood of publica¬ the precautions necessary for their rep¬ come. In other areas, the inherent ambi¬ tion? If one could, then that on which lication. This work ushered in a new guities of language lead to a constant our "knowledge" is based, the published type of report—one that described diffi¬ struggle to decipher the meaning or in¬ literature, is a biased representation of culties and errors. Thus, in the 1600s, tent of the written word. The US Con- knowledge as a whole. 1700s, and early 1800s, the usual scien¬ If the characteristics that determine tific report described not only the "posi¬ From the Department of Epidemiology, The Johns publication are related to study quality, tive" findings but also the "negative" or Hopkins University, School of Hygiene and Public then the selection bias incurred "nil" results. Health, Baltimore, Md. by Presented at The First International Congress on Peer studying only the published literature is Concerned about publication prac¬ Review in Biomedical Publication, Chicago, III, May 10\x=req-\ acceptable, even desirable. If, on the tices in the physical and life sciences, 12, 1989. other hand, the direction of re¬ lamented in 1661 that scientists Reprint requests to Department of Ophthalmology, study Boyle University of Maryland School of Medicine, 22 S Greene sults or the statistical significance ofthe did not write up single results but felt St, Baltimore, MD 21201 (Dr Dickersin). results is the reason for differential pub- compelled to refrain from publishing un-

Downloaded from jama.ama-assn.org at Johns Hopkins University on August 5, 2011 til they had a "system" worked out that Table 1 —Manuscript Ratings for Same Manuscript With Varying Presentations of Results or Discussion10 they deemed worthy of formal presen¬ tation: "But the worst inconvenience of Mean Ratings all is yet to be mentioned, and that No. of Data Scientific Publication is, Presentation Referees Methods That whilst this vanity of thinking men Presentation Contribution Merit Positive results to write either or noth¬ 4.3 obliged systems results ing is in many excellent notions Negative 2.4 2.6 2.4 1.8 request, Methods or experiments are, by sober and mod¬ only 4.5 3.4 "2 Mixed results, est .... men, suppressed Apparent¬ Positive discussion 13 2.5 1.3 1.6 0.5 ly, the notion ofgoing to press only if one Mixed results, has something "big" to present is not negative discussion 14 2.7 2.0 1.7 modern at all. By the mid-1800s, the style of scien¬ tific was in the of writing process chang¬ Table 2.—Studies of Publication Bias in Medicine ing to the terse, rather technical ap¬ proach with which we are familiar. Index Follow-up Limitations of time (as science began to Source, y Subject Source Method Results move quite rapidly), journal space, the Simes, Cancer Cancer Publications Published trials 1986 trials trials and register show increased development of groups of scientists register efficacy of and a written combined working together forging treatment document together, the response to Dickersin Randomized, File of Questionnaire Published trials peer review, and economic dependence etal, controlled randomized, favor test on a that rewarded quick suc¬ 1987" trials controlled treatment more system trials often cess were all factors that led to a change in scientific and The Sommer, Menstrual Society Questionnaire Published studies writing publishing. 1987" cycle membership more often change in style that has taken place over research statistically the years is not inherently bad. The significant is whether the increased brevi¬ Chalmers Perinatal ODPT* ODPT full Strength of results problem etal, trials abstracts reports in abstract not ty has resulted in lost information and 1989* associated with whether it represents biased reporting. full publication Evidence for Publication Bias *ODPT indicates Oxford Database of Perinatal Trials. Perhaps as a result of the difficulties ofdesigning studies to address the prob¬ another received a manuscript that de¬ known to exist and its etiology well un¬ lem, more has been written to complain scribed negative results. A third group derstood. "Investigators are more about publication bias than to report was asked to evaluate a manuscript on strongly motivated to offer positive re¬ results of studies undertaken to evalu¬ the basis of the "Methods" section and sults for publication rather than null re¬ ate it. Most research on publication bias relevance alone; no data were provided. sults. Many journal editors select pa¬ has been done in the psychology and The fourth and fifth groups received pers for publication on this very basis, education fields.3"10 manuscripts with "mixed" results, with some of them expecting to see P values Sterling3 was probably the first to em¬ either a positive or negative "Discus¬ less than 0.05. Published clinical trials phasize that the tendency to publish sion" section. The referees used a scale are inevitably a positively biased positive results and reject negative of 0 to 6 (low to high) to rate the manu¬ selection."12 findings is a serious problem. He re¬ scripts for five items: relevance, meth¬ Information regarding publishing viewed all articles published in four ods, data presentation, scientific contri¬ practices is not easily obtained or readi¬ journals during 1 year (1955 or 1956) and bution, and publication merit. The ly available; it is likely that this is the found that 97% of the articles that used referees' ratings are presented in Table reason so little research has been done. tests of significance rejected the null 1. Although studies with positive and One approach is to survey investigators hypothesis. Others in the social and be¬ negative results had identical "Meth¬ regarding their habits and experience. havioral sciences have found similar evi¬ ods" sections, referees rated the nega¬ The problems with this method are illus¬ dence for publication bias.4"9 tive results lower in the quality of meth¬ trated by a study by Hetherington et al13 Two experimental studies have been ods, as they did studies with mixed in which a one-page questionnaire re¬ done in this area (New York Times. Sep¬ results. Data presentation, scientific questing information regarding unpub¬ tember 27,1988)10; both found that when contribution, and publication merit lished perinatal trials was sent to ap¬ all other variables were held constant, scores were also scored lower when re¬ proximately 42 000 obstetricians and reviewers were highly influenced by the sults were negative or mixed. Negative pediatricians around the world. Of the direction and strength of the study re¬ studies received significantly lower 395 unpublished trials reported to the sults. The study by Mahoney10 is partic¬ scores for publication merit as well. investigators, only 18 were completed ularly illustrative of where biases may Little has been done to investigate more than 2 years before the survey. exist in the reviewing process. Seventy- the possibility of publication bias in the The rest were either ongoing (252) or five referees for one journal were ran¬ medical area.11 Despite the dearth of had ceased recruitment within 2 years domly assigned to receive one of five empirical evidence, it has been accepted of completion (125) and, thus, were con¬ similar manuscripts. All manuscripts as fact, rather than as a hypothesized sidered to be within the period needed were identical in the "Introduction" and problem in need of further study. Disre¬ for results to reach publication. Hether¬ "Methods" sections but varied in either garding the absence of good data, prom¬ ington et al concluded that it is not possi¬ the "Results" or "Discussion" sections. inent investigators have written arti¬ ble to estimate the size of publication One group of referees received a manu¬ cles in medical journals where they have bias by attempting to identify unpub¬ script that described positive results, referred to publication bias as if it is lished trials retrospectively.

Downloaded from jama.ama-assn.org at Johns Hopkins University on August 5, 2011 What data from the medical field sup¬ Table 3. Results of Published Randomized, Controlled Trials (RCTs) vs Results of Completed Unpublished port the notion that publication is relat¬ RCTs'8*- ed the direction and of to strength study Completed findings? Indirect evidence for publica¬ Published RCTst Unpublished RCTst tion bias has been several Trend or provided by Statistical % of Total With % of Total With studies.14"16 Chalmers reviewed 23 publi¬ Significance No. Trend Specified No. Trend Specified cations that provided fatality rates for Favors new serum hepatitis and found that reported therapy (P<05) 26 rates from 0.3% to 62%. The Trend favors new ranged 123 16.0 40 22.5 rates were associated with stud¬ therapy higher No difference ies that had smaller numbers of pa¬ between therapies 170 79 tients. The authors suggested that Trend favors control or these results may indicate an increased standard therapy 25 3.3 23 12.9 tendency on the part of the investiga¬ Favors control or standard tors to report unusual findings. therapy (P<.05)_ 3.4 10 5.6 Direct evidence for bias in Total No. of RCTs With publication Trend Specified 767 100.0 178 100.0 the medical area is shown in Table 2. No. of RCTs with trend of Simes17 compared results of published results not specified 274 26 trials and results of trials registered Total No. of RCTs 1041 204* with the International Cancer Research Data Bank. Trials were chosen for the 'Reprinted by permission of the publisher, copyright 1987 by Elsevier Science Publishing Co, Inc. tx2(4df) = 111-09;P<.001. therapeutic situations: (1) initial alkyl- tDoes not Include 34 completed, unpublished trials by one author. ating agents vs combination chemo¬ therapy for the treatment of advanced ovarian cancer and (2) alkylating agents Table 4.—Randomized, Controlled Trial (RCT) Status and Reasons for Not Publishing Completed RCTs18* or prednisone vs combination chemo¬ therapy for the treatment of multiple RCT Completed All trials were or myeloma. registered RCT RCT but Article Not and Article Response published before October 1983. The Status Stopped Submitted Submitted Blank Total No. (%) pooled results of the published trials of Article Intended, in in treatments for ovarian cancer demon¬ progress, or peer review 15 10 25 (12) strated a benefit statistically significant Results 16 35 of the combination while the negative 58 (28) therapy, Lack of 16 results of the interest 24 (12) pooled registered trials, Sample size problems 20 23 (11) which included some and published Poor methods 9 (4) some not show a unpublished trials, did Side effects 12 13 survival Similar¬ (6) significant advantage. External group problem 10 (5) a survival ad¬ ly, statistically significant Controversy 5 (2) was seen for combination ther¬ vantage Unknown or blank 26 1 37 (18) apy in the trials of treatment published Total No. (%) of RCTs 74 (36) 102 (50) 23 (11) 5 (2) 204f for myeloma, with a reduced, although still statistically significant, advantage »Reprinted by permission of the publisher, copyright 1987 by Elsevier Science Publishing Co, Inc. in the registered trials. fDoes not include 34 completed unpublished trials by one author. Additional evidence has come from a survey of 318 authors of published trials who were asked whether they had par¬ significant results. When only studies results published in abtracts are ulti¬ ticipated in any unpublished trials18 (Ta¬ that reported the statistical significance mately published in full. McCormick ble 3). The 156 respondents reported of the outcome were counted, these per¬ and Holmes23 found that pédiatrie ab¬ 271 unpublished and 1041 published tri¬ centages were 61%, 76%, and 40%, re¬ stracts submitted but not accepted for als. Completed unpublished trials fa¬ spectively. Sommer found that the best presentation reach subsequent full pub¬ vored the test therapy 14% of the time, predictor of publication status of the lication just 13% to 22% of the time, compared with 55% of the published tri¬ study was prior publication by the re¬ while 49% to 54% of selected abstracts als. The major reasons the authors gave sponding investigator. Investigators achieved full publication. This may be for not publishing were results not fa¬ with only one study under their belts an indication that the selected articles voring the test treatment and lack of tended not to publish (76%), while those represent studies of better quality, but interest (Table 4). It appears from the with two or more usually had one or it could just as easily indicate greater data that nonpublication resulted pri¬ more published studies (67%). Further¬ editorial interest in the findings pre¬ marily from a failure to write up trial more, if their first study was published, sented in selected articles. Chalmers et results rather than decisions on the part investigators were more likely to per¬ al24 followed up summary reports that of referees or editors. form subsequent studies (68%) than were contained in the Oxford Database Sommer19 surveyed all 140 members were investigators who had not pub¬ of Perinatal Trials and published be¬ of the Society for Menstrual Cycle Re¬ lished their first study (35%). tween 1940 and 1984. Search ofthe data¬ search and identified 73 published and In addition, several studies con¬ base, using authors' names, revealed 28 unpublished studies (response rate cerned with the complete publication of that approximately 37% were subse¬ was 67%). Thirty percent of 73 pub¬ studies initially published as abstracts quently published in full. Neither study lished studies, 38% of 42 reports in the have been reported.20"24 Data from these quality, as judged from the abstracts, publication pipeline, and 29% of the 28 studies have been remarkably consis¬ nor study results were associated with unpublished studies had statistically tent, showing that only 30% to 60% of final publication status. As this design

Downloaded from jama.ama-assn.org at Johns Hopkins University on August 5, 2011 Table 5.—Characteristics of Drug Trials by Publication Status27 random sample of nonpsychotropic in Finland were reviewed. For all Nonpsychotroplc Psychotropic drugs Drugs (n=69) Drugs (n = 234) years, 39% and 44% of the trials includ¬ Trial <-"- '-"- ed in in Finland and Swe¬ Characteristics Published Unpublished Published Unpublished applications were % den, respectively, not published. Controlled_47_26_47_52 related trials % "Good" Unpublished reports to of quality_23_46_35_37 more often con¬ % That had information psychotropic drugs regarding adverse tained information regarding patient effects_43_83_56_77 selection and exclusion criteria than did Mean sample size 62 48 83 76 published reports, although this was not so for unpublished studies of non¬ psychotropic drugs. Overall, the quality does not provide any information about scientific need for a publication of nega¬ of the published and unpublished re¬ the process between data analysis and tive or null-difference results is appar¬ ports seemed about equal. This study the decision to publish, the results do ent and the 'Journal of Negative Re¬ provides evidence that the quality of a not necessarily indicate the absence of sults' has been bandied about for many clinical trial funded by a pharmaceutical publication bias. years as an almost sick joke. Such a company is not a factor in publication It is difficult to estimate, even crude¬ journal would not only be decidedly dull decisions. the size of the of but also a financial ly, problem publication catastrophe."12 The Role of Size bias, given the available information. Maxwell12 went on to suggest that at Sample When data from investigations of the least editors should provide a register of Study size may play a part, either problem are used, the ratio of published negative results, listing the subject, au¬ directly or indirectly, in publication de¬ to unpublished studies ranges from thors' names and addresses, and title of cisions. The aforementioned study by 128:113 to 1:1," with the majority of the study, accessible by Index Medicus. me and my colleagues18 found that un¬ the ratios falling between 10:1 and This is an equally unsatisfactory solu¬ published trials performed by a specific 1:1.1,9'1S17"19 We are currently conducting tion to the problem. group of authors had a median sample a prospective study at The Johns Hop¬ Edward Huth, editor ofthe Annals of size of 24, whereas the "index" random¬ kins University, Baltimore, Md, that Internal Medicine, has stated that an ized, controlled trials (a sample of pub¬ should provide a better estimate of the electronic medical journal could be pub¬ lished randomized, controlled trials size of the problem. The project is de¬ lished at a lower cost than printed jour¬ used to generate the list of authors sur¬ signed to follow up studies approved in nals, and thus might be in a better posi¬ veyed) had a median sample size of 68. 1980 by institutional review boards at tion to publish negative or nil results. Chalmers and coworkers24 found that of our institution and clinical trials funded However, he warned, "it ought to be 176 abstracts that described perinatal by the National Institutes of Health, quite clear in the title or in the abstract trials, those with a sample size greater Bethesda, Md, in 1979, to see whether that the paper arrived at a negative con¬ than the median were more likely to be publication bias exists and, if so, what clusion, lest the authors or researchers published in full than those with a sam¬ the risk factors are. Potential risk fac¬ think they are getting positive data" ple size less than the median. (If finer tors are study design characteristics, (New York Times. April 29, 1986:C1, strata were used to categorize sample such as sample size, type of control C7). size, however, the association between group, and number of collaborating cen¬ The Journal ofthe American Medical sample size and publication was not sig¬ ters; investigator characteristics; fund¬ Association once had a section entitled nificant.) It is reassuring in some ways ing source; and strength of study "Negative Results" that a quick perusal that smaller studies may not be pub¬ findings. of JAMA volumes indicates was includ¬ lished as often, because sample size can ed as a somewhat be an indicator of a The Role of Journal Editors regular feature, ap¬ study's quality. proximately once a month, from 1962 However, small sample size may lead to How has publication bias come about? through 1968. The articles were 11/2 to an underpowered study that incorrectly There has been an assumption in medi¬ 2 pages long. Unfortunately, there is no fails to reject the null hypothesis. If cal literature that the bias for publishing information available regarding the ra¬ publication bias operates, the study striking results starts with the journal tionale for the start-up or continuation would go unpublished because of nil re¬ editors. There is some basis for this be¬ of this section of the journal, despite sults. These data support this hypothe¬ lief. The British Medical Journal stat¬ attempts to learn more (E. Knoll, PhD, sized continuum. ed in 1980 that their ideal article de¬ personal communication, March 1988). Berlin and Begg,28,29 in a review of 246 scribed "findings that will affect clinical published trials of treatments for can¬ The Role of Study Quality found a association between practice, . . . and findings in a common cer, strong disease that either improved prognosis So far, the potential role of the direc¬ sample size and treatment effect: stud¬ "26 or simplified management.... A tion of study findings on the publication ies with smaller sample sizes had larger 1983 piece in the "Views" section of the of results has been emphasized. There treatment effects. The trend is most same journal gave advice that "those are other potential risk factors for non- dramatic in randomized, as opposed to who seek rapid publication of a paper publication, most prominently quality, nonrandomized, studies. This implies (especially negative results)" should sample size, and the funding source of that small trials with large effects tend submit the article to a pay journal. They trials. to be published preferentially, while trials are be finished, ". . . as to how many people In 1980, Hemminki27 described the large likely to published will then see it . . . well, negative re¬ quality of information submitted to regardless of the outcome. sults have never made read¬ authorities and related riveting drug licensing The Role of Source ing."26 this to publication status (Table 5). Ap¬ Funding This situation has prompted discus¬ plications for the licensing of psychotro¬ Davidson30 reviewed 107 trials pub¬ sion as to whether it makes sense to pic drugs in Finland and Sweden for lished in 1984 and classified them based have a journal of negative results: "The 1965, 1970, 1974, and 1975 and for a on the direction of the results (favoring

Downloaded from jama.ama-assn.org at Johns Hopkins University on August 5, 2011 Table 6.—Direction of Results of Clinical Trials Published During 1984 in Selected Journals from the family/marital psychotherapy literature. Clin Psychol Rev. 1989;9:589-603. No. (%) 10. Mahoney MJ. Publication prejudices: an ex- perimental study of confirmatory bias in the peer Favoring Favoring review Ther Res. New Standard system. Cog 1977;1:161-175. 11. Berlin JA. Publication bias: a prob- Journal Treatment Treatment Total No. (%) Begg CB, lem in interpreting medical data. J R Stat Soc Se- Annals of Internal Medicine_12 (86)_2 (14)_14 (100) ries A. 1988;151(pt 3):419-463. Archives of Internal Medicine_11 (79)_3 (21)_14 (100) 12. Maxwell C. Clinical trials, reviews, and the lancet_30 (75)_10 (25)_40 (100) journal of negative results. Br J Clin Pharmacol. New Journal of (100) 1981;1:15-18. England Medicine_18 (67)_9 (33)_27 13. Hetherington J, Dickersin K, Chalmers I, American Journal of Medicine 5 (42) 7 (58) 12 (100) Meinert C. Retrospective and prospective identifi- cation of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediat- rics. 1989;84:374-380. 14. Chalmers TC, Koff RS, Grady GF. A note on fatality in serum hepatitis. Gastroenterology. new therapy vs favoring the standard on issues of quality and logical reasoning 1965;49:22-26. therapy) and the source of funding by the authors and not the direction and 15. Juhl E, Christensen E, Tygstrup N. The epide- (pharmaceutically supported vs "gener¬ strength of study results.33 Although miology of the gastrointestinal randomized clinical (71%) of this is the to the trial. NEngl J Med. 1977;296:20-22. ally" supported). Seventy-six simplest approach 16. Vandenbrouke JP. Passive smoking and lung the 107 trials favored the new of its univer¬ therapy problem publication bias, cancer: a publication bias? Br J Med. 1988;296:391\x=req-\ and 31 (29%) favored the standard ther¬ sal implementation is not likely to be 392. apy. Of those that favored the new ther¬ realized. 17. Simes RJ. The case for an international regis- ofclinical trials. J Clin Oncol. 33 (43%) were The most effective measure to try 1986;4:1529-1541. apy, pharmaceutically pre¬ 18. Dickersin K, Chan S, Chalmers TC, Sacks HS, supported, while of those that favored vent publication bias is the registration Smith H Jr. Publication bias and clinical trials. the traditional therapy, only 4 (13%) of all trials, perhaps all research stud¬ Controlled Clin Trials. 1987;8:343-353. were pharmaceutically supported. This ies, undertaken. Registers exist for 19. Sommer B. The file drawer effect and publica- translates into 89% (33/37) of the phar¬ several research areas,34 most notably tion rates in menstrual cycle research. Psychol and 61% the and Women Q. 1987;11:233-242. maceutically supported studies perinatal,35 cancer,36 acquired 20. Dudley HAF. Surgical research: master or ser- (43/70) of the generally supported stud¬ immunodeficiency syndrome37 fields. vant. Am J Surg. 1978;135:458-460. ies favoring the new treatment. The The Oxford Database of Perinatal Tri¬ 21. Goldman L, Loscalzo A. Fate of cardiology form. of articles that favored the als35 is one of the best and has research originally published in abstract N proportion developed Engl J Med. 1980;303:255-259. new vs the traditional varied as a basis for therapies been used methodological 22. Meranze J, Ellison N, Greenhow DE. Publica- considerably, depending on the journal research and hundreds of meta-an- tions resulting from anesthesia meeting abstracts. evaluated (Table 6). Davidson conclud¬ alyses.38 Although prospective trial reg¬ Anesth Analg. 1982;61:445-448. ed: "While it seems that con¬ istration is a considerable it is the 23. McCormick MC, Holmes JH. Publication of re- unlikely task, search at the AJDC. unfavorable re¬ presented pediatric meetings. spiracies to suppress logical imperative for the electronic age 1985;139:122-126. sults of clinical trials exist, a de facto in which we live. Prevention, not cure, 24. Chalmers I, Adams M, Dickersin K, et al. A exclusion of negative results may be oc¬ is the direction in which we should cohort study of summary reports of controlled tri- The of move. als. JAMA. 1990;263:1401-1404. curring." prospect conspiratorial ' of results has also been 25. 'The editor regrets Br Med J. 1980; suppression 280:508. Editorial. . . . raised31 and refuted.32 This work was supported by grant ROI HS 05523 26. Minerva. Br Med J. 1983;287:1886. Views. The that af¬ from the National Center for Health Services 27. Hemminki E. Study of information submitted possibility funding may Research/Health Care in are Technology Assessment, by drug companies to licensing authorities. Br Med fect the way which study results Rockville, Md. communicated extends the clini¬ J. 1980;280:833-836. beyond I thank Iain Chalmers, FRCOG, for reading 28. Berlin JA, Begg CB, Louis TA. An assessment cal trial setting.29 Because of concern several drafts of the manuscript critically and mak¬ of publication bias using a sample of published clini- about this issue and others having to do ing many helpful comments. cal trials. J Am Stat Assoc. 1989;84:381-392. with conflict of interest, full disclosure 29. Begg CB, Berlin JA. Publication bias and dis- References semination of clinical research. JNCI. 1989;81:107\x=req-\ of financial support is now required by 115. many journals, including JAMA, for all 1. Smith ML. Publication bias and meta-analysis. 30. Davidson RA. Source of funding and outcome published reports. Evaluation Educ. 1980;4:22-24. of clinical trials. J Gen Intern Med. 1986;1:155-158. 2. Hall MB. In defense of experimental essays. In: 31. Lauritsen K, Havelund T, Laursen LS, Rask\x=req-\ COMMENT Robert Boyle on Natural Philosophy. Blooming- Madsen J. Withholding unfavourable results in ton: Indiana University Press; 1965:119-131. drug company sponsored clinical trials. Lancet. It is probably not in our best interest 3. Sterling TD. Publication decisions and their pos- 1987;1:1091. to develop ways to "cure" the problem of sible effects on inferences drawn from tests of sig- 32. Nicholson PA. Information for drug trial par- bias. For retrieval nificance\p=m-\orvice versa. J Am Stat Assoc. 1959; ticipants. Lancet. 1987;2:396. publication example, 54:30-34. 33. Angell M. Negative studies. N Engl J Med. of unpublished data from trials requires 4. Smart RG. The importance ofnegative results in 1989;321:464-466. a great deal of effort and may not be psychological research. Can Psychol. 1964;5:225\x=req-\ 34. Dickersin K. Report from the panel on the case unbiased. Although this cure and oth¬ 232. for registers of clinical trials. Controlled Clin Tri- 5. Greenwald AG. of als. ers29 can useful additional infor¬ Consequences prejudice 1988;9:76-81. provide against the null hypothesis. Psychol Bull. 1975; 35. Chalmers I, Hetherington J, Newdick M, et al. mation for those evaluating the pub¬ 82:1-20. The Oxford Database of Perinatal Trials: develop- lished literature, they are not very good 6. White KR. The relation between socioeconomic ing a register of published reports of controlled remedies for publication bias. status and academic achievement. Psychol Bull. trials. Controlled Clin Trials. 1986;7:306-324. 36. Hubbard A measure somewhere between cure 1982;91:461-481. SM, Henney JE, DeVita VT Jr. A 7. Coursol A, Wagner EE. Effect of positive find- computer data base for information on cancer treat- and prevention is to insist that the scien¬ ings on submission and acceptance rates: a note on ment. NEngl J Med. 1987;316:315-318. tific community mend its ways. Investi¬ meta-analysis bias. Professional Psychol Res 37. Dutcher GA. NLM Technical Bull. Washing- gators should report the results of all Prac. 1986;17:136-137. ton, DC: US Dept of Health and Human Services; studies undertaken. Journal editors 8. Smith ML. Sex bias in counseling and psycho- 1989;243:17-27. therapy. Psychol Bull. 1980;87:392-407. 38. Chalmers I, Enkin M, Kierse M, eds. Effective should formalize editorial policy stating 9. Shadish WR, Doherty M, Montgomery LM. Care in Pregnancy and Childbirth. Oxford, En- that the decision to publish will be based How many studies in the file drawer? an estimate gland: Oxford University Press; 1989.

All in-text references underlinedDownloaded in blue from are jama.ama-assn.org linked to publications aton JohnsResearchGate, Hopkins letting University you access on August and read 5, 2011them immediately.