The Existence of Publication Bias and Risk Factors for Its Occurrence Kay Dickersin, PhD Publication bias is the tendency on the parts of investigators, reviewers, and lication, the bias in terms ofthe informa¬ editors to submit or accept manuscripts for publication based on the direction or tion available to the scientific communi¬ strength of the study findings. Much of what has been learned about publication ty may be considerable. The bias that is when of re¬ bias comes from the social sciences, less from the field of medicine. In medicine, created publication study sults is on the direction or three studies have direct evidence for this bias. Prevention of based signifi¬ provided publica- cance of the is called tion bias is both from the scientific dissemina- findings publica¬ important perspective (complete tion bias. This term seems to have been tion of knowledge) and from the perspective of those who combine results from a used first in the published scientific lit¬ number of similar studies (meta-analysis). If treatment decisions are based on erature by Smith1 in 1980. the published literature, then the literature must include all available data that is Even if publication bias exists, is it of acceptable quality. Currently, obtaining information regarding all studies worth worrying about? In a scholarly undertaken in a given field is difficult, even impossible. Registration of clinical sense, it is certainly worth worrying trials, and perhaps other types of studies, is the direction in which the scientific about. If one believes that judgments should move. about medical treatment should be community all (JAMA. 1990;263:1385-1389) made using good, available evidence, then one should insist that all evidence be made available. In reality, however, medical to have The human intellect. .is more moved stitution is a case in Science de¬ decisions, date, mainly point. been the individual clinician's and excited than on accurate, and guided by by affirmatives by pends clear, precise and Re¬ negatives. wording in the descriptions ofwork per¬ training personal experience. Francis Bacon, 1621 formed and results obtained. It is im¬ cently, there has been a change in the have been The rise perative that there be only one possible way decisions made. EVER SINCE about 1450, the stan¬ interpretation of what is written. of consensus conferences, decision anal¬ dard method of imparting information Moreover, to advance, science de¬ ysis, expert systems, using clinical tri¬ and of has been on both in als as a basis for policy, and meta-analy- acquiring knowledge pends complete reporting, sis has decisions use of the written word. Even before terms of what experiments or studies propelled regarding medical toward a more scien¬ Gutenberg's printing press, the bene¬ were conducted and in terms of how an treatment fits of writing things down was recog¬ experiment or study was conducted. tific approach. nized on or even (Moses did not rely oral tradi¬ Practically, it is not possible PUBLICATION BIAS tion to and on the Ten desirable that or ev¬ interpret pass every experiment Historical Commandments correctly). The value ery element of an experiment be report¬ Aspects of the written tradition is manifold: it ed. Yet, there seem to be no established There seem to be no formal guidelines preserves, as in the recording of histori¬ standards by which an investi¬ in science as to when study results cal information about families or com¬ gator decides what is worth reporting: should or should not be published. The munities; it provides a basis for common the decision to report one's findings and decision as to what to include in a publi¬ understanding, as in lawmaking; and it the manner in which they are reported cation and whether to publish is largely provides the vehicle by which we share are a matter ofjudgment. personal, although dictated by the fash¬ information deemed to be important, as The question of how and when study ion of the times to a certain extent. in reporting the news of the day or the results are reported is of interest be¬ When Robert Boyle, the chemist, pub¬ latest scientific findings. cause of potential selection bias: given a lished his experiments on air in 1680, he In artistic endeavors, reinterpreta- set of characteristics about a study— was credited with being the first to re¬ tion of the written word, such as in design, operation, and outcome—could port the details of his experiments and translations, is acceptable, even wel¬ one predict the likelihood of publica¬ the precautions necessary for their rep¬ come. In other areas, the inherent ambi¬ tion? If one could, then that on which lication. This work ushered in a new guities of language lead to a constant our "knowledge" is based, the published type of report—one that described diffi¬ struggle to decipher the meaning or in¬ literature, is a biased representation of culties and errors. Thus, in the 1600s, tent of the written word. The US Con- knowledge as a whole. 1700s, and early 1800s, the usual scien¬ If the characteristics that determine tific report described not only the "posi¬ From the Department of Epidemiology, The Johns publication are related to study quality, tive" findings but also the "negative" or Hopkins University, School of Hygiene and Public then the selection bias incurred "nil" results. Health, Baltimore, Md. by Presented at The First International Congress on Peer studying only the published literature is Concerned about publication prac¬ Review in Biomedical Publication, Chicago, III, May 10\x=req-\ acceptable, even desirable. If, on the tices in the physical and life sciences, 12, 1989. other hand, the direction of re¬ lamented in 1661 that scientists Reprint requests to Department of Ophthalmology, study Boyle University of Maryland School of Medicine, 22 S Greene sults or the statistical significance ofthe did not write up single results but felt St, Baltimore, MD 21201 (Dr Dickersin). results is the reason for differential pub- compelled to refrain from publishing un- Downloaded from jama.ama-assn.org at Johns Hopkins University on August 5, 2011 til they had a "system" worked out that Table 1 —Manuscript Ratings for Same Manuscript With Varying Presentations of Results or Discussion10 they deemed worthy of formal presen¬ tation: "But the worst inconvenience of Mean Ratings all is yet to be mentioned, and that No. of Data Scientific Publication is, Presentation Referees Methods That whilst this vanity of thinking men Presentation Contribution Merit Positive results to write either or noth¬ 4.3 obliged systems results ing is in many excellent notions Negative 2.4 2.6 2.4 1.8 request, Methods or experiments are, by sober and mod¬ only 4.5 3.4 "2 Mixed results, est .... men, suppressed Apparent¬ Positive discussion 13 2.5 1.3 1.6 0.5 ly, the notion ofgoing to press only if one Mixed results, has something "big" to present is not negative discussion 14 2.7 2.0 1.7 modern at all. By the mid-1800s, the style of scien¬ tific was in the of writing process chang¬ Table 2.—Studies of Publication Bias in Medicine ing to the terse, rather technical ap¬ proach with which we are familiar. Index Follow-up Limitations of time (as science began to Source, y Subject Source Method Results move quite rapidly), journal space, the Simes, Cancer Cancer Publications Published trials 1986 trials trials and register show increased development of groups of scientists register efficacy of and a written combined working together forging treatment document together, the response to Dickersin Randomized, File of Questionnaire Published trials peer review, and economic dependence etal, controlled randomized, favor test on a that rewarded quick suc¬ 1987" trials controlled treatment more system trials often cess were all factors that led to a change in scientific and The Sommer, Menstrual Society Questionnaire Published studies writing publishing. 1987" cycle membership more often change in style that has taken place over research statistically the years is not inherently bad. The significant is whether the increased brevi¬ Chalmers Perinatal ODPT* ODPT full Strength of results problem etal, trials abstracts reports in abstract not ty has resulted in lost information and 1989* associated with whether it represents biased reporting. full publication Evidence for Publication Bias *ODPT indicates Oxford Database of Perinatal Trials. Perhaps as a result of the difficulties ofdesigning studies to address the prob¬ another received a manuscript that de¬ known to exist and its etiology well un¬ lem, more has been written to complain scribed negative results. A third group derstood. "Investigators are more about publication bias than to report was asked to evaluate a manuscript on strongly motivated to offer positive re¬ results of studies undertaken to evalu¬ the basis of the "Methods" section and sults for publication rather than null re¬ ate it. Most research on publication bias relevance alone; no data were provided. sults. Many journal editors select pa¬ has been done in the psychology and The fourth and fifth groups received pers for publication on this very basis, education fields.3"10 manuscripts with "mixed" results, with some of them expecting to see P values Sterling3 was probably the first to em¬ either a positive or negative "Discus¬ less than 0.05. Published clinical trials phasize that the tendency to publish sion" section. The referees used a scale are inevitably a positively biased positive results and reject negative of 0 to 6 (low to high) to rate the manu¬ selection."12 findings is a serious problem. He re¬ scripts for five items: relevance, meth¬ Information regarding publishing viewed all articles published in four ods, data presentation, scientific contri¬ practices is not easily obtained or readi¬ journals during 1 year (1955 or 1956) and bution, and publication merit.