Essay Why Current Publication Practices May Distort Science Neal S. Young*, John P. A. Ioannidis, Omar Al-Ubaydli

his essay makes the underlying Summary the more extreme, spectacular assumption that scientific results (the largest treatment The current system of publication information is an economic effects, the strongest associations, T in biomedical provides commodity, and that scientific journals or the most unusually novel and a distorted view of the reality of are a medium for its dissemination exciting biological stories) may be scientific data that are generated in the and exchange. While this exchange preferentially published. Journals laboratory and clinic. This system can system differs from a conventional serve as intermediaries and may suffer be studied by applying principles from market in many senses, including minimal immediate consequences for the field of economics. The “winner’s the nature of payments, it shares the errors of over- or mis-estimation, but curse,” a more general statement of goal of transferring the commodity it is the consumers of these laboratory publication , suggests that the (knowledge) from its producers and clinical results (other expert small proportion of results chosen for (scientists) to its consumers (other scientists; trainees choosing fields publication are unrepresentative of scientists, administrators, physicians, of endeavour; physicians and their scientists’ repeated samplings of the real patients, and funding agencies). The patients; funding agencies; the media) world. The self-correcting mechanism function of this system has major who are “cursed” if these results are in science is retarded by the extreme consequences. Idealists may be severely exaggerated—overvalued imbalance between the abundance offended that research be compared to and unrepresentative of the true of supply (the output of basic science widgets, but realists will acknowledge outcomes of many similar . laboratories and clinical investigations) that journals generate revenue; For example, initial clinical studies and the increasingly limited venues for publications are critical in drug are often unrepresentative and publication (journals with sufficiently development and marketing and to misleading. An empirical evaluation high impact). This system would be attract venture capital; and publishing of the 49 most-cited papers on the expected intrinsically to lead to the defines successful scientific careers. effectiveness of medical interventions, misallocation of resources. The scarcity Economic modelling of science may of available outlets is artificial, based on yield important insights (Table 1). the costs of printing in an electronic age Funding: The authors received no specific funding The Winner’s Curse and a belief that selectivity is equivalent for this article. to quality. Science is subject to great In auction theory, under certain Competing Interests: The authors have declared uncertainty: we cannot be confident conditions, the bidder who wins tends that no competing interests exist. now which efforts will ultimately yield to have overpaid. Consider oil firms worthwhile achievements. However, Citation: Young NS, Ioannidis JPA, Al-Ubaydli O bidding for drilling rights; companies (2008) Why current publication practices may distort the current system abdicates to a small estimate the size of the reserves, and science. PLoS Med 5(10): e201. doi:10.1371/journal. number of intermediates an authoritative pmed.0050201 estimates differ across firms. The prescience to anticipate a highly average of all the firms’ estimates would This is an open-access article distributed under the unpredictable future. In considering terms of the Creative Commons Public Domain usually approximate the true reserve society’s expectations and our own declaration, which stipulates that, once placed in the size. Since the firm with the highest public domain, this work may be freely reproduced, goals as scientists, we believe that there estimate bids the most, the auction distributed, transmitted, modified, built upon, or is a moral imperative to reconsider otherwise used by anyone for any lawful purpose. winner systematically overestimates, how scientific data are judged and sometimes so substantially as to lose Neal S. Young is with the Hematology Branch, disseminated. money in net terms [1]. When bidders National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland, United are cognizant of the statistical processes States of America. John P. A. Ioannidis is with the of estimates and bids, they correct for Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, and the Biomedical the winner’s curse by shading their bids curse [5–8]. Indeed, the winner’s curse Research Institute, Foundation for Research and down. This is why experienced bidders was first proposed by oil operations Technology – Hellas, Ioannina, Greece; and the sometimes avoid the curse, as opposed researchers after they had recognised Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts, United States of to inexperienced ones [1–4]. Yet in aberrant results in their own market. America. Omar Al-Ubaydli is with the Department of numerous studies, bidder behaviour An analogy can be applied to Economics and the Mercatus Center, George Mason appears consistent with the winner’s scientific publications. As with University, Fairfax, Virginia, United States of America. individual bidders in an auction, * To whom correspondence should be addressed. the average result from multiple E-mail: [email protected] The Essay section contains opinion pieces on topics studies yields a reasonable estimate of broad interest to a general medical audience. Provenance: Not commissioned; externally peer of a “true” relationship. However, reviewed

PLoS Medicine | www.plosmedicine.org 1418 October 2008 | Volume 5 | Issue 10 | e201 Table 1. Economic Terms and Analogies in Scientific Publication Economic Term Meaning Analogy in Scientific Publication

Winner’s curse The winner in an auction tends on average to have overpaid, Scientific studies try to find true relationships, but none are especially when no participant is sure exactly how valuable certain of what these relationships are exactly. Published the item is. articles, especially in very competitive journals, have on average exaggerated results. Oligopoly A market where a few traders have the major share and each Very few journals with limited publication slots (compared oligopolist has significant power to influence the market. with geometrically increasing scientific data that seek publication) determine highly visible science. Herding “Follow-the-leader” behaviour: the actions of the first or Scientists may uncritically follow paths of investigation dominant player supersede the individual information and that are popularised in prestigious publications, neglecting actions of all the players in a market. novel ideas and truly independent investigative paths. Artificial scarcity Restrictions on the provision of a commodity above that Print page limits are an obvious excuse for failure to expected from its production cost. accept articles, and further the small number of major “high-impact” journals have limited slots; extremely low acceptance rates provide status signals to successful publications and their authors. Uncertainty Situation where the real long-term value of a commodity is For much (most?) scientific work, it is difficult or impossible largely unpredictable. to immediately predict future value, extensions, and practical applications. Branding Marking a product as valuable; of key importance when it is Publishing in selective journals provides evidence of value difficult to determine a product’s value prior to consuming it. of a research result and its authors, independent of the manuscript’s content. doi:10.1371/journal.pmed.0050201.t001 published in highly visible journals in at conferences or among colleagues, considerations of experimental design, 1990–2004, showed that a quarter of but surface more publicly only when execution, or importance. Much data the randomised trials and five of six dominant paradigms are replaced. are never formally refuted in print, non-randomised studies had already Sometimes, negative data do appear but most promising preclinical work been contradicted or found to have in refutation of prominent claims. eventually fails to translate to clinical been exaggerated by 2005 [9]. The In the “Proteus phenomenon”, an benefit [22]. Worse, in the course of delay between the reporting of an extreme result reported in the first ongoing experimentation, apparently initial positive study and subsequent published study is followed by an negative studies are abandoned publication of concurrently performed extreme opposite result; this sequence prematurely as wasteful. but negative results is measured in may cast doubt on the significance, years [10,11]. An important role of meaning, or validity of any of the Oligopoly systematic reviews may be to correct results [18]. Several factors may predict Successful publication may be more the inflated effects present in the initial irreproducibility (small effects, small difficult at present than in the past. studies published in famous journals studies, “hot” fields, strong interests, The supply and demand of scientific [12], but this process may be similarly large databases, flexible statistics) [19], production have changed. Across the prolonged and even systematic reviews but claiming that a specific study is health and life sciences, the number of may perpetuate inflated results [13,14]. wrong is a difficult, charged decision. published articles in Scopus-indexed More alarming is the general paucity In the basic biological sciences, journals rose from 590,807 in 1997 in the literature of negative data. statistical considerations are secondary to 883,853 in 2007, a modest 50% In some fields, almost all published or nonexistent, results entirely increase. In the same decade, data studies show formally significant unpredicted by hypotheses are acquisition has accelerated by many results so that no celebrated, and there are few formal orders of magnitude: as an example, longer appears discriminating [15,16]. rules for reproducibility [20,21]. A the current Cancer Genome Atlas Discovering selective reporting is not signalling benefit from the market— project requires 10,000 times more easy, but the implications are dire, as in good scientists being identified by their sequencing effort than the Human the “hidden” results for positive results—may be more powerful Genome Project, but is expected to trials [17]: in a recent paper, it was in the basic biological sciences take a tenth of the time to complete shown that while almost all trials with than in clinical research, where the [23]. In the current environment, “positive” results on consequences of incorrect assessment the distinction between raw data had been published, trials with of positive results are more dire. As and articles (telling for sure what “negative” results submitted to the with clinical research, prominent more an article has compared with US Food and Drug Administration, claims sometimes disappear over time raw data) can sometimes become with few exceptions, remained either [21]. If a posteriori considerations are difficult. Only a small proportion of unpublished or were published with met sceptically in clinical research, in the explosively expanded output of the results presented so that they would basic biology they dominate. Negative biological laboratories appears in the appear “positive” [17]. Negative or data are not necessarily different modestly increased number of journal contradictory data may be discussed than positive results as related to slots available for its publication, even

PLoS Medicine | www.plosmedicine.org 1419 October 2008 | Volume 5 | Issue 10 | e201 if more data can be compacted in the luxury items intentionally kept in short Manuscripts are assessed with a average paper now than in the past. supply, there is a motivation to restrict fundamentally negative bias: how they Constriction on the demand access [37,38]. may best be rejected to promote the side is further exaggerated by the Some unfavourable consequences presumed selectivity of the journal. disproportionate prominence of a very may be predicted and some are Journals closely track and advertise few journals. Moreover, these journals visible. Resource allocation has long their low acceptance rates, equating strive to attract specific papers, such as been recognised by economists as these with rigorous review: “Nature influential trials that generate publicity problematic in science, especially in has space to publish only 10% or so and profitable reprint sales. This basic research where the risks are the of the 170 papers submitted each “winner-take-all” reward structure [24] greatest. Rival teams undertake unduly week, hence its selection criteria are leaves very little space for “successful dubious and overly similar projects; and rigorous”—even though it admits that publication” for the vast majority of too many are attracted to one particular has a secondary role: “the scientific work and further exaggerates contest to the neglect of other areas, judgement about which papers will the winner’s curse. The acceptance rate reducing the diversity of areas under interest a broad readership is made by decreases by 5.3% with doubling of exploration [39]. Early decisions by Nature’s editors, not its referees” [43]. circulation, and circulation rates differ a few influential individuals as to the Science also equates “high standards of by over 1,000-fold among 114 journals importance of an area of investigation peer review and editorial quality” with publishing clinical research [25]. For consolidate path dependency: the the fact that “of the more than 12,000 most published papers, “publication” first decision predetermines the top-notch scientific manuscripts that often just signifies “final registration trajectory. A related effect is herding, the journal sees each year, less than 8% into oblivion”. Besides print circulation, where the actions of a few prominent are accepted for publication” [44]. in theory online journals should be individuals rather than the cumulative The publication system may operate readily visible, especially if open access. input of many independent agents differently in different fields. For However, perhaps unjustifiably, most drives people’s valuations of a example, for drug trials, journal articles published in online journals commodity [40,41]. Cascades arise operations may be dominated by the remain rarely accessed. Only 73 of when individuals regard others’ earlier interests of larger markets: the high the many thousands of articles ever actions as more informative than their volume of transactions involved extends published by the 187 BMC-affiliated own private information. The actions well beyond the small circle of scientific journals had over 10,000 accesses upon which people herd may not valuations and interests. In other through their journal Web sites in the necessarily be correct; and herding fields where no additional markets last year [26]. may long continue upon a completely are involved (the history of science is Impact factors are widely adopted as wrong path [41]. Information cascades perhaps one extreme example), the criteria for success, despite whatever encourage conventional behaviour, situation of course may be different. qualms have been expressed [27–32]. suppress information aggregation, The question to be examined is They powerfully discriminate against and promote “bubble and bust” cycles. whether published data may be more submission to most journals, restricting Informational analysis of the literature representative (and more unbiased) acceptable outlets for publication. on molecular interactions in Drosophila depending on factors such as the ratio “Gaming” of impact factors is explicit. genetics has suggested the existence of of journal outlets to amount of data Editors make estimates of likely such information cascades, with positive generated, the relative valuation of citations for submitted articles to momentum, interdependence among specialty journals, career consequences gauge their interest in publication. published papers (most reporting of publication, and accessibility of The citation game [33,34] has created positive data), and dominant themes primary data to the reader. distinct hierarchical relationships leading to stagnating conformism [42]. One solution to artificial scarcity— among journals in different fields. In digital publication—is obvious and scientific fields with many citations, Artificial Scarcity already employed. Digital platforms very few leading journals concentrate The authority of journals increasingly can facilitate the publication of greater the top-cited work [35]: in each of derives from their selectivity. The venue numbers of appropriately peer- the seven large fields to which the life of publication provides a valuable reviewed manuscripts with reasonable sciences are divided by ISI Essential status signal. A common excuse for hypotheses and sound methods. Indicators (each including several rejection is selectivity based on a Digitally formatted publication need hundreds of journals), six journals limitation ironically irrelevant in the not be limited to few journals, or only account for 68%–94% of the 100 modern age—printed page space. This to open-access journals. Ideally, all most-cited articles in the last decade is essentially an example of artificial journals could publish in digital form (Clinical Medicine 83%, Immunology scarcity. Artificial scarcity refers to manuscripts that they have received 94%, Biochemistry and Biology 68%, any situation where, even though and reviewed and that they consider Molecular Biology and Genetics 85%, a commodity exists in abundance, unsuitable for print publication Neurosciences 72%, Microbiology 76%, restrictions of access, distribution, or based on subjective assessments of Pharmacology/Toxicology 72%). The availability make it seem rare, and thus priority. The current privileging of scientific publishing industry is used for overpriced. Low acceptance rates create print over digital publication by some career advancement [36]: publication an illusion of exclusivity based on merit authors and review committees may in specific journals provides scientists and more frenzied competition among be reversed, if online-only papers with a status signal. As with other scientists “selling” manuscripts. can be demonstrated or perceived to

PLoS Medicine | www.plosmedicine.org 1420 October 2008 | Volume 5 | Issue 10 | e201 represent equal or better scientific Box 1. Potential Competing To exorcise the winner’s curse, the reality than conventional printed or Complementary Options quality of experiments rather than manuscripts. and Solutions for Scientific the seemingly dramatic results in a Publication minority of them would be the focus of Uncertainty review, but is this feasible in the current When scientific information itself is 1. Accept the current system as having reality? the commodity, there is uncertainty as evolved to be the optimal solution to There are limitations to our analysis. to its value, both immediately and in complex and competing problems. Compared with the importance the long term. Usually we do not know 2. Promote rapid, digital publication of the problem, there is a relative what information will be most useful of all articles that contain no flaws, paucity of empirical observations on (valuable) eventually. Economists irrespective of perceived the process of scientific publication. have struggled with these peculiar “importance”. The winner’s curse is fundamental to attributes of scientific information as 3. Adopt preferred publication of our thesis, but there is active debate a commodity. Production of scientific negative over positive results; require among economists whether it inhibits information is largely paid for by public very demanding reproducibility real environments or is more of a investment, but the product is offered criteria before publishing positive theoretical phenomenon [1,8]. Do free to commercial intermediaries, results. senior investigators make the same and is culled by them with minimal adjustments on a high-profile paper’s cost, for sale back to the producers and 4. Select articles for publication in value as do experienced traders on their underwriters! An explanation highly visible venues based on prices? Is herding an appropriate for such a strange arrangement is the quality of study methods, their model for scientific publication? Can the need for branding—marking the rigorous implementation, and astute we correlate the site of publication product as valuable. Branding may be interpretation, irrespective of results. with the long-term value of individual more important when a commodity 5. Adopt formal post-publication scientific work and of whole areas of cannot easily be assigned much downward adjustment of claims investigation? These questions remain intrinsic value and when we fear the of papers published in prestigious open to analysis and . exchange environment will be flooded journals. Even though its goals may sometimes with an overabundance of redundant, 6. Modify current practice to elevate be usurped for other purposes, science useless, and misleading product and incorporate more expansive is hard work with limited rewards [39,45]. Branding serves a similar data to accompany print articles or and only occasional successes. Its and complementary function to the to be accessible in attractive formats interest and importance should speak status signal for scientists discussed associated with high-quality journals: for themselves, without hyperbole. above. While it is easy to blame journal combine the “magazine” and Uncertainty is powerful and yet quite editors, the industry, or the popular “archive” roles of journals. insufficiently acknowledged when we press, there is scant evidence that pretend prescience to guess at the 7. Promote critical reviews, digests, and these actors bear the major culpability ultimate value of today’s endeavours. summaries of the large amounts of [46–49]. Probably authors themselves If “the striving for knowledge and the biomedical data now generated. self-select their work for branding search for truth are still the strongest [10,11,50–52]. 8. Offer disincentives to herding and motives of scientific discovery”, and if incentives for truly independent, “the advance of science depends upon Conclusions novel, or heuristic scientific work. the free competition of thought” [55], We may consider several competing 9. Recognise explicitly and respond we must ask whether we have created or complementary options about the to the branding role of journal a system for the exchange of scientific future of scientific publication (Box 1). publication in career development ideas that will serve this end. When economists are asked to analyse and funding decisions. a resource-allocation system, a typical 10. Modulate publication practices Supporting Information Text S1. assumption is that when information based on empirical research, which is dispersed, over time, the individual might address correlates of long- An extended version of this analytical essay actors will not make systematic errors term successful outcomes (such as Found at doi:10.1371/journal. in their inferences. However, not all reproducibility, applicability, opening pmed.0050201.sd001 (115 KB PDF). economists accept this strong version of new avenues) of published papers. rationality. Systematic misperceptions Acknowledgments in human behaviour occur with some Many colleagues have carefully read versions frequency [53], and misperceptions wasteful efforts of repetitive submission, of this work, and the authors express their review, revision, and resubmission? can perpetuate ineffective systems. gratitude in particular to John Barrett, Some may accept the current Preferred publication of negative over Cynthia Dunbar, Jack Levin, Leonid publication system as the ideal positive results has been suggested, Margolis, Philip Mortimer, Alan Schechter, culmination of an evolutionary with print publication favoured for Philip Scheinberg, Andrea Young, and process. However, this order is hardly all negative data (as more likely to be Massimo Young. divinely inspired; additionally, the true) and for only a minority of the References larger environment has changed over positive results that have demonstrated 1. Thaler RH (1988) Anomalies: The winner’s time. Can digital publication alleviate consistency and reproducibility [54]. curse. J Econ Perspect 2: 191-202.

PLoS Medicine | www.plosmedicine.org 1421 October 2008 | Volume 5 | Issue 10 | e201 2. Cox JC, Isaac RM (1984) In search of the 19. Ioannidis JPA (2005) Why most published 38. Becker GS (1991) A note on restaurant pricing winner’s curse. Econ Inq 22: 579-592. research findings are false. PLoS Med 2: e124. and other examples of social influences on 3. Dyer D, Kagel JH (1996) Bidding in common doi:10.1371/journal.pmed.0020124 price. Polit Econ 99: 1109-1116. value auctions: How the commercial 20. Easterbrook P, Berlin JA, Gopalan R, Matthews 39. Dasgupta P, David PA (1994) Toward a new construction industry corrects for the winner’s DR (1991) Publication bias in clinical research. economics of science. Res Pol 23: 487-521. curse. Manage Sci 42: 1463-1475. Lancet 337: 867-872. 40. Bikhchandani S, Hirshleifer D, Welch I 4. Harrison GW, List JA (2007) Naturally 21. Ioannidis JP (2006) Evolution and translation (1998) Learning from the behavior of others: occurring markets and exogenous laboratory of research findings: From bench to where? Conformity, fads, and informational cascades. J experiments: A case study of the winner’s PLoS Clin Trials 1: e36. doi:10.1371/journal. Econ Perspect 12: 151-170. curse. NBER Working Paper No. 13072 pctr.0010036 41. Hirshleifer D Teoh SH (2003) Herd behaviour 1-20. Cambridge (MA): National Bureau of 22. Contopoulos-Ioannidis DG, Ntzani E, Ioannidis and cascading in capital markets: A review and Economic Research. JP (2003) Translation of highly promising basic synthesis. Eur Financ Manage 9: 25-66. 5. Capen EC, Clapp RV, Campbell WM (1971) science research into clinical applications. Am J 42. Rzhetsky A, Lossifov I, Loh JM, White KP Competitive bidding in high-risk situations. J Med 114: 477-484. (2007) Microparadigms: Chains of collective Petrol Technol 23: 641-653. 23. National Human Genome Research Institute reasoning in publications about molecular 6. Cassing J, Douglas RW (1980) Implications of (2008) The Cancer Genome Atlas. Available: interactions. Proc Nat Acad Sci U S A 103: the auction mechanism in baseball’s free-agent http://www.genome.gov/17516564. Accessed 4 4940-4945. draft. South Econ J 47: 110-121. September 2008. 43. Nature (2008) Getting published in Nature: 7. Dessauer JP (1981) Book publishing: What it 24. Frank RH, Cook PJ (1995) The winner-take-all The editorial process. Available: http://www. is, what it does. 2nd edition. New York: R. R. society. New York: Free Press. nature.com/nature/authors/get_published/ Bowker. 25. Goodman SN, Altman DG, George SL (1998) index.html. Accessed 4 September 2008. 8. Hendricks K, Porter RH, Boudreau B (1987) Statistical reviewing policies of medical 44. Science (2008) About Science and AAAS. Information, returns, and bidding behavior journals: Caveat lector? J Gen Intern Med 13: Available: http://www.sciencemag.org/help/ in OCS auctions: 1954-1969. J Ind Econ 35: 753-756. about/about.dtl. Accessed 4 September 2008. 517-542. 26. Biomed Central (2008) Most viewed 45. Merton RK (1957) Priorities in scientific 9. Ioannidis JPA (2005) Contradicted and articles in past year. Available: http://www. discovery: A chapter in the sociology of science. initially stronger effects in highly cited biomedcentral.com/mostviewedbyyear/. Am Sociol Rev 22: 635-659. clinical research. J Am Med Assoc 294: Accessed 4 September 2008. 46. Olson CM, Rennie D, Cook D, Dickersin K, 218-228. 27. The PLoS Medicine Editors (2006) The impact Flanagin A (2002) Publication bias in editorial 10. Krzyzanowska MK, Pintilie M, Tannock IF factor game. PLoS Med 3: e291. doi:10.1371/ decision making. J Am Med Assoc 287: (2003) Factors associated with failure to journal.pmed.0030291 2825-2828. publish large randomized trials presented at 28. Smith R (2006) Commentary: The power of 47. Brown A, Kraft D, Schmitz SM, Sharpless an oncology meeting. J Am Med Assoc 290: the unrelenting impact factor—Is it a force for V, Martin C (2006) Association of industry 495-501. good or harm? Int J Epidemiol 35: 1129-1130. sponsorship to published outcomes in 11. Stern JM (2006) Publication bias: Evidence of 29. Andersen J, Belmont J, Cho CT (2006) gastrointestinal clinical research. Clin delayed publication in a cohort study of clinical Journal impact factor in the era of expanding Gastroenterol Hepatol 4: 1445-1451. research projects. Br Med J 315: 640-645. literature. J Microbiol Immunol Infect 39: 48. Patsopoulos NA, Analatos AA, Ioannidis 12. Goodman SN (2008) Systematic reviews are not 436-443. JPA (2006) Origin and funding of the most biased by results from trials stopped early for 30. Ha TC, Tan SB, Soo KC (2006) The journal equently cited papers in medicine: Database benefit. J Clin Epidemiol 61: 95-96. impact factor: Too much of an impact? Ann analysis. Br Med J 332: 1061-1064. 13. Bassler D, Ferreira-Gonzalez I, Briel M, Cook Acad Med Singapore 35: 911-916. 49. Bubela TM, Caulfield TA (2004) Do the print DJ, Devereaux PJ, et al. (2007) Systematic 31. Song F, Eastwood A, Bilbody S, Duley L (1999) media “hype” genetic research? A comparison reviewers neglect bias that results from trials The role of electronic journals in reducing of newspaper stories and peer-reviewed stopped early for benefit. J Clin Epidemiol 60: publication bias. Med Inform 24: 223-229. research papers. Canad Med Assoc J 170: 869-873. 32. Rossner M, Van Epps H, Hill E (2007) Show 1399-1407. 14. Ioannidis JP (2008). Why most discovered true me the data. J Exp Med 204: 3052-3053. 50. Polanyi M (1964) Science, faith and society. associations are inflated. Epidemiology 19: 33. Chew M, Villanueva EV, van der Weyden MB 2nd edition. Chicago: University of Chicago 640-648. (2007) Life and times of the impact factor: Press. 15. Kavvoura FK, Liberopoulos G, Ioannidis JPA Retrospective analysis of trends for seven 51. Dickersin K, Chan S, Chalmers TC, Sacks HS, (2007) Selection in reported epidemiological medical journals (1994-2005) and their editors’ Smith H Jr (1987) Publication bias and clinical risks: An empirical assessment. PLoS Med 4: views. J Royal Soc Med 100: 142-150. trials. Control Clin Trials 8: 343-353. e79. doi:10.1371/journal.pmed.0040079 34. Ronco C (2006) Scientific journals: Who 52. Callaham ML, Wears RL, Weber EJ, Barton 16. Kyzas PA, Denaxa-Kyza D, Ioannidis JP (2007) impacts the impact factor? Int J Artif Organs C, Young G (1998) Positive- and Almost all articles on cancer prognostic 29: 645-648. other limitations in the outcome of research markers report statistically significant results. 35. Ioannidis JPA (2006) Concentration of the abstracts submitted to a scientific meeting. J Eur J Cancer 43: 2559-2579. most-cited papers in the scientific literature: Am Med Assoc 280: 254-257. 17. Turner EH, Matthews AM, Linardatos E, Tell analysis of journal ecosystems. PLoS ONE 1: e5. 53. Kahneman D (2003) A perspective on RA, Rosenthal R (2008) Selective publication doi:10.1371/journal.pone.0000005 judgment and choice: Mapping bounded of antidepressant trials and its influence on 36. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, rationality. Am Psychol 58: 697-720. apparent efficacy. N Eng J Med 358: 252-260. Ioannidis JP (2005) Local literature bias in 54. Ioannidis JPA (2006) Journals should 18. Ioannidis JPA, Trikalinos TA (2005) Early genetic epidemiology: An empirical evaluation publish all “null” results and should sparingly extreme contradictory estimates may appear in of the Chinese literature. PLoS Med 2: e334. publish “positive” results. Cancer Epidemiol published research: The Proteus phenomenon doi:10.1371/journal.pmed.0020334 Biomarkers Prev 15: 185. in molecular genetics research and randomized 37. Ireland NJ (1994) On limiting the market for 55. Popper K (1959) The logic of scientific trials. J Clin Epidemiol 58: 543-549. status signals. J Public Econ 53: 91-110. discovery. New York: Basic Books.

PLoS Medicine | www.plosmedicine.org 1422 October 2008 | Volume 5 | Issue 10 | e201