Unpublished Research from a Medical Specialty Meeting: Why Investigators Fail to Publish

7. Chalmers T, Smith H, Blackburn B, et al. A Current status of academic emergency medicine vival Analysis and Logistic Regression. Charlot- method for assessing the quality of a randomized within academic medicine in the United States. tesville: Dept of Health Evaluation Sciences, School controlled trial. Control Clin Trials. 1981;2:31-49. Acad Emerg Med. 1994;1:41-46. of Medicine, University of Virginia; 1996. 8. Oxman AD, Guyatt GH. Validation of an index of 15. Garfunkel JM, Ulshen MH, Hamrick HJ, Law- 24. de Bellefeuille C, Morrison C, Tannock I. The the quality of review articles. J Clin Epidemiol. son EE. Effect of institutional prestige on review- fate of abstracts submitted to a cancer meeting: fac- 1991;44:1271-1278. ers’ recommendations and editorial decisions. tors which influence presentation and subsequent 9. Weber EJ, Callaham ML, Wears RL, Barton C, JAMA. 1994;272:137-138. publication. Ann Oncol. 1992;3:187-191. Young G. Unpublished research from a medical spe- 16. Olson CM. Publication bias. Acad Emerg Med. 25. Goldman L, Loscalzo A. Fate of cardiology re- cialty meeting: why investigators fail to publish. 1994;1:207-209. search originally published in abstract form. N Engl JAMA. 1998;280:257-259. 17. Simes RJ. Publication bias: the case for an in- J Med. 1980;303:255-259. 10. Garfield E. SCI Journal Citation Reports: A ternational registry of clinical trials. J Clin Oncol. 26. Meranze J, Ellison N, Greenhow D. Publica- Bibliometric Analysis of Science Journals in the 1986;4:1529-1541. tions reseulting from anesthesia meeting abstracts. ISI Database. Philadelphia, Pa: Institute for Sci- 18. DickersinK,MinYI.NIHclinicaltrialsandpub- Anesth Analg. 1982;61:445-448. ence Information Inc; 1991. lication bias. Online J Curr Clin Trials [serial on- 27. McCormick M, Holmes J. Publication of re- 11. Garfield E. SCI Journal Citation Reports: A line]. 1993;doc 50. search presented at the pediatric meetings. AJDC. Bibliometric Analysis of Science Journals in the 19. Rennie D, Flanagin A. Publication bias: the tri- 1985;139:122-126. ISI Database. Philadelphia, Pa: Institute for Sci- umph of hope over experience. JAMA. 1992;267: 28. Koren G, Graham K, Shear H, Einarson T. Bias ence Information Inc; 1992. 411-412. against the null hypothesis: the reproductive haz- 12. Garfield E. SCI Journal Citation Reports: A 20. BerlinJA.Willpublicationbiasvanishintheage ards of cocaine. Lancet. 1989;2:1440-1444. Bibliometric Analysis of Science Journals in the of online journals? Online J Curr Clin Trials [serial 29. Newcombe RG. Towards a reduction in publi- ISI Database; Philadelphia, Pa: Institute for Sci- online]. 1992;doc 12. cation bias. BMJ. 1987;295:656-659. ence Information Inc; 1993. 21. Begg CB, Berlin JA. Publication bias and dis- 30. Dickersin K. The existence of publication bias 13. Garfield E. SCI Journal Citation Reports: A semination of clinical research. J Natl Cancer Inst. and risk factors for its occurrence. JAMA. 1990;263: Bibliometric Analysis of Science Journals in the 1989;81:107-115. 1385-1389. ISI Database. Philadelphia, Pa: Institute for Sci- 22. Hosmer D, Lemeshow S. Applied Logistic Re- 31. Weintraub WH. Are published manuscripts rep- ence Information Inc; 1994. gression.NewYork,NY:JohnWiley&SonsInc;1989. resentative of the surgical meeting abstracts? an ob- 14. Gallagher E, Goldfrank L, Anderson G, et al. 23. HarrellFE.PredictingOutcomes:AppliedSur- jective appraisal. J Pediatr Surg. 1987;22:11-13.

Unpublished Research From a Medical Specialty Meeting Why Investigators Fail to Publish

Ellen J. Weber, MD; Michael L. Callaham, MD; Robert L. Wears, MD; Christopher Barton, MD; Gary Young, MD

Context.—It is not known whether peer review of research abstracts submitted UNDERREPORTING OF research is to scientific meetings influences subsequent attempts at publication. a well-recognized problem with serious Objective.—To determine why research submitted to a scientific meeting is not implications for clinical practice.1 Most unpublished research is never submit- subsequently published. We hypothesized that authors of abstracts rejected by a 2-6 meeting are less likely to pursue publication than those whose abstracts are ted to a journal for consideration. Pre- vious investigations suggest research- accepted, regardless of research quality. ers are more likely to attempt to publish Design and Participants.—Blinded review of abstracts submitted to a medical studies with positive outcomes (publica- specialty meeting in 1991 and not published as full manuscripts within 5 years. In tion bias).2-4 Much of the research sub- 1996, authors of 266 unpublished studies were asked to complete questionnaires. mitted to scientific meetings is never Main Outcome Measures.—Submission of a full manuscript to a journal published,5-12 yetitisnotknownwhether between 1991 and 1996; failure to submit a manuscript to a journal because the acceptance or rejection of the abstract investigator believed it would not be accepted for publication. Results.—A total of 223 (84%) of the unpublished investigators returned the From the Division of Emergency Medicine (Drs questionnaire. Only 44 (20%) had submitted manuscripts to a journal. Manuscript Weber, Callaham, and Young), University of Cali- submission was not associated with abstract quality (odds ratio [OR], 1.16; 95% fornia, San Francisco; the Division of Emergency Medicine, University of Florida Health Center, Jack- confidence interval [CI], 0.80-1.64), positive results (OR, 0.75; 95% CI, 0.31-1.57), sonville (Dr Wears); the Department of Emergency or other study characteristics. Having an abstract accepted for presentation at the Medicine, University of North Carolina, Chapel Hill (Dr Barton); and the Department of Emergency meeting weakly predicted submission of a manuscript to a journal (OR, 1.88; 95% Medicine, Highland Hospital, Alameda County CI, 0.84-4.10). Authors of accepted abstracts were significantly less likely to believe Medical Center, Oakland, Calif (Dr Young). Dr Young is now with Sacred Heart Medical Center, a journal would not publish their manuscript than were authors of rejected abstracts Eugene, Ore. (OR, 0.23; 95% CI, 0.0001-0.61). Presented at the Third International Congress on Conclusions.—Study characteristics do not predict attempts to publish research Peer Review in Biomedical Publication, Prague, Czech Republic, September 19, 1997. submitted to a scientific meeting. Investigators whose research is rejected by a Corresponding author: Ellen J. Weber, MD, Division meeting are pessimistic about chances for publication and may make less effort to of Emergency Medicine, University of California, San Francisco, Box 0208, San Francisco, CA 94143-0208 publish. (e-mail: [email protected]). JAMA. 1998;280:257-259 Reprints not available from the authors.

Downloaded From: https://jamanetwork.com/ on 09/25/2021 Table 1.—Relationship of Meeting Decision and Table 2.—Reasons for Failure to Submit a Manu- Study Population Study Characteristics to Submission of a Manuscript script to a Journal* to a Journal* No. Choosing Unpublished Studies Manuscript Submitted Reason Given for Response (N = 266) to a Journal, Nonsubmission (% of Respondents) Variable OR (95% CI)† Not enough time 74 (42) Questionnaires Returned Accepted by SAEM‡ 1.88 (0.84-4.10) Thought journals unlikely to 35 (20) (n = 223, 84%) Quality 1.16 (0.80-1.64) accept Originality 0.99 (0.45-2.24) Results not important enough 21 (12) Returned From Returned From Randomized 0.80 (0.18-2.50) Too much trouble with 16 (9) Authors of Accepted Authors of Rejected Sample size 1.00 (1.00-1.00) coauthors Not worth the trouble 13 (7) Abstracts Abstracts Positive results 0.75 (0.31-1.57) Institutional ranking 0.94 (0.68-1.25) Other papers with similar 11 (6) (n = 64, 84%) (n = 159, 84%) ﬁndings *N = 211 questionnaires. Statistical analysis not positive 7 (4) †Odds ratios (ORs) and 95% conﬁdence intervals Other reason 40 (22) A total of 266 authors whose studies were not pub- (CIs) for the association of potential predictors with the lished as full manuscripts within 5 years of the 1991 outcome of whether a full manuscript was ever submit- *N = 179 questionnaires. Number of responses is SAEM meeting were surveyed, 223 of whom com- ted to a journal. greater than number of questionnaires because many ‡SAEM indicates Society for Academic Emergency subjects chose more than 1 response. pleted questionnaires. Accepted indicates abstract Medicine. accepted for presentation at meeting; rejected, abstract submitted to meeting but not accepted for presentation. mitting institution’s ordinal ranking in within 3 months were sent another copy federal grant dollars.16 The presence or of the questionnaire. After 2 attempts, absence of positive results was noted for a meeting influences subsequent ef- the questionnaire was sent to another for studies with a hypothesis and con- forts to publish a full manuscript. author of the abstract and repeated in 3 trol group. Results were considered The purpose of this study was to de- months if necessary. positive if the author reported that the termine what factors influence an inves- Research was considered unpub- intervention was more effective than tigator’s attempts to publish research lished if an article was not found in the the control2,6,13,17; statistical significance that was submitted to a scientific meet- search and either the responding au- was not required. The statistical analy- ing. We hypothesized that authors of ab- thor confirmed that the study was not sis was done using S-Plus, Version 3.3 stracts rejected by the meeting were published or no questionnaire was re- (MathSoft Inc, Seattle, Wash). less likely to pursue publication than turned. Published studies, whether those whose abstracts had been ac- identified through the database search RESULTS cepted, regardless of the study quality. or the questionnaire, were excluded Of the 492 studies submitted to from further analysis. SAEM in 1991, 266 (55%) were never METHODS Unpublished abstracts were ran- published (Figure). Investigators com- We obtained copies of all abstracts domly assigned to 2 of the investigators pleted questionnaires for 223 (84%) of submitted to the 21st annual meeting of for classification of study characteris- the unpublished studies. Response the Society for Academic Emergency tics.9 Reviewers were blinded with re- rate did not differ for authors of ab- Medicine (SAEM) held in 1991. In No- spect to author, submitting institution, stracts rejected by the meeting and vember 1995 and again in March 1996, and whether the abstract had been ac- those accepted (PϾ.99). Abstracts of we searched MEDLINE, EMBASE, cepted for presentation at SAEM. Dis- respondents and nonrespondents were and the Cochrane Collaboration to iden- agreements regarding study character- similar in quality score (2.48 vs 2.33, tify which of these abstracts had been istics were resolved by a third investi- P = .33) and originality (1.58 vs 1.56, subsequently published in a peer-re- gator (M.L.C.). Reviewers assigned glo- P = .79). viewed journal. A peer-reviewed publi- bal ratings for scientific quality (overall cation was defined as a full-length manu- scientific solidity) on a 5-point Likert Pursuit of Publication script (not letters or editorials) appear- scale and originality on a 3-point A full manuscript had been submitted ing in a journal that uses a process of scale.14,15 The scores of the 2 reviewers to a journal for 44 (20%) of the 223 un- external review. Multiple search strat- were averaged. The intraclass correla- published studies, with a mean of 1.67 egies were used, beginning with several tion for scientific quality was 0.44 (95% (SD, 0.95) submissions per study. authors, and then, if necessary, combi- confidence interval [CI], 0.37-0.51) and No association was found between nations of authors, title, and keywords. for originality was 0.29 (95% CI, 0.21- manuscript submission and study char- Online abstracts were compared with 0.37). acteristics (Table 1). There was a trend the original abstract submitted to the The main outcomes were whether or suggesting that investigators whose ab- SAEM meeting in 1991 when necessary not a manuscript was submitted to a stracts had been accepted for presenta- to confirm the identity of the publica- journal (pursuit of publication) and tion at the SAEM meeting were more tion. whether or not authors who failed to likely to submit full manuscripts to a In October 1996, a questionnaire6,13 submit a manuscript stated that a jour- journal than those whose abstracts had was mailed to the first author of each nal would be unlikely to accept it for been rejected by the meeting (odds ratio abstract for whom no publication was publication (pessimism). Potential pre- [OR], 1.88; 95% CI, 0.84-4.1). found in the search. The questionnaire dictors were assessed by fitting a sepa- asked whether the study had been pub- rate logistic regression model to each Reasons for Failure to Publish lished, and asked the author to provide a of these outcomes. The predictors Among the 179 investigators who citation if it had been published. For un- were whether the abstract had been never submitted a full manuscript to a published research, authors were asked accepted or rejected for presentation journal, the most common reason se- whether they had submitted a manu- at the meeting, quality and originality lected was lack of time (Table 2). Only script to a journal, and if not, they were scores, whether or not the study was 7 investigators said they did not sub- asked to select a reason for not doing so. randomized, sample size, presence or mit a manuscript because the statisti- Investigators who did not respond absence of positive results, and sub- cal analysis was not positive, even

Downloaded From: https://jamanetwork.com/ on 09/25/2021 though 43 controlled studies had nega- To our knowledge, the current study dition, abstracts of respondents and non- tive results. is the first to investigate the relation- respondentsweresimilarinquality,origi- Two variables predicted whether or ship among study characteristics, meet- nality, and acceptance by SAEM. We not an investigator selected the re- ing decision, and an author’s efforts to identified with certainty the publication sponse “thought journals unlikely to ac- publish research submitted to a scien- fate of 92% of all submitted abstracts. The cept.”Authorsofabstractsthathadbeen tific meeting. Authors whose abstracts SAEM meeting is comparable with the accepted for the meeting chose this op- were rejected from the meeting were meetings of 31 other specialty societies tion significantly less frequently than significantly more pessimistic about the with regard to attendance, number of ab- those whose abstracts had been rejected chances of publication, and there was stracts submitted, and subsequent publi- (OR, 0.23; 95% CI, 0.0001-0.61). Authors also a trend suggesting that authors of cation rate.18 The 5-year interval from from institutions ranking higher in fed- rejected abstracts were less likely to submission may have influenced our rat- eral grant dollars chose the response pursue full publication. There was no as- ings and the authors’ responses, but was “journals unlikely to accept” (OR, 1.71; sociation between publication efforts necessary to allow ample time for publi- 95% CI, 1.2-2.9) more frequently than and study quality, originality, sample cation.5-7,9 those from lower-ranking institutions. size, design, or results. For research submitted to scientific Study quality, originality, design, sam- Unlike previous studies, we found no meetings, subsequent publication ef- ple size, and the presence of a positive evidence of publication bias among our forts are not predicted by the specifics of outcome did not predict whether or not investigators.Thisismostlikelybecause the research, but may be affected by the an investigator chose this response. of the difference in study populations.2-4 meeting’sdecisiontoacceptorrejectthe Previous investigations focused on fully abstract.Investigatorsappeartobeeas- COMMENT funded research projects from a single ily discouraged by rejection at both the Why are the results of many studies institution or fully funded randomized meeting and journal stages of publica- never published? Our findings confirm controlled trials, and are therefore rep- tion. Because failure to publish com- prior reports that most unpublished resentative of only a minority of unpub- pleted research affects medical practice, research is never submitted to a jour- lishedresearch.Thestudiesinouranaly- it is imperative that specialty societies nal for review.2-6 Only 20% of the un- sis came from 144 different institutions understand the potential impact of their published studies originally submitted and included many projects that were decisions and make additional efforts to to the SAEM meeting were later sub- not funded. Additionally, our population encourage all investigators, not just mitted as a full manuscript to a jour- of researchers had all undergone the re- those whose abstracts are accepted for nal. Moreover, investigators were eas- view process for a scientific meeting. presentation, to pursue full publication. ily dissuaded, submitting a manuscript, The 84% response rate for our ques- on average, to fewer than 2 journals tionnaireequalsorsurpassesthatofother The authors wish to thank Fran Malespin for her before giving up. studiesofunpublishedresearch.2-6,13 Inad- assistance in conducting the survey.

References 1. Chalmers I. Underreporting research is scien- 8. McCormick M, Holmes J. Publication of research fluencing publication of research results. Follow-up tific misconduct. JAMA. 1990;263:1405-1408. presented at the pediatric meetings. AJDC. 1985; of applications submitted to two institutional re- 2. Dickersin K, Min YI. NIH clinical trials and pub- 139:122-126. view boards. JAMA. 1992;267:374-378. lication bias. Online J Curr Clin Trials [serial on- 9. Callaham ML, Wears RL, Weber EJ, Barton 14. Oxman AD, Guyatt GH, Singer J, et al. Agree- line]. April 28, 1993; doc 50. C, Young G. Positive-outcome bias and other ment among reviewers of review articles. J Clin 3. EasterbrookPJ,BerlinJA,GopalanR,Matthews limitations in the outcome of research abstracts Epidemiol. 1991;44:91-98. DR. Publication bias in clinical research. Lancet. submitted to a scientific meeting. JAMA. 1998;280: 15. Nylenna M, Riis P, Karlsson Y. Multiple blinded 1991;337:867-872. 254-257. reviews of the same two manuscripts: effects of ref- 4. Dickersin K, Chan S, Chalmers TC, Sacks HS, 10. Gorman RL, Oderda GM. Publication of pre- eree characteristics and publication language. Smith H Jr. Publication bias and clinical trials. Con- sented abstracts at annual scientific meetings: a JAMA. 1994;272:149-151. trol Clin Trials. 1987;8:343-353. measure of quality? Vet Hum Toxicol. 1990;32: 16. Gallagher E, Goldfrank L, Anderson G, et al. 5. De Bellefeuille C, Morrison CA, Tannock IF. The 470-472. Current status of academic emergency medicine fate of abstracts submitted to a cancer meeting: fac- 11. Yentis SM, Campbell FA, Lerman J. Publica- within academic medicine in the United States. tors which influence presentation and subsequent tion of abstracts presented at anaesthesia meetings. Acad Emerg Med. 1994;1:41-46. publication. Ann Oncol. 1992;3:187-191. Can J Anaesth. 1993;40:632-634. 17. Chalmers I, Adams M, Dickersin K, et al. A co- 6. Scherer RW, Dickersin K, Langenberg P. Full 12. JuzychMS,ShinDH,CoffeyJ,JuzychL,ShinD. hort study of summary reports of controlled trials. publication of results initially presented in ab- Whatever happened to abstracts from different sec- JAMA. 1990;263:1401-1405. stracts: a meta-analysis. JAMA. 1994;272:158-162. tions of the association for research in vision and 18. Wuerz RE, Holliman CJ. Attendance and re- 7. Goldman L, Loscalzo A. Fate of cardiology ophthalmology? Invest Ophthalmol Vis Sci. 1993; search abstract activity at the 1993 annual meetings research originally published in abstract form. 34:1879-1882. of the academic medical societies [abstract]. Acad N Engl J Med. 1980;303:255-259. 13. Dickersin K, Min YI, Meinert CL. Factors in- Emerg Med. 1994;1:A59.

Downloaded From: https://jamanetwork.com/ on 09/25/2021