26 Adoption of PubMed Commons as a forum for post-publication peer review Paul Lane Director of Social Media and Web-Based Information, Envision Pharma Group, Horsham, UK

Abstract Results Objective: To assess the level of adoption of PubMed Commons, a pilot system allowing comments to be added to publications listed • A total of 1771 publications have received comments in the PubMed by anyone who is an author on a PubMed- Figure 4A. Proportion of comments by of indexed publication, since launching in October 2013. since the launch of Commons; they were almost all comment design and : A search was conducted using the English (99%) and approximately half were term “has_user_comments[filter]” to capture all items with comments available as free full text (49%; Figure 1). 9 8.5 within the PubMed database between the launch and 21 September 8 2014. Publications receiving comments were then sorted by the type of publication (as designated by PubMed filters – clinical trials, reviews and 7 free full text), publication year and journal. Figure 1. Proportion of articles with comments by 6 5.7 Results: A total of 1607 publications have received comments since publication type 5 the launch of Commons; they were almost all English language 4 (99%) and approximately half were available as free full text (49%). 120 Although publications as early as 1973 received comments, most 3 2.8

Comments (%) 2.1 were from 2013/2014 (58%). Over half were human studies (54%), 1.8 100 2 1.4 but those classified as clinical trials and systematic reviews/meta- 1.4 1.1 0.7 analyses accounted for only 8% and 9%, respectively. The top five 1 0.7 0.4 0.7 journals for publications with comments were PLoS One (4.1%), New 80 0 England Journal of Medicine (3.2%), (2.2%), Nature (2.0%) and Proceedings of the National of U S A (1.9%). 60 author Conclusions: Levels of adoption of PubMed Commons have been very .gov issue poor and only 565 of the 903,614 publications added to PubMed in 40 Correction Articles with CT 2014 garnered comments. Publications garnering comments often had comments (%) Consort issue Congratulations open-access availability or were published in top-tier journals. 20 Omitted citations Acknowledgment Comment by Highlight retraction

Request for clarification 0 Suggestion for experiments

Response to a comment by author FFT PMC Letter Review English Editorial Introduction Guideline Clinical trial Meta- Other animals Human studies • The most common (22.4%) type of comment was a • Scientific publication has always been based on detailed, referenced critique of the research, akin to a letter a system of review and comment on published to the editor. More general, non-referenced comments FFT, free full text; PMC, PubMed Central. research via pre-publication peer-review and were the second most common (16.7%), with links to post-publication “letters to the editor” supporting research (8.2%), conflicting research (6.0%), • Over half were human studies (57%), but those classified and blogs (7.1%) the third most common (Figure 4B). • Post-publication review now occurs much as clinical trials and systematic reviews/meta-analyses more rapidly, via social media (eg,Twitter), accounted for only 8% each. online scientific communities (eg, Faculty of Figure 4B. Proportion of comments by nature of • Although articles published as early as 1956 received 1000 or Researchgate) or commenting forums comment comments, most were from 2013/2014 (58%; Figure 2). within the journal (eg, PLoS One or 25 22.4 New England Journal of Medicine). Number of articles with comments by year 20 • However, there are inherent problems with this Figure 2. 16.7 system: anonymity, comments appearing across 15 multiple disparate venues, and no requirements 600 that those commenting are qualified to do so. 10 8.2 500 7.1 • PubMed Commons is a pilot system launched Comments (%) 6.0 4.3 5 in October 2013 that seeks to address 2.1 400 2.1 these problems by allowing comments to 0.4 0.7 0.4 appear directly beneath abstracts within the 0 PubMed database. 300

• If adopted in a wholesale fashion, PubMed 200 Link to blog COI disclosed Number of articles Link to full text Commons would allow centralization of Link to software General comment comments in the world’s largest searchable 100 Link to related editorial Removed by moderator database of biomedical literature. Link to supportingLink to conflicting paper paper Detailed, referenced critique Link to supplementary data • However, unlike other venues, a comment can 0 COI, conflict of interest. 2014 2012 1998 1996 1994 1992 1990 only be added by an individual who is an author 2010 2008 2006 2004 2002 2000 on at least one PubMed-indexed article. • Other useful purposes of posted comments were linking • This means that comments cannot be added to free full text of the articles, supplementary data sets, or anonymously and that the person commenting • The top six journals publishing articles that elicited the software used for a particular analysis. should be qualified to do so with a degree of comments were PLoS One (4%), New England Journal of • Potential problems highlighted by comments were scientific rigor. Medicine (3%), Proceedings of the National Academy of Sciences (3%), Science (2%), Nature (2%) and Journal of the corrections to the published research, omission of important • Purpose: To assess the level of adoption American Medical Association (1.5%; Figure 3) citations, CONSORT issues, Clinicaltrials.gov issues (lack of and type of comments added via the NCT registry or posted results), plagiarism, and retractions. PubMed Commons forum since launching in Figure 3. Number of articles with comments by • A general indicator of the quality of the comments was October 2013. journal that only one of the 281 (0.4%) comments sampled was removed by the site moderator. However, conflicts 70 of interest were only declared for two (0.7%) of the

Methods 60 comments in the sample.

• A search was conducted using the term “has_ 50 user_comments[filter]” to capture all items 40 Conclusions with comments within the PubMed database • Levels of adoption of PubMed Commons have been between the launch of PubMed Commons and 30 very poor, and <0.05% of publications added to 30 November 2014. 20 PubMed in 2014 garnered comments. Number of articles • Publications receiving comments were sorted 10 • Publications garnering comments often had open- by the type of publication (as designated by access availability or were published in top-tier PubMed filters – eg, clinical trials, reviews and 0 journals. However, the comments that were posted free full text), publication year, and journal. BMJ generally displayed scientific rigor or fulfilled a JAMA Nature Lancet Science scientifically useful purpose. Acids Res • The type of comment was assessed by a PLoS One Pediatrics J Neurosci J Biol Chem N Engl J Med Nat Commun Nat Neurosci random sampling of 10% of the publications Ann Intern MedBioinformatics • If PubMed Commons is to fulfil its proposed role in Ann Emerg Med using a random number generator (random.org) NucleicJAMA Intern Med the post-publication evaluation of scientific research, to pick entries from the chronological listing of Proc Natl Acad Sci USA levels of adoption must improve and commenters

all articles with comments. Cochrane Database Syst Rev must disclose any pertinent conflicts of interest. • Comments on these manuscripts were assigned to several non-exclusive groups (eg, “Link to • The random sampling of 177 articles had 281 individual Acknowledgments and Disclosures supplementary data” or “Correction”) based on comments; the highest number of comments received by any single article was 17, but the vast majority of articles I would like to thank Joanne King and Jason Pelc of the Envision Pharma Group for their the content of the comments and the person assistance in the production of this poster. who posted them (eg, author of the article or had ≤2 comments. Please scan this QR code with your smartphone app to view an electronic non-author); totals for each type of comment • Only 8.5% of comments were made by an author of the version of this poster. were then expressed as a percentage of the article in question and these were mostly (66.0% of cases) If you do not have a smartphone, access the poster via the Internet at overall number of sampled comments. in response to questions by non-authors (Figure 4A). http://bit.ly/1DDT14F

Presented at the International Society for Medical Publication Professionals European Meeting (January 20-21, 2015; London, UK) and Annual Meeting (April 27-29; Arlington, VA, USA)