sign in sign up
<<
Home , Altmetrics, BioMed Central, Impact factor, PubMed

#4 Measuring publication impact using article-level metrics ()

Ted Huang, PhD1; Janet Davies, PhD2; Kate Weatherall, PhD2; Sunil Joseph, PhD, CMPP2; and Matthew Wadyka, CMPP3

1 Nucleus Global, Hamilton, NJ, USA; 2 Nucleus Global, , UK; 3 Genentech Inc., San Francisco, CA, USA

Figure 2. Impact levels for drug A as assessed by Impactstory.org and .com scores for A) articles and B) reviews A. Articles

Objective: Recent in and online access of publications have led to new ways of assessing Impactstory.org Altmetric.com Journal * Score publication impact, using altmetrics. This study is a semi-quantitative comparison of two commonly used altmetric tools. Highly saved 1 (2010) 19 23.92 Research design and : As part of a publications gap analysis, altmetrics were assessed using Altmetric.com and cited and Impactstory.org. Here, we focus on publications for one drug in Phase 3 . Additionally, activity 2 (2009) 9 8.31 (tweet volume/sentiment) was assessed using Topsy.com. 3 (2009) 1 23.92 Results: †4 (2012) 7.1 Top 10 primary research articles, 2009–2013 Highly cited 5 (201 1) 2 4.17 Publication Journal Impactstory.org scores* Altmetric.com (year) impact PubMed Highly Highly Highly saved 6 (2012) 21 7.58 † score factor citations readers cited saved 7 (2012) ? 4.65 23.92 79 — 88 X X 19 1 (2010) 8 (2010) ? 4.17 2 (2009) 8.31 80 — 97 X X 9 9 (2013) 2.93 3 (2009) 23.92 — 50 35 X X 1 4 4 (2012) 7.11 4 — 28 X X — 10 (2013) 1 14.48 5 (2011) 4.17 6 — 13 X X 2 0 20 40 60 80 100 6 (2012) 7.58 14 42 — X — 21 Impactstory.org: PubMed:pmc_citations_reviews PubMed:pmc_citations_editorials PubMed:pmc_citations Mendeley:readers 7 (2012) 4.65 1 — 8 — — — Altmetric.com scores: F1000 blogs Twitter No activity 8 (2010) 4.17 17 — 27 — — —

9 (2013) 2.93 1 — 6 — — 4 B. Reviews

10 (2013) 14.48 1 — 6 — — 1 Impactstory.org Altmetric.com * Score Impact Factor th th 1 (2010) 9.63 *Impactstory.org focuses on academic metrics, including academic readers and citations. Highly saved = in the 75 –100 percentile of Highly saved 1 th th articles added to users’ Mendeley libraries vs. all articles published the same year. Highly cited = in the 75 –100 percentile of articles and cited 2 (2009) ? 3.97 cited on PubMed Central or Scopus vs. all articles published the same year. †Altmetric.com focuses on “public” social media, providing results including more investor-related blogs and posts. A weighted score is produced, factoring in relevant impact of each online forum. 3 (2009) 3.77 Articles with a score ≥ 3 are in the top 25% of all articles ranked by attention; a score ≥ 15 puts the article in the top 5% of all articles †4 (2012) 15 3.35 ranked by attention. Articles classified as highly cited/saved by Impactstory.org could have no or low scores on Altmetric.com and scores did not always correlate across the two tools. 5 (201 1) ? 4.39 6 (2012) 2 3.37 Conclusion: Altmetric.com and Impactstory.org report different aspects of publication impact. Further, traditional †7 (2012) 3.68 impact measures, such as journal impact factor, did not equate to Impactstory.org or Altmetric.com scores and calls Highly cited 8 (2010) 2 3.35 into question their value as qualitative impact tools. Topsy analysis highlighted that Twitter activity focused around †9 (2013) 1.01 congress presentations and press releases rather than publications (data not shown). Further research is required 10 (2013) 2 15.52 to assess the utility of altmetrics in tracking publication impact. Highly saved 11 (2012) 1 3.35 12 (2012) ? 32.44 13 (2010) 2 14.60 14 (2013) 3 9.25

Objective 15 (2013) 2 2.96 0 20 40 60 80 100

• Recent trends in social media and online access to publications have led to new Impactstory.org: PubMed:pmc_citations_reviews PubMed:pmc_citations_editorials PubMed:pmc_citations Mendeley:readers 1 ways of assessing publication impact using article-level metrics (altmetrics). Altmetric.com scores: Science blogs Facebook Twitter Reddit No activity

• Altmetrics measure usage, captures/downloads, social media mentions and *Highly saved = in the 75th–100th percentile of articles added to users’ Mendeley libraries vs. all articles published the same year. Highly citations. Two commonly used tools include Impactstory.org and Altmetric.com. cited = in the 75th–100th percentile of articles cited on PubMed Central or Scopus vs. all articles published the same year. †Article was not available via Altmetric.com. Each publication, the year of publication and journal impact factor is also shown. o Impactstory.org is a free-to-use, open-source tool that: Figure 3. Lack of correlation between Impactstory.org and Altmetric.com scores – Uses both academic metrics (scholars) and altmetrics (public) – Normalizes metrics based on a sample of articles published the same year Highly saved 2R 1R 2A 1A AND highly with results presented as raw data and percentiles compared with other 4A 6R 4R cited articles (Figure 1a). 5R 3A

o Altmetric.com is a proprietary tool adopted by several publishers including Highly saved 5A Springer, Group, Scopus () and BioMed Central that: OR highly 8R 6A cited – Focuses on “public” social media, such as blog posts, news pieces, tweets Impactstory.org 10R and likes. score 13R Saved 7A – Scores each article based on three main factors: number of individuals 10A AND/OR 12R 15R 9A mentioning the paper, where the mentions occurred and how often the cited 8A 11R 14R author of each mention talks about scholarly articles (Figure 1b). 2A Article (Impactstory.org ranking see Fig 2a) • As part of a publications gap analysis exercise, Impactstory.org and Altmetric.com Articles with Bottom 75% Top 25% Top 5% were used to measure the influence and reach of publications in a specific therapy 2R Review (Impactstory.org no score articles articles articles ranking see Fig 2b) (< 3 score) (≥ 3 score) (≥ 15 score) area. The present study is a semi-quantitative comparison of the two altmetric tools in assessing publication impact for a compound (drug A) in Phase 3 development. Altmetric.com score • In addition Topsy (analytics), from Topsy.com, was used to analyze Twitter and • Publications impact for a range of drugs in the same therapy area as drug A assess the social media noise of drug A as shown by real-time tweet volume and (pipeline and launched) was also conducted using Impactstory.org and sentiment. (Note: at the time of the present analysis, conducted between October Altmetric.com. Results were similar to those reported for drug A, with a lack of and December 2013, access to a free trial of Topsy was available but this service correlation between Impactstory.org and Altmetric.com results (data not shown). has since been withdrawn). • Topsy analysis of Twitter highlighted that ‘noise’ surrounding drug A focused Figure 1. Examples of detail pages from a) Impactstory.org and b) Altmetric.com around its congress presentations and associated press releases, pharma news showing the donut visualization, score and breakdown of metrics and investor news, rather than publications (Figure 4). A. Impactstory.org B. Altmetric.com Figure 4. Drug A Topsy analysis

Tweeted by 263 700 Highly cited On 1 Facebook Page 600 Mentioned in 22 Google+ posts Highly recommended Reddited by 1 500 Picked up by 1 news outlet Highly saved 400 Blogged by 9 Discussed 0 readers on Mendeley 300

0 readers on Connotea tweets of Number

Viewed 0 readers on CiteULike 200

100

0 Research design and methods 2008 2009 2010 2011 2012 2013

Congress abstract Press release Article • PubMed searches were performed on drug A and were limited to English publications (primary articles and reviews), with a custom date range set from January 2008 to December 2013. Conclusions • PubMed search results were manually checked to confirm relevance to drug A, and manually divided into primary articles and reviews. • As seen in the present study, two of the most commonly used tools, Impactstory.org • For drug A, the list of publication IDs (DOIs) for primary publications and reviews were entered into Impactstory.org to generate a dataset of Impactstory.org metrics and Altmetric.com, report somewhat different metrics. However, which of the two for each group. tools provides the more meaningful results and which reflects more ‘scholarly’ • From this dataset, the top 10 primary articles and top 15 reviews for drug A were impact? then individually assessed using the free Altmetric.com bookmarklet. o Public social media is characterized by a short lifespan and rapid decay, reflecting the transient nature of popularity. o The Altmetric.com bookmarklet produces a weighted score, factoring in the relevant impact of each online forum. Articles with scores ≥ 3 and ≥ 15 are in o In the present study, looking at the sources of blogs and tweets highlighted the top 25% and 5% of all publications ranked by attention, respectively. their variable nature – mainly investor sites, pharma sites, and Facebook. • For Topsy analysis, searches were conducted on drug A from 2008 to 2013 and How ‘scholarly’ are these blogs and tweets? Comments posted on these included both original tweets and retweets. sources may not reflect scientific/scholarly impact, but rather social impact/attention. o Altmetrics noise may not always be “good noise”. Results from a failed trial could get a high altmetrics score due to the negative attention. o Longer-term online metrics such as downloads, readers and comment Results numbers, while collected more slowly, may be more meaningful and more scholarly. • in reporting are apparent between the two altmetric tools. • A limitation of the present study is the small number of publications assessed. Impactstory.org appeared to focus on academic metrics, including academic However, similar results were observed for additional compounds in the same, and readers and citations, whilst Altmetric.com results focused on “public” social media, in different, therapy areas. In addition, both Impactstory.org and Altmetric.com including more investor-related blogs and posts (Figure 2). For example: have well-documented limitations.1,2 o Impactstory.org article 1 had an Altmetric.com score of 19, and was reviewed by the (http://f1000.com/), discussed in a • In conclusion, altmetric tools such as Impactstory.org and Altmetric.com are gaining medicine blog “Brain Posts” (http://brainposts.blogspot.co.uk/) and a Nature popularity as new measures of publications impact. However, further research is Medicine blog required to understand and define the meaning/context of the various social media o Article 2 had an Altmetric.com score of 9 and was discussed in a pharma blog metrics, their scoring and their relationship to scholarly impact vs. social (http://pipeline.corante.com) media impact. o Article 6 had the highest Altmetric.com score and was mentioned in the Corante pharma blog, a blog, and up to December 2013 had been mentioned in 12 tweets from 12 accounts with an upper bound of 33,169 combined followers. References • Impact levels did not correlate across Impactstory.org and Altmetric.com (Figure 3). Several articles or reviews defined as highly saved and highly cited by 1. http://www.altmetric.com. Accessed March 2014 Impactstory.org had low or no scores by Altmetric.com, and did not feature in the top 5% or 25% of publications by attention. 2. http://impactstory.org/faq. Accessed March 2014 • Equally, there was no clear correlation between journal impact factors and either tool; articles with high journal impact factors had low or no scores on Impactstory.org or Altmetric.com (Figure 2).

10th Annual Meeting of ISMPP - April 7–9, 2014 - Arlington, VA