Measuring the Societal Impact of Open Science – Presentation of a Research Project
Total Page:16
File Type:pdf, Size:1020Kb
InformaatiotutkimusKATSAUS 34(4), 2015 Holmberg et al: Measuring... 1 Holmberg, K.*,1, Didegah, F. 1, Bowman, S. 1, Bowman, T.D. 1, & Kortelainen, T.2 Measuring the societal impact of open science – Presentation of a research project Address: 1Research Unit for the Sociology of Education, University of Turku 2Information Studies, University of Oulu Email: *[email protected] Introduction data, and 2) investigate novel quantitative indica- tors of research impact to incentivize researchers Research assessment has become increasingly in adopting the open science movement. important—especially in these economically challenging times—as funders of research try to Assessing impact of research identify researchers, research groups, and uni- versities that are most deserving of the limited Two approaches have traditionally been used to funds. The goal of any research assessment is to assess research impact: (1) assessments based on discover research that is of the highest quality and citations and (2) assessments based on publica- therefore more deserving of funding. As quality is tion venues. Both approaches have issues that are very difficult and time-consuming to assess and well-documented (e.g., Vanclay, 2012). Citations can be highly subjective, other approaches have have been found to not always reflect quality or been preferred for assessment purposes (especially intellectual debt as they can be created for many when assessing big data). Research assessments different reasons, some of which do not reflect usually focus on evaluating the level of impact a the scientific value of the cited article (Borgman research product has made; impact is therefore & Furner, 2002). In addition citations can take a used as a proxy for quality. Impact can, however, long time to accumulate, which suggests that one be difficult to identify, track, and quantify, and is evaluating past impact and past success when as the current methods to assess research impact assessing research based only on citations. Using are being increasingly criticized, new methods the publication venues as the unit of analysis as- and data sources need to be investigated. For this sumes that an article published in a high impact purpose, the altmetrics movement is investigating journal (or a journal that has in some other way what the online mentions of research products can been judged as more important or “better”) has disclose about the impact research has had and who gone through a more rigourous review process has been influenced by research. The focus of the and that it therefore has to be of higher quality. research project presented here will be to examine While this approach may sound logical in theory, online mentions of research products and to de- in practice it may not be the case that an article velop methods and tools to evaluate the potential published in an important journal will have higher of these mentions for measuring societal impact impact. In addition, neither of the two approaches of Finnish research in Finland and beyond. This investigates the actual, current scientific contribu- research will 1) map the current state of research tions of a specific article, as both approaches use in Finland using altmetric research methods and indirect methods to examine proxies of impact. 2 Holmberg et. al: Measuring... Informaatiotutkimus 34 (4), 2015 Informaatiotutkimus 34(4), 2015 Holmberg et al: Measuring... 3 Another problem with both of these approaches and results openly through various online tools is that they only assess scientific impact—the im- including their online social networks, which allow pact on research publications or on other scientists. their network connections to comment, discuss, and Science can, however, have a much wider impact share them further. Just as Google ranks websites on many different audiences, evidence of which based on the amount of links they receive, the is increasingly demanded by research funders. attention various research products receive in While science can inspire artists, change what various contexts and by various audiences could be is being taught at schools, increase people’s used for ranking (Priem et al., 2010). For instance, environmental consciousness, lead to technological events tracked in online contexts—so-called breakthroughs, have economic impact, benefit “altmetrics”—could potentially point to research culture, public policy or services (Research that has had higher impact, more influence, and has Excellence Framework 2014) and possibly lead to received greater attention from a wider audience behavioral changes, none of these impacts outside beyond the closed silos of academia. of scientific impact can be mapped or assessed Although altmetrics does not yet have a widely through citations, nor through investigating the accepted definition, the idea and potential with publication venues. Various types of impact have altmetrics is that the mentions and other indicators been discussed and some have previously examined of visibility and awareness (e.g., number of tweets, approaches to measure the broader societal impact comments, blog entries, and social bookmarks) of science (e.g., Wolf, Lindenthal, Szerencsits, a research article and other research products Holbrook, & Heß, 2013; Bornmann & Marx, receive on the web and in social media could tell 2014; Walter, Helgenberger, Wiek, & Scholz, something about the impact or influence of that 2007), yet there are currently no widely accepted research. Shema, Bar-Ilan & Thelwall (2014) try to methods of measuring this wider societal impact capture this as they define altmetrics as “web-based of research that would be demonstrable (Impact metrics for the impact of scholarly material, with toolkit 2015). Therefore there is a great need an emphasis on social media outlets as sources of for new data sources and methods to aggregate data.” The term “altmetrics” is being used in two and analyze different sources of data related to ways: (1) to describe the metrics resulting from wider scientific output beyond academia. The the traces of the use and production of research data should be able to demonstrate economic products made in online environments, and (2) or societal, instrumental, conceptual impact or as an umbrella term used to identify the research impact on capacity building. Instrumental impact field investigating the meaning and application of may mean influencing the development policy, these metrics and events. These altmetric events are practice or service provision or sharing legislation. useful in that they trace the use and production of Conceptual impact contributes to the understanding research outside of academia. Scholars examining of policy issues or reframing debates. Capacity altmetric data collect and investigate events building means impact through technical and surrounding research products from various personal skill development (ESRC). As scholarly online sources including social media sites, news communication is increasingly moving online and outlets, Wikipedia, and blogs, with the assumption becoming more transparent, many researchers have that these events can identify something about turned to monitor various online sources in order the wider impact of research or what types of to examine the digital traces left behind from acts impact (Priem, 2014) research is having. A vast, including the discussion, comments, and sharing wide-ranging audience (i.e., not only scholars) is of research products. responsible for altmetric events, hence, altmetrics It is the 350th birthday of the scientific journal may be able to provide a more nuanced view of in 2015 and it seems that it has not changed much the impact research has made on society (Liu & during this time, while scholarly communication Adie, 2013; Piwowar, 2013). These new metrics has undergone significant changes over the last should not, however, be considered as alternatives couple of decades, primarily because of the use to citations for demonstrating scientific impact, of online tools such as social media. Although but rather they should be investigated for their publications and citations are still fundamental potential to demonstrate other forms of societal for scholarly communication, it can now be said impact including educational, technological, that they are no longer irreplaceable. Researchers environmental, economical, and cultural. But as can share and disseminate their ideas, thoughts, the validity and meaning of some of the altmetrics 2 Holmberg et. al: Measuring... Informaatiotutkimus 34 (4), 2015 Informaatiotutkimus 34(4), 2015 Holmberg et al: Measuring... 3 acts are still unclear, more research is required by making the research process public and open to fully understand both what altmetrics actually for anyone to join, and efforts to make science reflect and which metrics are stable and reliable more available to the public through 1) writing in enough to be used as real indicators of impact a manner that is understandable even outside of outside the academia. academia, 2) including the public in the research We know that research has impact far beyond process through “citizen science”, and 3) ensuring citations as less than 1% of the article views in the open access to scientific literature. In addition, the open access journal PLoS result in later citations authors believe that a wider range of quantitative (Lin & Fenner, 2014). Thus the impact of research indicators of a broader range of impact can be articles is never fully captured. In addition,