Are Altmetrics a panacea for citation delays? A critical view of the journey of the citing article through the research and impact cycles
Omwoyo Bosire Onyancha University of South Africa Department of Information Science
Department of Information Studies (Univ of Zululand) Conference, 3-5 September 2014 Agenda
Introduction Interpretation of citations in scholarly communication Problems with citations Citation delays The concept Statistical data on time delays Causes of citation delays Altmetrics – an introduction Methods and materials Findings and discussions Conclusions and recommendations
Introduction
the scientific tradition of citing or bibliographic referencing (the act of scientists, while documenting their own research, refer to earlier works that relate to the subject matter of their reported work) is as old as science itself (Nicolaisen 2008: 609) The use of citations in research evaluation – “supposed to identify those earlier researchers whose concepts, theories, methods equipment and so on, inspired or were used by the author in the process of conducting and presenting his or her own research” (Nicolaisen 2008: 610) How should citations be interpreted?
Citations are signposts left behind after information has been utilized (Smith, 1981) Citations are frozen footprints in the landscape of scholarly achievement ... which bear witness to the passage of ideas (Cronin, 1981) Citations are one important form of use of scientific information within the framework of documented science communication (Glänzel and Schoepflin 1999) Citations may be used as a formalised account of the information use and can be taken as a strong indicator of reception at this level (Glänzel and Schoepflin 1999) Citing Smith (1981) and Merton (1973), Bornmann & Daniel (2008: 48) express the use of citation counts (or citation analysis) as follows:
The use of citation counts as an indicator for research impact is appropriate only when the citation of a document means that the citing author used that document; the citation of a document reflects the merit (quality, significance, impact) of that document; and citations are made to the best possible works (Smith, 1981)… the normative theory, following Robert K. Merton's – the founder of the modern sociology of science – sociological theory of science (Merton, 1973), basically states that scientists give credit to colleagues whose work they use by citing that work Some problems with citations and citation analysis
Assumptions (Wallace 1989) The citing author has actually used the cited work Citing a given work is an indicator of its quality The citing author has provided references to the best works the content of the citing work is significantly related to the content of the cited works that all citations are of equal value Peters (2006: 786) argues that authors receive citations for ‘bad’ as well as ‘good’ work; there are] problems with accuracy, consistency and reliability in citation data collection; [there are] problems of self-citation Wang, Wang & Xu (2013: 718) observe that
“It is parochial to regard impact just as citations, since some influential theories, such as the Merton Miller theorem and Mendelian genetics, are widely accepted but seldom cited”
Kostoff (1998: 28) argues thus:
I could write a paper including Lotka's law without providing a reference to Lotka's law, or without even mentioning the name 'Lotka's law'
Citations to an article happen or are normally counted once the citing article has been published and, more often, subsequently indexed in the same indexing service Citation delay – the concept
The citation delay is sometimes referred to as response time (to the cited document) or citation time lag or citation speed or delayed recognition Diodato (1994: 157) defines it as “the number of months or years between the publication of a document and the publication of the first item that cites it”. Although many journals publish articles online before going print, the online-to-print time lag has continued to increase over time Tort, Targino & Amaral (2012) observed that a journal’s impact factor is still reliant on the publication of the print version of its articles. The implication of the above is that citation delays will still occur in scholarly communication, thereby leading to “important distortions, with concrete effects on policy decisions concerning the evaluation of journals and/or individual scientists” (Tort, Targino & Amaral 2012: 5) Causes of citation delay
Wang, Wang & Xu (2013: 718) have observed that “the publication of a scientific paper requires months to execute the review process, and as a result, significant publication delay will cause citation delay, and thus cause delay in the current research trend analysis.
Publication delay has also been identified as one of the possible causes of citation delays by Aman (2013; Amat (nd); Bornmann and Daniel (2010); and Wang, Wang & Xu (2013)
van Els, Janz, and Pair (1989) opine that the number of citations that an author or article misses may be due to the format of disseminating the research findings. The authors argue, for instance, that one may miss citations because “the results of his/her research are made public in the form of an instrument instead of via an article in a journal”.
Indexing time lag, defined as the “number of months or years between the publication of an article and the publication of an index that refers to the article” Diodato (1994: 157), can delay citations Studies on citation delays
Citation delays have been studied under different contexts (see Bornmann and Daniel, 2010; van Els, Janz, and Pair, 1989; RINIA, et al, 2001)
21.5% of all paper published in 1980 have not received any citation between the publication year and 2000. Most of these papers will probably never be cited (Glazel, Schlemmer, &Thijs, 2003: 574).
The number of articles published without a previous preprint in this time period (1996-2009) is 424, of which 53 remained uncited in their first three years (12.5%). The number of articles with a foregoing preprint is 287, of which 21 remained uncited in their first three years (7.3%). To include the uncited articles into the examination, their citation delay was set to 1,095 days (Aman, 2013: 39).
The median value of the citation delay for articles having a preprint is 304 days. Articles without a preprint receive their first citation on average three months later; the median is 395 days (Aman, 2013:39). Altmetrics – an introduction
Altmetrics is sometimes called social web metrics (Costas, Zahedi & Wouters, nd) and/or influmetrics (Cronin & Weaver, 1995; Rousseau & Ye , 2013) Refers to the use of social media, particularly Web 2.0 media, in assessing the influence of researchers on all type of users (Rousseau & Ye, 2013:1) Refers to mentions of scientific outputs in social web tools such as Facebook, Twitter, blogs, news media or online reference management tools (Costas, Zahedi & Wouters, nd) Some of the metrics that have been identified as constituting Alt-metrics include downloads, tweets, likes, posts on Facebook, blogs, bookmarks, views, visits, However, Altmetrics go beyond raw usage factors (such as downloads or click-through rates) in that they focus on readership, diffusion and reuse indicators that can be tracked via blogs, social media, peer production systems, collaborative annotation tools (including social bookmarking and reference management services). Konkiel (nd) believes that Altmetrics is concerned with assessing:
How many times an output article, website, blog, dataset, grey literature, software, etc
has been: Viewed (Publisher websites, Dryad) Downloaded (Slideshare, publisher websites, Dryad) Cited (PubMed, CrossRef, Scopus, Wikipedia, DOI, Web of Science) Reused/Adapted (Github) Shared (Facebook, Twitter) Bookmarked (Mendeley, CiteULike, Delicious) Commented upon (Twitter, Mendeley, blogs, publisher websites, Wikipedia, Faculty of 1000)
Research question
Are Altmetrics a panacea for citation delays? Methods and materials
For us to answer this question, we first sought to revisit as well as establish the citation delays using the citing ‘work’ as opposed to the cited ‘work’ as the unit of analysis A closed-ended question containing 9 questions (8 of which were closed-ended) was sent to18 academics in LIS schools in South Africa The 18 researchers were purposively selected on the basis of having obtained a PhD degree So far, we have received only 14 completed questionnaires We also obtained data on the file visits and views of the top most visited or viewed documents in academic institutional repositories and compared it with the Google Scholar citations to determine whether or not Altmetrics could have a role to play in citation delays Findings and discussions At what stage of the research cycle do researchers often use literature to inform their research?
Sometime Most of Never Rarely Always s the Time Before identifying of the research idea 2 5 8 Identification of the research idea 1 2 9 Identification of theory 2 12 Identification of empirical entities 1 3 9 Data acquisition 1 3 4 4 2 Data conversion 2 2 4 5 1 Data analysis 1 4 5 4 Publication 1 4 5 3 At what stage of the research cycle do researchers often cite literature to inform their research?
Most of Never Rarely Sometimes the Time Always Before identifying the research idea 1 3 4 2 2 Identification of the research idea 4 3 6 Identification of theory 5 9 Identification of empirical entities 2 3 7 Data acquisition 4 4 2 3 Data conversion 2 4 2 4 Data analysis 2 3 2 6 Publication 2 3 3 5 Which of the following best describes how researchers cite the sources that they have consulted when writing a research publication (i.e. article, chapter in a book, conference paper, etc)?
a. I list the sources even before I have started writing the publication [ 1 ] b. I cite the sources as I write the publication [ 13 ] c. I cite the sources after I have completed writing the publication [ 0 ] d. Any other, please specify………………………………………… [ 0 ] How long, on average, did it take the researchers to complete the research proposal, in terms of the number of months, for the following categories/purposes of research?
No of respondents Average time (in months) Research article 8 1,8 Chapter in Book 5 1,3 Masters Dissertation* 10 4,5 Doctoral thesis 10 7,5 Technical reports 5 1,6 Conference papers 9 1,4 Application for funding 11 2,6
Average completion period 3.3 months Now, let us compare citation opportunities missed along the research process vis-à-vis the altmetrics opportunities Identification of Mind maps research problem
Reference management Identification of theory Literature search Concept and Identification of operationalization empirical methods
Identification of empirical entities Compilation of question
Interview procedure Data acquisition
Transcription Data conversion Qualitative data analysis
Quantitative data analysis Data analysis Scientific writing Figures Publication Identification of Mind maps research problem
But if not published 4.7 months ‘citation’ period is lost
If shared, an Altmetric data is created instantly
You can share your mind map and citations can be picked by social media If the literature you have read and identified as relevant to your study is not shared (or published) a citation period of about 2.6 months is lost
Reference management Identification of theory Literature search
You can share the literature so far identified by sharing what you have read
Mendeley for example informs your followers in Tweeter what you are reading Reference management
Literature search Concept and Identification of operationalization empirical methods These processes take an average of 3.5 months which can be missed opportunity for citation if the outcomes are not Identification of published in mainstream publication outlets empirical entities
The illustration can be shared through social media – with citations supporting its formulation
The sharing may be immediately upon its completion On average, data acquisition takes about 2.1 months to complete
Compilation of question
Interview procedure Data acquisition
Some respondents indicated that they do cite sources while collecting data
If the product so far developed as at the data collection stage is not published, another citation opportunity is lost for ‘cited sources’
One can share information about the methods of data collection, population determination, sampling procedures, etc on social media – instant citation impact for cited source realized Transcription Data conversion
Researchers spend about 2.7 months to complete data conversion
NB: out of 14 respondents, only 2 who indicated that they do not cite consulted sources at the time of data conversion (e.g. transcription).
If the product of data conversion is not shared through publication, another 2.7 months of citation period are missed.
At this stage, one can share data (with cited sources) through social media Data analysis techniques, analyzed data, presentation Average time it takes a LIS skills, can be shared among researcher to complete data researchers as research analysis = 2.1 months progresses
A few researchers indicated that they cite sources during this stage of the research process
If any product resulting from this stage is not published, the cited author will not be recognized before a total of 17.7 months elapse Qualitative data analysis
Quantitative data analysis Data analysis According to the 14 respondents investigated for this paper, publication of various types of research products takes an average of 7.8 months
Publication can be online or print or both.
Online takes a shorter period but citation delay is still long (Aman, 2013)
We all know that publication of a manuscript takes much longer when done in print Scientific writing Figures Publication Identification of Mind maps 4.7 research problem
Reference management 2.1 Identification of 2.6 3.2 theory Literature search 2.9 4.0 Concept and Identification of operationalization empirical methods
Identification of 3.5 empirical entities Compilation of question 1.6
Interview procedure 2.8 Data acquisition 2.1 Transcription 2.7 Data conversion 2.7 Qualitative data analysis 2.5
Quantitative data analysis 1.7 Data analysis 2.1 Scientific writing 6.1 7.8 Publication of manuscript 9.7 Publication What does the above illustration imply?
Although the citing author (Author A or product A) has mentioned (cited Author B or Article B) in his/her work at or before point A, only after his/her work reaches point B does the citation get recognized in traditional citation analysis. 25.5 months t0 tt Article B was Article B receives first ‘used’ by its first citation in author A Article A Identification of a Publication of research idea for Article A Article A Are Altmetrics the answer to citation delays? Citations vs altmetrics (downloads, visits and views) Least Greatest Median average Total intenger intenger Citations 0 14,32 859 0 750 File visits 809 2177,02 130,621 25 28,452 Views 2571,5 3327,8 199,668 313 24,979
40
35 Uncited 30 documents (60%) 25 20 15 10 5
Number of citations per item per citations of Number N = 60 docs 0 0 1 2 3 4 5 7 17 27 750 Number of items 36 4 3 9 1 2 2 1 1 1 Empirical results – IR downloads and views vs Google citations
Citations File Visits Views
35000
30000
25000
20000
15000
10000
5000
0 a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j DUT UP NWU Unisa UJ Wits Conclusion – are altmetrics a panacea for citation delays?
YES Altmetrics can appear more rapidly than citations (Thelwall, Haustein, Lariviere, & Sugimoto, 2013; Eysenbach, 2011: abstract) Piwowar & Priem (2013: 11), argue that citations take years to accrue… “This delay is a big problem for graduate students who are applying for jobs soon after publishing their first papers and for those promotion candidates whose most profound work is published only shortly before review” Altmetrics considers all the stages and products of scholarly research from ‘‘[…] social literature search via Facebook to discussion of published results via Twitter, including any impact a publication or author may have on other people […]’’ (Bar-Ilan et al. 2012: 2 in Hammarfelt, 2014: nd) BUT Altmetrics should be aimed at complementing and improving the limitations of both traditional (i.e. bibliometrics) and web based (e.g. download and usage data) impact metrics and giving new insights to the analysis of impact (Galligan & Dyas-Correia, 2013 as cited in Costas, Zahedi & Wouters, nd) In your opinion, what would your preference be in terms of the usage of these metrics in supporting the measurement of research impact?
Neither Strongly Strongly Agree agree nor Disagree agree disagree disagree Altmetrics should be used to supplement traditional metrics (e.g. 6 4 3 1 citations and citation impact) Altmetrics should be used as replacement of traditional metrics 3 4 7 (e.g. citations and citation impact) Altmetrics should be cautiously used 3 7 2 2 to measure research impact Altmetrics should NEVER be used to 1 1 7 5 measure research impact I prefer Altmetrics to traditional 1 6 6 1 citation metrics I prefer traditional citation metrics to 7 5 1 1 Altmetrics END
Thanks